By Brian Lucas
This one is dedicated to “little Buddy” who passed on before he got a chance to live and “little Bear” in the hope he will live long and in the infinite happiness of puppy-hood all his life.
“Oh, thou hast a damnable iteration, and art indeed able to corrupt a saint. Thou hast done much harm upon me…” – William Shakespeare
One of the most complained about aspects of agile is the iterative process1. It is an anathema to traditionalists. They see it as inefficient and a waste of coding effort and producing hacked and patched code instead of cleanly designed logic. Too often their complaint is valid! Yet there is a way to remove the negative aspects of iteration from the equation by following a simple concept. It will reduce overall effort and make your upfront agile kickoff be more efficient and effective.
The quote is Falstaff’s, from Henry IV Part 1: Act 1, Scene 2, as he is speaking to Prince Henry. He is saying that Henry, who he calls Hal, has a wicked talent for quoting scripture out of context for the purpose of ruining Falstaff’s innocence2. Just like Shakespeare’s Prince Henry, with a remarkably detrimental frequency, interpreters of original ideas often derive negative results from an author’s original positive intent. Why is this? Their sins fall into two categories: misunderstanding of the original concept and using the idea to restrict thought.
Here is a little philosophy and psychology, so please bear with me. Misunderstanding the original concept is a forgivable mistake. After all, it is a new concept or an old concept applied in a new way. People are different. They learn at different rates and will always have different levels of understanding. Some can assimilate a radically different concept instantaneously and even extrapolate it beyond its original premise. Others will struggle to gain even a rudimentary level of understanding and sometimes require exhaustively repetitive explanations3.
Honest obliviousness4 or at least an incomplete level of understanding is not something that should be condemned, providing the person is trying to learn. The cure for this is patient teaching and creative approaches in explaining the subject in question.
On the other hand, using an idea to restrict thought is a mortal sin to me. It is the favorite tool of those who seek greater levels of command and control over everything. These are the people that worship process for its own sake, because it reflects their own narrow and rigidly defined thinking. No true thought leader ever wishes their idea to restrict thinking, regardless of how proud they are of their theory.
Instead, they revel in the fact that they spurred new thoughts and unleashed new creativity in others. They relish the moment, when it comes, that their idea was used as either a foundation or a springboard for an even more advanced hypothesis. My underlying message here is that for my part, I do not wish anyone to use any of the ideas I put forward to restrict the creativity of anyone’s thinking. If you are a CEO, keep rigid thinkers out of positions of authority, if you want an agile enterprise.
The reason I went through this seeming tirade is that I am going to be giving you a specific interpretation of Niklaus Wirth’s5 stepwise refinement6 that embraces some top-down design7 concepts. In top-down design, each level must represent the entire solution in scope, if not in detail. However, the interpretation I am going to give you of stepwise refinement, does not. At least not until the following layer is developed, earlier work can be revised at any time to keep it inclusive of its lower layers. Frankly, my interpretation of stepwise refinement occurred to me when I first saw the title, “Program Development by Stepwise Refinement” in the Journal of the ACM years ago when I was first studying computer science. It was already an old article, but I love trolling old copies of journals7 looking for treasures of thought.
To me, Wirth’s use of stepwise meant I could get some elements of the solution correct in the first step and add more right ones in the second, while subtracting the erroneous ones. This is clearly a more iterative flavored approach than top-down. Even when I went through his example of the 8 queens9, I had this thought in mind, especially since he mentioned trial and error methods. Here is where the light bulb goes off. Often people use the trial and error method blindly. I hate to use the infinite numbers of monkeys typing on an infinite number of Android Tablets10 will eventually produce a Shakespeare’s Prince Henry, but at times it seems like that is the case.
Wirth talks about strategy of preselection, stepwise construction of trial solutions, introduction of auxiliary data and recursion. Further Wirth states,
“Every refinement step implies some design decisions. It is important that these decisions be made explicit, and that the programmer be aware of the underlying criteria and of the existence of alternative solutions. The possible solutions to a given problem emerge as the leaves of a tree, each node representing a point of deliberation and decision. Subtrees may be considered as families of solutions with certain common characteristics and structures. The notion of such a tree may be particularly helpful in the situation of changing purpose and environment to which a program may sometime have to be adapted.”
This to me shouts of a recursive, iterative process where intelligent hypothesizes are tested, failures discarded and successes used as a stepping stone to further the final solution. Most importantly, Wirth’s mathematical formula implies concentrating on the more discrete variables first and using their correct derivatives to help narrow down the more “exhaustive set of candidates.”
Let’s now jump to user stories. If you were following top-down design strictly, your epics would have to be perfect before you decomposed them into subsequently more detailed user stories. Each of these user stories would also have to be perfect. However, if you are following my interpretation of stepwise refinement, you start with an imperfect epic. You improve this epic with effort. This effort can take the form of story boarding or wire framing the epic to get a visual representation. You can also break the epic down into user stories. It can even be as simple as creating additional epics. This can help you gauge the level of function in each individual epic as compared to the body of epics. It can also redefine the overall boundaries of adjacent epics in story mapping. In both cases, the feedback from these iterations will improve the precision of the language you are using in the epics and even the user stories.
This is all happening before coding begins. So you are not wasting development effort. Don’t misinterpret11 me here. I am not calling for six months of business and systems analysts arguing in endless meetings coming up with the perfect solution. It is a working session of usually the whole development team functioning in virtually a JAD12 fashion to gain an understanding of a potential release. It is part of the release planning activities that I always recommend to both novice and experienced development teams alike. Even though I advocate using Gherkin13 language, it is very hard to get this perfect in the first pass, especially for an inexperienced team.
A number of pleasant and beneficial things happen when you follow this simultaneous activity refinement14 concept. First, it is a great way to get the entire development team on board and get them communicating. Second, it gets you somewhere positive quicker because you concentrate initially on the known elements letting the unknown elements surface as needed. Third, it takes a lot of pressure off the user to formulate “perfect” user stories to feed the development team, which never happens anyway. Fourth, since I advocate the early use of story boarding or wire framing it helps the user interface people get their ideas together ahead of time instead of having the developers waiting for them. Fifth, it does wonders for the quality of your code, because developers are not hacking their code every time the user story changes. The code begins to reflect the stability of the epics and their subsequent user stories. It is more defined in the solid areas and more open-ended in the less defined areas. Developers will be able to know where to decouple the logic because the user stories are correctly structured that way.
One of the few good lessons of the waterfall era15 was that a problem found in design, cost a $1 to fix. A problem found in development, cost $10 to fix. A problem found in testing, cost $100 to fix. A problem found in production, cost $1000 to fix. This is still true in agile. If you apply a bit of stepwise refinement thinking up front you can save your developers a lot of wasted coding effort. Everyone will be happier. Remember till next time, keep agile!
1 See Boehm, Barry W.( May 1988) “A Spiral Model of Software Development and Enhancement,” Computer, IEEE
2 I would rather avoid the metaphysical argument as to the original intent of what I am referring to as non-denominal scripture. Let us assume it was positive.
3 Heaven knows I have often had to repeat myself over and over again, ad nauseam until some of the more complex ideas (at least others thought they were difficult) I was trying to convey would sink in. I remember once when I define a highly stratified and layered architecture for a rather unique situation. To me, it was simplicity itself. However, even though I created diagrams with write-ups and even some colorful documentation, I had to explain it over and over again to a rather large project team I was directing at the time. I finally accepted the fact that they needed a daily refresher and began an early morning stand-up briefing on the functions of each layer and answered questions. I could not conceive of how they could not understand the concept since it was so self-evident to me.
4 Those who are defiantly and belligerently ignorant and proud of it have always been the bane of mankind.
7 Is the software design technique that describes functionality in a series of levels of detail each object on the higher level expands into another more detailed representation of itself in a subsequent level.
8 Among my favorites are Nature, Science News, Journal of the ACM, Reviews of Modern Physics, Scientific American, Academy of Management Review, Trends in Ecology and Evolution, The Lancet and Journal of Applied Psychology. Fun reading!
9 Which I admit I appreciated being a chess player.
10 The infinite monkey theorem states that a monkey hitting keys at random on a typewriter keyboard for an infinite amount of time will type a given text, such as the complete works of William Shakespeare. I have upgraded it from typewriters to Android Tablets and would be satisfied with a copy of Henry IV.
11 Sorry I could not resist that rather dry pun.
12 See Davidson, E.J. (1999). Joint application design (JAD) in practice. Journal of Systems & Software. I realize this is a reference to an old technique, but it is in keeping with the theme that if you are agile and don’t let you tinking be boxed in you can get a lot more millage out of new ideas.
14 I hope I don’t read an article in the ACM called SAR in a few months.
15 God help those of you who are still there.