By Brian Lucas

This one is dedicated to “little Buddy” who passed on before he got a chance to live and “little Bear” in the hope he will live long and in the infinite happiness of puppy-hood all his life.

“Oh, thou hast a damnable iteration, and art indeed able to corrupt a saint. Thou hast done much harm upon me…” – William Shakespeare

One of the most complained about aspects of agile is the iterative process¹.  It is an anathema to traditionalists.  They see it as inefficient and a waste of coding effort and producing hacked and patched code instead of cleanly designed logic.  Too often their complaint is valid!  Yet there is a way to remove the negative aspects of iteration from the equation by following a simple concept.  It will reduce overall effort and make your upfront agile kickoff be more efficient and effective.

The quote is Falstaff’s, from Henry IV Part 1: Act 1, Scene 2, as he is speaking to Prince Henry.  He is saying that Henry, who he calls Hal, has a wicked talent for quoting scripture out of context for the purpose of ruining Falstaff’s innocence².  Just like Shakespeare’s Prince Henry, with a remarkably detrimental frequency, interpreters of original ideas often derive negative results from an author’s original positive intent.  Why is this? Their sins fall into two categories: misunderstanding of the original concept and using the idea to restrict thought.

Here is a little philosophy and psychology, so please bear with me.  Misunderstanding the original concept is a forgivable mistake.  After all, it is a new concept or an old concept applied in a new way.  People are different.  They learn at different rates and will always have different levels of understanding.  Some can assimilate a radically different concept instantaneously and even extrapolate it beyond its original premise.  Others will struggle to gain even a rudimentary level of understanding and sometimes require exhaustively repetitive explanations³.

Honest obliviousness⁴ or at least an incomplete level of understanding is not something that should be condemned, providing the person is trying to learn.  The cure for this is patient teaching and creative approaches in explaining the subject in question.

On the other hand, using an idea to restrict thought is a mortal sin to me.  It is the favorite tool of those who seek greater levels of command and control over everything.  These are the people that worship process for its own sake, because it reflects their own narrow and rigidly defined thinking.  No true thought leader ever wishes their idea to restrict thinking, regardless of how proud they are of their theory.

Instead, they revel in the fact that they spurred new thoughts and unleashed new creativity in others.  They relish the moment, when it comes, that their idea was used as either a foundation or a springboard for an even more advanced hypothesis.   My underlying message here is that for my part, I do not wish anyone to use any of the ideas I put forward to restrict the creativity of anyone’s thinking.  If you are a CEO, keep rigid thinkers out of positions of authority, if you want an agile enterprise.

The reason I went through this seeming tirade is that I am going to be giving you a specific interpretation of Niklaus Wirth’s⁵ stepwise refinement⁶ that embraces some top-down design⁷ concepts.  In top-down design, each level must represent the entire solution in scope, if not in detail.  However, the interpretation I am going to give you of stepwise refinement, does not.  At least not until the following layer is developed, earlier work can be revised at any time to keep it inclusive of its lower layers.  Frankly, my interpretation of stepwise refinement occurred to me when I first saw the title, “Program Development by Stepwise Refinement” in the Journal of the ACM years ago when I was first studying computer science.  It was already an old article, but I love trolling old copies of journals⁷ looking for treasures of thought.

To me, Wirth’s use of stepwise meant I could get some elements of the solution correct in the first step and add more right ones in the second, while subtracting the erroneous ones.  This is clearly a more iterative flavored approach than top-down.  Even when I went through his example of the 8 queens⁹, I had this thought in mind, especially since he mentioned trial and error methods.  Here is where the light bulb goes off.  Often people use the trial and error method blindly.  I hate to use the infinite numbers of monkeys typing on an infinite number of Android Tablets¹⁰ will eventually produce a Shakespeare’s Prince Henry, but at times it seems like that is the case.

Wirth talks about strategy of preselection, stepwise construction of trial solutions, introduction of auxiliary data and recursion.  Further Wirth states,

“Every refinement step implies some design decisions. It is important that these decisions be made explicit, and that the programmer be aware of the underlying criteria and of the existence of alternative solutions. The possible solutions to a given problem emerge as the leaves of a tree, each node representing a point of deliberation and decision. Subtrees may be considered as families of solutions with certain common characteristics and structures. The notion of such a tree may be particularly helpful in the situation of changing purpose and environment to which a program may sometime have to be adapted.”

This to me shouts of a recursive, iterative process where intelligent hypothesizes are tested, failures discarded and successes used as a stepping stone to further the final solution.  Most importantly, Wirth’s mathematical formula implies concentrating on the more discrete variables first and using their correct derivatives to help narrow down the more “exhaustive set of candidates.”

Let’s now jump to user stories.  If you were following top-down design strictly, your epics would have to be perfect before you decomposed them into subsequently more detailed user stories.  Each of these user stories would also have to be perfect.  However, if you are following my interpretation of stepwise refinement, you start with an imperfect epic.  You improve this epic with effort.  This effort can take the form of story boarding or wire framing the epic to get a visual representation.  You can also break the epic down into user stories.  It can even be as simple as creating additional epics.  This can help you gauge the level of function in each individual epic as compared to the body of epics.  It can also redefine the overall boundaries of adjacent epics in story mapping.  In both cases, the feedback from these iterations will improve the precision of the language you are using in the epics and even the user stories.

This is all happening before coding begins.  So you are not wasting development effort.  Don’t misinterpret¹¹ me here.  I am not calling for six months of business and systems analysts arguing in endless meetings coming up with the perfect solution.  It is a working session of usually the whole development team functioning in virtually a JAD¹² fashion to gain an understanding of a potential release.  It is part of the release planning activities that I always recommend to both novice and experienced development teams alike.  Even though I advocate using Gherkin¹³ language, it is very hard to get this perfect in the first pass, especially for an inexperienced team.

A number of pleasant and beneficial things happen when you follow this simultaneous activity refinement¹⁴ concept.  First, it is a great way to get the entire development team on board and get them communicating.  Second, it gets you somewhere positive quicker because you concentrate initially on the known elements letting the unknown elements surface as needed.  Third, it takes a lot of pressure off the user to formulate “perfect” user stories to feed the development team, which never happens anyway.  Fourth, since I advocate the early use of story boarding or wire framing it helps the user interface people get their ideas together ahead of time instead of having the developers waiting for them.  Fifth, it does wonders for the quality of your code, because developers are not hacking their code every time the user story changes.  The code begins to reflect the stability of the epics and their subsequent user stories.  It is more defined in the solid areas and more open-ended in the less defined areas.  Developers will be able to know where to decouple the logic because the user stories are correctly structured that way.

One of the few good lessons of the waterfall era¹⁵ was that a problem found in design, cost a $1 to fix.  A problem found in development, cost $10 to fix.  A problem found in testing, cost $100 to fix.  A problem found in production, cost $1000 to fix.  This is still true in agile.  If you apply a bit of stepwise refinement thinking up front you can save your developers a lot of wasted coding effort.  Everyone will be happier.  Remember till next time, keep agile!