Turning Damnable Iteration into Elegant Stepwise Refinement in Agile

By Brian Lucas

This one is dedicated to “little Buddy” who passed on before he got a chance to live and “little Bear” in the hope he will live long and in the infinite happiness of puppy-hood all his life.

“Oh, thou hast a damnable iteration, and art indeed able to corrupt a saint. Thou hast done much harm upon me…” – William Shakespeare

One of the most complained about aspects of agile is the iterative process1.  It is an anathema to traditionalists.  They see it as inefficient and a waste of coding effort and producing hacked and patched code instead of cleanly designed logic.  Too often their complaint is valid!  Yet there is a way to remove the negative aspects of iteration from the equation by following a simple concept.  It will reduce overall effort and make your upfront agile kickoff be more efficient and effective.

The quote is Falstaff’s, from Henry IV Part 1: Act 1, Scene 2, as he is speaking to Prince Henry.  He is saying that Henry, who he calls Hal, has a wicked talent for quoting scripture out of context for the purpose of ruining Falstaff’s innocence2.  Just like Shakespeare’s Prince Henry, with a remarkably detrimental frequency, interpreters of original ideas often derive negative results from an author’s original positive intent.  Why is this? Their sins fall into two categories: misunderstanding of the original concept and using the idea to restrict thought.

Here is a little philosophy and psychology, so please bear with me.  Misunderstanding the original concept is a forgivable mistake.  After all, it is a new concept or an old concept applied in a new way.  People are different.  They learn at different rates and will always have different levels of understanding.  Some can assimilate a radically different concept instantaneously and even extrapolate it beyond its original premise.  Others will struggle to gain even a rudimentary level of understanding and sometimes require exhaustively repetitive explanations3.

Honest obliviousness4 or at least an incomplete level of understanding is not something that should be condemned, providing the person is trying to learn.  The cure for this is patient teaching and creative approaches in explaining the subject in question.

On the other hand, using an idea to restrict thought is a mortal sin to me.  It is the favorite tool of those who seek greater levels of command and control over everything.  These are the people that worship process for its own sake, because it reflects their own narrow and rigidly defined thinking.  No true thought leader ever wishes their idea to restrict thinking, regardless of how proud they are of their theory.

Instead, they revel in the fact that they spurred new thoughts and unleashed new creativity in others.  They relish the moment, when it comes, that their idea was used as either a foundation or a springboard for an even more advanced hypothesis.   My underlying message here is that for my part, I do not wish anyone to use any of the ideas I put forward to restrict the creativity of anyone’s thinking.  If you are a CEO, keep rigid thinkers out of positions of authority, if you want an agile enterprise.

The reason I went through this seeming tirade is that I am going to be giving you a specific interpretation of Niklaus Wirth’s5 stepwise refinement6 that embraces some top-down design7 concepts.  In top-down design, each level must represent the entire solution in scope, if not in detail.  However, the interpretation I am going to give you of stepwise refinement, does not.  At least not until the following layer is developed, earlier work can be revised at any time to keep it inclusive of its lower layers.  Frankly, my interpretation of stepwise refinement occurred to me when I first saw the title, “Program Development by Stepwise Refinement” in the Journal of the ACM years ago when I was first studying computer science.  It was already an old article, but I love trolling old copies of journals7 looking for treasures of thought.

To me, Wirth’s use of stepwise meant I could get some elements of the solution correct in the first step and add more right ones in the second, while subtracting the erroneous ones.  This is clearly a more iterative flavored approach than top-down.  Even when I went through his example of the 8 queens9, I had this thought in mind, especially since he mentioned trial and error methods.  Here is where the light bulb goes off.  Often people use the trial and error method blindly.  I hate to use the infinite numbers of monkeys typing on an infinite number of Android Tablets10 will eventually produce a Shakespeare’s Prince Henry, but at times it seems like that is the case.

Wirth talks about strategy of preselection, stepwise construction of trial solutions, introduction of auxiliary data and recursion.  Further Wirth states,

“Every refinement step implies some design decisions. It is important that these decisions be made explicit, and that the programmer be aware of the underlying criteria and of the existence of alternative solutions. The possible solutions to a given problem emerge as the leaves of a tree, each node representing a point of deliberation and decision. Subtrees may be considered as families of solutions with certain common characteristics and structures. The notion of such a tree may be particularly helpful in the situation of changing purpose and environment to which a program may sometime have to be adapted.”

This to me shouts of a recursive, iterative process where intelligent hypothesizes are tested, failures discarded and successes used as a stepping stone to further the final solution.  Most importantly, Wirth’s mathematical formula implies concentrating on the more discrete variables first and using their correct derivatives to help narrow down the more “exhaustive set of candidates.”

Let’s now jump to user stories.  If you were following top-down design strictly, your epics would have to be perfect before you decomposed them into subsequently more detailed user stories.  Each of these user stories would also have to be perfect.  However, if you are following my interpretation of stepwise refinement, you start with an imperfect epic.  You improve this epic with effort.  This effort can take the form of story boarding or wire framing the epic to get a visual representation.  You can also break the epic down into user stories.  It can even be as simple as creating additional epics.  This can help you gauge the level of function in each individual epic as compared to the body of epics.  It can also redefine the overall boundaries of adjacent epics in story mapping.  In both cases, the feedback from these iterations will improve the precision of the language you are using in the epics and even the user stories.

This is all happening before coding begins.  So you are not wasting development effort.  Don’t misinterpret11 me here.  I am not calling for six months of business and systems analysts arguing in endless meetings coming up with the perfect solution.  It is a working session of usually the whole development team functioning in virtually a JAD12 fashion to gain an understanding of a potential release.  It is part of the release planning activities that I always recommend to both novice and experienced development teams alike.  Even though I advocate using Gherkin13 language, it is very hard to get this perfect in the first pass, especially for an inexperienced team.

A number of pleasant and beneficial things happen when you follow this simultaneous activity refinement14 concept.  First, it is a great way to get the entire development team on board and get them communicating.  Second, it gets you somewhere positive quicker because you concentrate initially on the known elements letting the unknown elements surface as needed.  Third, it takes a lot of pressure off the user to formulate “perfect” user stories to feed the development team, which never happens anyway.  Fourth, since I advocate the early use of story boarding or wire framing it helps the user interface people get their ideas together ahead of time instead of having the developers waiting for them.  Fifth, it does wonders for the quality of your code, because developers are not hacking their code every time the user story changes.  The code begins to reflect the stability of the epics and their subsequent user stories.  It is more defined in the solid areas and more open-ended in the less defined areas.  Developers will be able to know where to decouple the logic because the user stories are correctly structured that way.

One of the few good lessons of the waterfall era15 was that a problem found in design, cost a $1 to fix.  A problem found in development, cost $10 to fix.  A problem found in testing, cost $100 to fix.  A problem found in production, cost $1000 to fix.  This is still true in agile.  If you apply a bit of stepwise refinement thinking up front you can save your developers a lot of wasted coding effort.  Everyone will be happier.  Remember till next time, keep agile!

1 See Boehm, Barry W.( May 1988) “A Spiral Model of Software Development and Enhancement,” Computer, IEEE

2 I would rather avoid the metaphysical argument as to the original intent of what I am referring to as non-denominal scripture.  Let us assume it was positive.

3 Heaven knows I have often had to repeat myself over and over again, ad nauseam until some of the more complex ideas (at least others thought they were difficult) I was trying to convey would sink in.  I remember once when I define a highly stratified and layered architecture for a rather unique situation. To me, it was simplicity itself.  However, even though I created diagrams with write-ups and even some colorful documentation, I had to explain it over and over again to a rather large project team I was directing at the time.  I finally accepted the fact that they needed a daily refresher and began an early morning stand-up briefing on the functions of each layer and answered questions.  I could not conceive of how they could not understand the concept since it was so self-evident to me.

4 Those who are defiantly and belligerently ignorant and proud of it have always been the bane of mankind.

7 Is the software design technique that describes functionality in a series of levels of detail each object on the higher level expands into another more detailed representation of itself in a subsequent level.

8 Among my favorites are Nature, Science News, Journal of the ACM, Reviews of Modern Physics, Scientific American, Academy of Management Review, Trends in Ecology and Evolution, The Lancet and Journal of Applied Psychology.  Fun reading!

9 Which I admit I appreciated being a chess player.

10 The infinite monkey theorem states that a monkey hitting keys at random on a typewriter keyboard for an infinite amount of time will type a given text, such as the complete works of William Shakespeare. I have upgraded it from typewriters to Android Tablets and would be satisfied with a copy of Henry IV.

11 Sorry I could not resist that rather dry pun.

12 See Davidson, E.J. (1999). Joint application design (JAD) in practice. Journal of Systems & Software. I realize this is a reference to an old technique, but it is in keeping with the theme that if you are agile and don’t let you tinking be boxed in you can get a lot more millage out of new ideas.

14 I hope I don’t read an article in the ACM called SAR in a few months.

15 God help those of you who are still there.

About Brian Lucas

In his life, Brian Lucas has been a coach, farm worker, forester, health care advocate, life guard, general contractor, mechanic, mixologist, musician/singer (in a rock group), salesman and teacher. Brian has worked as a project manager, technical marketer, methodologist, manager, software architect, systems designer, data modeler, business analyst, systems programmer, software developer and creative writer. These efforts include over a hundred hi-tech initiatives in almost every business and industrial sector as well as government and military projects. Among them, he designed and developed a quality assurance system for the first transatlantic fiber optic communications network, a manufacturing system for a large computer manufacture’s seven manufacturing centers, a data mining system for steel production, an instrumentation system for cable systems, defined requirements for government’s information systems and designed and developed human performance management systems. Brian has educated and mentored many over the years, designing programs to discover and develop talent. He has also lectured extensively to a variety of audiences. Brian is currently devoting as much time as possible to the innovation of business agility and human capital management along with the next generation of agile software development. As an amateur theoretical physicist he is working on joining general relativity and quantum mechanics through a multidimensional time corollary on string theory and negating the uncertainty principle with Louis de Broglie’s wave/particle hypothesis. He is also an avid blue-water sailor and wilderness backpacker. He enjoys billiards, boxing, chess, cooking, famous battle reenactments and war gaming, fencing, flying, gardening, horseback riding, martial arts (particularly Ninjutsu), philosophy and psychology, playing musical instruments (7 so far), poker, rapid-fire target shooting, reading (he tries to read a new book every night), painting with oils, scuba diving, skiing and recently writing novels.
This entry was posted in Agile and Strategic Planning, Agile Arguments, Agile for Beginners, Agile for Software Development, Agile in the Enterprise and tagged , , , , , , , , . Bookmark the permalink.

20 Responses to Turning Damnable Iteration into Elegant Stepwise Refinement in Agile

  1. Jack says:

    Wow Brian! I am speechless!! This just blows me away and I will have to read it several more times to understand it all, because it is very deep. This is similar to some of the advice you gave in email stream that Jim, Benson, Willis and I were sharing. It is a much more scientific here and I appreciate all the references you have. They always make for good reading. Thanks for sharing your VAST knowledge with us mere mortals, my friend! – your loyal friend Jack

  2. Elliot says:

    Brian I take it this concept works with your story maping you describe in your previous post. My question is does this refinement take place before, during or after the story mapping?

  3. codedemomslayer says:

    Dude this is a heavy and righteous share…thx man

  4. Ahmad Hassan CSM, PMP says:

    This is brilliantly conceived, yet quite feasible! Wisdom from a master!

  5. Sally says:

    Brian, I am so sorry to hear about little Bear! Once again you have posted an absolute gem, both scholarly and colorful. -Sally

  6. Rick-K says:

    This is a good guideline. We often either jump into coding and prototyping too soon or spend too much time in analysis paralysis for our own good. We never seem to hit it right. I like the emphasis an talking or story-boarding through the first iterations and refining them as you go before coding. My big question is when do you know when you are done refining the epics and user stories? And is that the point you start coding? Do you use an absolute cut off like a percentage of the iteration? I have read that 15% is a working number for time spent before coding do you agree? Thanks for posting such a valuable and professional blog and sharing your expertise!

  7. Jaysom Platt, CSM says:

    Refreshing focus on what is important and when not to rush into a solution. I am forwarding this to all my entire teams as required reading!

  8. John Kisch says:

    Another solid and information rich post on the beginning activities of a release. I appreciate the emphasis on quality and not rushing to coding. I always view top-down and step-wise as the same thing. Your argument is convincing and it convinced me. I did read the references and followed your link to Wirth’s original paper and read it for the first time. I have to second Rick-K’s questions. When do you stop refining? And when do you start coding. And let me add – does any of this thinking apply to spikes?
    John Kisch, CSM, PMP

  9. Alfred Richter, Phd says:

    I was forwarded this blog by Ahmad a former student of mine. Speaking as a retired computer science teacher, I find this a scholarly article told with clarity and a touch of dry humor. It speaks well of reason and the principles of thoughtful adaptation of ideas. Despite its somewhat theoretical basis, it is surprisingly understandable. Mr. Lucas is a thought leader deserving of widespread recognition for his uniquely universal interpretation of agile principles. He is well deserving of the title Master Agilist. Bravo sir!

  10. Ellen Lindermann says:

    This fantastic title caught my attention. Junk blogs about agile are everywhere on the net. This is NOT one of them. This post far exceeded any expectations I had when I started. It is intelligently written, yet easy to understand – theoretical, yet practical – well referenced, yet flows readily without the footnotes – philosophical, yet scientific – general in theme, yet specific in advice. I will have to read the rest of Mr. Lucas’s blog to see if they are all such gems!

  11. John Chandler says:

    This is quite well written. What is of far more importance is it’s practicality. I have experienced this issue with development teams as a scrum master. Your solution is clear and simple. Your reference to Shakespeare’s Falstaff is delightful.

  12. ElktonK says:

    Damn elegantly written 🙂 and informative about solving a serious problem in agile. Cheers!

  13. Jackie Tanaka says:

    You are a true master and teacher! You take the work of another master (Wirth) and apply it to a new situation (agile) in a manner consistent with its original intent. You teach the underlying reason and not just the surface instruction. I will follow your blog from now on. Thank you!

  14. Simon Henderson says:

    Very nice exploration of the subject!!!! I will follow your blog from now on.

  15. Betty Garnett says:

    Nice post Brian! Sometimes you loose me a bit, but I always feel smarter after I read your blog! lol 🙂 Betty

  16. Dr. Hanson Walker says:

    I would like to use this as an example in my class on agile development. I note in your rules that you permit this and will provide you and your blog with full credit. Thank you for an exceptionally competent and informed treatment on refinement in agile.

  17. Simone Rochelle says:

    Elegant post Brian! I would love to exchange thoughts with you. May I email you directly? Simone

  18. Paul Fortner, PMP says:

    This is a great explanation of iteration and stepwise refinement! I am distributing it to all my project teams. Thanks for sharing Brian!

  19. Darin Lawton, PMP CSM says:

    Brian thanks for posting this. We were struggling with some of the concepts of refinement and subsequent refactoring of code. This helped clear up some of our confusion. I have a few questions I would like to ask you. Can I email them to you?

  20. Arvid Ingesson says:

    Mr. Lucas, I appreciate your unified approach to agile. It must be a consuming passion of yours as it shows in all your writings. The manner in which you blend all the artifacts in an agile project lends strength and cohesion to the effort. I have come to have a tremendous amount of respect for your thinking!
    Arvid Ingesson
    Chief Innovator, [Company name deleted by administrator]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s