By Brian Lucas
“Facts are stubborn things; whatever may be our wishes, our inclinations, or the dictates of our passions, they cannot alter the state of facts and evidence.” – JOHN ADAMS
What can you do when someone honestly doesn’t see the real underlying differences between agile and waterfall concepts? What do you do when you have tried all of your standard explanations and you both still can’t see things from a common perspective? The answer, of course, is – adapt! If what you are saying isn’t convincing; then try something else. Inject something small that’s different into the dialog and see what impact it has. Facts are notoriously unrelenting and will be acknowledged by any reasonable person. Never let your emotional frustration provide an answer that should have come from your dispassionate logic. You need to understand your subject and respect the person with whom you are dialoging. You will find that articulating proofs of this nature will actually strengthen your knowledge; much like defending a doctoral thesis. So to prove that I practice what I preach; I am including the following email stream. It’s a dialog I am having with a very good person and friend of mine named Jim. He is both very intelligent and a phenomenal worker and just happens to be an agile “semi-skeptic”.
Jim: “So what is agile? Is it a methodology or a framework?”
Brian: “Agile is, in its deepest and most profound sense, a philosophy of intelligent adaptation to constantly acquired knowledge and changing circumstances. Agile is solely driven and measured by business success. Agile is not simply a development methodology. Ultimately, all aspects of the enterprise from strategic planning to the most atomic level tasks must embrace agile for optimal effect. People that are looking for a fixed methodology or a heavily controlled process for agile will always realize less benefit – if not experience downright failure. The key words here are adaptation and success.”
Jim: “I don’t know, I just see agile as waterfall with a twist just in smaller iterations.”
Brian: “It is unfortunate that you see agile as waterfall with a twist, since the agile concept is actually the antithesis of waterfall principles. I am sure your view will change over time and with exposure.”
Jim: “Both Agile and Waterfall, to me, seem to have a large number of small implementations vice one big one. Both establish a time box and ask “what can you do within this constraint?” I think that Agile compresses the time box and addresses one of the main problems with waterfall — when you are ready to deliver, the business or marketplace could have changed so much that it is no longer relevant. Waterfall puts the emphasis on documentation up front under the basis that it costs more to redesign and fix a solution than it does to build it correctly in the first place. Agile says it costs more to build an outdated solution. I think it also de-emphasizes (but does not eliminate) documentation – always a popular notion. (Whenever I get the urge to document, I lie down until it goes away…) Interestingly enough, when you Google “similarities between agile and waterfall methodologies” you find an article on the IBM website that describes Agile as “Waterfall 2.0”… As with all methodologies, neither is correct for every project and both will probably continue to have their place alongside the other myriad of methodologies.”
Brian: “Jim, I am sure you are aware that all-that-is-written-is-not-gold! Actually waterfall and agile are not similar in the way you might think. Waterfall does not normally embrace the concept of a time box except for gross purposes. Estimation techniques like function point analysis are based on having a detailed requirements document completed. Estimates are often not made before then.
Another big difference is that who is doing what and when changes dramatically between waterfall and agile. In waterfall methods, analysts create requirements up front, followed by designers who transmogrify that output into more documentation, followed by developers who further interpret that documentation into code and then add more documentation to the code, followed by testers that interpret all the previous documentation and create more documentation, followed by help writers who – guess what – and last but not least the implementers. This is a massive amount of documentation that is virtually never kept in sync and up-to-date and interpreting the written word always leads to confusion.
Putting that aside for now, in agile, a team does most of the activity together – not an individual. The team directly shares knowledge and gains a common understanding, first hand, rather than through documentation interpretation. The team, usually an analyst, a user, a developer, a tester and a UI designer, work together on user stories, acceptance criteria, UI design (usually wireframes) and test cases. When the developer actually starts coding, the need and the proposed solution are so well understood in a common contextual framework that miscues are rare. Even if they occur they are limited in scope to the duration of at most a single iteration and usually are caught in the daily scrums.
Furthermore, agile really concentrates effort on a single unit of work-in-process. This focus increases the efficiency of the effort and generally the quality of the results. But let’s move right on to methodology coexistence. The fact is waterfall is dying a not-so-slow-death. Agile beats waterfall by almost a 3 to 1 margin. Enterprises have abandoned waterfall in favor of agile according to a Forrester Global Developer Survey in 2009 and the disproportionality of the numbers is increasing all the time as you can see in the following figure:
My last argument (for now) is that knowing you as I do, I believe you actually always operate and operated in an agile fashion and really don’t think in any other way. I have observed that some people, who are natural agilists, just like some people who are naturally object-oriented, can’t really conceive of another way of thinking. So I suspect that your view of waterfall is skewed by your own natural agile tendencies.“
Jim: “In ancient times – I’m thinking of the early 1990’s — I participated with a group of people to do an in-depth study of various methodologies. I came away from this with a belief that DATA is the key to it all. A company needs to know how profitable it is. No matter what company and what product you are talking about, the answer is the same. PROFIT = SELLING-PRICE – COST-TO-PRODUCE. It is the definition of these terms for the specific company that makes all the difference. The cost-to-produce may include all kinds of indirect things, for example – just to be a little silly about it – the cost of putting batteries in the automatic paper towel dispensers. If you don’t catch that with a robust design, over the course of the years this dollar leak will accumulate until a lot of money is missing and no one will know where it went. To make my point, I purposefully chose a silly example (unless of course your business is to replace batteries in paper towel dispensers.) But my feeling is that one of the shortcomings of Agile is a robust data design up front. It would seem to me that you would always be updating it as new data points are added. This issue is nothing new in the IT industry, but Agile makes it happen more often. More changes mean more problems means more disruption to my users. And what do I do with my 100K rows of sales data when a new “required” field is suddenly added??.”
Brian: “Jim, I agree with you that data is important it is one of the critical defining factors of a system. However, agile does not mean no data design; it merely means that you don’t spend a tremendous time up front attempting to figure out what your data design should be when you know the least amount about the solution.
I have seen a tendency in data modelers that like to work in a waterfall fashion to stuff in everything including the kitchen sink in their data models in an attempt to make them encompass even bizarre user circumstances. Just for fun I once did an inventory of data bases at a company I was consulting at to see how much data was in each field and furthermore did a database analysis to see how often it was accessed. Well I found that only about 50% of the structures had most of the data, but the real kicker was when I looked at the actual usage statistics. Sure enough – only 22% of the fields were accessed on a regular basis; 41% were accessed infrequently and 27% were virtually never accessed after original population. The Pareto Principle struck again.
Another direction I have seen is where modelers design an overabundance of abstraction into the data model. This pushes basic relational database management system (RDBMS) functions up into the application program layer. The resulting program code then ends up doing what a RDBMS was designed for which complicates the code and generally results in poor performance. With this approach there is also a marked lack of functionality in the application and the user ends up having to do a significantly greater amount of work per transaction. You mention disruption to your users – nothing is more disruptive to your users than a solution delivered late and with a bloated design that requires day in and day out more effort on the user’s part than is absolutely necessary.
From a data design perspective, I have been working on (in the last several years) a “functional engine design concept” for the data layer of agile developed systems. This builds robustness into the solution without sacrificing performance or ease of use. It enables customer driven extensibility within the domain of the solution. This means the RDBMS can do what it was designed for, program code can focus on business logic and the user gets to have things their way. Best of all, from an agile perspective, It can be introduced in an iterative fashion and grow virally into an existing system taking over a dysfunctional design in a piecemeal fashion. So to sum up, agile doesn’t mean no data design up front; it means a smarter data design that evolves as your knowledge of the solution increases.”
James has taken over Jim’s skeptical side of the discussion. He is not the same person.
James: “Brian, I am glad you agree that data is an important and even critical defining factor of a system. The question in agile is where you start and how you proceed? In agile how do you go about identifying what data you need? Data modeling gives designers and developers a guide to analyzing the data needs of the system. This helps to make the development requirement more complete. In agile you must be always adding new data that you end up needing that you missed in your initial pass. This requires model changes and code rework.”
“James, first of all let me welcome you to the forum and thank you for taking over Jim’s position. Let me deal with your arguments point by point.
- The question in agile is where you start and how you proceed?
Answer: You start identifying data in agile based of the functional needs of the user story.
- In agile how do you go about identifying what data you need? Data modeling gives designers and developers a guide to analyzing the data needs of the system. This helps to make the development requirement more complete.
Answer: As you progress this thru wireframes, you actually get something very concrete to work with rather than having esoteric conversations with users about their data needs. I am sure you will have to admit that users can never guess all of their data needs off the top of their heads. It is when they actually start working with a solution that the needs become clearer. This is one of agile’s great advantages.
- In agile you must be always adding new data that you end up needing that you missed in your initial pass. This requires model changes and code rework.”
Answer: If you take the life of an agile project and compare it to the life of a similar function sized non-agile one you will find that actual rework is less in the agile one. This is because most changes happen when there is less code and it is simpler. It also is a result of the fact that there is much less unneeded code developed in an agile project than a traditional one. This code can be as high as 80% of the code base and even though it is unneeded once created it still needs to be maintained.”