By Brian Lucas
Agile concepts, techniques and technologies are much in demand today. Many organizations have astounding success with agile, while others experience failure, sometimes horrifically. Usually agile fails in these cases from improper application of the concepts or lack of supporting environments. While agile necessarily implies “rapid”, at least as far as iterations are concerned; it doesn’t mean poor quality. However, constant code changes to improve quality can bring a development effort to its knees; particularly when you are dealing with large scale efforts involving multiple teams. Refactoring[1], a concept that is embraced in agile, can include very negative maintenance aspects. After all you want to spend your resources developing vital new business functions not maintaining existing code without changing function. How can you build longevity and scalability into your agile efforts while minimizing refactoring and insulating them from technology changes? The answer is you need to adopt agile architectures!
There is a notion that strategic architectures and agile concepts are incompatible and that the I/T professionals in each camp are so different they cannot work together. On the surface this seems valid, but if you think outside the box, the solution is really only a matter of perspective and lies in the need to impose the requirement of supporting agility on these architectural constructs.
What is the concept of an architecture and why are architectures important. Let’s take a simple example. How many of you have ever put a puzzle together? Did you just start putting pieces together in any old fashion? Did you select a good location to put it together in? One where you could work and not have the puzzle constantly disturbed? Did you have help and were they helpful[2]? How long did it take you? Did you have to move pieces around a lot? Did this approach take a long time and was it frustrating[3]? Did you finally give up and throw the puzzle away[4]? Or instead, did you try to find all the straight pieces first and assemble the border? Did you sneak a peek at the picture on the box? Did you try to find all the pieces for the red barn and put them together next? If you didn’t put the barn together in the spot where it belonged, did it fall apart when you moved it and you ended up having to put major parts of it back together? If you put the barn and the stream together in the right spots didn’t it save you a lot of time and effort?
This is what good architectures are all about, particularly agile ones. They put the borders together to establish your systems’ flight envelops. They sneak peeks at the picture on the box to establish strategic enterprise vision for your information systems. They intelligently think about the environment where you are going to put the puzzle together to select the development and production environments and technologies that make your efforts easier and more efficient. Architectures are strategic guides which show you where elements should reside and shape your development efforts without causing onerous restrictions[5].
Architectures are generally wrapped in an enterprise architectural (EA) methodology. There have been a great many attempts at establishing various EAs since the concept was born over 20 years ago by John Zachman. The vast majority of firms today use either the Zachman Framework for Enterprise Architectures (ZFEA), the Open Group Architectural Framework (TOGAF), the Federal Enterprise Architecture or the Gartner Methodology (GM). Each has their own advantages and disadvantages. Not being a purist, I like to use a combination of three of them: the ZFEA which has the value of great multiple perspectives, the TOGAF which complements the ZFEA nicely and to which I add a fifth architectural construct I call a system architecture, and the GM which embraces the concept of where you are going is more important than where you are. For now just accept the fact that you can be eclectic with architectures and pick the good points of more than one and combine them. That is after all being adaptive, which is a principle agile concept.
In TOGAF terms, there are four complimentary architectures your organization needs to make any of your development successful. They are a business architecture, an application architecture, a data architecture and a technical architecture. I add a fifth called a system architecture. Put simply, a system architecture is the background or foundation in which you detail the structural implementation of an application architecture in order to meet the business architecture needs through technical and data architectures. Business, application, data and technical architectures are familiar to most CIOs and I/T personnel. It is the agile requirement and indeed imperative that makes them dramatically different from their historical representations. Agile is this case doesn’t mean that the architectures have to change constantly with the changing business needs. It means that the architectures are flexible enough to readily accommodate changing business needs without changing themselves. This gives stability to the development effort and minimizes or eliminates negative refactoring.
An agile architecture limits total cost of ownership by enhancing flexibility as a means of evolving its contents with changing business needs without changing its structure in order to eliminate negative refactoring.
A brief comment on technology; too many confuse technology and architecture. It is a fact that technology can provide a tremendous advantage and keeping current is generally a good idea. However, too many reach for technology as a solution to all ills, including bad design and weak architectures. Throwing more hardware at the problem or upgrading the software technology is a Band-Aid on the cancer that will eventually kill your application, your information system’s viability or worse your entire organization. Further compounding the problem, a poor architecture will often prevent you from maximizing the benefits of the latest technology.
So now you understand the importance of architecture and know that strategic architectures did not go out the window with agile development. In fact, they became more important and unfortunately more difficult to construct. Maximizing your success with agile software development means addressing agile concepts from an architectural perspective. To learn about the technical aspects of an agile system architecture read my article, Agile System Architectures. I will cover the other aspects for a complete formulation of an agile enterprise architecture in additional articles, and till next time, keep agile!
[1] Code refactoring in its purest form is a highly disciplined technique for restructuring an existing body of code, modifying the internal structure without changing any external behavior. This is driven by the need to improve aspects of the software that are not directly related to business function. This is done through refactorings, which are independent discrete changes to the code designed to correct an undesirable technical issue. Philosophically, refactoring’s advantages include improved code readability, reduced complexity, better maintainability and a more demonstrable internal architecture. I call this “positive refactoring”. Unfortunately, refactoring often is not discrete, involves large changes, propagates change across large numbers of objects or classes and spills into functional areas. When people say refactoring, they often include this second set of descriptions in their definition. I call this “negative refactoring”.
[2] Having the right organization structure or organization architecture is covered in my article, The Imperative of having an Agile Organization Structure.
[3] Does this sound like traditional software development?
[4] According to The Standish Group’s Chaos Report which surveyed 13,522 projects, unqualified project successes are only 34 percent to be exact and outright failures, where projects were abandoned midstream, are at 15 percent.
[5] Architectures that restrict development and don’t facilitate the rapidity and prototyping that are essential to agile are one of the reasons agile efforts fail. You are trying to be agile in a non-agile environment.
Keeping Agile™ with Brian Lucas 
This was article was spot on!!! I was forwarded it by a fellow employee. I have now forwarded it to our CIO. I am very impressed with your blog. You are the most refreshing writer I have read in a long time! – Sincerely Thomas Brighton
Thank you Thomas! I appreciate the compliment. When someone forwards my blog link or a post to a friend I take that as high praise.
This is pretty heady stuff. I would like to know how you develop these architectural layers if you call them layers. How they unfold and interrelate and so on if you are willing to share. The agile concept in architectures is a good concept, but I am unclear as to the mechanics. Still I must agree with my friend Tom that you have a very impressive blog. You should post more though.
As a professional enterprise architecture consultant, I am always on the lookout for new approaches. Agile has been a particular thorn is my side for a number of years. I really like your approach to dealing with agile and your slices of models that you use. I think the approach you are taking is genuinely brilliant. It is simple and its simplicity makes it work. I would like to buy any books or other whitepapers you have on the subject. If they exist can you direct me? Also if you have any webinars or podcasts I would like to view them please let me know. With appreciation – J. McCormic
John – Thank you for commenting. It is always good to hear from a fellow architect! I haven’t broken down and written a book yet on enterprise architectures. It is really an issue of time constraint. I realize that agile can be difficult for architects if you approach it traditionally. Let me give you an analogy. If you built a classic north east brick or stone home, designed to be strong in a powerful storm, in California it would collapse in the first bad earthquake. One of the ways that buildings are designed to survive earthquakes is a partial suppression of the seismic energy flow into the superstructure known as seismic or base isolation. Low friction pads are inserted into or under all major load-carrying elements in the base of the building which substantially decouple the superstructure from its substructure that rests on the potentially shaking ground. This is in fact an old technique with its first known use in Pasargadae, a city in ancient Persia in the 6th century BC. Decoupling connections in an architecture through mediation patterns works much in the same way. I would like to do a webinar on this subject if I get the time. The more people that demand one the quicker I will get to it so keep your requests coming! -Brian
Brian – Can you explain how your proposed structure contrasts with a matrix managed one? I am a bit confused. Thanks in advance!
Sorry for any confusion I will explore the differences between hybrid agile architectures and matrix ones in a coming post. In the meantime you can email me if you have specific questions. I try to answer them. -Brian
So very few blogger would take the time to email replies with such thought and detail as you sent to Sally. I wish I worked with someone as generous with his thoughts and ideas as you are. May you live a long life and prosper Brian. You are helping us all prosper. I respect your company for allowing you the platform and freedom to do this.
I had the same question as Brown did. Your reply to Sally cleared it up for me. Thanks for taking the time. This helped me out in the new architectural work I am doing. -Martin
Brian – why don’t you essentially post at least part of the email you sent me. It was a great explanation. -Sally
Good idea Sally here it is in brief.
I am going to throw out a quick blurb on this that will unfortunately be oversimplified. There are 5 types of basic management philosophies and structures (actually there are a lot more, but let us concentrate on just these): They are functional, matrix, composite, lattice and hybrid.
Functional (I generally call this territorial) management has been the most common type of organizational management. Here the organization is divided by specialty of function such as Sales, Production, Engineering Design, Finance and so on. This is a very vertically stacked organization that is purely hierarchical. Often this company is structured around product and/or geographic sales managers at the C level. The C level does all the planning and thinking with command and control meetings. Everyone else in the comp-any must be managed. Communication generally occurs downwards in a single department. Coordination of information and sharing of resources happens at the department head level. These structures tend to work in a waterfall method. Here information sharing is weakest. Political intrigue is usually abundant here.
Matrix management pools people with similar skills for servicing work assignments. Graphic Designers may all work in a graphics department and report to a graphical design manager. Workers are assigned to projects as utility pooled resources and any one employee can have many different managers assigning him work. Matrix organizations really are sheep in wolf’s clothing since they follow more traditional structure concepts with modifications based on project units. Here information sharing is somewhat improved. Conflict for resources and confusion of priorities and political intrigue are still common here and the environment must be closely managed.
Now it is becoming popular for organizations to have characteristics of both functional and matrix concepts. These are defined in the Guide to the Project Management Body of Knowledge (PMBOK) 4th Edition as composite organizations. Often this takes the form of a special project team structure that is assigned to high profile projects. These Project-centered organizations are structured around project or product teams as the functional units. Information is shared reasonably well within the team, but poorly across teams.
A lattice organization is structured so that work is directed, coordinated and managed by individual workers. There are no many layers of organization and management as in hierarchies. Workers have all the responsibility. This provides flexible and fluid work paths which are by their nature are remarkably efficient. It also provides a multitude of career paths. These structures allow workers to gain much knowledge and maximize their contribution to the organization. They have 2 overt problems in that they place great emphasis on the quality of the individual and have difficulty dealing with day-to-day functional aspects of an enterprise. There are some similarities with a results-only work environment (ROWE), but a lattice generally has issues revolving around fair compensation practices.
A hybrid organization is kind of like Chinese philosophy – it takes the best from a number of beliefs and leaves the baggage behind. It uses the strength of a hierarchy to run day-to-day operations and deal with personnel issues. It uses a product and services matrix concept to service work needs. Information flows through the structure mush like a true lattice organization. Confusion and lack of long term vision are eliminated by compositing long term visioning with real time planning and measurement in a composite fashion where strategic planning maintains its vibrancy with day-to-day tasks and activities.
Your approach to agile architectures is completely convincing and brilliant. This is what has been missing from so many agile efforts I have worked on. I have worked on a large projects controlled by taking a scrum of scrums approach. These would have benefited greatly from your architectural concepts. I would have a few questions for you if you would permit me to email you directly. Regards Gunther
Gunther – by all means email me and we can start a dialog. -Brian
Gunter and Brian – would you mind copying me on your email stream? -Sally
Hello Brian,
This is a fantastic post it strikes just the right level for the CIO. Your explanation brings much clarity to what is often a confusing subject. I know firsthand that refactoring can be a serious problem in agile. At first I thought that when you stated, “Put simply, a system architecture is the background or foundation in which you detail the structural implementation of an application architecture in order to meet the business architecture needs through technical and data architectures.”, you were displaying some dry humor since it is a complex sentence. When I thought about it more, I realized that you really were explaining all of enterprise architecture in the simplest terms. That was quite perceptive and deep. Your other statement, “An agile architecture limits total cost of ownership by enhancing flexibility as a means of evolving its contents with changing business needs without changing its structure in order to eliminate negative refactoring.”, I thought was quite profound, but I would like to know how specifically you are accomplishing this. I looked for your article, Agile System Architectures, but I could not find it anywhere. Can you please let me have a copy? It would be greatly appreciated.
Sincerely Laura
Thank you Laura – I did put some considerable thought and effort into the wording of those statements. I have emailed you the draft copy of my whitepaper Agile System Architectures. Please email me back your thoughts and questions. -Brian
Hey can I get in on reviewing that draft? It sounds like it would be a great read!
Brian – another great technical post from the IT side of agile and one that I can understand. I love how you are able to take a complex subject like this and explain it in a way that a person like myself who is not a developer can understand. I am getting quite an education reading your blog and appreciate the contribution you are making to the effort to transform outdated organization structures into agile ones. – Sally
Sally – If all Sally – If all consultants had the tenacity and grasp you have the world’s organizations would be in much better shape. Keep leading the charge in changing monolithic and archaic hierarchies into hybrid agile ones. Time and external market forces are all on your side! And thank you for your very kind email! -Brian
I much perfer informative articles like this to that high brow literature.
Can I just say what a relief to find semoone who actually knows what theyre talking about on the internet. You definitely know how to bring an issue to light and make it important. More people need to read this and understand this side of the story. I can’t believe you’re not more popular because you definitely have the gift!
Brian have you developed this subject any further? Specifically I am looking for how you develop these models and how they fit into the user stories.
Hey Brian as a fellow software architect, I have to tell you this is really good stuff!!! Do you have any more on this subject? Have you done and webinars or have any books in the works?
This was very informative for me as a CIO struggling with enterprise architectures and agile development. We have developed such a disconnect with our architectural group and the agile development since we started over 6 years ago, that our architectures are no longer relevant. Neither executive management nor the user community want to give up the rapid development for architectural compliance. I did find your puzzle analogy effective in explaining the need for enterprise architecture to both the user community and the executive management. But your post here doesn’t give an actual example of how to formulate an agile architecture and that is what we are struggling with right now. Any help would be appreciated.
I think I get this you are using to follow your puzzle analogy a global picture to imply where applications probably based on data should reside in the enterprise and a technical what you call system architecture to frame your utility component and add abstraction to that layer of the software. Is that right?
Nice exploration on making architectures agile. Wish you’d write more, but I guess we would have to pay for it. Still I have to say you have more here about agile architectures than I have read anywhere else so thanks for sharing with us fellow software architects.
I had not seen this approach before anywhere. As a CIO, I find this type of information valuable since it can directly affect the bottom line. Thank you for sharing.
I find your approach very progressive and illuminating to architecture. Enterprise Architecture has lost some of its relevancy in recent years because it is not responsive. You appear to have built responsiveness into the design. That is a brilliant solution. My work as an architect is somewhat paralleling yours, though I believe your thinking is further developed than mine. I would appreciate the opportunity to email you and begin a discussion on your concepts of agile architectures. May I contact you?
Superlative points about how to approach agile architecture in a strategic fashion. I hear too many claims from the ignorant that agile can’t be be in some projects. Agile can be used anywhere if you have really mastered its basic principles. Fantastic post Brian!
Brian do you have any more written on this subject? I would very much like to learn more about this for the architectural work I am doing for our company.