Software development projects need a starting plan,"Managing software development -- where plans are inadequate but planning is indispensable. Today's problems come from yesterday's solutions." -- Elmer Wheeler
and also a process to appropriately adjust that plan.
Why is the software industry, which has a collective success rating equivalent to most batting averages, slow to change how it develops software?
Much has been written in the last few years about lightweight and agile software development methodologies and the increased effectiveness they provide. Based on my 20 years' experience developing and managing software projects, I know that these approaches work. I have used them in one form or another long before they were identified and named. I know they work because in the end, it's the only way to develop and deliver software. I'm puzzled by the lack of enthusiasm in adopting them. For complex projects, I know the "waterfall" approach, complete with one-size-fits-all bloated processes, doesn't work. I have "lived the lie" of doing my job in "lightweight" fashion while simultaneously acting as if I was using a "heavyweight" methodology.
As I continue to "live the lie," I feel compelled to understand why proven methods are slow to gain acceptance. It is not my intent to restate the definition of new development methodologies but instead to make some comparative examples of old and new, to gain insight about what makes these hard to accept. (If you are not familiar with lightweight methodologies, I recommend the excellent book by Alistair Cockburn, Agile Software Development.)
The lie I live is complying with the overfed Capability Maturity Model (CMM) type processes, and derivatives thereof, which model 19th and 20th century industrial engineering processes. These processes are based on sound physical principles like the law of physics and the theory of relativity. For example, there are three preexisting conditions that support the practice of developing a detailed architectural design of a bridge before it is built. The first condition is that everything that needs to be known about what it will take to build it, how it will be built and how it will be used, is known beforehand. Second, construction is significantly more expensive than design and the cost of construction mistakes can be enormous. So it makes sense and saves money to design it first.
Third, the design is reviewed and proven based upon well-understood engineering principles and can be used to develop a rigid, step-by-step construction plan. The plan in turn can be followed, without deviation, to measure progress and manage construction to completion. The plan is essentially the same for all bridge building. This works when building something completely physical, based solely on laws of physics. The people and workers are important but they are subordinate to, and driven by, the process.
Comparatively, software development requirements of gathering, analysis, design and coding are all discovery-based exercises that lack absolute physical laws of nature to impose constraints, boundaries or guidelines. They are uniquely human, ephemeral activities that are unpredictable, immeasurable, ill defined and non-repeatable. There are rules of thumb, best practices and experience to guide the undertaking but nothing can be "proven" until the software runs. Every development effort will have varying dynamics requiring commensurate adjustments in methods used to manage them.
The flexible methodologies are structured to recognize, accept and feed information uncovered during discovery back into the process. They are all about flexibility, response to change, and enabling the creative process. They model collaborative planning, employ some form of time boxing, and have controls. But they are structured to support organic growth by responding to change continually until the natural process is complete. The barometer of progress and success is calibrated based on working software as opposed to document creation. Lightweight methodologies are inherently flexible enough to be tailored to each project depending upon the characteristics of each.
So why have I found it difficult to implement these methodologies? My guess is management, in general, doesn't get the same "comfort level" they are accustomed to with well-defined linear project plans. I believe there is a perception that agile processes lack stability. A detailed step-by-step project plan, embodying a sequential list of contingent activities that cumulatively map to a desired outcome, is clear, concrete and certain. It provides assurance and the perception of certainty.
In practice however, the existence of such detailed plans has not guaranteed success. That is because the probabilities of success are significantly weighted towards the human capabilities of metastasizing a business need into software. Capabilities, which are singularly more important than all the processes, plans and systems, put together, are also unpredictable enough to render linear planning unreliable.
Lightweight methodologies are perceived as providing little control -- the ontology of reinventing rules, response to change, adjusting the plan and being "agile" gives the impression of being "willy-nilly." The mindset of continual response to change gives the sense that things are unplanned, or results are unpredictable, unreliable and uncertain. It is this perception of less control that makes it harder for the new methodologies to take hold.
The truth, however, is that software development is encircled by uncertainty, and the best form of control is to recognize the uncertainty, accept it, manage it and use it to navigate your way to completion. We all know that significant change in behavior usually cannot be realized until you are willing to give something up. Hiding uncertainty with an overstuffed project plan is what we should be willing to give up.
We could learn from what former President Eisenhower recognized over 50 years ago. In speaking of D-Day, he said the invasion was planned for years. The purpose of planning was to assess alternatives, approach, resources and timing. He said the most important aspect of the plan was how to structure the attack so as to keep lines of communication open from the foxholes to headquarters.
The reason that was so important, he said, was because as soon as the first shot was fired, the battle would take its own course and the rest of the plan would be history. He knew events would be unpredictable and the only hope of success was his ability to know what was happening, to be able assess the situation, and to respond accordingly.
Like D-Day, software development projects need a starting plan, then a process to appropriately adjust that plan. The leader(s) must be informed continually by great communications and have the wisdom and judgment to know when and how to react to inevitable changes.
I think we know this. But knowledge is only useful, when put to use.
Ed Carroll is the director of U.S. Development at Trintech Inc. [email protected]