Is Agile a Fad?

I attended today's Agile Denver meeting - this time in Boulder - to hear Mary Poppendieck's presentation, Is Agile a Fad?

I'll summarize some of her material here, mostly in reverse order, starting with the key points and conclusions of the talk, followed by some of the contextual info she presented leading up to those conclusions.

The key to successful development organizations

The key to a successful development is for engineers (developers) to have a deep understanding of their customers - both internal and external. When new engineers start at Toyota, they spend their first 6 months on the production line assembling cars, so they fully understand their internal customers. Then, they spend 6 months working for a dealership - selling cars - so they know what customers really want.

Another key is building a culture that retains quality people for the 6-10 years it takes to build true expertise, and growing leaders from within the organization.

Fads vs. enduring principles

Why do we have these fads that fail?

• silver bullet thinking. there is no silver bullet
• trying to apply 1 solution in different contexts. different contexts require different solutions.
• essential tensions in software. Don't swing too far toward one side or the other; rather find a solution that solves the valid concerns on both sides of the issue.

The principles behind systems engineering are robust over time. The concepts in project management are fragile over time.

Systems Engineering	Project management
low dependency architecture	complete requirements
quality by design	quality by testing (at the end)
technical disciplines	maturity levels
respect for complexity	scope control
skilled tech leaders	resource mgmt
learning & feedback cycles	timeboxes
success = accomplishing system's objective	success = accomplishing planning scope, cost, schedule

5 essential tensions in development

• People. self-managed vs. managers. Answer: the servant leader facilitating self-organization.
• Process: empirical vs. defined.Solution: relentless improvement, rigorous process for effective improvement. Identify the true root cause of problems, hypothesize solution, determine how to measure if it succeeds.
• Product: development team vs. customers & biz operations. Solution: whole team philosophy. team talks to customers so they understand the problem deeply.
• Planning: evolving plans vs. predictability. Solution: pull scheduling, set-based design (build multiple options), clear technical vision
• Performance: concern only for the next iteration vs. long-term scope, schedule & cost. Solution: a team with pride & passion that delights customers - and has a deep knowledge of it's customers' needs, sustainable profit, breakthrough innovation

A brief history of software methodologies and the seeds of agile

What happened to all those methodology buzzwords? RAD, lean, structured programming, etc.? Sprinkled throughout the history of software, various people discovered and promoted practices that we call agile & lean today. They also promoted various practices that were unsuccessful fads.

1968

NATO conference on the software engineering crisis. Edsger Dijkstra said that programming became a problem in relation to the size and complexity of computer hardware. Douglas Ross of MIT said the most deadly thing is the assumption that you can specify what you're going to do, and then do it. The solution (compared to assembly languages): high level languages! (Cobol, Fortran, etc.). This removed drudgery, but increased the level of complexity possible, which led to the same problem all over again(Dijkstra).

1972

New York Times software project

• structured programming made software more readable. Dijkstra proposed quality by design, as opposed to reliance on testing.
• Dave Parnas devised information hiding, the concept of objects.
• Top down programming was introduced by Terry Baker - basically this was the concept of continuous integration.
• The "chief programmer team" concept introduced the tech lead, design review, pair programming, common code ownership. Result was 100 times more productivity (measured in LOC) and higher quality than typical at the time.

1976

Barry Boehm proposed that software maintenance was becoming the most expensive part of systems, and that the cost of changes got exponentially greater in later phases of the lifecycle. This famous (infamous?) curve was the key reason that everyone tried to nail down all the requirements at the beginning.

1982

• Daniel McCracken & Michael Jackson wrote that the lifecycle concept (waterfall) was harmful and perpetuates failure by constraining thinking, and ignoring the reality that needs inevitably change over time.
• James Martin wrote the 4th gen languages would allow application development without programmers. (A gross oversimplification of the inherent complexity.)

1984

• Scott Schultz at DuPont introduced timebox development. 30 days for analysis & design, 90 days to develop. He called it rapid iterative production prototyping.

1988

• Boehm introduces the spiral lifecycle model. More evolutionary model, but it's still a project management model, not a systems engineering model.
• Watts Humphrey introduced software process maturity model (CMM). Attempt to bring in statistical process control and mandate maturity assessments. Focus on project management practices over system engineering practices.

1991

• James Martin wrote book on RAD, facilitated by CASE tools. (Where are those CASE tools today? Anyone?) Problem: RAD often produced un-maintainable code. Didn't live up to the hype.

1995

• Internet booms. J.C.R. Licklider serves as the technical visionary for several key internet organizations. Standards were developed.