Modern Software Development: An Agile Failure

SR-71 Assembly @ The Skunk Works (Wikipedia)

Over the last 35 or so years I have seen a lot of strategies for developing software come and go.

When I was in my twenties many of these strategies were encapsulated in books by "famous" authors.  Perhaps what started it all (circa 1975) and turned out to be the most famous was "The Mythical Man Month" by Fred Brooks.

Brooks wrote about his observations at IBM during the creation of OS/360 - the computer operating system for IBM mainframes introduced with the IBM 360 series of computers.

A key observation by Brooks was "adding more programmers to a project that's already late will make it later," i.e., adding man power does not necessarily get things done quicker.  (Those familiar with, for example, roofing find this counter-intuitive.  I have have 20 squares of shingles to install and I have two roofers I can easily add a third roofer and get more done.)  But this is not necessarily so with software.

The reason is that unlike roofing each piece of software to some degree "interacts" with the other.  So imagine shingles not organized as "all the same" but instead as a puzzle.  Now I cannot simply pull any old shingle out of any old package of shingles and nail it in place.  I now have to inspect that shingle, inspect other shingles, inspect the work and placement of already installed shingles, all before putting my piece in place.

In this context adding a person may make it take longer as anyone who has had inept family members work on their jigsaw puzzle will tell you.  Usually someone takes the lead, develops the strategy (perhaps of sorting pieces by the edge type or color), organizes the work, and gets the job done.

Software is actually an even more complex process in that pieces of the puzzle may be unknown in some way - how they will fit in, what exactly they should do, and so on.

Resolving these issues requires communication between the various parties involved.  However, sometimes a party might make an "assumption" about what another party is doing - either explicitly or implicitly (perhaps without even realizing it).  When its time to connect two pieces about which each side has made assumptions we often find the assumptions don't match up and the pieces don't fit.

Over the last 35 years various strategies have been developed to resolve these issues and make software development more "effective."

During the last sixty or seventy years we can look at similar engineer processes, e.g., airplane building.

Airplanes are as large and complex as any software system (the old adage is that no modern airplane ever built was strong enough to lift off the ground carrying its own documentation).  In the 1940's, 50's and 60's airplanes were built largely by hand - with wood frames for bending metal, slide rules to do the calculations, and drafting boards to create blueprints.

By today's standards this seems primitive but these technologies built airplanes like the XB-70, Concorde, SR-71 Blackbird, F-104 Star Fighter, X-15, 747, and many, many others.  Airplanes which in many cases provided advancement and/or features that today's airplanes would be hard pressed to match both in terms of capability as well as time frame from initial concept to flying prototype.

How can this be so today?  Everyone has advanced degrees, computers in their pocket that exceeded the capacity of all the computers on the planet in 1957.

What's wrong?

If we look at software engineering we see the current "model du jour" of software engineering is "Agile Development."  The idea with this is we "iterate" over some model of what the software should do very quickly.  Work a week or two based on some loose specifications and deliver a prototype.  Get some feedback, work a week or two more, deliver another prototype, and so on until the job is complete!

Basically development by committee.

Early modern jet aircraft, as a counter example, were designed by often by a individual who was able to fully envision the task the plane was too address.  By "fully envision" I mean to literally draw out on a few sheets the entire concept and get more or less "right" the first time.  Further, the process to build the plane was tightly controlled by a strong, able leader.  (See this for the rules of the Lockheed Skunk Works.)

These people had all the experience to innately "know" they were designing something that could be built and would fly.  Innate experience that comes from "working your way up the ladder" in a place like the Skunk Works: knowing the right kind of metal existed and would work, understanding stress innately rather than only through some simulation, and so on.

But today we no longer have this kind of strength or depth in knowledge or leadership - particularly in software.

Today specialization is so strong that many programmers never speak to a customer - or have no idea how things will be used - or have no concept of "product life cycle."  So while they may be "highly educated" they know little about what needs to be done.

Often they are more concern with what others will think or feel about the work being done than actually doing the work right.  They are more concerned with their own careers than properly performing the work for the project.

Things like Agile Development simply hide this from the customers view.

Ultimately the real problem is that software engineers are just that - software engineers.  What do they know about, for example, flap controllers on an airplane.  For a software engineer to build a piece of software to do the "right thing" for the flaps on the 747 they have to "know" as much as an aerodynamic engineer.

If they don't and don't take into account complex issues like, for example, stall speed, then the functionality of their software will be deficient.

So in truth the only way to write good software is to actually understand what you are doing in the domain the software will work in, e.g., airplanes, or music, or print, or whatever.

But today certainly no younger people have this depth of skill.  They've only learned one thing at school: writing software.

So now they have to learn about what they are doing in some other field in order to properly create software for that field.

And that's where things are going bad, at least from my perspective.

Modern software "kids" don't understand any sort of "big picture."

And the products they build suffer because of it.

And "strong leaders" are discouraged as well - they stand out and often are at "odds" with the community development concepts - strong leaders have strong ideas about how things should be done.  Typically this involves "vision" - something a committee does not usually have in abundance.

With the explosion of computers in our lives (installed in everything from refrigerators to cars to razors) there is a constant demand for "programmers."  But what's turned out by schools is usually not up to snuff in terms what industry needs.

I think that "Agile Development" is the epitome of this failure.

Hire someone to run your project that knows what they are doing rather than iterate over "guesses" about what should be done.

Often in Agile environments people with actual knowledge of the problem to be solved are viewed as impediments to the Agile process.   This is because actual knowledge may point to a solution that does not fit rapid iteration, e.g., the problem requires complex software to be developed based on something like the laws of physics.  NO amount of iteration will change these laws so you have to bite the bullet and deal with it.

If no one knows enough to run the project then someone will have to learn.

This is yet another massive failure of the US educational system - probably on par with the same failures I have written about in medical research.

We need to wake up and smell the coffee...