Category: Lean Software Development

Deconstructing Agile

To really understand something, we need to read beyond the surface, beyond the rituals and understand the underlying motivations. The Agile Manifesto for developing software isn’t prescriptive about methodologies. The manifesto only has four points:

  1. Individuals and interactions over processes and tools
  2. Working software over comprehensive documentation
  3. Customer collaboration over contract negotiation
  4. Responding to change over following a plan

There are many methodologies that teams can adapt to develop software in an Agile manner, Scrum, XP and Lean are amongst some of the more popular methods. Often teams adopt practices they like with varying degrees of success.

Software development is often over-complicated. The goal of creating software is to build something that people want. To determine what people want, we need to get customer feedback. Time spent creating things that the customer doesn’t value are a waste of time and effort.

The whole point of Agile is to release software more frequently because the more frequent software is released the sooner we can get customer feedback.

The objective of Agile is to optimize for customer feedback.

Without loss of generality, Scrum is really a compressed waterfall, one compressed into a 1 to 2-week “sprint”. When time is compressed, it puts constraints on a team, this is by design. By constraining the amount of time allotted to sprint planning, the team is forced to find other ways of describing and estimating work. They no longer have time to collect detailed requirements, write comprehensive design docs, or come up with detailed work plans and effort-estimates. The team is forced to optimize for what is deemed high-value for the customer. At the end of the sprint, the team gets feedback from the customer. At worse, they will have worked on something for 2-weeks that the customer doesn’t want, they can then pivot the very next sprint. This is pretty good, it’s unequivocally better than working on something for months or years only have the customer reject it.

If you try to optimize for customer feedback, you’ll notice that there’s still a potential 2-weeks worth of wasted time. The thing that gates customer feedback is a team’s frequency of releasing software. By releasing even more frequently, the team can get feedback even sooner. The feedback enables customer collaboration, a key value of Agile.

Lean can push this to the extreme. Assume developers are expected to deploy to production what they started on that morning. Now instead of releasing only at the end of a sprint, they release changes every day so that customer feedback can be solicited even sooner. This constraint means the changes deployed to production need to be small, small changes also limit deployment risk. Instead of a worst-case scenario of wasting 2-weeks, the worst case only a wasted day.

Doing stand-ups, sprint planning, kanban, kaizens, demos, velocity tracking, retrospectives, are just rituals, they don’t necessarily make your team Agile. I find many of these so-called Agile rituals downright wasteful. Understand the rationale behind the rituals and what to optimize for and it will lead to appreciation and enlightenment.

One Piece Flow

Many people don’t realize that Lean Software Development is based on the Toyota Production System (TPS). Toyota’s way of engineering cars is radically different than how most companies build cars. Instead of batch and queue (i.e. mass-production assembly-line), their manufacturing process is optimized for one-piece flow and pull-processes. Toyota’s way of building cars has demonstrated to produce higher quality products, increased productivity, faster time to market, and happier more accountable employees. Whereas assembly-line mass-production often results in over-production, lots of waste, and demoralized employees.

Achieving one-piece flow is about figuring out how to create a single quality unit as fast as possible while minimizing inventory, wait-time, transportation, hand-offs, etc. Through this process, the overall time from beginning to end is minimized to contain only all value-add steps. When there are defects in the manufacturing, the entire process must halt and the problem immediately fixed to ensure quality is built-in and there are no defects in the end product. Many organizations have problems implementing a true one-piece flow, it requires constant improvement, or kaizen.

Continuous Integration & Continuous Delivery (CI/CD) is the Toyota Production System’s concept of Flow applied to software development. At Toyota, the one-piece they’re making is a single car. In software, the analogous one-piece is a deployable software feature/task. Assuming trunk-based development and a modern containerized CI/CD setup, a developer typically writes code, runs unit tests locally and then commits it to a pull-request. As the pull-request is committed against, the new code is continously integrated and tested by the continuous integration system (i.e. built-in quality). After the pull-request is reviewed by a peer it’s permitted by the system to be merged into the master branch. At this point, a container is automatically built by the build system. Once built, the container is sent to a staging environment, and service tests are run against it. If they pass the tests, the container is blessed and deployed to production. Deployments to production must take minimal time but be fail-safe (i.e. blue-green), if they fail, they must rollback automatically. This process is a true one-piece flow, the build contains only the feature that is asked. If there’s something that breaks during any of the processes the developer is right there to fix it immediately. They don’t move onto another task until the feature is deployed and running successfully in production, they stay engaged the entire time.

There are pitfalls that people sometimes fall into while developing software. Sometimes people bundle multiple features into that deployable one-piece. Batching actually increases risk, when there’s more new code it increases the chance that something is defective. When something is defective, it now requires rolling back the batch of features instead of the single feature.

Sometimes people cut corners, they don’t write unit tests, integration tests or service tests. This is a false economy, you can’t have built-in quality if you don’t employ the right processes and safety nets. The engineers at Toyota say, “the right process will create the right results”. At Amazon.jp warehouses, they have a saying plastered everywhere, “safety is #1”. The same concepts apply to software development. The tests are the safeties.

There’s a lot of tooling that needs to be invested in to make one-piece flow work, sometimes people don’t spend the time investment to put the right machinery in place or continue to improve to make the processes more efficient (reducing waste). Good hygiene and exercise is necessary if you don’t want to get slow and fat.

Software can be complicated, the process of developing software doesn’t need to be.