Deconstructing Agile

To really understand something, we need to read beyond the surface, beyond the rituals and understand the underlying motivations. The Agile Manifesto for developing software isn’t prescriptive about methodologies. The manifesto only has four points:

  1. Individuals and interactions over processes and tools
  2. Working software over comprehensive documentation
  3. Customer collaboration over contract negotiation
  4. Responding to change over following a plan

There are many methodologies that teams can adapt to develop software in an Agile manner, Scrum, XP and Lean are amongst some of the more popular methods. Often teams adopt practices they like with varying degrees of success.

Software development is often over-complicated. The goal of creating software is to build something that people want. To determine what people want, we need to get customer feedback. Time spent creating things that the customer doesn’t value are a waste of time and effort.

The whole point of Agile is to release software more frequently because the more frequent software is released the sooner we can get customer feedback.

The objective of Agile is to optimize for customer feedback.

Without loss of generality, Scrum is really a compressed waterfall, one compressed into a 1 to 2-week “sprint”. When time is compressed, it puts constraints on a team, this is by design. By constraining the amount of time allotted to sprint planning, the team is forced to find other ways of describing and estimating work. They no longer have time to collect detailed requirements, write comprehensive design docs, or come up with detailed work plans and effort-estimates. The team is forced to optimize for what is deemed high-value for the customer. At the end of the sprint, the team gets feedback from the customer. At worse, they will have worked on something for 2-weeks that the customer doesn’t want, they can then pivot the very next sprint. This is pretty good, it’s unequivocally better than working on something for months or years only have the customer reject it.

If you try to optimize for customer feedback, you’ll notice that there’s still a potential 2-weeks worth of wasted time. The thing that gates customer feedback is a team’s frequency of releasing software. By releasing even more frequently, the team can get feedback even sooner. The feedback enables customer collaboration, a key value of Agile.

Lean can push this to the extreme. Assume developers are expected to deploy to production what they started on that morning. Now instead of releasing only at the end of a sprint, they release changes every day so that customer feedback can be solicited even sooner. This constraint means the changes deployed to production need to be small, small changes also limit deployment risk. Instead of a worst-case scenario of wasting 2-weeks, the worst case only a wasted day.

Doing stand-ups, sprint planning, kanban, kaizens, demos, velocity tracking, retrospectives, are just rituals, they don’t necessarily make your team Agile. I find many of these so-called Agile rituals downright wasteful. Understand the rationale behind the rituals and what to optimize for and it will lead to appreciation and enlightenment.

One Piece Flow

Many people don’t realize that Lean Software Development is based on the Toyota Production System (TPS). Toyota’s way of engineering cars is radically different than how most companies build cars. Instead of batch and queue (i.e. mass-production assembly-line), their manufacturing process is optimized for one-piece flow and pull-processes. Toyota’s way of building cars has demonstrated to produce higher quality products, increased productivity, faster time to market, and happier more accountable employees. Whereas assembly-line mass-production often results in over-production, lots of waste, and demoralized employees.

Achieving one-piece flow is about figuring out how to create a single quality unit as fast as possible while minimizing inventory, wait-time, transportation, hand-offs, etc. Through this process, the overall time from beginning to end is minimized to contain only all value-add steps. When there are defects in the manufacturing, the entire process must halt and the problem immediately fixed to ensure quality is built-in and there are no defects in the end product. Many organizations have problems implementing a true one-piece flow, it requires constant improvement, or kaizen.

Continuous Integration & Continuous Delivery (CI/CD) is the Toyota Production System’s concept of Flow applied to software development. At Toyota, the one-piece they’re making is a single car. In software, the analogous one-piece is a deployable software feature/task. Assuming trunk-based development and a modern containerized CI/CD setup, a developer typically writes code, runs unit tests locally and then commits it to a pull-request. As the pull-request is committed against, the new code is continously integrated and tested by the continuous integration system (i.e. built-in quality). After the pull-request is reviewed by a peer it’s permitted by the system to be merged into the master branch. At this point, a container is automatically built by the build system. Once built, the container is sent to a staging environment, and service tests are run against it. If they pass the tests, the container is blessed and deployed to production. Deployments to production must take minimal time but be fail-safe (i.e. blue-green), if they fail, they must rollback automatically. This process is a true one-piece flow, the build contains only the feature that is asked. If there’s something that breaks during any of the processes the developer is right there to fix it immediately. They don’t move onto another task until the feature is deployed and running successfully in production, they stay engaged the entire time.

There are pitfalls that people sometimes fall into while developing software. Sometimes people bundle multiple features into that deployable one-piece. Batching actually increases risk, when there’s more new code it increases the chance that something is defective. When something is defective, it now requires rolling back the batch of features instead of the single feature.

Sometimes people cut corners, they don’t write unit tests, integration tests or service tests. This is a false economy, you can’t have built-in quality if you don’t employ the right processes and safety nets. The engineers at Toyota say, “the right process will create the right results”. At Amazon.jp warehouses, they have a saying plastered everywhere, “safety is #1”. The same concepts apply to software development. The tests are the safeties.

There’s a lot of tooling that needs to be invested in to make one-piece flow work, sometimes people don’t spend the time investment to put the right machinery in place or continue to improve to make the processes more efficient (reducing waste). Good hygiene and exercise is necessary if you don’t want to get slow and fat.

Software can be complicated, the process of developing software doesn’t need to be.

Thinking Distributed

The best analogy I can think of for how to think about scaling an organization to think distributed is the contrast between communism and capitalism methods of government.  Objectively speaking, one is not necessarily better than the other, but in practice one scales better in our current world.  

Communism is centralized command-and-control, it is a non-distributed organization.  In theory, if you had perfect knowledge and ability to react in real-time, there would be minimal waste and demands would always be met with the right amount of supply.  In practice, it hasn’t played out this way, probably because we don’t live in a world of perfect data and instantenous data processing abilities.  Capitalism in contrast, relies on that invisible hand, it argues that in efficient markets, supply-and-demand will balance itself through price discovery.  There might be some degree of loss in the beginning but it will come to an equilibrium fast and the overall oversupply or undersupply will be minimized.  

In capitalism, when the government wants people to spend money, they lower interest rates.  Instead of directing, they guide.  They create incentives for the market to move in the desired direction but they don’t direct.  In communism, there’s a required a feedback loop to central command so it can process the data and tell each type of producer the next steps.  This often leads to either over-production or under-production because it’s easy to have incomplete information or data processing delay. 

Leading a distributed organization is similar to working with monetary policy.   We want to create the right incentives, we want the default thing to be the obvious thing.  We want to build the necessary tools and safety-nets so that people can focus on their objectives and not worry about duplicating work.  We want teams to operate independently, instead of relying on upper-management (central-command) for next steps. 

Leadership cannot scale to have perfect knowledge of everything teams do, otherwise the leadership team ends up having massive reporting apparatus just to track work output and teams will end up idle waiting for the next instructions. Instead leadership needs to empower teams to want to act in their own best interests because those interests.  Leadership can guide teams by helping them clearly define KPIs, but how they go about achieving those KPIs should be left in their hands, doing anything differently will only disempower.  

Disagree and commit

We live in a world of inter-subjective realities.  Humans created religions, economic models, monetary systems, government, corporations, and moral values.  None of these things exist in nature, they were created from our imagination and we applied human value to them.  

When a person joins a new company, they will often discover that the company has a set of values.  These values were chosen by the leaders of the company because they want these values to guide the company’s actions.  These values are frequently repeated and made highly visible to keep people aligned.  The company’s values might actually differ from an employee’s own personal values, this leads to some degree of cognitive dissonance.  The employee may disagree with some of the values, but the company doesn’t need them to agree but to commit to it.  

Humans are masters of overcoming cognitive dissonance.  When someone converts from one religion to another, seldom do they buy-in 100%.  There are likely beliefs, values, or rituals they don’t agree with or understand, but they still converted to this new religion and go through th rituals because they’ve committed to doing so.  

Imagine someone immigrating to a new country.  This person might immigrate because they like the way of life, the natural landscape, the employment opportunities, the people or the social benefits.  They might not agree with the taxation system and how that money is spent (cognitive dissonance) but the government still expects them to commit to paying their taxes. 

As a leader, it’s important to recognize the difference between personal emotional comfort vs operational need.  We often want people to agree with our values when it is more imperative that we get people to commit and deliver.  

Becoming a leader

Being a leader is a lot different than being a manager.  A manager is an operator, they follow processes and are expected to produce the desired output.  

A leader is not necessarily a manager.  Individual contributors can be great leaders.  Leaders are people who have strong values, exemplify those values, attract a following or inspire others to do the same.  People work for a manager because they are expected to.  People follow a leader because they want to. 

Leaders are inspirational people, have a vision, the will and ability to make it happen.  Of course, they can’t do it alone.  In a casual conversation, Mike Katchen mentioned to his friend that I got a slew of people at the office to start doing HIIT workouts and intermittent fasting.  His friend noted, “you’re like a cult-leader!” I chuckled and replied, “being a CTO is kind-of-like a cult-leader, but it’s also good for them”.  Post, I realized there’s some truth to his statement, people follow because they want to, I didn’t have to force them to.  

We live in an inter-subjective world.  Values are what we as individuals deem to be valuable.  As a leader, we create a narrative around why we value a way of doing things, why something is worth doing, and communicating potential benefits in the far-off distance.  We don’t have all the solutions.  As a leader, we want others to be inspired by our vision, follow us and come up with solutions that aid in achieving our vision.

As a technical leader, we live in a world where there’s endless chaos, no matter where we look there’s the opportunity for things to be better.  The most important aspect of being a strong leader is to acknowledge this chaos, embrace it and provide a vision and plan for how to achieve the end-goal.  I remember a quote from Bruce Lee, “A goal is not always meant to be reached, it often serves simply as something to aim at.”

Hiring a technical leader

One of the most challenging things for startups is hiring a really great technical leader. 

Early-stage tech companies that don’t have a technical founder are often pressured by investors to land this crucial hire.  Investors know that hiring this individual is like finding and capturing a proverbial unicorn; they’ve seen this play out across numerous companies.

A great tech leader will guide the team in terms of architecture, operations, engineering practices, employee growth, retention, etc.  All these things allow for the product and company to grow and scale.  This person is generally well-respected within the engineering community, this, in turn, attracts top talent to the company. 

They are rare because there are several forces at play. In years past, technical leaders at startups typically spent a few years in software development or middle-management at large tech companies and decided to seek out the startup life.  The last few years have been really good to large tech.  Their middle-management and senior engineering talent have been rewarded handsomely, usually not enough to retire, but enough that they’ve grown complacent.

Someone who’s spent considerable time at Amazon, Google, Netflix, Facebook, etc have likely gotten used to their internal tools, the support infrastructure, their peculiar ways of doing things and that work-life balance.  When one finds themselves outside that ecosystem, they can suddenly feel lost at sea.  Some may even feel entitled and not willing to do things they feel are beneath them.  What they don’t realize is how good they had it, many of the problems were already solved for them years ago at these companies.  They are cogs in a machine.  They are just expected to operate.  Life is very different at startups. 

Startups the past few years have also been doing quite well.  Some are startups only in name (Uber, AirBnB, etc).  These people aren’t likely going to leave a rocket ship or don’t really want to rough it out again, they’re waiting for their payday.  The rate of new startups being created has actually slowed down in past years, of course, many have failed.  Developers and engineering leaders find it a lot less risky (and lucrative enough) to join an established company. There are many engineering leaders that have failed, the really great ones that can execute and build engineering teams are an elite few.  This makes for a very tight talent pool for experienced senior technical leadership.  

Many teams make the mistake of not hiring this individual and they make lots of engineering mistakes, eventually, they fail to execute.  You can’t expect a developer with 2-3 years experience to fill this role, without years of making mistakes and mentoring.  Technology is a very expensive place to make mistakes.  Many startups have mountains of technical debt, monolithic codebases and no rational plan or experience for managing it.  Investors push founders to hire someone with pedigree (i.e. has a big name on their resume) because it’ll make other investors at ease that they have someone experienced at the helm, but often this person is not the right person for the all the aforementioned reasons. 

Finding this talent is hard. When they do, startups should be prepared to pay for it in cash and equity.  If you’re early stage and cash-strapped, perhaps tie create a year-end bonus correlated to company metrics.

In any case, good luck!

Lack of meeting rooms

At Wealthsimple’s Toronto office we have a serious lack of meeting rooms.  We have 2 large rooms that can accommodate up to 20 people.  Four smaller meeting rooms that are good up to 6 people and then four cabins which are only good for 2 people.  We have about 170 people at our Toronto office.  The ratio is not good.  

Needless to say, we have a supply & demand problem when it comes to meeting rooms.  1-on-1 meetings became walks in the park.  Sometimes people use the stairway to conduct meetings.  I’ve even once sat in the accessibility bathroom to do a video call.  Eventually, I had to explain why there was toilet sounds every so often interrupting the call. 

Let me say, tough times, tough times…

Jetlagged

It’s been a few years since I’ve been to Asia.  I’ve never handled jetlag well, when I go to the west coast, I don’t bother switching time zones, I just wake up at 4:30am every day and go to bed at 8pm. 

I recently got back from Hong Kong on a Sunday evening and went to work the next day.  First day back at the office was fine, but the rest of the week was beyond horrible.  I powered through and stayed up till 11pm but I’d wake up at 3am and find myself unable to sleep and eventually giving up hope.  I took short power naps in the office to get through the rest of the day, but also slept through a meeting.

By Friday, I finally had a good night’s rest.  But that night I stayed out late, woke up early the next day.  By 1pm I couldn’t stay awake anymore, so I figured I’d take a short nap but then didn’t wake up till 8pm.  Now I’m re-living my week of jet lag hell all over again.  

Technical Debt

Technical debt shares many similar characteristics to financial debt.  Financial debt can be responsible; borrowing money responsibily can enable individuals and companies to achieve their goals (e.g. home ownership, transportation mobility, education, mergers, equipment purchase, etc). Typically, financial debt is often slowly paid off, amortized, consolidated, rolled over, or paid down in lump sums.  As long as the financial debt can be serviced without undue hardship to cash flow and provides a positive cost-benefit utility, the debt is deemed being used responsibly.  Financial debt can be debilitating, unpaid balances with high interest rates can quickly pile on more debt.  For many people, financial literacy is low and understanding the nature of debt and managing it effectively is often confounded with emotion and stigma.

Technical debt provides utility.  Let’s remember that software is a means to an end to help achieve a business need or accomplish a task at hand.  As a technical leader, it is important to frame technical debt and understand how you will service it and how it is serving you.  You can pay down the technical debt slowly, each time you’re in that area of code, make it a little better than before.  You can choose to consolidate it and pay it off in a lump sum either by writing a new service or bypassing it strategically.  No matter what the strategy is, you need to be crisp on the plan and the communication around it.  

It’s important to remind ourselves, there’s no perfect piece of software.  We’re always making tradeoffs and these tradeoffs are okay.  What’s great software today is tomorrow’s legacy software.  Learning how to frame, manage and accept technical debt as a means to an end will give you leverage and productivity as you grow.

Amazon HQ2

The Toronto tech scene is hot.  Last year, more tech jobs were created in Toronto than San Francisco, Seattle, and NYC combined.  By making Amazon’s HQ2 shortlist, Amazon essentially vetted Toronto as a great tech hub.  Amazon saved all these companies thousands of hours of research, cities put pitch decks together for all to see, and now tech companies are moving in fast.

In the past few months, Bay Area startup heavyweights and big tech-behemoths have announced they’ll be recruiting large engineering teams in Toronto. Uber announced they’re investing $200M to build an engineering hub in Toronto.  Microsoft announced they’re moving their Canadian HQ to Toronto. If Toronto becomes the site for HQ2, we should rename Toronto to Seattle2.  Amazon already has over 700 tech employees in Toronto working on Alexa, Supply Chain, and Fulfillment systems.  Their footprint here is already sizeable.  The Bay Area heavyweights are all talking about building 200+ person engineering teams in Toronto.  

This will have both positive and negative effects for sure.  It’s going to drive up the cost of talent, poaching talent away from promising startups to join more established companies.  I honestly believe that life as a software developer is vastly different at Amazon vs a startup.  But companies like Amazon and the more established startups serve an important part of this ecosystem, they are finishing schools for new grad talent.  I’m convinced after they’ve worked a few years in a big-tech company, they’ll itch to join a startup and have more impact.  

In the short term, the biggest losers will be early-stage startups that are trying to find product-market fit and have limited funding because the loss of talent will be felt most acutely. Time will tell, but I believe these developments are good for Toronto, but there will be growing pains.