Nobody wants to deliver late, over-budget software. I don’t know a single software developer who wakes up in the morning and thinks “I’d like to do a rubbish job today. How can I cost my employer more money?” And yet, so many software projects don’t go well. And with every new project, there seems to be more and more pressure to go faster. So, if we’re in the business of making software, what do we do? How do we go faster without compromising quality?

Despite more than 50 years of history and countless methodologies, advice and books, IT projects keep failing.

—Susan Moore ¹

Now, I’m not writing here as some kind of expert. I have never run my own software company. I am not delivering wisdom distilled from copious academic studies or controlled experiments. I am writing this to organise my own thoughts as I try to make sense of what I see going on around me.

To think this through properly, we need to start with why. What is the point of all this software production? Why are we even making software in the first place? Let’s leave open source as the elephant in the room for the moment and talk about commercial software. Let’s start with business.

Business is about reducing customer pain.

As I understand it, to run a successful business, we first find something that causes people pain. It might be a metaphorical or literal pain (though usually, it is metaphorical). Then, we offer a way to reduce that pain in exchange for money. For example, people find it difficult (painful) to learn to code. So, there is a market for books and classes that teach programming. Some people dislike their physical appearance. So, there are whole industries in exercise, cosmetics, beauty therapy, and so on. A business delivers value to customers to the degree that they reduce that customer pain (or the perception of it). And if people are confident that we can reduce their pain, then they will be glad to pay us money. 

In a software product business, software is the thing we offer to reduce customer pain. In this type of business, software development is the key activity delivering value. Customers buy (or subscribe to) the product, and software development creates it. Of course, this applies only to product businesses. If we are selling consulting services, or IT as a support function, then things are different. But where the core business is a software product, then development is what gets it done.

This is not to say that development is the only activity that adds value. For example, if nobody knows our product exists, then it may as well not exist. So sales and marketing activities are essential. We also have to make sure that our product actually addresses real customer pain points. If not, we’re wasting our time. So market research (whether formal or ad-hoc) is also vital. We also need user experience (UX) and graphic design activities to reduce friction. Friction in our product gets in the way of solving our customers’ problems. All these activities (marketing, sales, market research, UX, design) are important. And if you squint a little, then they all start to look similar. They’re like facets of the same core activity: understanding people. But, in the end, all these activities deliver only plans and promises for customer value. It is software development that turns the plans and promises into a product.²

It all works better when you embrace the idea that “product,” “design,” and “engineering” are just different perspectives on the same thing.

— Greg Veen ³

Minimising lead time to business impact

If we’re doing all this “understanding people” stuff right, then it’s an ongoing activity. As we go, we learn more about the problems we’re trying to solve. So we start devising better solutions. So we need the software product we’re creating to change as well. For this to work, we need a nimble development team. A team that is able to deliver value rapidly, and able to respond quickly to change. This is the core goal of software development practice. As Dan North puts it:

“The goal of software development is sustainably minimising lead time to business impact”

—Dan North ⁴

So, having a nimble development team is important. But how do you get a nimble development team? Do you:

• Pay your developers like kings?
• Buy them super-fast, expensive computers?
• Send them to whatever crazy tech conferences they want to attend?

We could make a good argument for any of these things. If you want to keep your nimble development team, then give serious thought to each one. Fast computers and good tech conferences will improve developer performance. This investment will pay off over time. But these things are more relevant to retaining good developers. We want to think about building a nimble team.

So if the answer is not giving developers whatever they want, what do we do then? The short answer is, ask the developers. But, ask them at the right time, in the right way. The thing to understand about developers is that they tend to be natural problem solvers. Good developers like their jobs. They like their jobs because they get to solve interesting complex puzzles all day and get paid for it. Good developers revel in taking complex challenges and finding elegant solutions. So they should be able to come up with great ideas for becoming more nimble. But many organisations encourage developers to focus on the wrong problems. This encouragement may be neither deliberate nor conscious, but it happens nonetheless.

Focussing on the wrong problems

How does this happen? How do we end up asking developers to focus on the wrong problems, without even knowing we’re doing it? This happens because we distance developers from the customers. As soon as a project gets to be any reasonable size, we bring in project managers and business analysts.⁵ And we bring these people in for a very good reason—developers can’t do everything. Software projects are complicated. The code is complicated enough, but on top of that, there’s all the work of deciding what to build, planning the development phases, structuring the roll-out and deployment plans, liaising with customers… the list goes on. The developers have enough to worry about with the code. So we need these extra people to help.

But, what happens is, these extras become the developers’ interface to the world. The project manager and business analyst mediate communication with external stakeholders. The project manager, in particular, cares about delivering the project. Project managers report to management. And management cares about:

• How much is it going to cost?
• How long is it going to take?
• Why is it costing so much?
• Why is the project so late?
• Why isn’t it finished already?
• My goodness, we’re burning through how much per day on this late project?!

It’s understandable then, that project managers become focussed on predictability. They want plans, structure, estimates. They want to know what is happening and when. Predictability and measurement allow them to sound competent when they’re reporting to management. So they talk to developers about estimates and reports and deadlines. So then, developers begin to focus on deadlines and reports and estimates. They focus on estimation and predictability to keep the project manager happy.

But there’s an unfortunate issue with this. The issue is that estimation and predictability are impossible problems to solve. Every time a developer starts a new task they face an uncomfortable reality. Any given task may or may not contain an enormous sinkhole of hidden complexity. We hope the task is simple. But it might not be. You never know. And then Hofstadter’s law comes into play:

Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.

—Douglas Hofstadter ⁶

Consider this scenario: A project manager asks an inexperienced developer for an estimate. The inexperienced developer gives an estimate they think is reasonable. And then the project manager goes away and turns that into a deadline and a plan. A good project manager will even add a little ‘fat,’ to be on the safe side. But then the inevitable happens—the project falls behind. So the developer starts working longer hours to meet the deadline. But working longer hours means the developer gets tired. They start making more mistakes. And it’s still not enough. The project is still behind. The project manager is demanding to know what’s taking so long. So the harried developer starts cutting corners. And in the process, they start shipping bugs. So now the product is not only late, but also buggy.

This situation delivers negative customer value. Sure the late, buggy product might still solve some amount of customer pain. But the bugs introduce new pain and take time to fix. The customer loses confidence in our ability to help them. This makes them less inclined to pay us money. Everybody loses.

Experienced developers know the estimates game is rigged, so they try hard not to play it. Imagine, a project manager comes to an experienced developer asking for estimates. What they do is give a number that is large enough to sound ridiculous. But also small enough that the project won’t be cancelled immediately. Next, the project manager (or salesperson) comes back to challenge this ridiculous figure. “That estimate seems a bit larger than we were hoping for. Is there any chance we could maybe squeeze things a little and bring that number down?” At this point, the experienced developer asks: “What kind of figure do we need to get down to?” The salesperson gives a number back. Then the experienced developer rubs her chin and says, “Well, it will be tight, but we’ll see what we can do. We’ll have to take some requirements out and deliver only the most basic features.” Then she estimates how little they can promise to deliver without appearing incompetent. And she commits to only that. This way, when she delivers much more than promised, everyone is happy. But even in this scenario, Hofstadter’s Law will still raise its ugly head. And soon enough we’re back to scrambling to meet deadlines and shipping buggy code.

Estimates are typically a necessary evil in software development. Unfortunately, people tend to assume that writing new software is like building a house or fixing a car, and that as such the contractor or mechanic involved should be perfectly capable of providing a reliable estimate for the work to be done in advance of the customer approving the work. […] With custom software, however, a great deal of the system is being built from scratch, and usually how it’s put together, how it ultimately works, and what exactly it’s supposed to do when it’s done are all moving targets. It’s hard to know when you’ll finish when usually the path you’ll take and the destination are both unknown at the start of the journey.

—Steve Smith ⁷

My point here is not to complain about software estimation. Everyone knows it’s a necessary evil. But it’s a necessary evil that ends in a vicious cycle. We cut corners and ship poor quality code to meet the deadline. And all the while we assure each other that we’ll come back and fix it later. But ‘later’ never comes. We’re already behind on the next phase because we had to go back and fix those bugs. And we’re now building on top of brittle, hacked-together code that isn’t built for rapid change. And once locked in this cycle a developer’s focus shifts away from solving customer pain. Instead, they focus on problems like these:

• What’s the quickest possible way we can mark this feature ‘done’ and get the project manager off my back?
• How can I touch this brittle, fragile code as little as possible? Because the more I touch, the more likely it is to break.
• How can I eke out one tiny piece of code that I’m proud of amongst this giant steamy pile of technical debt?
• How can I get better at justifying my decisions to people who don’t have a clue about what I do or how complicated it is?
• How can I blame someone else when the customer starts complaining about the bugs I didn’t have time to fix?
• How can I get some good buzzwords on my CV so I can get another job somewhere where everything isn’t such a mess?

Now, no developer I’ve ever met wants to deliver late, buggy software. But we pressure developers to give short estimates because we want it sooner.⁸ The developers comply because they want to please. But then they’re stuck because the estimates are always wrong. So they’re now under pressure to deliver. They want to please so they work longer hours, and cut corners. They compromise on quality because everyone is asking them ‘Is it done yet?‘. But nobody is happy. The software is still late and buggy.

So, most developers I know, are doing the best they can. But they’re stuck. They’re too busy trying to catch up to even think about going ‘faster’. And so, they focus on the wrong problems. They’re focussed on survival. It’s hard to focus on saving for retirement when you’re about to die of starvation. It’s also hard to work out how to work smarter when you’re working seven days a week on a late project. So the first step is to acknowledge that going faster requires investment. And if things are bad, it will need both a financial/time investment and an emotional investment.

Disrupting the cycle

Earlier, I suggested asking the developers how to reduce lead time to business impact. But, when developers are in ‘catch up’ mode we’re unlikely to get great responses from them. When we come into this environment and say, “How can we go faster?” we’ll likely get one of two types of response:

1. Burn it with fire. “We need to go away for two years and rewrite everything from scratch.” This happens when developers are completely overwhelmed by technical debt. So much so that they feel only way out is to declare bankruptcy. They may have a point, too. But at the same time, we may not have the budget to do that, and the market definitely won’t sit still while we rebuild.
2. Outrage. “We are going faster. I can’t believe you think you can fix this complicated issue with a half-hour brainstorm! How dare you?!” This happens when developers feel forced to ship poor quality code.  And then they feel they receive the blame when customers complain about bugs. And they may well be justified in their outrage. Developers in this mindset won’t help us at all until we can show that we hear them. They need to know we understand their concerns. We also need to show that we’re serious about changing things.

In both cases, the developer’s concerns are valid, but they are inward-focussed. We want to create a situation where everyone is working to minimise lead time to business impact. It won’t happen while developers are stuck in this mindset. Step zero is to show that we’re serious about changing things. That will usually involve finding some way to reduce pressure. Even if it’s only temporary.

But even then, unless something changes, developers will still be inward-focussed. They will have plenty of ideas on how to improve what they’re doing. Some of them might be great ideas. But there’s a lot of risk. We need the developers to focus on minimising lead time to business impact. We need to get their focus off dealing with internal pressures. We need to expose them to customer pain.

Exposing developers to customer pain

So, how then do you expose developers to customer pain? Plenty of other people have written at length on this, so I will only skim the surface. Here are three ideas in order of least-effective to most-effective:

1. Get developers to use the product they’re making as part of their day-to-day work. In the industry, this is known as drinking your own champagne or eating your own dog food. The advantage of doing this is that it turns developers into users of the product. So any glaring bugs or issues will now cause pain for the developers too. The problem with this approach is that developers aren’t typical users (most of the time). The way developers use software is often different from most customers. So, while this may help developers fix major bugs, it may not provide great insight into typical use cases. Also, this is not always practical. For example, imagine we’re producing a SaaS product for dental hygienists. It may be difficult to for developers to integrate thisinto their daily workflow.
2. Get developers to do rotations on support teams. A better approach is to encourage developers to take part in some kind of support roster for the product. (They may need quite strong encouragement.) This way developers get to experience customer pain first-hand. So, as they answer phone calls and email (or tweets, or whatever) customers tell them about problems. If developers do this long enough then they will also start to observe patterns of common issues. They’ll see things that come up over and over again. Not having to hear that same complaint again makes a good motivator to fix usability issues. Unfortunately, people rarely contact support to tell you what’s working well. So the feedback is somewhat biased.
3. Get developers to sit with and watch people using the software on a regular basis. This is the most inconvenient option as it requires the most organisation. But it is also likely to bring the best results. With this approach, developers get to see how real people use the software in real life to do real stuff. They get to see the good, the bad, and the ugly.

Doing these kinds of things with consistency is hard work. It takes effort and organisation. And most developers will have a natural disinclination for it. I feel awkward writing this because I don’t do this as often as I ought. But I believe it’s worth the effort.

Exposing developers to customer pain is an exercise of deliberate effort to overcome cognitive bias. Which is a long way of saying “it’s a way to learn some humility.” We developers are prone to think we’re clever. And many developers are clever. But we don’t know everything. Maybe I’ve finally figured out how monadic bind operations relate to functional composition. That’s great, but it doesn’t mean I know a thing about what our customers are facing as they use our software every day. Exposing myself to customer pain reminds me of how little I really know.

In my experience, the more isolated the developers, the worse the resulting end product. It doesn’t help that most teams have a layer of business analysts who feel it is their job it to shield developers from users, and vice-versa. It’s dangerous to create an environment where developers have no idea who the users are.

—Jeff Atwood ⁹

Now, there is a glaring problem with all this customer-facing warm-fuzziness. In short, it doesn’t make developers go faster. In fact, it takes time away from coding, so it arguably slows them down. So why would I be arguing for it then? The short answer is that going faster doesn’t do a lick of good if you’re running in the wrong direction. Exposing developers to customer pain is about direction rather than speed.

Ask the developers

We want to sustainably minimise lead time to business impact. My hypothesis is that if you point developers in the right direction, then you can ask them for ideas on how to do that. If we then empower them to implement those ideas then we should start to see results.

Ideally, this is an ongoing process. We ask developers if they have any ideas on how to go faster. And then we try them out. Then come back after a couple of weeks and ask how that went. Then ask them again. And keep on asking them until every time you approach their work area you don’t even have to ask them. They start saying things like: “That refactor we did of the routeing engine is really paying off. But, I think we’ll be able to rip things in and out more quickly if we move some of that logic back out into the microservices layer.” You may have no idea what that means, but if we see fewer bugs and happier customers then everyone wins.

How you ask your particular team of developers is up to you. Some people like brainstorming workshops, while others prefer surveys or one-on-one interviews. Each approach will have different strengths and weaknesses. But whichever approach you choose, be sure to make any constraints clear. If you have only a very small budget, then say so. If there’s no flexibility to push out any deadlines, let the developers know. Assuming you have smart, capable developers, they can take these things into account. And if they don’t get it, even after you’ve explained it many times, then you’ve learned something…

Do be careful when communicating constraints though. If we tell developers that there’s no budget; deadlines are fixed; and there’s no wiggle room at all… then they will rightly reply that they can’t help. You have to be very careful in this situation. Quality software, produced fast, costs money. The developers need to see that we are willing to invest in them and their tools. If there is no budget; no room to move on deadlines; and no sign that this will ever change… then a smart developer will look elsewhere. And I would applaud them for it. That is a no-win situation. This is where emotional investment comes in. Show the developers that we care and we’re willing to invest in them in the future. And explain that we are severely resource constrained right now. Then, they may be willing to come up with some creative solutions to get us out of the current pickle.

Assumptions

I’m making a somewhat large assumption here. My assumption here is that your developer are smart enough to understand the constraints when you explain them. The biggest and most obvious constraint is that we do not have an infinite bucket of money to play with. Making software costs a lot of money. More than most people expect or realise. And good software developers are not cheap to employ. My big assumption here is that you have at least one or two smart developers who are capable of understanding this.

The sad fact is that some developers just don’t get it. What do you do then? Well, there’s no simple answer. But I suspect that the reason some developers don’t get it is because they have never been exposed to the bigger picture. They’ve just been asked for unrealistic estimates and told to go faster. They don’t often see things from the customer’s point of view or from the point of view of the person who pays their salary. And the only way they will start to get it is if someone shows them.

The other big assumption I am making is that the developers can be trusted not to embarrass the company if we put them in front of clients. And yes, I have been in plenty of meetings with clients where developers have said stupid things or vented their frustrations in front of clients. Not everyone is ready to be put in front of a powerpoint deck and asked to give a sales pitch. But if a developer can be trusted to just politely shake a hand and say hello, then surely they can at least sit in a corner and quietly watch people use software? ¹⁰ Maybe they need someone to go with them at first. But how else will someone learn to be a good ambassador for the organisation if they are never given a chance?

But, I digress. Back to going faster…

Safety belts and engine upgrades

Let’s assume your team is full of clever developers. When you ask them for ideas, they may come up with some things that seem counter-intuitive at first. Things like:

• Test Driven Development (TDD)
• Continuous Integration
• Pair programming or mob programming
• Code reviews

All these techniques will slow development down… at first. TDD looks a lot like writing double the amount of code to produce the same outcome. Pair programming sounds like taking two productive developers and halving their output. I can understand some scepticism. But these are not just trendy buzzwords (and most of these techniques have been around for decades). There are good reasons for all these things.

Let me try to explain with an analogy. When you drive a car, you wear a seatbelt. And these days, we expect our cars to have airbags and crumple zones. But, when you want to drive really fast, you wear a racing harness and helmet and fireproof clothing. To the car, we add a roll cage and a spoiler and sticky tires. It’s not a perfect analogy but hopefully, you see what I’m getting at. At first, things like TDD and code reviews appear to slow you down. They can be awkward and difficult to get used to. But these are the very things that enable the team to go faster safely.

What is pretty certain is that TDD saves time & money as maintenance costs get factored in — a lot of time & money.

—Eric Elliott ¹¹

Techniques like TDD and continuous integration are about improving software quality. This means releasing fewer bugs into production. Catching bugs before release means less re-work, less embarrassment and happier customers. They are usually quicker (and less expensive) to fix. Over time, the time not spent fixing bugs adds up. What’s more, these techniques also tend to produce code that is more flexible. Code that’s easier to change or re-use. This means that we spend less time fighting against a brittle codebase. And we can spend more time adding new features or modifying functionality. The net result is better software, faster.

Tightening Feedback Loops

The point of all this is to shorten the time between writing some code and getting it into the hands of customers. Once there, then the developers can observe how well this new code is reducing customer pain. Armed with this feedback, they can then improve the code further… and so on. We create a virtuous cycle.

What has been transformative for us is the massive reduction in the amount of time to get feedback from real users.

—Phil Wills ¹²

If you’ve been following IT trends over the last few years, then this virtuous cycle will sound familiar. It sounds very much like continuous delivery. But the buzzword isn’t the point. Continuous delivery is just a label for a set of practices. Together, these practices provide tight feedback loops. The feedback loops allow us to decrease risk while increasing velocity.

There is a good reason for this. The environment in which we build software is not just complicated, it is complex. A complicated system has many parts. So many parts in fact, that it takes an expert to understand how everything fits together. But in a complex system, there are not just many parts, but all the parts are connected and react to each other. So, when you change one small thing, then the entire system might change in response. A classic example of this is the cobra effect:

The British government was concerned about the number of venomous cobra snakes in Delhi. The government therefore offered a bounty for every dead cobra. Initially this was a successful strategy as large numbers of snakes were killed for the reward. Eventually, however, enterprising people began to breed cobras for the income. When the government became aware of this, the reward program was scrapped, causing the cobra breeders to set the now-worthless snakes free. As a result, the wild cobra population further increased. ¹³

With complex systems, it is very difficult to predict what the effect of a given change might be. This is because making the same change twice may have completely different consequences. The first change causes the system to respond in such a way that it reacts very differently the next time. This can lead to unintended consequences and makes planning and estimating problematic.

[The] way to understand complexity is that acting in the space causes the space to change, and cause and effect can only be understood in retrospect.

—Liz Keogh ¹⁴

How then do we manage to get anything done in a complex environment? What the experts suggest is to “probe, sense and respond.” In other words, create tight feedback loops to gauge whether something is working or not. Then we iterate as quickly as possible. We keep the changes small and the cycle short. As a result, the risk associated with failures is also kept small, and it is cheaper to recover. We make lots of small experiments, keep the ones which work, and revert the ones that fail.

In a complex environment, you probe, sense and respond. You do something that can fail, safely, and it tells you things about the environment which you respond to, changing the environment. This is the land of high-feedback, risk and innovation.

—Liz Keogh ¹⁵

Conclusion

We cannot build a high-performing development team simply by applying ‘best practice’. Unfortunately, there are very few silver bullets in software development. But there are patterns that work well when we have the humility to admit we don’t know everything. Exposing the developers to customer pain closes a feedback loop. This allows us to make sure that if we are going fast, we’re going fast in the right direction. Once this is in place, we can work on continual improvement in a way that suits our given circumstances.