Software Estimation — Some Things I Should’ve Done Differently

• Why
• Estimates, Targets, and Commitments.
• We’re all bad at estimating
• Agile business, rigid estimates
• An estimate comes with uncertainty

I’ve been on a few projects that came in on time, and far more that didn’t. When I reflect critically on this, I’ve definitely made my fair share of mistakes, but somehow I never felt at any given time that I was making a mistake while I was doing it. I wanted to dive deeper to understand the reasonings behind my past mistakes, so I recently read Steve McConnell’s Software Estimation: Demystifying the Black Art. I really enjoyed it, and it was very easy to read — short chapters, concise information, persuasive arguments, and a healthy dose of data showing the effects of good and bad practice.

Estimates, Targets, and Commitments

Estimates are needed by the business to help plan their next moves. It’s easy to overlook some of the language that gets used. I’ve seen tons of exchanges go like this:

• Business Executive: “How long should that take to build?”
• Developer: “Emm… about a week”
• Business Executive: “Ok great”

There are a lot of factors here at play that have set the stage for potential failure with a conversation like this. The developer has just made an off the cuff estimate, because she was asked a question, and probably felt pressured into giving an answer. Off the cuff estimates are normally quite bad — this estimate was made in her head in a fraction of a second, how accurate can that really be? The business executive asked how long it should take, not how long it will take — this kind of language sets the stage for the developer to exclude the scenario where something goes wrong. The executive also asked how long it will take to build, which sets the stage for the developer to mentally exclude the other tasks involved in actually getting the feature into a usable state into the hands of the users and stakeholders, such as discovering any finer details, testing the feature, performing code-review, deploying it to the various environments safely, and anything else needed.

But most egregiously, the software developer has portrayed an estimate, but what the business executive was really after here, was a commitment to have something done by a given timeframe. When you start to look critically at your day to day estimations, think carefully about the language you use.

• An estimate is a calculation or judgement call, presented with some level of uncertainty, to indicate the effort required to deliver a given feature. This is entirely a technical effort.
• A target is a business goal — some deadline that’s important for the business to have developed some feature by. This is entirely a business effort.
• A commitment is a promise by a team or individual, to have delivered some feature by a given date. This a collaborative agreement between the technical team and the business.

You cannot negotiate the size of an estimate with a stakeholder. If you provide an estimate of 2 weeks to a stakeholder, they can’t say “that’s unreasonable, can you make it 1 week?”. They’ve asked for your judgement of how long something will take, and then utterly rejected it. That’s like asking your taxi driver how long it will take to travel 30 miles through the city, then telling her that her answer is nonsense. You can change the parameters around the work, like adding extra resource, or removing features, in order to deliver something in a shorter timeframe, if 2 weeks really isn’t feasible.

If you do little else, you should strive to make sure that you’re not giving out estimates that are being heard as commitments. Developers hearing “We can do that in one week”, hear that it might be technically possible to complete the task with 7 days worth of effort, at some point in the future. On a Monday morning, an executive hearing “We can do that in one week”, might reasonably expect it to be delivered by Friday afternoon. Make sure you’re communicating clearly what you mean, because the language used here can easily be misconstrued.

We’re all bad at estimating

There are a lot of factors that can eventually contribute to initial estimates coming out nowhere near reality for a software project. They come from 4 generic sources:

• Inaccurate info about the project being estimated. The requirements are not all there, or are unclear, or aren’t quite what is really needed.
• Inaccurate info about the capabilities of the organisation doing the work. About the level of skill available for different roles, and the general project management capability maturity.
• Too much chaos in the project to actually support accurate estimation
• Inaccuracies the come from the actual process of estimating.

This last one stings, because we all like to think we’re good at our jobs, and that we have lots of experience doing it, so we can tell how long it will take. Unfortunately, there’s a lot of human error involved in estimating, and there are a few reasons we can be bad at providing good estimates.

• We tend to be overly optimistic without just cause. We have flawed memories of past projects, we remember more fondly our successes, and gloss over our failures. We also over-estimate our ability to reflect on past mistakes and say “well, we just won’t make that mistake again this time, and it’ll be done quicker!”. This excludes the possibility of any other error cropping up, and is a dangerous proposal.
• We’re prone to forget some component of what we’re estimating. When you say it’ll take x days to build a feature, building the feature on its own is of practically no use, other than showing your loved ones. You also probably need to test it, and certainly need to deploy it. We tend to forget all the other steps, and simply focus on the building aspect. When it comes to entire projects, we can have missing requirements, and missing non-development activities, which if forgotten, can inflate the entire project estimation.
• Being an expert in doing the work, doesn’t necessarily make you an expert estimator of that work. Individual contributors’ productivity can vary by a factor of 10, so an individuals judgement based on their own experience could be wildly inaccurate for the team.

One of the largest factors in estimating a project, is the capability of the business analyst actually gathering the requirements feeding into the project. But after that, we need to size up what they’ve deemed is required. Here’s where most teams go wrong through, they look at the user stories or task provided, and give a quick estimate as to how long it will take, based on memory. Developers often use their personal judgement, even though a better estimate might be possible. Historical project data on past similar tasks is likely to provide a more realistic estimate based on unbiased experience. Counting these past results of effort, and computing a new estimate based on the historical average, is a far more realistic estimate.

NASA has an extremely mature ecosystem for software development processes. When a new project is to begin, they have pretty good idea of how many sub-systems it requires, they immediately multiply that by the average number of lines of code for a sub-system historically, in order to start working towards a proxy for total effort. They’ve got so much detailed data, that nobody ever asks “how long will that take?”, because the numbers are staring them right in the face, free of bias, and rooted in historical experience.

Unfortunately in my software development career, I’m yet to see someone refer to a database of past tasks of estimates, or an average time to complete a kind of task, even though this has been proven to be a better mechanism. A lot of teams don’t put in the effort to record their past estimates, and the actual effort applied to given user stories or tasks. What’s more likely, is developers size their user stories, as a means to gauge how much work they can do in a sprint, and call it a day with their estimation efforts.

Agile business, rigid estimates

Every loves agile, or at least loves to hate some aspect of it. Regardless of your feelings, odds are more likely that someone in a large business somewhere, is doing some flavour of agile. One key focus comes into play here — quoting directly from the agile manifesto, “We prefer customer collaboration over contract negotiation”. The value here being the ability to incorporate changes in requirements into our work stream. However, one of the problems, is that change to our plans almost unilaterally increases the amount of work that needs to be done — but rarely have I seen estimates revised to account for this. My past experience has shown that when a team is approaching a deadline, and a last minute change arrives, this merely increases pressure on the development team — the expectation that the team will somehow stick to their original estimate seems to remain.

This is something that has haunted a few past projects for me. When changes continually arrive as the project continues, I felt a lot of pressure to deliver what I had originally committed to. If I’ve promised an app in 3 months time, I’d bust my ass to get it done, but this becomes more and more difficult as more changes are added. Even if my initial estimate was good, if I had delivered exactly what was in mind at the start of the project, I would have failed in the eyes of the business, because its needs changed, or the requirements became clearer as time progressed. Unfortunately, I promised an app, not “feature x, y and z”, and the other people I’ve communicated that promise to, have gone away feeling reassured that by the deadline, they’ll have what they want. They will be quick to forgot how many changes the project took along the way, and quick to remember my promise to deliver.

I wish I’d had clearer communication along the way, that while their changes can certainly be incorporated, that each changes comes with a cost in effort, and that continually adding changes, adds to the timeframe required to complete.

An estimate comes with uncertainty

If estimates were always spot on, we’d apply those skills to lottery numbers and retire. In software, our estimates are always out somewhat. With tasks we’ve done plenty of times before, like providing a report of recent business stats, the level of error is pretty low. You might provide an estimate like “3 hours, give or take an hour”. But so often I’ve seen this recorded as a single measure.

One of the best pieces of advice I’ve read in Steve’s book is the give estimates as ranges. Give the best case, worst cases, and most likely case, and this can start to form the basis of statistical analysis for your software estimation. The IEEE even states,

Software Developers have a professional responsibility to include uncertainty in their estimates

This can start to provide some sort of standard deviation around the duration of a collection of estimated tasks, to provide a more rigid framework for estimations. For 20 tasks, with an average historical effort of 3 staff-days, and a standard deviation of 0.75 staff-days, you can say there’s a 50% chance of getting it done within 60 staff days, or a 60% chance of getting it done in under 70 staff days. Either way, you begin to work towards a more reliable model for giving estimates to the business, and providing the uncertainty in a useful format that the business understands — risk. Risk of not meeting some deadline, and business impact. They can then factor in these risks with their other business decisions, which might ultimately affect what the team providing the estimate ends up doing.

I’ll admit, I’ve never seen anyone give estimates in this format, but I would have been thoroughly impressed with their professionalism on the spot if someone had ever said “I think this will take 2 days. Best case, one and a half, worst cast if x happens, might take 4–5 days”. I’ve never seen someone do that, but that level of insight to a business executive is immediately useful.

This capability then leads into models around what happens when the business needs to alter the schedule provided by current estimates. When a team provides an estimate, saying “We think there is a 45% chance this will be done within 80 staff weeks, and we have 4 staff”, and this presents a level of risk the business isn’t willing to take on, they’re ultimately saying they want a higher level of certainty in completing the task sooner.

This leads straight into a discussion of the dynamics of diseconomies of scale when building software. A really good read on this comes from the book, the mythical man month, which I would wholeheartedly recommend. Essentially, throwing more staff at the problem will only get you so far. Increased staff requires increased communication, increased training, and increased co-ordination. There is an upper limit to how much work a team can produce in a finite timescale, regardless of resources. Even Jeff Bezos couldn’t be on the moon by tomorrow afternoon if you tell him now. No amount of resources can co-ordinate that kind of operation on that time-scale. So it is with software, adding more resources eventually reaches a hard limit, but having an understanding of these dynamics is definitely worthwhile for anyone working on larger projects

So what should I have done differently?

I shouldn’t allow stakeholder’s targets to pressure me into commitments. When expressing my estimates, if a stakeholder tries to pressure me into doing it sooner, instead of agreeing to put that pressure on myself or my team, I should explain why the estimate is what it is, and explain to the stakeholder that we can only commit to their target if they’re willing to remove something else from our backlog of work due before that target.

I should have collected a lot more data. If you’ve used JIRA, especially an on-premise one, you might be aware that it is extremely customisable. You can record almost anything in there, all to be stored in a PostgreSQL database and queryable. But often I’ve seen teams throw an estimate in there, and rarely look at how accurate those estimates were, they’re just loose estimates to figure out how much can be done in a sprint. But with enough of a history, we can actually use this data to create baseline estimates for us, for each task. This is something I’d love to see in action, providing some insight into how accurate our estimates are.

I shouldn’t allow changes in requirements to slide into the backlog without changes to estimates, and possibly commitments. Changes to work, whether already completed or ready to work on, have a tangible cost, and ignoring this cost will make it harder to meet the initial estimate, but leave me with a more difficult commitment to adhere to. I’m not advocating to stop changes or reduce the number of changes, agile says we should be open to working with our stakeholders and incorporating those changes. But I should make it clear to the product owner that these changes will add to our estimates, and we will need to re-estimate if we get enough changes.

I should definitely give estimates in the format of best-case, most-likely and worst case. When a stakeholder asks for an off the cuff estimate, it’s tempting to provide a quick answer, but we’re often optimistic about things, and have a natural tendency to give the best case answer. But the best case really only happens a small fraction of the time. If you give a stakeholder a best case estimate for 5 tasks, the probability of meeting all estimates compounds to get smaller and smaller. Avoid this problem by telling the user the best and worst case. I would love to be able to browse a back catalog of tasks that me and my team have completed in the past, and be able to give estimates based on tasks we’ve done before, so that I can say “Based on past data, there’s a 70% chance we’ll be able to complete this by your target deadline” — this uncertainty is represented to the business and allows them to plan with these targets in mind.

Conclusion

If you’ve never though critically about your software development estimation process, never inspected the accuracy of your estimates, or tried to improve this, I’d thoroughly recommend reading deeper into the theory behind how top-performing software teams do this. There is a lot to learn, and tons of practical information that can help teams of any size, and individuals. I’d definitely recommend reading Software Estimation: Demystifying the black art (I’m not an affiliate).

I’ve also posted my handwritten notes on github. Have a look here: https://github.com/Mark-McCracken/handwritten-notes

Enjoyed this? Follow along to get notified for the next post. It’ll be about the practical applications fo nested and repeated fields in bigquery, and will be out before the end of next week. I’ll also be writing about specifics of software/data engineering techniques, professional topics like agile methodologies, and anything else I think readers might find interesting related to the intersection of software and business.