The Planning Fallacy: Why Experience Never Fixes Your Time Estimates

I have written the sentence "this should take about two days" more times than I can count. I could not tell you the last time it did.

What is strange is not that I am wrong. What is strange is the confidence. Every single time, the estimate arrives feeling like a measurement rather than a hope. I can see the work. I can see the steps. Two days. And then it is Thursday of the following week and the thing is nearly done, and I feel mildly embarrassed, and I attribute the overrun to the specific thing that went wrong — the dependency that broke, the message that took three days to come back, the child who came home from school with a temperature. Each of those explanations is true. Each of them is also, in a way that took me years to see, completely beside the point.

Because something went wrong last time too. And the time before. And the something was different every time, which is exactly why I never learned anything from it.

The textbook that took eight years

The tidiest illustration of this belongs to Daniel Kahneman, and part of what makes it useful is that he is the one who comes out of it badly.

In the 1970s he was part of a team writing a new high-school curriculum and textbook. About a year in, with real progress made, he asked everyone in the room to write down privately how long they thought the remaining work would take. The estimates clustered between eighteen and thirty months.

Then he turned to a colleague on the team who had seen many such projects and asked how long comparable teams had taken. The colleague went quiet for a while, and then said something nobody wanted to hear: of the groups he could think of, about forty per cent never finished at all, and none of the ones that did had taken less than seven years.

Here is the part I find genuinely uncomfortable. The room now had two numbers. One was a feeling. The other was a base rate drawn from actual comparable projects, delivered by a person with no incentive to be pessimistic. They discussed it briefly, decided their situation was different, and carried on.

The book was finished eight years later. By then the agency that had commissioned it had lost interest, and it was never used.

What that story establishes is not that people are bad at estimating. It is that the correct answer can be sitting on the table, in the room, spoken aloud, and lose anyway — because it does not feel like it applies to us.

Why doing it twenty times does not help

The intuitive fix for bad estimates is experience. Do the thing enough times and the numbers should converge on reality.

They mostly do not, and the reason is structural rather than a matter of attention.

When you estimate a task, you do not consult your history. You build a scenario. You picture the work: first this, then that, then the review, then done. This is what Kahneman and Dan Lovallo called the inside view — reasoning from the specific features of the case in front of you. And the scenario you build is necessarily a clean one, because a scenario has to be constructible. You cannot picture the specific unknown obstacle that will eat next Tuesday. Unknown obstacles are, definitionally, not available for imagining. So the plan you construct is the plan where nothing surprising happens, and you time that plan.

Meanwhile the past — which is full of the actual obstacles — gets systematically neutralised by explanation. The reason last quarter's project slipped was the vendor. That was a one-off. It is not going to happen again, because vendors don't usually do that. Every overrun in your history has a named, local, non-recurring cause attached to it, so none of them accumulate into a base rate. You end up with twenty data points and zero evidence.

Two smaller mechanisms make it worse. Memory compresses: people asked to recall how long a past task took reliably shorten it, so even deliberate consultation of your own history returns numbers that are already too small. And attention narrows — you estimate the task as if it were the only thing in the week, when in fact it will be sharing a week with everything else you have already promised.

There is one result in this literature I keep coming back to. Roger Buehler and colleagues found that when people predicted how long someone else would take, they were markedly more accurate — and more pessimistic. Watching a friend say "I'll have it done by Friday," you think: no you won't, you never do. Turned on yourself, that same knowledge is unavailable, because you have access to your own intentions and they feel like evidence. From the outside, all anyone can see is your track record. That is precisely why the outside view works.

Reality beats your worst case

The finding that reframed this for me came from a 1994 study by Buehler, Dale Griffin and Michael Ross, on students finishing their honours theses.

Students were asked for three numbers rather than one: their realistic best guess, how long it would take if everything went as well as it possibly could, and how long it would take if everything went as badly as it possibly could.

Only about thirty per cent finished by their best guess. That much you would expect.

The uncomfortable number is the third one. On average, students took longer than the estimate they had produced when explicitly asked to imagine everything going wrong. Their worst case was optimistic.

Sit with that, because it disposes of the most common self-remedy. The standard response to knowing about the planning fallacy is to pad: estimate two days, say three, feel virtuous. But padding is applied to an inside-view number, and the study suggests that even asking your imagination for the disaster scenario does not get you out to reality. Your pessimism is anchored to your optimism. You are not adding a buffer to the truth; you are adding a buffer to a fiction.

Which means the fix cannot be a better feeling. It has to be a different input.

What professional planners actually do

The correction has a name — reference class forecasting — and it exists because the stakes eventually got large enough that somebody had to solve it.

Bent Flyvbjerg's work on large infrastructure documented cost and schedule overruns that were not occasional but close to universal, and consistent across decades, countries and project types. The Sydney Opera House is the canonical case: estimated at around seven million Australian dollars, delivered a decade late for over a hundred million. Not a scandal — a genre.

What emerged from that work is a method blunt enough to be almost anticlimactic:

1. Identify a class of past projects that your project belongs to.
2. Find the actual outcomes for that class — what really happened, not what was planned.
3. Position your project inside that distribution.

Notice what has been removed. There is no step where you reason about your project. The specific features — your excellent team, your clearer brief, your better tooling — are exactly the inputs that produce the error, so the method excludes them on purpose. The UK Treasury eventually formalised a version of this as mandated optimism-bias uplifts: percentage adjustments applied to project estimates by category, imposed from outside rather than negotiated by the people doing the estimating.

That last detail is the one worth stealing. The correction has to be structural, because if it is left to judgment, judgment will apply it to everyone else's project.

Where it shows up when there is no project at all

Megaprojects are a vivid illustration and a slightly misleading one, because they make this look like a workplace problem. Most of the damage is domestic.

It shows up in the Saturday morning list of six errands that comfortably fills the day and eats Sunday. Each one takes twelve minutes in your head and none of them include parking, or the queue, or the fact that the shop moved.

It shows up worst in the yes you give to something four months out. A commitment in November is not evaluated against November's actual calendar — it is evaluated against November's emptiness, because the ordinary competing obligations of that week have not been scheduled yet and are therefore invisible. Distance does not make you busier; it makes the future look emptier than any real week has ever been. Then November arrives full, as every month does, and you are surprised, and you honour the commitment by taking the time from sleep or from the people you live with.

It shows up in the reading pile, the side project, the "quick" home repair, the fortnight of holiday that was going to include finally sorting out the garage.

And the cost is not really the missed deadline. It is that a chronically over-optimistic plan quietly converts into an obligation to work at an unsustainable pace to protect a number you invented while feeling cheerful. The estimate was a guess. The guilt is real. That trade is a bad one and almost nobody notices they made it.

Building a buffer that survives contact

Three things have actually helped, as distinct from things that sound sensible.

Separate effort from elapsed time, and estimate them separately. Most overruns are not effort overruns. The work really was six hours. It just took three weeks, because it was waiting on a reply, or on a decision, or on a Saturday where nothing else was happening. Estimating "six hours" and then reporting it as a duration is the single most common way an honest person produces a wildly wrong forecast. Ask two questions: how many hours is this, and what is the earliest it can realistically finish given everything else in the calendar.

Use your own ratio instead of a round number. Padding by fifty per cent is arbitrary. Your actual multiplier is discoverable, and for most people it lands somewhere between 1.5 and 3 depending on the kind of work. Once you know yours, the adjustment stops being a moral question about whether you are being pessimistic and becomes arithmetic.

Buffer the calendar, not the task. Adding slack inside a task gets consumed silently, because work expands and nobody sees the slack disappear. Leaving genuinely unallocated days in the week is visible: if they get eaten, you can see what ate them. This one changed more for me than any estimation technique, and it is not really an estimation technique at all.

One caution about buffers. If a buffer is announced, it will be spent. Deadlines communicated with an explicit "but there's slack in there" arrive at the true deadline every time. The buffer has to be structural — a real gap in a real calendar — rather than a private intention.

A reference-class worksheet for your next project

Fifteen minutes, on paper, before you commit to anything. The order of the steps matters more than it looks, because step one anchors you and step three is designed to break the anchor.

1. Write your gut estimate first, and seal it. You will not be able to un-know it later, so capture it now, honestly, before anything below contaminates it. This is your control.

2. Name the class, not the task. Not "this article." Rather: "articles of roughly this length that I have written." Not "this kitchen." Rather: "rooms in this house we have renovated." The class has to be broad enough to have members and narrow enough to be genuinely comparable. If you cannot find five members, widen it until you can — a rough reference class beats none.

3. Find five real instances, with real dates. This is the actual work, and it is mostly retrieval rather than thinking. Your calendar, your sent mail, your commit history, your bank statements, your photo roll all record when things actually started and actually finished. Write down the start date and the finish date. Not your memory of them — memory shortens.

4. Take the median and the worst. Two numbers, not an average. The median is your realistic case. The worst one in your set is not a disaster scenario; it is a thing that already happened to you, which makes it a legitimate forecast.

5. Compute your multiplier. Divide the median actual by what you would have guessed for those same past jobs. That ratio is more useful than any of the individual numbers, because it transfers to work you have never done before.

6. Adjust only for differences you can name and defend. You are allowed to adjust downward, but you must write the reason on the page, and it must be a structural difference rather than a mood — "the scope is genuinely half" counts; "I'm more focused these days" does not. Cap the total adjustment at twenty-five per cent in either direction. The cap exists because this step is where the inside view sneaks back in.

7. Commit to the outside-view number. Say that one out loud. Put that one in the email. The gut estimate from step one stays in the notebook, where it belongs, as a record of how far off you were before you looked.

8. Log the actual when it finishes. One line: what it was, what you predicted, what it took. This is the only step that compounds — it is how you stop needing to reconstruct a reference class from scratch every time. Within a year you have a table that answers the question in thirty seconds and cannot be argued with.

Questions worth answering

Doesn't this just make me slower and more cautious?
It makes your stated dates slower. It does not change how long the work takes, which was never under your control in the first place. What it changes is whether the gap between plan and reality gets paid for out of your evenings. Accurate forecasting is not pessimism; it is the thing that lets you say yes to fewer things and actually deliver them.

What if I genuinely have no comparable past projects?
Then borrow someone else's class, or widen yours until it has members — "things of this general size" is a weak reference class but still an empirical one. And notice that "this is unlike anything I have done" is itself the strongest available signal for a large multiplier, because novelty is where unknown obstacles live.

Isn't a deliberately tight deadline sometimes useful?
Yes, as a forcing function on scope, which is a real and different tool. The danger is conflating the two. A tight deadline used to compress scope is a decision. A tight deadline arrived at by underestimating is an accident that you will later experience as personal failure. Know which one you are doing.

My manager wants a number now. What do I say?
Give a range with the basis attached: "Comparable work has taken us three to five weeks; I'd plan on four." The basis is what makes a range hold up under pressure. A bare number is infinitely negotiable, because there is nothing behind it but confidence — and confidence is exactly the thing that got us here.

Does knowing about the planning fallacy fix it?
No, and this is well established. Kahneman knew and it cost him eight years. Douglas Hofstadter put it best: "It always takes longer than you expect, even when you take into account Hofstadter's Law." Awareness does not correct the bias; it only motivates you to install a procedure that does.

The thing I keep circling back to is that the estimate always feels like the honest one. It never feels like optimism from the inside — it feels like arithmetic. Which is why the only reliable defence is to stop consulting the feeling and go and look at what actually happened last time, in writing, before you open your mouth.