Has something like this ever happened to you? You start work on a project after estimating the time and budget for its completion. You consider all the pertinent information, come up with projections, and get going on the project. As you get closer to the project’s completion, you find that you had grossly underestimated the time and/or money needed for the project’s completion. You are desperately behind schedule and over budget.

Why is this scenario so common in real life? Why do we fail to project reasonable time and cost estimates? And if you think it is only individuals who make these errors, and not large organizations, think again. Big corporations and governments are equally susceptible to these errors. Some classic examples of these errors are the Sydney Opera House (10 years behind schedule and 14 times over budget), Denver International Airport (16 months behind and $2 billion—about Rs7,950 crore today—over budget), and Boston’s Big Dig (several years behind and almost $10 billion over budget).

Researchers call this inability to reasonably estimate the time needed to complete a task the “planning fallacy". While in the case of governments and public projects, red tape and bureaucratic delays may also contribute to delays, research has convincingly demonstrated that individuals working entirely on their own are particularly prone to making these mistakes.

An important insight into this phenomenon was provided by a study where researchers first asked participants to give realistic, most likely estimates for a scenario. They were then asked to give “best case" estimates for the same scenario. Interestingly, the two estimates were practically the same. In other words, people unconsciously make the best case assumptions while working on the “most likely" scenario.

Thus, as we plan ahead, we are overly optimistic in our assumptions and fail to account for factors (either low probability or outside the direct realm of the project), which may still have an impact on the project’s on-time completion.

Will it help if we visualized minute details of the project as we estimate the time it might take to complete it? Not necessarily. While it may help us uncover some hidden aspects of the project that we have not planned for previously, we still don’t compensate enough for unexpected “overheads".

The best way to overcome this problem is to seek an “outside view" instead of depending on the “inside view". By “outside view" we don’t just mean seeking the opinion of a neutral third party but, rather, deliberately avoiding bringing the unique and specific features into your estimates and instead using completion benchmarks for broadly similar projects completed in the past.

Although, intuitively, you may want to factor in your unique knowledge of the project details as you prepare your estimate, doing so may actually make your estimate more optimistic (and therefore less accurate). It may increase your confidence in your estimate, but it may do nothing to improve its accuracy.

If you do have access to outside experts who have experience with similar projects and can easily recall those experiences, then it may also help to seek their estimates of completion time. They won’t have access to the project’s details, but their estimates will be more realistic as they are based on actual completion times for similar projects in the past.

So, as you plan, keep in mind that there are always unpredictable forces and factors that keep you from faster completion of a project. Since the factors are generally unpredictable, even controlling for factors that delayed your last project won’t protect you from new factors that could delay your current project. Mere knowledge of details of a project does not mitigate the delaying effect of unforeseen circumstances.

Also, if you think that you are more determined this time around, think back to when you were planning for your last project. Were you any less determined then? Determination and enthusiasm are never a problem in the planning stage. It is at the execution stage that things happen and you start wondering why your forecasts were so optimistic!

Send your comments to driversseat@livemint.com

Praveen Aggarwal is an associate professor of marketing at the Labovitz School of Business & Economics at the University of Minnesota Duluth and Rajiv Vaidyanathan is a professor of marketing and director of MBA programmes at the University of Minnesota Duluth.