How to successfully estimate software projects

One of the most common mistakes, which derails the timely delivery of software engineering projects is the lack of thorough breakdowns and estimations of the implementation details.

In a junior world, engineers dive into implementation first, peeling away one layer at a time. Each layer presents its own challenges, which may have been minimized if all layers were understood ahead of implementation. Architectural decisions made in the first layer can lead to major refactoring a few layers into the project.

In a senior world, planning time is spent on system architecture and high-level design, with many of the implementation nuances assumed to be trivial. Those nuances are not always trivial and take significantly longer than an engineer initially expects.

A great approach, which I have seen successfully work for delivering many projects, within reasonable margin of chronological error, is to break down all the work into the smallest possible tasks: target having no task estimated to take longer than one hour; a task should also be no smaller than 15 minutes. If a task is estimated to take longer than one hour, it should be broken down into smaller sub-tasks. Estimating the time to complete each small task becomes significantly simpler and more precise. While the entire project may be new territory, chances are, the small tasks have been completed for previous projects.

Let’s go through an example of breaking down and estimating a basic user sign-up form. We will not worry about scalability or advanced security in this scenario.

Product Requirements: build a sign-up form, which takes in a user’s first name, last name, email, and password, to create a user account and be logged into the application.

The most common approach to breaking this down is:

• Front-end (1/2 day) … because how long does it really take to make a simple form that calls an API?
• Back-end (1/2 day) … because how long does it take to make a simple API that takes in user info and stores to the DB

So the quick estimate for the project is 1 day.

What does a detailed breakdown look like?

Front-end (9 hours)

• routing for sign-up page (30 min)
• generic template layout (1 hour)
• first name form field (1 hour) // with reusable styling for other fields
• last name form field (15 min)
• email form field (15 min)
• password form field (15 min)
• submit button (15 min)
• client-side input validation: general logic (30 min)
• client-side input validation: first name (15 min)
• client-side input validation: last name (15 min)
• client-side input validation: email (30 min)
• client-side input validation: password (30 min)
• integration with API (1 hour)
• handle success case for API response (30 min) // confirmation UI
• handle failure case for API response (1 hour) // identify errors and present to user
• redirect user to logged-in dashboard (30 min)
• QA time (30 min)

Back-end (8.5 hours)

• DB migration to create users table (1 hour)
• User model (1 hour)
• routing for API url (30 min)
• controller with stub action method (1 hour)
• create user via model using API inputs (15 min)
• server-side input validation: general logic (30 min)
• server-side input validation: first name (15 min)
• server-side input validation: last name (15 min)
• server-side input validation: email (30 min)
• server-side input validation: password (30 min)
• create user session (1 hour)
• successful response (15 min)
• failed response (1 hour) // simulating various test scenarios creating user takes time
• QA time (30 min)

The detailed breakdown adds up to 17.5 hours, which makes this roughly a 2-day project. Many of the listed items may take less than the estimated time, yet any one of them has the potential of taking significantly more.

In the real world, this estimate is in engineering time. This means it’s the time to actually do the work. Add to this distractions and context switches that come up during a typical day for an engineer and its safe to double engineering time to get the calendar time for a project. The project above will take four calendar days. Add to this the time needed to code-review, merge, and deploy. Now you’re looking at a week of time from breaking ground to getting the project into production — a big gap between the quick estimate of 1 day and the reality of 1 week.

If you don’t have the luxury of time, then the work needs to be parallelized and the context switching explicitly minimized. The above project can be worked on by two engineers in parallel (front-end and back-end). Though be careful — not all projects can be sped up by more resources. Two engineers can do two days of engineering work in one day. Eight engineers will not be able to do same work in two hours.

The user sign up project can be worked on by two engineers, be code-complete by end of the first day; integrate, QA, and iron out the issues by end of the second day; code review, merge in, and release by end of the third day.

The key takeaway is to think through all the details and nuances first. Estimating small tasks is significantly more precise than ball-parking a full project.