Why we suck at estimates

Posted on by Sean

Let’s start with a waterfall cautionary tale

Bob was a software engineer in the mid 90’s.  He was working for one of those really large companies that had the lion’s share of the market.  In theory, they released a new product once a year.  In practice, they released product once every 18 months or two years.  Bob’s team had just finished v4 of their popular document creation and management software.  This iteration was only 7 months late from when it was promised.  All in all, not too far off of their average.

The team was getting together to discuss version 5 with all of the key stakeholders.  This time was going to be different.  This time they were going to account for everything and release on time and under budget.  All of the managers vowed that they had learned from their mistakes and they ‘had this’.  Bob inwardly rolled his eyes but did his best to hop on board and toe the company line.  He sat through all of the meetings and gave his input on estimates, research required, design methodologies and architecture.  He thought they were underestimating many of the innovative new elements and said so.  The team increased those estimates by a token amount and signed off on the yearly project plan.

The first four months seemed to be going ok.  They were spending a little too much time on fixing defects coming in from v4 out in the field.  They had already released three hot fixes to deal with the more serious ones.  They were also struggling with two new technologies that they were trying to bring into the product.  One was a new open source library and the other was based on a functional language that the team didn’t have any experience with.  Their architect, Roger, said they were only days away from cracking both of those particular nuts so the atmosphere was still  optimistic.

Going into month 6, the road started to get a lot bumpier.  Turns out, the open source library they chose doesn’t scale.  They were going to have to either scrap all the functionality that relied upon it or rewrite those pieces of the library themselves.  The functional programming piece, on the other hand, was going wonderfully.  They put one of their younger aces on it and he was well ahead of schedule.  Unfortunately, he liked coding in this new language so much that he had just accepted a new position where he could code in it full time.  It was going to take at least three months to replace him and the team only had two weeks to do the knowledge transfer.

Bob’s immediate manager had put on at least 10 pounds and looked like he was working hard on his first ulcer.  When the sales team came in month 8 to say that their competitors had just released a new menu system that was killing it in the marketplace, the team knew they were in for a major scope change.  Those silver tongue bastards in sales had convinced the leadership team that they had to have this functionality or risk ruin.  The CEO, a salesman by trade, agreed.  Bob’s manager was told that they could scrap one of the smaller pieces of functionality to replace it with this.   Bob’s manager’s arguments of, the technology was not built to support the new menu system, fell on deaf ears.  The sales team got their way.

Bob and the team worked diligently to duct tape this new functionality on to the existing architecture.  They were building mountains of technical debt, but who cared, they were getting close to go time.  They hit their code complete date at month 11.  Everybody celebrated for about a day until QA got their hands on it.

Come month 15, QA declared the product a disaster.  Some of the bugs they were finding were so unwieldy that the team had to rewrite major core elements of the code base that had worked just fine in v4.  There were also rumors that the new menu system was not doing as well in the market as they had first heard and there was a large group of customers who hated it.  They were so committed at this point to making it work that they pushed through anyway.

The product was released in month 22 to mixed reviews.  95% of the customers who bought it turned off the new menu system in the config screen.  The sales guy who championed the idea left six months ago so there was no one to point the finger at but the leadership team.  Most of the developers working on the project left because they were so demoralized about spending so much time on something no one used.

Well duh, that’s why everyone has switched to agile methodologies, right?

Sadly, agile does not totally fix this problem.  It certainly doesn’t if implemented poorly.

Jump forward 15 years and meet Karen, an engineer for an ERP company.  Her company had just hired a new VP of Engineering for the sole reason of bringing agile into the company.  Their product was an enterprise product that still had to be installed on premise.  They had distant plans to move it to a SaaS product but not anytime soon.

This ERP company had about as good of a track record as Bob’s company when it came to delivering on time.  Their new VP of Engineering had a ton of energy and was well liked by everyone.  He was hugely successful in his last job where he built a bunch of custom websites for Fortune 500 companies.

Karen’s last product release was over two years late, so the team was ready for a change.  They soaked up the idea of agile and its two week sprints like sponges out of water.  The simple user story was such a relief after wading through 200 page design docs for so long.  Plus, QA was going to be working right next to them the entire time!  This had to work, right?

The trouble began in preparing for the very first release.  Their typical release schedule was every two years, plus or minus a year.  The VP fought for monthly releases and lost.  He then fought for quarterly releases and lost.  Our customers will never take releases that often, he was told.  Finally, he convinced the leadership team to do single year releases instead of every two years.  He explained to the team that they would be doing quarterly releases internally.

Sadly, the product team did not report to our new VP of Engineering.  So the product team built a yearlong product spec that did not have any adaptability for change.  The spec also required some serious innovation, new architecture and design patterns that the team had never used before.  The product team was also unwilling to split this spec into quarterly parts knowing that would probably require them to trim some of the functionality set that they had labored so hard over.

In the initial product meetings, the VP of Engineering got bullied into certain estimate limitations by the ‘experts’ in the company who were so adept at delivering product a year or two late.  This bullying happened after he and his team said the initial spec would take at least 18 months.  He got product to cut a couple of big features but no more.  His estimates still had them at 16 months.  He finally acquiesced when the CEO paid him a visit and said that all change had to happen gradually, even agile.

After the first quarterly internal release it was clear that there was no way they were going to make their annual release.  The new architecture was still very unsteady and had a host of unknowns.  Our VP was praised by the leadership team when he brought up these issues this early in the process.  “Most of the time we don’t find out about these type of problems until a month before the release,” said the CEO.  The VP of Engineering gave him a pained grin.

Come the second quarterly release, the team had figured out the architectural issues but in twice the time they had estimated.  The product team was already pissed that this meant a lot of feature cuts.  However, they were exuberant that they already had something to play with so early.  Naturally, they took the MVP out in the field and got some responses.  Not all of it was positive so they started changing scope.

Our VP of Engineering was excited about the changed scope because this was how agile was supposed to happen!  However, he lost the leadership team when he said that the team would have to reevaluate all of their previous estimates.  He tried again for monthly releases, cautioning that there would be even more scope changes as the customer feedback came in, so why not get rid of the whole idea of an annual release?  This just would not sink in with the old guard.

Our VP was being courted by other companies throughout and when he lost this last battle he gave up and took another position.  Karen and four other members of the team left with him.  They heard a year later that the product was released only 5 months late.  The company was celebrating that as a win.

These failures are so common in software development and a lot of that failure comes down to our reliance upon estimates.

Why is estimation so hard?

There are several principles of estimation that I have found to be universal.  If you don’t account for these principles, chances are you’re going to pay for it later.

  • Almost all estimates are given as best case scenarios
    • When we are asked how long something will take, we always think, “how long with this take if I have no interruptions?”
    • Having no interruptions is a fantasy.  We are constantly being put into context switching scenarios where we rarely find the ‘zone’.
    • Story points are your friend here.  Story points are relative based off of past history of the team and they do a good job of accounting for interruptions.
    • If you are forced into giving hourly estimates, try to push for days as your smallest unit of time and pad those estimates based on previous results.
  • We can only estimate as if the world around us is static
    • If there is one constant to this existence it is that change will happen.  Entropy is the way of the universe.
    • We can take a bunch of guesses about what may happen over a long period of time, but they will just be guesses and most of them will be wrong.
    • The further you look in the future, the more wrong you will be.  Keep your releases as short as possible.
  • We cannot estimate things we have never done before with any degree of accuracy
    • If you have unknowns or are working with technology that you or your team has never worked with before, your only true answer to how long it will take is: I don’t know.  It’s OK to say I don’t know.
    • Don’t fake it.  You need to de-risk these unknowns by prototyping or bringing in experts that have done these things before.
  • People who don’t understand technology assume that all technology is similar
    • Many execs assume that if they can talk intelligently about a product, they know how long it should take to build it.  This is so painfully wrong.
    • There is some willful ignorance here but most of it is that tech folks do a poor job of educating others.
    • Chances are you will hear this question many times: that program took only a month to build, why does this one take a year?  Don’t take offense, answer it honestly.
    • There is also an assumption that all technologists are the same.  This has gotten better over the last couple of years but it’s amazing how many people believe that a full stack developer means that that person knows about every technology.
    • There’s enough tech out there these days to require specialists.  Asking a UI developer to build a data layer is like asking an ophthalmologist to perform open heart surgery.  They could probably figure it out given enough time, but would you want them to?
    • Educate these folks as often as possible.  Don’t assume they understand technology like you do.
  • Your code will be far buggier than you think
    • Getting QA involved early and often helps dramatically
    • Unit test, unit test, unit test.  Look at TDD if the team can stomach it.
    • Automated testing helps a ton here too.
    • You have to budget for fixing defects.  So many developers give estimates as if everything will work perfectly the first time.
  • No one thinks about nonfunctional requirements while estimating until reminded
    • Nonfunctional requirements are things like scalability and performance.  Nobody thinks about this until the product is delivered and the CEO asks, “is it normal for the loading screen to take 5 minutes?”
    • Optimization takes a lot of time.  Plan for it.

Regardless of how progressively agile your company is, chances are you will be asked for estimates sometime along the way. Keep in mind that all estimates are guesses and that the further out you are the more likely you are to be wrong.  If you keep these estimation principles in mind, you will have the best chance of not making a complete ass of yourself.

Is Agilefall stealing your soul?

Posted on by Sean

Throw away the crystal ball

Almost everyone has acknowledged that long term project estimation needs to be a thing of the past.  If you are still doing waterfall development where you are trying to estimate projects 9 months or a year in the future, I’m sorry.  You are stuck in a pointless time loop.  It’s time we call these long term plans what they really are, guesses.  Why is a long term plan a guess?  In that period of time, things will change.  That is a guarantee.   Your customers’ needs will change.  The technologies you are building with will change.  Market forces will change.  You may be able to predict some of these changes but why would you want to?  Predictions are guesses.  You’re much better off setting yourself up to adapt to change.  If you can roll with the punches, chances are you will exit the ring standing up.

This is why agile development has become so widely accepted.  The core premise behind agile is that it allows a company to embrace change rather than fight it.  Conversely, this is why a lot of anachronistic big companies struggle so much to innovate.  When they take the waterfall approach and finally release product, the product they release would have been relevant a year ago (or two years depending on how late it was) but is not relevant today.

The onset of AgileFall

Even most big companies have started to recognize that waterfall is not getting the job done.  They have seen the studies.  They know they are losing more and more to innovators.  So, they decide to give the whole agile thing a try.  But, they don’t go all in.  It’s like watching Grandma adopt a cellphone.  Sure, she figures out how to use it, but she never picks the damn thing up when you call her.  Even worse, she leaves it sitting on the kitchen counter anytime she goes somewhere.  Might as well leave it plugged into the wall.

There’s many reasons why companies struggle with committing to agile.  Many of these reasons are good ones, at least to management.  Here’s the big three I’ve heard again and again:

  1. The folks on top don’t get it and aren’t that interested in learning
  2. ‘My enterprise customers will never take releases more than once a year’
  3. We need the time to build buzz and marketing campaigns around the upcoming release

For those of you that came out of the gate with SaaS, count your blessings.  You’ve never had to deal with this crap and probably never will.  For those of you with legacy code bases or working for larger companies, this is most likely the norm.

Break big things into smaller things and ship the smaller things

You’re chances of going from annual releases to releasing every day or every sprint are effectively zero.  Your best bet to get to real agile is by doing it piecemeal.  Treat it like any other engineering problem: break the big problem into smaller ones.  If you are doing annual releases, shift to quarterly releases.

This step by step approach allows you to combat the three most common pushbacks.  Convincing the folks on top that this is the way to go becomes a hell of a lot easier once you start hitting dates.  The logic here is sound and easy to manage up: if we’re doing less, there is less risk and less change.  Therefore, our predictability will increase dramatically.  If you take this approach, make damn sure you hit your dates or watch it all blow up in your face.  More on that in another post.

DIA waterfall development
DIA

Another bit of ammo to use in convincing those at the top is the well-known project failure.  Take your pick, there’s a mighty graveyard of failed projects at your fingertips.  Keep it topical.  I’m in Colorado so it was easy for me to reference DIA (Denver International Airport) which opened sixteen months late at a cost overrun of $2 billion.  Talk about your big project failure.  If you’re on the east coast, use the Big Dig.  The argument becomes even stronger if you can reference a similar, smaller project success.  For us, this was the second round of the light rail once Denver’s Union Station was completed.  They didn’t try to launch it with all lines opening at once.  Instead, they first launched the train to the plane.  Then they started opening up another line every quarter.  To my knowledge, they haven’t missed a date.  Each successful line opened is another feather in the cap that builds trust.

Union Station Agile Development
The New Union Station

As for enterprise customers not taking releases more than once a year, this is not a tough argument either.  Make sure that these customers don’t HAVE to take each quarterly release.  If they still want to take just one release a year, fine, let them take every fourth release.  The trick to this approach is in making sure that each release builds on the previous one and you are using only one code line.  The minute you break into multiple code lines, you’re done.  The support nightmares will take down your company.

Somewhat surprisingly, your sales and marketing teams will quickly become your allies.  With the quarterly release cycle, you’ve now given them a lot more to do and a lot more to talk about.  This is again only true if you hit your dates.  You miss your date once, they may forgive you but more than once and the trust is most likely permanently broken.  This happens and folks lose faith in the product and you can wave goodbye to your best sales folks.

Agile Development Quarterly Release
What Quarterly Releases Look like

Do it once and doing it again is a LOT easier

Once you successfully switch over to a quarterly release, a monthly release is the next logical step.  If you had any success in the quarterly release cycle, the monthly release cycle is a much easier conversation.  You have already established yourself as an expert.  Chances are your value to the company at this point is pretty high and you don’t have to convince the folks up top.

Where the battle gets more interesting here is with sales, marketing and your deployment teams.  They’re going to feel overwhelmed.  Those teams that cheered you on when you hit your quarterly releases will turn on you.  Take a minute to put yourself in their shoes.  All of a sudden they have to constantly learn new features.   I can hear all of you SaaSers giggling but keep in mind that these folks came from annual releases.  Many of them have never experienced things like quick feedback from their customers and pivots based off of that feedback.  This is a sea change that requires their roles to change.  That’s scary.

The pain of this transition can be eased by making the first two monthly releases of the quarter ‘internal’.  You still cut a build and move on to the next one but you don’t actually ship it.  This gives you time to continue building trust around the monthly release.  It gives them time to come to grips with how their roles need to change.  They are now selling and marketing an ever-evolving product instead of incremental releases.   It also allows your deployment teams the time to turn into devOps teams.  It takes time to make those process changes.

Once the other teams have become comfortable with monthly release cycles, moving to releasing every sprint is a very easy step.  Now you’re doing agile!

The horrors of context switching (and how to beat them) – Part 2

Posted on by Sean

Part Deux

The first post on this topic covered many of the ways that context switching can destroy the productivity of your day but most of the post was aimed at developers.  This one reaches out to everyone that works in an office.  Scott Adams has made a career out of ridiculing the pointlessness of workplace politics.  In no setting is this ridicule more pronounced than the pointless meeting.

Agile best practices

Multitasking meetings?

Have you ever been in a meeting and looked around the table while someone is giving a presentation?  What do you see?  Too many times I would see a scene like this:  John is looking at his phone, Betty is trolling the web while Michael is writing on a post-it in front of Margaret.  Sound familiar?  Exactly, the workplace has regressed back to fifth grade.

Agile best practices

Even worse, I once had a CEO that would regularly take phone calls from his wife while we were in a meeting and would talk with her on the phone for a good 5-10 minutes while everyone would be sitting there waiting.  Sometimes, he would even lapse into baby talk.  It was a fantastic use of time.  This happened in executive leadership meetings all the time.  Let’s look at the economics behind this.  If he took a call with eight of us in the room, each of us making an average of $250/hr, each 10 minute phone call he took would cost the company roughly $500.  This accounts for the 5 minutes it would take for us to get back to where we were in the meeting.  Granted, this guy was a world class asshole but these types of things happen all the time in meetings across the globe.

Luckily, there is a cure for the pointless meeting.

7 tips for a productive meeting

The best way to deal with distractions if these are your meetings is to be honest with your teams.  Be very clear that you are not going to waste their time but be just as clear that you will not tolerate them wasting yours or other people’s time.  When people aren’t paying attention, most of the time it is because they are not engaged.  That’s on you as a leader.  These are 7 tips that I have found to be incredibly effective for holding a highly engaged meeting:

  1. Have an agenda and stick to it
  2. ONLY invite people that can contribute to the topics on the agenda
  3. Lids down.  Phones down.
  4. Start on time
  5. Don’t take up the full meeting time if you don’t have to
  6. At the end of the meeting ask the attendees if they thought it was productive or not
  7. Write down action items and send them out with a summary after the meeting

Make it about respect

The strongest statement you can make that the meeting will be productive is to have an agenda.  Having an agenda means that you have enough respect for your team that you prepared in advance.  If it is not your meeting, it is perfectly acceptable to ask what the agenda is.  If the leader of the meeting doesn’t have one, make sure you get that person to cover the points they hope to get done in the meeting before the meeting starts.  I’m at a stage in my career that I will typically leave a meeting if there is no agenda.  The end result of this action is that I either don’t get invited to, in my opinion pointless, meetings or the leader of the meeting puts together an agenda before inviting me.

Always run the numbers

Run the numbers.  If somebody is not going to be contributing in the meeting and you invite them to it anyway, ask yourself, how much do they make an hour?  If they’re not contributing am I willing to burn $100 of the company’s money to have them there doing nothing?  Many times people are invited to meetings not to contribute but just so they don’t miss out on information being passed in the meeting.  This is a terrible use of their time.  A simple summary email once the meeting is over is far more effective and will not require context switching.

Once the agenda is briefly covered, expect all participants of the meeting to be engaged.  The best way to enforce this is by asking for lids down and phones down.  It is far more difficult to be engaged if the distraction of a laptop or a phone is staring you in the face.

Start your meetings on time.  Do not wait for stragglers that are rolling in late.  Start on time and either end on time or end early.  Respect your people by treating them like adults.  If somebody rolls in late, let them catch up on the topic in their own time.  Do not keep stopping the meeting to fill folks in on what they missed.  That’s a waste of everyone’s time.  Even if it is somebody who is higher than you in the corporate hierarchy.  If somebody asks for a summary after showing up late, you can handle this with tact.  Say something along the lines of, “I’ll be happy to fill you in later but with respect to everyone’s time we need to stay on topic.”

I used this approach with a former COO of mine.  This COO would roll into every meeting 5 to 10 minutes late.  I could tell he was annoyed with my approach but, very professionally, he waited until the meeting was over to let me know.  He explained that he was incredibly busy and that most other meetings he had ran over.  I let him know that I understood that frustration but then I went to the numbers.  If I interrupted the meeting every time someone came in late, I not only risked disengaging the rest of the team, but we would be forced to go off topic, then context switch back to the topic on hand.  If there were 10 people in the room each making roughly $100 an hour, that 5 minute interruption costs almost $100.  If we do that twice a meeting, it starts to get very expensive.  The other point I made was that if I only were to do this for him because he outranked me, I would be showing my team that we have two different standards in the company in regards to respecting people’s time.  We would basically be saying that it’s ok to waste other people’s time as long as you outrank them.  To his credit, he just nodded.  I would love to say that after that conversation, he was never late to a meeting but that just wasn’t true.  He never asked me for a summary again however.

When you get through your points, end the meeting

Once you are through with the agenda, END THE MEETING.  Nothing is more infuriating than having a leader of a meeting look at the clock and say, “Oooh look, we still have 20 minutes left, let’s try to get this done too!”  Why is that so infuriating?  Because of tip #2.  You put the attendees of that meeting together ahead of time to contribute to the topics on the agenda.  Once you go off of those topics, somebody, if not everybody will feel like you are wasting their time.  Ending early is a sign of efficiency and your team will respect you for it.  Throwing pointless filler in at the end of a meeting is a waste of resources.

Incorporate the feedback loop.  At the end of the meeting ask the group if they thought the meeting was an effective use of their time.  Make sure you get an answer from each attendee.  It is critical that you are open to this feedback or else it just comes across as lip service.  If an attendee says they think that it was an ineffective use of their time, make sure you get them to give you a couple of ideas of how they would make the time more effective.  If the majority of the attendees say the meeting was an ineffective use of time, you need to be ok with cancelling similar future meetings especially if they are scheduled weekly.

Finally, make sure to write down the action items that come up during the meeting.  When the meeting is over, take five minutes to write up a summary that gets sent out to the attendees and anyone else that needs to know what was discussed in the meeting.  This summary should also contain the action items that came out of the meeting.  This is critical because this is a written record of the commitments that people made in the meeting.

Conclusion

I would say that at least 60% of all meetings are a complete waste of time.  Not only do you lose the time that is spent in the pointless meeting, you also lose all the time around the meeting when you have to context switch and try to get back in the flow of what you were actually working on.  It doesn’t have to be this way:

Agile best practices