Stop Blowing Up Sprints!!

Posted on by Sean

If you are using scrum successfully, this is probably obvious to you.  I’ve been doing a fair amount of consulting lately and I’ve worked with a lot of teams where blowing up sprints is common practice.  Let me define what I mean by blowing up a sprint.

A sprint blow up is when you have a fixed length sprint, we’ll call it two weeks, and midway into the sprint you decide to change scope.  This could be due to a defect coming in from the field.  This could be because the CEO wants to work on something different.  It could be because the product owner changed their mind about a feature.

Regardless of the reason for the blow up, it is typically done with little forethought.  It’s almost never done with maliciousness.  The primary reason for a sprint blow up is ignorance.  The person blowing up the sprint will typically ask, why can’t I just replace this 3 point story with that 3 point story?  Pretty reasonable question right?

As software engineers, it is up to us to educate why it is such a bad idea to blow up a sprint.  There are two overarching reasons why sprint blow ups are so disruptive.  The first reason is cultural, the other(s) is technical.

When business leaders decide to use agile, they are often told that one of the benefits is that their developers will commit to hitting each sprint.  For business leaders that have any memory of the old waterfall days where being six months over estimate was more common than inappropriate yoga pants at a Walmart, this sounded like a pretty good deal.  The problem was, no one told these leaders the full story.

A sprint is a contract

Scrum-Contract

The folks building and testing product are committing to hitting a sprint.  In return, the rest of the company is committing to not changing the contents of that sprint for the duration of that sprint.  That is the contract.  I can tell almost immediately when I walk in to a dev shop where blowing up sprints is commonplace.  The developers and QA all have a defeated, hang dog look about them and the rest of the company has very little respect for them.  The rest of the company ends up blaming the development team for not hitting their commitments without the slightest understanding that they are the ones in breach of contract.

You know what happens to good developers in this type of toxic culture?  Yep, they leave.  Nobody wants to work in a culture where everyone is blaming them for failure.

Before we get to the technical challenges around sprint blow ups let’s first answer the question – why can’t I just swap out this 3 point story with that 3 point story?  The answer to that is estimation is not an exact science.  Story points estimates are no more accurate than t-shirt sizes.  We’ve all experienced putting on an XL t-shirt that wears like body paint, while putting on another XL that fits like a moo moo.  I have anyway.  When the team is doing their estimates, everybody in the room knows which story is a heavy 5 and which story is a light 3.  When they commit to the sprint, they’re all doing the inner calculus that says ‘I may fall a little behind on that heavy 5 but I’m sure I can pick it up on that light 3.’

When you blow up the sprint, that inner calculus disappears.  You have invalidated the development team’s commitment.  Once the sprint is violated, those commitments can no longer be honored.

What if we have no choice?

The most common complaint I get when passing on this advice is – well what about issues coming in from the field?  We can’t just leave our customers high and dry.  I agree, you can’t leave your customers hanging especially if you’re dealing with high priority defects.

The best answer to this is to have a team that is dedicated to customer issues that is outside of the sprint development cycle.  This is typically only one dev and one QA person and that QA person can even be part time.  When that dev is not fielding high priority issues from the field, they are fixing defects or refactoring tech debt.  This should be a rotating position.  When doing two week sprints, dev support should only be for two sprints at a time.  This is a great way to spread the burden, so you don’t get one developer who ends up being the go to guy for all defects coming in.  It is also a great way to make sure everyone is familiar with the majority of the code base.  There’s no better way to train new developers than to throw them into that hot seat.  Sure, they’ll get help from other members of the team, but when it’s your ass on the line you will learn that code quickly.

What happens if your team is not big enough to field even one developer to handle these issues coming in from the field?  Please don’t try to half ass this.  I’ve worked with a couple of teams where they said – “we just budget x% of the sprint to manage defects coming in.”  This will never work.  You can’t budget for chaos.  When defects come in, they might take up 5% of the sprint, or more commonly, they end up taking more like 50% of the sprint.  Not to mention that defects are damn near impossible to estimate in the first place.  If the devs knew what the problem was, they wouldn’t have written the defect into the code in the first place.  Either way, the defect kills that inner calculus and the sprint is destroyed.

The short answer is if your team is too small to handle field issues outside of the sprint, you shouldn’t be using scrum.  Scrum only works when the contract is upheld.  I’d recommend Kanban instead.

What are the engineering reasons you shouldn’t blow up a sprint?

I’ll list several more reasons here if you still need to articulate why blowing up a sprint is a bad idea.  This list could easily be expanded and I recommend you to do so.  It’s cathartic.  When swapping out stories or throwing in defects to blow up a sprint, outside of the estimation issue, you are also not accounting for:

  • The work already started (half of story B is left unfinished)
  • The de-risking required for the new stories (which is normally just skipped and you wing it)
  • Context switching
  • The environmental set-ups (things like supporting data sets, integration points, etc.) required to build certain stories
  • The branching required to quarantine started work
  • The branching required to start new work
  • New dependencies introduced by the new work
  • New integration testing and merging required by the new work
  • Changes in release strategies if necessary

You want to bring dignity back to a development environment and culture?  Stop blowing up those sprints!

Refactor or start over?

Posted on by Sean

The refactor vs rewrite question

“Why don’t we just scrap all of it and start over?  This legacy code blows.” – pretty much every developer at least once.

RefactorvRewrite.jpg
Credit to bonkersworld.net

Has this question ever come up from one of the engineers on your team?  Many times, it is brought up by one of your best engineers.  It will certainly be brought up by most of your younger engineers.  It’s a fair question.  Especially from a developer’s perspective.  Let’s look at why:

  • Legacy code is notoriously difficult to learn
    • It makes ramping up new developers difficult
  • Legacy code is hard to maintain
    • Very often the devs who wrote it originally are no longer there
    • It’s very rare that you find good comments in legacy code
  • Legacy code is often a mess
    • The comment thing again
    • Trying to wade through functions hundreds of lines long that don’t seem to have much of a point
    • It rarely follows the code standards the current teams follow
  • A lot of legacy code feels hopelessly outdated
    • This is where you find things like custom string types and pointers that aren’t thread safe
    • After the first forays into legacy code, many younger devs will come out of it as if they just walked in on their parents having sex –  baffled, disgusted and with a tinge of confused wonder.

Were all past devs really that bad?

Having been a developer for many years it’s hard to miss the pervasive attitude that all developers not on the current team pretty much suck.  I contributed to that attitude when I was the one fixing ‘other people’s messes’.  It’s also really easy to point at old code and blame all of your current problems on it.  It’s certainly easier to point the finger at someone no longer there than to accept responsibility that you are your team are probably the ones at fault.  You and your team are certainly the ones that have to deal with it.

RefactorvRewriteDilbert

Try to nip this behavior at the bud when you can.  All developers that came before you don’t necessarily suck.  Chances are they were put in the same situations where they had to sacrifice the right way of doing something to deliver on a deadline.  Or, the technologies that you are so familiar with now were new to them at the time.  One of the best ways to nip this in the bud is to say something along the lines of: ‘one day, the code you’re writing now will be legacy code too!  Remember that some snotty nose kid will be making the same comments about you.”

Another reason to change this behavior is to eliminate a culture of finger pointing.  You want your devs to be taking responsibility for the whole code base.  They might not have made the messes that exist in the legacy code but they own it now.  Blaming someone else doesn’t fix these problems.

How to answer the refactor vs rewrite question

Once the team accepts that they own the mess, the more often you will hear the question.  Why don’t we just scrap it all and start over?  This question is a trap.  Be very careful how you answer it.

Why not give in to the devs and start over?  There are normally a ton of business reasons why you can’t do this.  You have to get the next product out the door so you don’t have the runway to start over.  Engineers understand this but the ivory tower dev mind in each of us has a hard time accepting the business reason alone.  The question becomes, if you did have the runway to scrap everything and start over, would it be a good idea then?

The first time I managed a team of developers, I wanted to answer this question with more than just the business reasons.  I knew that scrapping all previous code seemed like a bad idea but I struggled to articulate why.  So I went on the Google and ended up consulting the seer of Stack Exchange, Joel Spolsky.  If you haven’t checked out JoelOnSoftware – https://www.joelonsoftware.com/ , start reading his blog now.  His posts are insightful and incredibly well written.

Joel had the best answer that I’ve heard to this question and he used NetScape as an example.  His answer: the biggest benefit of legacy code is that it is production tested.  In many cases, it has thousands, if not millions of hours of customers pounding on that software.  All of those crazy little subroutines that you found that you have no idea what they do were built to fix a defect or to accommodate for a workflow that you would never have thought of in a million years.  What you naively think looks like a legacy code mess is actually a fount of knowledge gathered from many user years of production testing.

Joel then goes on to tell the story of Netscape.  Netscape was the top web browser in the nineties and early two thousands.  Netscape made the wrong choice.  They decided that with everything they had learned in their first versions of Netscape, they could start over and make a way better browser.  This was a disastrous decision.  They lost all of that production knowledge and eventually lost the browser wars because of it.  They started re-struggling with a bunch of problems that they had already solved in their previous versions and they never caught up to their competitors again.

The exceptions to the rule

It’s always ok to rewrite prototype code.  Prototype code doesn’t have any production knowledge behind it in most cases (at least it shouldn’t).  The purpose of prototype code is to research topics before they become production code.

In some cases it is also ok to rewrite modules of legacy code that can’t be optimized.  In most cases, these are prototypes or demos that somehow made it into the production code base that had no business being in there.  Be careful that this doesn’t become a convenient excuse though.  Devs will start calling all code prototype code if it means that they get to rewrite it.

The general rule of thumb for legacy code – always try to refactor before you rewrite.  If nothing else, this will force you and your team to try to understand what all of that production testing has to teach before you start tearing it down.

Why we suck at estimates

Posted on by Sean

Let’s start with a waterfall cautionary tale

Bob was a software engineer in the mid 90’s.  He was working for one of those really large companies that had the lion’s share of the market.  In theory, they released a new product once a year.  In practice, they released product once every 18 months or two years.  Bob’s team had just finished v4 of their popular document creation and management software.  This iteration was only 7 months late from when it was promised.  All in all, not too far off of their average.

The team was getting together to discuss version 5 with all of the key stakeholders.  This time was going to be different.  This time they were going to account for everything and release on time and under budget.  All of the managers vowed that they had learned from their mistakes and they ‘had this’.  Bob inwardly rolled his eyes but did his best to hop on board and toe the company line.  He sat through all of the meetings and gave his input on estimates, research required, design methodologies and architecture.  He thought they were underestimating many of the innovative new elements and said so.  The team increased those estimates by a token amount and signed off on the yearly project plan.

The first four months seemed to be going ok.  They were spending a little too much time on fixing defects coming in from v4 out in the field.  They had already released three hot fixes to deal with the more serious ones.  They were also struggling with two new technologies that they were trying to bring into the product.  One was a new open source library and the other was based on a functional language that the team didn’t have any experience with.  Their architect, Roger, said they were only days away from cracking both of those particular nuts so the atmosphere was still  optimistic.

Going into month 6, the road started to get a lot bumpier.  Turns out, the open source library they chose doesn’t scale.  They were going to have to either scrap all the functionality that relied upon it or rewrite those pieces of the library themselves.  The functional programming piece, on the other hand, was going wonderfully.  They put one of their younger aces on it and he was well ahead of schedule.  Unfortunately, he liked coding in this new language so much that he had just accepted a new position where he could code in it full time.  It was going to take at least three months to replace him and the team only had two weeks to do the knowledge transfer.

Bob’s immediate manager had put on at least 10 pounds and looked like he was working hard on his first ulcer.  When the sales team came in month 8 to say that their competitors had just released a new menu system that was killing it in the marketplace, the team knew they were in for a major scope change.  Those silver tongue bastards in sales had convinced the leadership team that they had to have this functionality or risk ruin.  The CEO, a salesman by trade, agreed.  Bob’s manager was told that they could scrap one of the smaller pieces of functionality to replace it with this.   Bob’s manager’s arguments of, the technology was not built to support the new menu system, fell on deaf ears.  The sales team got their way.

Bob and the team worked diligently to duct tape this new functionality on to the existing architecture.  They were building mountains of technical debt, but who cared, they were getting close to go time.  They hit their code complete date at month 11.  Everybody celebrated for about a day until QA got their hands on it.

Come month 15, QA declared the product a disaster.  Some of the bugs they were finding were so unwieldy that the team had to rewrite major core elements of the code base that had worked just fine in v4.  There were also rumors that the new menu system was not doing as well in the market as they had first heard and there was a large group of customers who hated it.  They were so committed at this point to making it work that they pushed through anyway.

The product was released in month 22 to mixed reviews.  95% of the customers who bought it turned off the new menu system in the config screen.  The sales guy who championed the idea left six months ago so there was no one to point the finger at but the leadership team.  Most of the developers working on the project left because they were so demoralized about spending so much time on something no one used.

Well duh, that’s why everyone has switched to agile methodologies, right?

Sadly, agile does not totally fix this problem.  It certainly doesn’t if implemented poorly.

Jump forward 15 years and meet Karen, an engineer for an ERP company.  Her company had just hired a new VP of Engineering for the sole reason of bringing agile into the company.  Their product was an enterprise product that still had to be installed on premise.  They had distant plans to move it to a SaaS product but not anytime soon.

This ERP company had about as good of a track record as Bob’s company when it came to delivering on time.  Their new VP of Engineering had a ton of energy and was well liked by everyone.  He was hugely successful in his last job where he built a bunch of custom websites for Fortune 500 companies.

Karen’s last product release was over two years late, so the team was ready for a change.  They soaked up the idea of agile and its two week sprints like sponges out of water.  The simple user story was such a relief after wading through 200 page design docs for so long.  Plus, QA was going to be working right next to them the entire time!  This had to work, right?

The trouble began in preparing for the very first release.  Their typical release schedule was every two years, plus or minus a year.  The VP fought for monthly releases and lost.  He then fought for quarterly releases and lost.  Our customers will never take releases that often, he was told.  Finally, he convinced the leadership team to do single year releases instead of every two years.  He explained to the team that they would be doing quarterly releases internally.

Sadly, the product team did not report to our new VP of Engineering.  So the product team built a yearlong product spec that did not have any adaptability for change.  The spec also required some serious innovation, new architecture and design patterns that the team had never used before.  The product team was also unwilling to split this spec into quarterly parts knowing that would probably require them to trim some of the functionality set that they had labored so hard over.

In the initial product meetings, the VP of Engineering got bullied into certain estimate limitations by the ‘experts’ in the company who were so adept at delivering product a year or two late.  This bullying happened after he and his team said the initial spec would take at least 18 months.  He got product to cut a couple of big features but no more.  His estimates still had them at 16 months.  He finally acquiesced when the CEO paid him a visit and said that all change had to happen gradually, even agile.

After the first quarterly internal release it was clear that there was no way they were going to make their annual release.  The new architecture was still very unsteady and had a host of unknowns.  Our VP was praised by the leadership team when he brought up these issues this early in the process.  “Most of the time we don’t find out about these type of problems until a month before the release,” said the CEO.  The VP of Engineering gave him a pained grin.

Come the second quarterly release, the team had figured out the architectural issues but in twice the time they had estimated.  The product team was already pissed that this meant a lot of feature cuts.  However, they were exuberant that they already had something to play with so early.  Naturally, they took the MVP out in the field and got some responses.  Not all of it was positive so they started changing scope.

Our VP of Engineering was excited about the changed scope because this was how agile was supposed to happen!  However, he lost the leadership team when he said that the team would have to reevaluate all of their previous estimates.  He tried again for monthly releases, cautioning that there would be even more scope changes as the customer feedback came in, so why not get rid of the whole idea of an annual release?  This just would not sink in with the old guard.

Our VP was being courted by other companies throughout and when he lost this last battle he gave up and took another position.  Karen and four other members of the team left with him.  They heard a year later that the product was released only 5 months late.  The company was celebrating that as a win.

These failures are so common in software development and a lot of that failure comes down to our reliance upon estimates.

Why is estimation so hard?

There are several principles of estimation that I have found to be universal.  If you don’t account for these principles, chances are you’re going to pay for it later.

  • Almost all estimates are given as best case scenarios
    • When we are asked how long something will take, we always think, “how long with this take if I have no interruptions?”
    • Having no interruptions is a fantasy.  We are constantly being put into context switching scenarios where we rarely find the ‘zone’.
    • Story points are your friend here.  Story points are relative based off of past history of the team and they do a good job of accounting for interruptions.
    • If you are forced into giving hourly estimates, try to push for days as your smallest unit of time and pad those estimates based on previous results.
  • We can only estimate as if the world around us is static
    • If there is one constant to this existence it is that change will happen.  Entropy is the way of the universe.
    • We can take a bunch of guesses about what may happen over a long period of time, but they will just be guesses and most of them will be wrong.
    • The further you look in the future, the more wrong you will be.  Keep your releases as short as possible.
  • We cannot estimate things we have never done before with any degree of accuracy
    • If you have unknowns or are working with technology that you or your team has never worked with before, your only true answer to how long it will take is: I don’t know.  It’s OK to say I don’t know.
    • Don’t fake it.  You need to de-risk these unknowns by prototyping or bringing in experts that have done these things before.
  • People who don’t understand technology assume that all technology is similar
    • Many execs assume that if they can talk intelligently about a product, they know how long it should take to build it.  This is so painfully wrong.
    • There is some willful ignorance here but most of it is that tech folks do a poor job of educating others.
    • Chances are you will hear this question many times: that program took only a month to build, why does this one take a year?  Don’t take offense, answer it honestly.
    • There is also an assumption that all technologists are the same.  This has gotten better over the last couple of years but it’s amazing how many people believe that a full stack developer means that that person knows about every technology.
    • There’s enough tech out there these days to require specialists.  Asking a UI developer to build a data layer is like asking an ophthalmologist to perform open heart surgery.  They could probably figure it out given enough time, but would you want them to?
    • Educate these folks as often as possible.  Don’t assume they understand technology like you do.
  • Your code will be far buggier than you think
    • Getting QA involved early and often helps dramatically
    • Unit test, unit test, unit test.  Look at TDD if the team can stomach it.
    • Automated testing helps a ton here too.
    • You have to budget for fixing defects.  So many developers give estimates as if everything will work perfectly the first time.
  • No one thinks about nonfunctional requirements while estimating until reminded
    • Nonfunctional requirements are things like scalability and performance.  Nobody thinks about this until the product is delivered and the CEO asks, “is it normal for the loading screen to take 5 minutes?”
    • Optimization takes a lot of time.  Plan for it.

Regardless of how progressively agile your company is, chances are you will be asked for estimates sometime along the way. Keep in mind that all estimates are guesses and that the further out you are the more likely you are to be wrong.  If you keep these estimation principles in mind, you will have the best chance of not making a complete ass of yourself.

Is Agilefall stealing your soul?

Posted on by Sean

Throw away the crystal ball

Almost everyone has acknowledged that long term project estimation needs to be a thing of the past.  If you are still doing waterfall development where you are trying to estimate projects 9 months or a year in the future, I’m sorry.  You are stuck in a pointless time loop.  It’s time we call these long term plans what they really are, guesses.  Why is a long term plan a guess?  In that period of time, things will change.  That is a guarantee.   Your customers’ needs will change.  The technologies you are building with will change.  Market forces will change.  You may be able to predict some of these changes but why would you want to?  Predictions are guesses.  You’re much better off setting yourself up to adapt to change.  If you can roll with the punches, chances are you will exit the ring standing up.

This is why agile development has become so widely accepted.  The core premise behind agile is that it allows a company to embrace change rather than fight it.  Conversely, this is why a lot of anachronistic big companies struggle so much to innovate.  When they take the waterfall approach and finally release product, the product they release would have been relevant a year ago (or two years depending on how late it was) but is not relevant today.

The onset of AgileFall

Even most big companies have started to recognize that waterfall is not getting the job done.  They have seen the studies.  They know they are losing more and more to innovators.  So, they decide to give the whole agile thing a try.  But, they don’t go all in.  It’s like watching Grandma adopt a cellphone.  Sure, she figures out how to use it, but she never picks the damn thing up when you call her.  Even worse, she leaves it sitting on the kitchen counter anytime she goes somewhere.  Might as well leave it plugged into the wall.

There’s many reasons why companies struggle with committing to agile.  Many of these reasons are good ones, at least to management.  Here’s the big three I’ve heard again and again:

  1. The folks on top don’t get it and aren’t that interested in learning
  2. ‘My enterprise customers will never take releases more than once a year’
  3. We need the time to build buzz and marketing campaigns around the upcoming release

For those of you that came out of the gate with SaaS, count your blessings.  You’ve never had to deal with this crap and probably never will.  For those of you with legacy code bases or working for larger companies, this is most likely the norm.

Break big things into smaller things and ship the smaller things

You’re chances of going from annual releases to releasing every day or every sprint are effectively zero.  Your best bet to get to real agile is by doing it piecemeal.  Treat it like any other engineering problem: break the big problem into smaller ones.  If you are doing annual releases, shift to quarterly releases.

This step by step approach allows you to combat the three most common pushbacks.  Convincing the folks on top that this is the way to go becomes a hell of a lot easier once you start hitting dates.  The logic here is sound and easy to manage up: if we’re doing less, there is less risk and less change.  Therefore, our predictability will increase dramatically.  If you take this approach, make damn sure you hit your dates or watch it all blow up in your face.  More on that in another post.

DIA waterfall development
DIA

Another bit of ammo to use in convincing those at the top is the well-known project failure.  Take your pick, there’s a mighty graveyard of failed projects at your fingertips.  Keep it topical.  I’m in Colorado so it was easy for me to reference DIA (Denver International Airport) which opened sixteen months late at a cost overrun of $2 billion.  Talk about your big project failure.  If you’re on the east coast, use the Big Dig.  The argument becomes even stronger if you can reference a similar, smaller project success.  For us, this was the second round of the light rail once Denver’s Union Station was completed.  They didn’t try to launch it with all lines opening at once.  Instead, they first launched the train to the plane.  Then they started opening up another line every quarter.  To my knowledge, they haven’t missed a date.  Each successful line opened is another feather in the cap that builds trust.

Union Station Agile Development
The New Union Station

As for enterprise customers not taking releases more than once a year, this is not a tough argument either.  Make sure that these customers don’t HAVE to take each quarterly release.  If they still want to take just one release a year, fine, let them take every fourth release.  The trick to this approach is in making sure that each release builds on the previous one and you are using only one code line.  The minute you break into multiple code lines, you’re done.  The support nightmares will take down your company.

Somewhat surprisingly, your sales and marketing teams will quickly become your allies.  With the quarterly release cycle, you’ve now given them a lot more to do and a lot more to talk about.  This is again only true if you hit your dates.  You miss your date once, they may forgive you but more than once and the trust is most likely permanently broken.  This happens and folks lose faith in the product and you can wave goodbye to your best sales folks.

Agile Development Quarterly Release
What Quarterly Releases Look like

Do it once and doing it again is a LOT easier

Once you successfully switch over to a quarterly release, a monthly release is the next logical step.  If you had any success in the quarterly release cycle, the monthly release cycle is a much easier conversation.  You have already established yourself as an expert.  Chances are your value to the company at this point is pretty high and you don’t have to convince the folks up top.

Where the battle gets more interesting here is with sales, marketing and your deployment teams.  They’re going to feel overwhelmed.  Those teams that cheered you on when you hit your quarterly releases will turn on you.  Take a minute to put yourself in their shoes.  All of a sudden they have to constantly learn new features.   I can hear all of you SaaSers giggling but keep in mind that these folks came from annual releases.  Many of them have never experienced things like quick feedback from their customers and pivots based off of that feedback.  This is a sea change that requires their roles to change.  That’s scary.

The pain of this transition can be eased by making the first two monthly releases of the quarter ‘internal’.  You still cut a build and move on to the next one but you don’t actually ship it.  This gives you time to continue building trust around the monthly release.  It gives them time to come to grips with how their roles need to change.  They are now selling and marketing an ever-evolving product instead of incremental releases.   It also allows your deployment teams the time to turn into devOps teams.  It takes time to make those process changes.

Once the other teams have become comfortable with monthly release cycles, moving to releasing every sprint is a very easy step.  Now you’re doing agile!

The horrors of context switching (and how to beat them) – Part 2

Posted on by Sean

Part Deux

The first post on this topic covered many of the ways that context switching can destroy the productivity of your day but most of the post was aimed at developers.  This one reaches out to everyone that works in an office.  Scott Adams has made a career out of ridiculing the pointlessness of workplace politics.  In no setting is this ridicule more pronounced than the pointless meeting.

Agile best practices

Multitasking meetings?

Have you ever been in a meeting and looked around the table while someone is giving a presentation?  What do you see?  Too many times I would see a scene like this:  John is looking at his phone, Betty is trolling the web while Michael is writing on a post-it in front of Margaret.  Sound familiar?  Exactly, the workplace has regressed back to fifth grade.

Agile best practices

Even worse, I once had a CEO that would regularly take phone calls from his wife while we were in a meeting and would talk with her on the phone for a good 5-10 minutes while everyone would be sitting there waiting.  Sometimes, he would even lapse into baby talk.  It was a fantastic use of time.  This happened in executive leadership meetings all the time.  Let’s look at the economics behind this.  If he took a call with eight of us in the room, each of us making an average of $250/hr, each 10 minute phone call he took would cost the company roughly $500.  This accounts for the 5 minutes it would take for us to get back to where we were in the meeting.  Granted, this guy was a world class asshole but these types of things happen all the time in meetings across the globe.

Luckily, there is a cure for the pointless meeting.

7 tips for a productive meeting

The best way to deal with distractions if these are your meetings is to be honest with your teams.  Be very clear that you are not going to waste their time but be just as clear that you will not tolerate them wasting yours or other people’s time.  When people aren’t paying attention, most of the time it is because they are not engaged.  That’s on you as a leader.  These are 7 tips that I have found to be incredibly effective for holding a highly engaged meeting:

  1. Have an agenda and stick to it
  2. ONLY invite people that can contribute to the topics on the agenda
  3. Lids down.  Phones down.
  4. Start on time
  5. Don’t take up the full meeting time if you don’t have to
  6. At the end of the meeting ask the attendees if they thought it was productive or not
  7. Write down action items and send them out with a summary after the meeting

Make it about respect

The strongest statement you can make that the meeting will be productive is to have an agenda.  Having an agenda means that you have enough respect for your team that you prepared in advance.  If it is not your meeting, it is perfectly acceptable to ask what the agenda is.  If the leader of the meeting doesn’t have one, make sure you get that person to cover the points they hope to get done in the meeting before the meeting starts.  I’m at a stage in my career that I will typically leave a meeting if there is no agenda.  The end result of this action is that I either don’t get invited to, in my opinion pointless, meetings or the leader of the meeting puts together an agenda before inviting me.

Always run the numbers

Run the numbers.  If somebody is not going to be contributing in the meeting and you invite them to it anyway, ask yourself, how much do they make an hour?  If they’re not contributing am I willing to burn $100 of the company’s money to have them there doing nothing?  Many times people are invited to meetings not to contribute but just so they don’t miss out on information being passed in the meeting.  This is a terrible use of their time.  A simple summary email once the meeting is over is far more effective and will not require context switching.

Once the agenda is briefly covered, expect all participants of the meeting to be engaged.  The best way to enforce this is by asking for lids down and phones down.  It is far more difficult to be engaged if the distraction of a laptop or a phone is staring you in the face.

Start your meetings on time.  Do not wait for stragglers that are rolling in late.  Start on time and either end on time or end early.  Respect your people by treating them like adults.  If somebody rolls in late, let them catch up on the topic in their own time.  Do not keep stopping the meeting to fill folks in on what they missed.  That’s a waste of everyone’s time.  Even if it is somebody who is higher than you in the corporate hierarchy.  If somebody asks for a summary after showing up late, you can handle this with tact.  Say something along the lines of, “I’ll be happy to fill you in later but with respect to everyone’s time we need to stay on topic.”

I used this approach with a former COO of mine.  This COO would roll into every meeting 5 to 10 minutes late.  I could tell he was annoyed with my approach but, very professionally, he waited until the meeting was over to let me know.  He explained that he was incredibly busy and that most other meetings he had ran over.  I let him know that I understood that frustration but then I went to the numbers.  If I interrupted the meeting every time someone came in late, I not only risked disengaging the rest of the team, but we would be forced to go off topic, then context switch back to the topic on hand.  If there were 10 people in the room each making roughly $100 an hour, that 5 minute interruption costs almost $100.  If we do that twice a meeting, it starts to get very expensive.  The other point I made was that if I only were to do this for him because he outranked me, I would be showing my team that we have two different standards in the company in regards to respecting people’s time.  We would basically be saying that it’s ok to waste other people’s time as long as you outrank them.  To his credit, he just nodded.  I would love to say that after that conversation, he was never late to a meeting but that just wasn’t true.  He never asked me for a summary again however.

When you get through your points, end the meeting

Once you are through with the agenda, END THE MEETING.  Nothing is more infuriating than having a leader of a meeting look at the clock and say, “Oooh look, we still have 20 minutes left, let’s try to get this done too!”  Why is that so infuriating?  Because of tip #2.  You put the attendees of that meeting together ahead of time to contribute to the topics on the agenda.  Once you go off of those topics, somebody, if not everybody will feel like you are wasting their time.  Ending early is a sign of efficiency and your team will respect you for it.  Throwing pointless filler in at the end of a meeting is a waste of resources.

Incorporate the feedback loop.  At the end of the meeting ask the group if they thought the meeting was an effective use of their time.  Make sure you get an answer from each attendee.  It is critical that you are open to this feedback or else it just comes across as lip service.  If an attendee says they think that it was an ineffective use of their time, make sure you get them to give you a couple of ideas of how they would make the time more effective.  If the majority of the attendees say the meeting was an ineffective use of time, you need to be ok with cancelling similar future meetings especially if they are scheduled weekly.

Finally, make sure to write down the action items that come up during the meeting.  When the meeting is over, take five minutes to write up a summary that gets sent out to the attendees and anyone else that needs to know what was discussed in the meeting.  This summary should also contain the action items that came out of the meeting.  This is critical because this is a written record of the commitments that people made in the meeting.

Conclusion

I would say that at least 60% of all meetings are a complete waste of time.  Not only do you lose the time that is spent in the pointless meeting, you also lose all the time around the meeting when you have to context switch and try to get back in the flow of what you were actually working on.  It doesn’t have to be this way:

Agile best practices

The horrors of context switching (and how to beat them) Part 1

Posted on by Sean

Context Switching

Context switching is the bane of productivity.  For those of you not familiar with the term: context switching is when one is forced to switch from one topic or work item to the next.  This is also known as multi-tasking.  Sadly, too many people believe they are excellent multi-taskers.  This is nonsense.  Nobody is an excellent multi-tasker.

The brain is single threaded.  We are not processors that can be running two tasks in parallel.  We don’t have the internal CPU for such a job.  Every time we switch tasks, we have to rebuild our mental architecture.  We have to tear down the thought processes for our current task and replace them with the thought processes for the new task.  This takes time when we are working in a vacuum.  But when’s the last time you worked in a vacuum?  Think about how much more time this takes when we are getting interrupted on a regular basis.

This is the famous chart that Gerry Weinberg put together to illustrate the waste caused by context switching.

Context Switching Chart 1

If the numbers aren’t working for you in the first chart, that is because several assumptions are being made.  The chart below explains those assumptions in a little more detail.

Context Switching Chart 2

If you don’t find these graphs shocking, you’re not reading them right.  If you are working on two projects at one time, then you are losing a full 20% of your productivity.  If you are working on 5?  75% of your productivity goes straight in the shitter.  To say that another way, when you are working five projects, each project gets 1/20th of your time.  Rebuilding your mental architecture each time gets 3/4ths of your time.

That doesn’t apply to me, I’m a great multi-tasker

Oh yeah?  If you are such a good multi-tasker do these two simple tasks for me:  recite the lyrics of your favorite song and calculate the square root of 2,116.  The hitch?  You have to do these two tasks at the same time.  Start the lyrics now and do not pause.  Before you finish those lyrics, come up with the answer to the math problem.

In my personal experiments, one of two things happen here.  The person gives up OR they come up with the right answer to the math problem but keep losing their place in the song.  Humans are NOT capable of multi-tasking.  Unless you are a cyborg or have figured out how to split your brain, please stop claiming that you are a good multi-tasker.  My crappy personal experimentation aside, the science is in.  You can start here for more info: http://www.npr.org/templates/story/story.php?storyId=95256794.

There is always some wise-ass at this point that asks the question: if that is true how can I drive and hold a conversation?  OR, how can I chew gum and walk at the same time.  The answer: habits don’t count.  The things that you have done so many times where your lizard brain takes over and the action gets moved to auto pilot don’t rely on higher level thought.  We are only talking about those tasks that take higher level thought.  Which, hopefully, is most of the tasks you do every day at work.

Interesting sidebar edit:  It took me an hour and a half to write 80% of this column.  I received a phone call when I was almost done writing that I had to take.  It took me another hour and a half to write the final 20%.  Context switching in action.

So what do we do about this loss of productivity?

One of my old bosses had a great philosophy: you want to move faster?  Do less at one time.  This is another way of stating the old maxim that less is more.

Let’s get specific and address one of two primary issues where context switching is a huge killer: software development.  We’ll cover the other productivity killing issue, meetings, in the next post.

Cut your WIP (work in progress) ASAP

This is where the less is more rubber hits the road.  When it comes to agile, one of the questions we need to be asking ourselves all the time is how much we can cut our work in progress.  This questioning is where Kanban can be incredibly effective.  Kanban translates to ‘visual signal’ and was pioneered on the floors of Toyota plants back in the fifties.  Most people will use Kanban to give the team and folks outside of the team insight on the progress of work.  Boards like the one below can be seen in almost every agile shop out there:

Kanban Board 1

Unfortunately, for most teams, that’s where the utilization of Kanban ends.  This is a shame.  If you end with just the visual representation of the work moving from stage to stage you are missing out on the critical improvements that Kanban can offer you and your team.  As you dive deeper you realize that much of the benefit of Kanban comes from identifying and eliminating bottlenecks.

One of the ways to tell if you have bottlenecks is to identify where most of your cards are backing up on your Kanban board.  If most of your cards are in your backlog (to do) or sitting in Done, this isn’t much of a problem for your team.  If however, most of your cards end up sitting in develop or test for most of the sprint right up until the last day: you’ve got WIP problems.

WIP problems typically fall into two categories: context switching and process problems.  These categories have a ton of overlap, so we’ll talk about both together.

Set fixed limits for WIP

In a previous life, one of the teams I worked with had six devs, three QA and a product owner.  Our rule for this group was the team could only have six WIP cards that were between the backlog and the done swim lanes.

The reaction to this early on was unexpected.  The developers complained that they would be sitting on their hands a lot waiting.  Granted, there was some hand sitting but not nearly as much as you would think.  Instead, the team gelled better than ever before.  A lot of agile teams pay lip service to the idea that no matter the task, the whole team pitches in to move product through the pipeline.  So dev will help with QA, QA will help with requirements definition, etc.  The sad reality is that these teams almost always get siloed.  Developers just work on dev and QA just works on quality.

What happened with our team after instituting very strict WIP limits is that the devs got a lot more involved with the QA process and helped test other developer’s stories.  In doing that testing, they gained a much better understanding of how QA was writing test cases.  Most QA folks can’t code so QA didn’t implement any stories but the QA team got much more involved in the definition of these stories.  Our devs also learned how valuable a good unit test is.

In the end, the numbers told the final story.  That team went from an average push rate (work items being pushed at the end of a sprint) of 17% to under 5%.  And their overall velocity increased by 10%.  What we found was that keeping the whole team focused on a small number of items at one time caused us to move a lot faster.  Less is more when you limit context switching.

You will need to play around with what your WIP limits are but err on the side of less work items at first.  Even though it may feel like some people are sitting around waiting to work, people generally like to work.  Chances are, you will find them helping out with other jobs and understanding the process as a whole a lot better.  You will also quickly identify your bottlenecks.  For us, the bottleneck was QA.  Having developers finish more stories was only backing up the QA machine in the production line.  Having dev help out with QA made everyone faster.

Segregate the issues coming in from the field

Another issue that can completely destroy your team’s productivity is adding work items to the sprint after the sprint has started.  This is agile 101.  Most of you know this and know that most product owners would never dare to add a new work item mid sprint.  But what happens when a defect comes in from the field?  This issue turns out to be bad enough that the customer is genuinely pissed off and now you have your CEO breathing down your neck.

The most effective way I’ve found to deal with this issue is to build a dev support team.  We had a rotating dev support team that was made up of one developer with another on back up.  The person on the dev support team was only on that team for two sprints at a time.  Then they would move to the backup position.  When they were the primary support dev, their only job was to handle issues coming in from the field.  When there were no issues coming in from the field, they would fix defects.  They were never assigned story work.

This is another great anti-silo approach.  This forces all of your developers to become familiar with the entire codebase.  It also shows you as a leader, how your different devs react to stress.

What if I’m on a custom software team?

If you’re building custom software for clients, context switching comes with the territory.  It can be managed but not eliminated.  The most effective custom or outsource teams that I have worked with manage this in a couple of ways.  First, they budget for the context switching costs.  Most of these teams have been doing this long enough that they understand the impact of context switching WAY better than product development teams so they incorporate it into their estimates.  They also mitigate some of the risk by trying to keep their resources on one project and only one project as long as they can.  When that is not a possibility, the best outsource teams that I have worked with try to limit their resources to working on only one project per day.  It’s easier to rebuild that mental architecture if you only have to do it once per day.

Conclusion

None of us can multi-task effectively which is why context switching is such a killer.  Acknowledging that multi-tasking is the problem is the first step in the healing process.  The more you can eliminate context switching from your life and the lives of your team, the more productive you will be.  This is part one of the context switching topic wherein we covered the mechanics of context switching in the agile product cycle.  In the next post we will cover the other big context switching time thief, meetings.

Feed the machine a healthy diet

Posted on by Sean

Requirements and the agile SDLC

Everyone is familiar with the expression, ‘garbage in equals garbage out’ by now.  This simple statement neatly articulates the necessity for quality software requirements.  Before you roll your eyes about yet another post on how to create software requirements, know that we are going to take a slightly different approach.

If you’re building a simple web or mobile app a basic knowledge of any of the agile philosophies is all you need.  They will have you putting together solid user stories to get you to a reliable end result.  What happens when you are dealing with far more complexity, like enterprise software?  How does this change the agile paradigms, if at all?

Let’s look at this through the lens of an enterprise R&D team.  Some of the questions that R&D regularly deals with:  how do you build software requirements when you don’t even know if what you want to build is technically possible?  Or, how do you reliably build requirements for a new component to an enterprise product when you don’t know what the integration between the new component and the existing components looks like?  When does product management  even get involved with these types of early stage modeling?

The root of the issue is managing unknowns.  More specifically, big unknowns.  These unknowns are too large for a single research spike.  For those of you not familiar, a research spike is a timeboxed user story with the goal of answering a question or gathering information rather than shipping product.  Typically, these are created when a user story is generated that cannot be well estimated until the dev team does some work to try to resolve a design issue or a technical question.

Another problem that arises fairly often when unknowns are present is research spike breeding.  One research spike is created, the answers are not sufficient, so that research spike creates three more.  Each of those create more and very quickly your sprint has been taken over by these cunicular little beasts.

One of the keys to a successful agile sprint is small stories that can be completed in one sprint.  Where a ton of agile teams, especially new ones, fall on their faces is adding stories that can’t be completed in a single sprint.  This causes the dreaded story push at the end of the sprint.  Or, even worse, extending the sprint.  More often than not, these stories are pushed again and again.  Sound familiar?

So what can you do about it?  The best way to get over this hurdle is through de-risking.

One of the best descriptions of scrum at an enterprise level that I’ve seen came from one of my mentors, Dr. Tom Ngo.  He created the diagram below.  Each block represents a potential story or epic story.  The black vertical lines of blocks at the bottom represent sprints or iterations of these stories.  This diagram represents the progression of risky stories to stories that can actually be completed in one sprint.

Enterprise Scrum
Agile design model

CURBed is an acronym that we made up that stands for Complex, has Unknowns, Research required or too Big.  These red blocks are raw ideas and are typically high risk.  It is up to your product and architecture teams to eliminate this risk and make these estimable by your dev and QA teams.  We’ll get into some guidelines of how you do that in a minute.  Once the team estimates these stories, only then can they be committed to a sprint.

Compare this to what a lot of teams do in this situation.  They don’t go through a thorough de-risking process.  Instead, they come up with a lot of cool ideas, get a ballpark estimate on each idea from one of their developers, then add it to a sprint.  That looks a little like this:

Chaos Scrum
Poor agile design model

The result is typically chaos.  You end up with a lot of pissed off stakeholders wondering why you can’t seem to ever hit a single commitment.  Stories end up on sprints with dependencies, research spikes are multiplying like rabbits and your PO is in a constant panic.  Not a fun way to work.

De-risking in the agile SDLC

Enter de-risking stage left.  This is what the path of de-risking looks like:

De-risking
De-risking process

Proposed ideas enter in raw form.  The product team and the architecture team prioritize and start de-risking.  The hardest step is going from CURBed to estimable.  Once you get to estimable, dev and QA can do some estimate poker and quickly get you to an estimated or committable state fairly quickly.  The queue of committable stories becomes your product backlog.  Then it’s up to your PO to decide when these items get thrown on the sprint.

9 critical steps of agile design

Every team is going to have to deal with getting from high risk, or CURBed stories, to something that is estimable in their own way.  Building a plan and sticking to it is the most important step you can make to start reigning in the chaos.  This is a methodology that worked for us and that has also worked for other teams with whom I’ve consulted.  This is what I call the 9 Critical Steps of Design or how to get from CURBed to estimable.

9 step design process
9 critical steps of agile design

It starts with figuring out the science of the problem.  If the team has never used an element of the tech stack or proposed design pattern, you have to prototype the hell out of it.  This includes bringing in outside expertise if necessary.  This is just for tough problems that the team has never worked on before.  If it’s just a coding exercise and you know the science, you can skip the prototype phase.  Don’t get complacent though because these prototypes are critical to future success.  If you have the time, always prototype.

Once the science is worked out, you can start on the high level stories.  We are talking very high level: ‘I am an administrator and I want to be able to log in to another user’s account’ as an example.  Don’t get more detailed yet.  Steps three and four are critical.  This is where the team gets aligned.  I’ll do another post on use case diagrams and sequence diagrams.  The goal of these diagrams is to eliminate confusion but also to set your devs up for success.  Every dev should have the information to implement any of these stories once these diagrams are built.

Once those diagrams are in place, the QA team can start building out test case stubs.  QA is a huge weapon in the design process.  Developers and architects think about how to build the product.  QA thinks about how to break it.  If you are not using their destructive nature in the design and de-risking process, you’re doing it wrong.

After a review, the product team can then start filling out the details of things like acceptance criteria.  Step 8, estimate and commit, really happens after the estimable phase and is what takes the team to committable.  Use whatever tools are in your arsenal to do the estimate, then add those stories to your backlog.

The feedback loop is important here too.  Many teams that I have worked with started well until change was introduced.  This change typically came in as a customer request or customer feedback.  The teams then fell back into their old patterns of trying to throw those items directly on the sprint once again without design.  The end result: chaos.

The overhead of going through a process like this looks high until you look at the alternative.  Nobody wants chaos in their organization.  Yes, agile’s primary goal is to adapt to change.  This does not mean that you have to sacrifice all predictability.  You may not know what the product looks like six sprints down the road but you should know what it looks like at the end of the current sprint.  Without de-risking, this is almost never true.

Are you ready to end the chaos in your product delivery org?  Start de-risking.