Context Switching

Context switching is the bane of productivity. For those of you not familiar with the term: context switching is when one is forced to switch from one topic or work item to the next. This is also known as multi-tasking. Sadly, too many people believe they are excellent multi-taskers. This is nonsense. Nobody is an excellent multi-tasker.

The brain is single threaded. We are not processors that can be running two tasks in parallel. We don’t have the internal CPU for such a job. Every time we switch tasks, we have to rebuild our mental architecture. We have to tear down the thought processes for our current task and replace them with the thought processes for the new task. This takes time when we are working in a vacuum. But when’s the last time you worked in a vacuum? Think about how much more time this takes when we are getting interrupted on a regular basis.

This is the famous chart that Gerry Weinberg put together to illustrate the waste caused by context switching.

Context Switching Chart 1

If the numbers aren’t working for you in the first chart, that is because several assumptions are being made. The chart below explains those assumptions in a little more detail.

Context Switching Chart 2

If you don’t find these graphs shocking, you’re not reading them right. If you are working on two projects at one time, then you are losing a full 20% of your productivity. If you are working on 5? 75% of your productivity goes straight in the shitter. To say that another way, when you are working five projects, each project gets 1/20th of your time. Rebuilding your mental architecture each time gets 3/4ths of your time.

That doesn’t apply to me, I’m a great multi-tasker

Oh yeah? If you are such a good multi-tasker do these two simple tasks for me: recite the lyrics of your favorite song and calculate the square root of 2,116. The hitch? You have to do these two tasks at the same time. Start the lyrics now and do not pause. Before you finish those lyrics, come up with the answer to the math problem.

In my personal experiments, one of two things happen here. The person gives up OR they come up with the right answer to the math problem but keep losing their place in the song. Humans are NOT capable of multi-tasking. Unless you are a cyborg or have figured out how to split your brain, please stop claiming that you are a good multi-tasker. My crappy personal experimentation aside, the science is in. You can start here for more info: http://www.npr.org/templates/story/story.php?storyId=95256794.

There is always some wise-ass at this point that asks the question: if that is true how can I drive and hold a conversation? OR, how can I chew gum and walk at the same time. The answer: habits don’t count. The things that you have done so many times where your lizard brain takes over and the action gets moved to auto pilot don’t rely on higher level thought. We are only talking about those tasks that take higher level thought. Which, hopefully, is most of the tasks you do every day at work.

Interesting sidebar edit: It took me an hour and a half to write 80% of this column. I received a phone call when I was almost done writing that I had to take. It took me another hour and a half to write the final 20%. Context switching in action.

So what do we do about this loss of productivity?

One of my old bosses had a great philosophy: you want to move faster? Do less at one time. This is another way of stating the old maxim that less is more.

Let’s get specific and address one of two primary issues where context switching is a huge killer: software development. We’ll cover the other productivity killing issue, meetings, in the next post.

Cut your WIP (work in progress) ASAP

This is where the less is more rubber hits the road. When it comes to agile, one of the questions we need to be asking ourselves all the time is how much we can cut our work in progress. This questioning is where Kanban can be incredibly effective. Kanban translates to ‘visual signal’ and was pioneered on the floors of Toyota plants back in the fifties. Most people will use Kanban to give the team and folks outside of the team insight on the progress of work. Boards like the one below can be seen in almost every agile shop out there:

Kanban Board 1

Unfortunately, for most teams, that’s where the utilization of Kanban ends. This is a shame. If you end with just the visual representation of the work moving from stage to stage you are missing out on the critical improvements that Kanban can offer you and your team. As you dive deeper you realize that much of the benefit of Kanban comes from identifying and eliminating bottlenecks.

One of the ways to tell if you have bottlenecks is to identify where most of your cards are backing up on your Kanban board. If most of your cards are in your backlog (to do) or sitting in Done, this isn’t much of a problem for your team. If however, most of your cards end up sitting in develop or test for most of the sprint right up until the last day: you’ve got WIP problems.

WIP problems typically fall into two categories: context switching and process problems. These categories have a ton of overlap, so we’ll talk about both together.

Set fixed limits for WIP

In a previous life, one of the teams I worked with had six devs, three QA and a product owner. Our rule for this group was the team could only have six WIP cards that were between the backlog and the done swim lanes.

The reaction to this early on was unexpected. The developers complained that they would be sitting on their hands a lot waiting. Granted, there was some hand sitting but not nearly as much as you would think. Instead, the team gelled better than ever before. A lot of agile teams pay lip service to the idea that no matter the task, the whole team pitches in to move product through the pipeline. So dev will help with QA, QA will help with requirements definition, etc. The sad reality is that these teams almost always get siloed. Developers just work on dev and QA just works on quality.

What happened with our team after instituting very strict WIP limits is that the devs got a lot more involved with the QA process and helped test other developer’s stories. In doing that testing, they gained a much better understanding of how QA was writing test cases. Most QA folks can’t code so QA didn’t implement any stories but the QA team got much more involved in the definition of these stories. Our devs also learned how valuable a good unit test is.

In the end, the numbers told the final story. That team went from an average push rate (work items being pushed at the end of a sprint) of 17% to under 5%. And their overall velocity increased by 10%. What we found was that keeping the whole team focused on a small number of items at one time caused us to move a lot faster. Less is more when you limit context switching.

You will need to play around with what your WIP limits are but err on the side of less work items at first. Even though it may feel like some people are sitting around waiting to work, people generally like to work. Chances are, you will find them helping out with other jobs and understanding the process as a whole a lot better. You will also quickly identify your bottlenecks. For us, the bottleneck was QA. Having developers finish more stories was only backing up the QA machine in the production line. Having dev help out with QA made everyone faster.

Segregate the issues coming in from the field

Another issue that can completely destroy your team’s productivity is adding work items to the sprint after the sprint has started. This is agile 101. Most of you know this and know that most product owners would never dare to add a new work item mid sprint. But what happens when a defect comes in from the field? This issue turns out to be bad enough that the customer is genuinely pissed off and now you have your CEO breathing down your neck.

The most effective way I’ve found to deal with this issue is to build a dev support team. We had a rotating dev support team that was made up of one developer with another on back up. The person on the dev support team was only on that team for two sprints at a time. Then they would move to the backup position. When they were the primary support dev, their only job was to handle issues coming in from the field. When there were no issues coming in from the field, they would fix defects. They were never assigned story work.

This is another great anti-silo approach. This forces all of your developers to become familiar with the entire codebase. It also shows you as a leader, how your different devs react to stress.

What if I’m on a custom software team?

If you’re building custom software for clients, context switching comes with the territory. It can be managed but not eliminated. The most effective custom or outsource teams that I have worked with manage this in a couple of ways. First, they budget for the context switching costs. Most of these teams have been doing this long enough that they understand the impact of context switching WAY better than product development teams so they incorporate it into their estimates. They also mitigate some of the risk by trying to keep their resources on one project and only one project as long as they can. When that is not a possibility, the best outsource teams that I have worked with try to limit their resources to working on only one project per day. It’s easier to rebuild that mental architecture if you only have to do it once per day.

Conclusion

None of us can multi-task effectively which is why context switching is such a killer. Acknowledging that multi-tasking is the problem is the first step in the healing process. The more you can eliminate context switching from your life and the lives of your team, the more productive you will be. This is part one of the context switching topic wherein we covered the mechanics of context switching in the agile product cycle. In the next post we will cover the other big context switching time thief, meetings.

Requirements and the agile SDLC

Everyone is familiar with the expression, ‘garbage in equals garbage out’ by now. This simple statement neatly articulates the necessity for quality software requirements. Before you roll your eyes about yet another post on how to create software requirements, know that we are going to take a slightly different approach.

If you’re building a simple web or mobile app a basic knowledge of any of the agile philosophies is all you need. They will have you putting together solid user stories to get you to a reliable end result. What happens when you are dealing with far more complexity, like enterprise software? How does this change the agile paradigms, if at all?

Let’s look at this through the lens of an enterprise R&D team. Some of the questions that R&D regularly deals with: how do you build software requirements when you don’t even know if what you want to build is technically possible? Or, how do you reliably build requirements for a new component to an enterprise product when you don’t know what the integration between the new component and the existing components looks like? When does product management even get involved with these types of early stage modeling?

The root of the issue is managing unknowns. More specifically, big unknowns. These unknowns are too large for a single research spike. For those of you not familiar, a research spike is a timeboxed user story with the goal of answering a question or gathering information rather than shipping product. Typically, these are created when a user story is generated that cannot be well estimated until the dev team does some work to try to resolve a design issue or a technical question.

Another problem that arises fairly often when unknowns are present is research spike breeding. One research spike is created, the answers are not sufficient, so that research spike creates three more. Each of those create more and very quickly your sprint has been taken over by these cunicular little beasts.

One of the keys to a successful agile sprint is small stories that can be completed in one sprint. Where a ton of agile teams, especially new ones, fall on their faces is adding stories that can’t be completed in a single sprint. This causes the dreaded story push at the end of the sprint. Or, even worse, extending the sprint. More often than not, these stories are pushed again and again. Sound familiar?

So what can you do about it? The best way to get over this hurdle is through de-risking.

One of the best descriptions of scrum at an enterprise level that I’ve seen came from one of my mentors, Dr. Tom Ngo. He created the diagram below. Each block represents a potential story or epic story. The black vertical lines of blocks at the bottom represent sprints or iterations of these stories. This diagram represents the progression of risky stories to stories that can actually be completed in one sprint.

CURBed is an acronym that we made up that stands for Complex, has Unknowns, Research required or too Big. These red blocks are raw ideas and are typically high risk. It is up to your product and architecture teams to eliminate this risk and make these estimable by your dev and QA teams. We’ll get into some guidelines of how you do that in a minute. Once the team estimates these stories, only then can they be committed to a sprint.

Compare this to what a lot of teams do in this situation. They don’t go through a thorough de-risking process. Instead, they come up with a lot of cool ideas, get a ballpark estimate on each idea from one of their developers, then add it to a sprint. That looks a little like this:

The result is typically chaos. You end up with a lot of pissed off stakeholders wondering why you can’t seem to ever hit a single commitment. Stories end up on sprints with dependencies, research spikes are multiplying like rabbits and your PO is in a constant panic. Not a fun way to work.

De-risking in the agile SDLC

Enter de-risking stage left. This is what the path of de-risking looks like:

Proposed ideas enter in raw form. The product team and the architecture team prioritize and start de-risking. The hardest step is going from CURBed to estimable. Once you get to estimable, dev and QA can do some estimate poker and quickly get you to an estimated or committable state fairly quickly. The queue of committable stories becomes your product backlog. Then it’s up to your PO to decide when these items get thrown on the sprint.

9 critical steps of agile design

Every team is going to have to deal with getting from high risk, or CURBed stories, to something that is estimable in their own way. Building a plan and sticking to it is the most important step you can make to start reigning in the chaos. This is a methodology that worked for us and that has also worked for other teams with whom I’ve consulted. This is what I call the 9 Critical Steps of Design or how to get from CURBed to estimable.

9 step design process — 9 critical steps of agile design

It starts with figuring out the science of the problem. If the team has never used an element of the tech stack or proposed design pattern, you have to prototype the hell out of it. This includes bringing in outside expertise if necessary. This is just for tough problems that the team has never worked on before. If it’s just a coding exercise and you know the science, you can skip the prototype phase. Don’t get complacent though because these prototypes are critical to future success. If you have the time, always prototype.

Once the science is worked out, you can start on the high level stories. We are talking very high level: ‘I am an administrator and I want to be able to log in to another user’s account’ as an example. Don’t get more detailed yet. Steps three and four are critical. This is where the team gets aligned. I’ll do another post on use case diagrams and sequence diagrams. The goal of these diagrams is to eliminate confusion but also to set your devs up for success. Every dev should have the information to implement any of these stories once these diagrams are built.

Once those diagrams are in place, the QA team can start building out test case stubs. QA is a huge weapon in the design process. Developers and architects think about how to build the product. QA thinks about how to break it. If you are not using their destructive nature in the design and de-risking process, you’re doing it wrong.

After a review, the product team can then start filling out the details of things like acceptance criteria. Step 8, estimate and commit, really happens after the estimable phase and is what takes the team to committable. Use whatever tools are in your arsenal to do the estimate, then add those stories to your backlog.

The feedback loop is important here too. Many teams that I have worked with started well until change was introduced. This change typically came in as a customer request or customer feedback. The teams then fell back into their old patterns of trying to throw those items directly on the sprint once again without design. The end result: chaos.

The overhead of going through a process like this looks high until you look at the alternative. Nobody wants chaos in their organization. Yes, agile’s primary goal is to adapt to change. This does not mean that you have to sacrifice all predictability. You may not know what the product looks like six sprints down the road but you should know what it looks like at the end of the current sprint. Without de-risking, this is almost never true.

Are you ready to end the chaos in your product delivery org? Start de-risking.

Agility in the workplace

Work in progress

The horrors of context switching (and how to beat them) Part 1