How to Manage Scope Creep in an Agile Project?

How do you manage scope creep in an agile project?

Welcome to the Amazing Apps show -- the podcast for you because you want to build amazing, agile Microsoft Business Applications that everyone will love.

Hi, I'm your host Neil Benson. My goal on this show is to help you slash your project budgets, reduce your delivery timelines, mitigate technical risks and create amazing, agile Microsoft Dynamics 365 and Power Platform applications.

Welcome to another Q&A episode. In this episode, Stephen Price asks about managing scope in an agile business apps project. Stephen is a digital solution architect at ITK Consulting in Calgary in Canada. Over to you Stephen.

Stephen’s asking about scope creep within a sprint as opposed to adding scope to a backlog. That’s an important distinction, so let’s start there.

Just before we get into Stephen’s question, I wanted to give a quick shout out to one of the Customery Academy students, Gavin Embley. Gavin is a CRM Consultant at 365 InHouse and joined my Scrum for Microsoft Business Apps course and went on to achieve his Scrum certification with Scrum.org. You can join Gavin by getting started with my free Agile Foundations for Microsoft Business Apps course at customery.com/foundations. Discover the basics and benefits of an agile approach and prepare to learn Scrum and build amazing applications.

The scope of our product is defined by the product backlog. The product backlog defines all the work we’d like to include in our business application. The product backlog is usually bigger than our project scope because projects have an end date or a budget that limits their scope whereas product backlogs are inherently bottomless, limitless. The scope of a project is usually defined by a line somewhere down the backlog that says everything about this line is in the scope of our project, and everything below it is not.

The product backlog will continue to exist and often continue to grow even after our project is over. Even if there is no follow-on project, subsequent phase or funding to keep going. Microsoft will keep shipping new releases that we should be testing and adopting. Users will keep finding bugs and suggesting enhancements. Leaders and managers will keep evolving their strategies and operations in response to new business opportunities and pressures. All this activity will add ideas to our product backlog whether or not there are developers turning those ideas into new increments.

If you’re with me, on the distinction between the product backlog and your project scope, then the next thing we need to discuss is: who is responsible for the product backlog and who is responsible for the project scope.

If you’re using the Scrum framework to build your business applications, there’s no doubt about this: the product owner is the single person with the responsibility and accountability for the product backlog. Anyone can contribute an idea to it, but the product owner is solely responsible for ordering it and can disregard any item she regards as insufficiently valuable. Even if your CEO asks the developers to work on something, we capture that request, add it to the product backlog and the product owner assesses its value. Our application’s stakeholders need to make their case to the product owner, they can’t overrule the product owner and get the developers working on their favourite ideas even if they are the CEO. It might be a career-limiting move to frequently ignore your CEO, but that’s the product owner’s prerogative.

In my Dynamics 365 and Power Apps projects, the product owner also manages the project scope. After all, the project scope is just a subset of the product backlog which the product owner is responsible for.

The scope of the project and the product backlog is the sole responsibility of the product owner. But what about the sprint backlog?

Again, the Scrum Guide is clear. It’s the developers’ responsibility to choose which items from the product backlog to include in the current sprint. Usually, we choose the items from the top of the product backlog because those are the items that the product owner has prioritised and flagged as most valuable. But if there are other items slightly further down that the developers want to include, that’s their prerogative. It might be a career-limiting move to frequently ignore your product owner, but that’s the developers’ prerogative. When developers include items from further down the backlog it’s usually because of a technical dependency and I’d always recommend explaining our reasons to the product owner and gaining their agreement so that trust is maintained within our team. Great sprint backlogs are negotiated between developers and the product owner (with the scrum master’s facilitation, if necessary) and once agreed, we’re accountable for it as a scrum team.

Once you’ve set the sprint goal and agreed on the sprint backlog, what happens if someone needs to or wants to change the sprint backlog?

Here are five occasions when I see requests to change to the sprint backlog:

1.       A critical new item is added to the product backlog. 

2.       We learn something new about one of the items in our sprint backlog.

3.       An item in the current sprint fails an acceptance test.

4.       We’ve got less (or more) capacity than we expected at the beginning of the sprint.

5.       The product owner has changed her priorities.

Let’s discuss each of these:

#1 A critical new item is added to the product backlog. It’s urgent and important. It’s a higher priority than some of the other work in the sprint backlog. The product owner asks the developers to start working on it immediately. This situation is infrequent, but it can happen. Developers need to be empathic to the product owner’s situation and make adjustments to the sprint backlog to accommodate the critical item. Usually, that means estimating the critical item and pulling out work of a similar size to make way for the critical item. The items we’re pulling out are, ideally, not started so that we’re not wasting any effort.

Expecting the developers, at any time, to take on more work than they forecast they can get done in the sprint by working harder or longer or expecting some kind of miraculous productivity improvement, is an anti-pattern and one than your scrum master needs to watch out more. The pressure doesn’t always come from the product owner. It can come indirectly from other stakeholders or from the developers’ line managers. Stay frosty.

#2 We learn something new about one of our items. This reason is usually the most common reason for changing the scope of a sprint. It usually happens when we’re reviewing or testing a product backlog item that’s almost done when someone spots an issue such as additional acceptance criteria or a scenario that was originally included.

In these situations, I encourage the developers to make a decision: Can I include the additional work without busting the original estimate. For example, if we originally estimated the item to be a 3 story point item, will the additional work push it up to a 5 story point item? And will the additional work jeopardise the completion of any other item in the sprint backlog or the sprint goal? If it’s a significant amount of work or will put the sprint goal or other items at risk then I recommend adding a new item to the product backlog, otherwise, the developer will normally include the extra work this sprint. 

Doing the extra work now is the least expensive option because it’s fresh in our minds. Deferring it until later will require context switching, especially if the work is picked up by a different developer. It’s always better to do it now if we can. That’s why I encourage my teams to include slightly less work than they have the capacity for in their sprint backlog. It enables teams to say ‘sure, we can do that’ more often than, ‘sorry, maybe next sprint’.

#3 A failing acceptance test. If your product backlog items have acceptance tests, and I recommend that you write acceptance tests just before you estimate the item or immediately you start working on it, then what do you do when an acceptance test fails?

If the test fails in the same sprint that you’re building the feature, then the two people building and testing the feature should work together to resolve the issue. Don’t create bugs, defects, issues or any other kind of product backlog item. Just have a chat and get it fixed. Tracking the issue is work that doesn’t add any value, and should be avoided where possible. 

If you are performing acceptance tests after the sprint when you built the feature, that’s an anti-pattern that you need to address. It’s an anti-pattern for two reasons. Firstly, you need to do some work to track the bug, describe the steps to reproduce, triage it, prioritise it, retest and resolve it, and none of that defect management work creates any value. Acceptance testing in the sprint we built the feature avoids most of that busy work. Secondly, is the cost of context switching. There’s a good chance that the developer who’s fixing the bug is not the developer who built the feature. And while there is a small knowledge transfer benefit here, it’s overshadowed by the cost of context switching. Picking up a new feature, learning how it works, reading the documentation, finding the root cause and fixing it all takes much longer when you’ve never seen the feature before.

I don’t consider failing acceptance tests a change in scope. I consider them part of our quality management process that needs to get done. As a scrum master, I’m trying to get a sense of how many acceptance tests are failing so that the team can discuss the problem and find ways to address it. What I’m trying to avoid is testers creating issues when they don’t have to, and sometimes that’s because developers ask them to so that the developer can get on with a new item instead of fixing their work.

#4 Changes in capacity. During sprint planning, my teams declare their expected capacity for the upcoming sprint. Usually, we’re available for 10 days in a two-week sprint, but that’s not true if you work part-time, have other work commitments, have planned leave or there’s a public holiday. But we take the team’s known capacity into account when selecting when items to include in our sprint backlog.

Our capacity can change during the sprint. It can go up if we complete some items faster than expected. Or it can go down if there’s an unplanned absence or the work takes longer than expected. I’ve seen teams struggle with both.

What do you do if you’ve completed some items faster and you’ve got unexpected capacity at the end of a sprint? I think that’s a call for the product owner. It’s her responsibility to maximise the value created by the developers, so it’s her choice: pay down technical debt, resolve bugs, or start a new item. If she asks the team to start a new item, I strongly advise selecting a small item that the team is almost certain it can get done in the sprint. Don’t start a large item now and plan to continue working on it in the next sprint.

If you’ve got less capacity than expected due to unplanned absence or some items taking a lot longer than expected then, again, the developers need to consult with the product owner about removing items from the sprint backlog. 

This is never a fun conversation to have, but’s it’s vital for our team’s transparency. Immature teams will duck the conversation and hope that no one will notice. Great teams have the conversation as early as possible and ensure there are no surprises at the sprint review.

#5 Product owner has changed her priorities. OK, if you thought that changes in the developers’ capacity was a difficult conversation, this one is even tougher.

Here’s how it normally goes. The product owner has been working hard to refine the product backlog and it’s in good shape at sprint planning. At sprint planning, the team devises a sprint goal and the developers select a bunch of the backlog items that will achieve the goal and which they can get done this sprint. 

The developers start working on the items. They’re writing acceptance tests and unit tests, developing features, testing them, reviewing them with users and getting them done. 

Partway through the sprint, the product owner brings the team together and tells them that she’d like them to stop working on the remaining items and pull in a different set of items instead. As many as they think they can get done in the remaining time.

The cause for this kind of handbrake turn during the sprint, is often that one of her stakeholders has demanded a change in direction, and for whatever reason, the product owner has decided she needs to accommodate the demand. 

It doesn’t really matter what the reason or motivation behind the shift in priority is. The outcome is that the team is now in a tough spot. 

Earlier versions of the Scrum Guide asked the product owner to commit to no changes during the sprint and required the product owner to cancel the sprint if the sprint goal became obsolete. There was a paragraph or two of rules for how to cancel a sprint. 

In the Scrum Guide 2020, cancelling a sprint gets a one-line mention. “A Sprint could be cancelled if the Sprint Goal becomes obsolete. Only the Product Owner has the authority to cancel the Sprint.”

Cancelling a sprint is an option. But it’s not one that I often recommend.

Where possible, I recommend we do nothing. Stick to our current course and complete the current sprint. If there are one or two critical items, then refer to scenario 1, but don’t write off the remainder of the sprint with a dramatic change in direction.

Partway through any sprint and the developers usually have a lot of work in production. They are focused and dialled-in on their sprint goal and sprint backlog. 

Product owners can terminate the sprint or negotiate massive changes to the remainder of the sprint backlog, but in either case, the developers’ productivity will crater. They are partway through the sprint. They are dialled-in on the current sprint goal and sprint backlog and their work in progress is at its high point. Significant changes will cause most of that work in progress to be wasted. And they’ll lose more time trying to switch context and adjust their focus to the new priorities.

In these situations, scrum masters need to provide strong guidance to the product owner. Help her calculate the hard costs of the lost productivity. Ensure that the costs and consequences are understood and accepted by her and the application’s other stakeholders.

Sure, we’re agile, but we benefit from a few weeks of stability so that we can get valuable work done and released into production.

Changing our minds every couple of days isn’t agile. That’s chaos.

Just for a quick recap, here are the scenarios and my recommendations:

1.     New, critical items. Handle these by taking out other unstarted items of equal size.

2.     Changes to an item. Unless significant, handle these within the sprint; that’s why we don’t overcommit in our sprint backlog.

3.     Failing tests. Not really a scope change, but can look like new scope if it leads to new items being added to the product backlog, and definitely something we should handle within the sprint.

4.     Changes in capacity. If capacity is more than expected, consult with the product owner and add a small item that you can get done in the remainder of the sprint. If capacity is less than expected, consult with the product owner and remove items from the sprint if necessary.

5.     Change in the product owner’s priorities. Coach your product owner and stakeholders than agility requires stability for a week or two, but if absolutely necessary then cancel the sprint.

Stephen, I hope that helps you and your team consider some of the scenarios of scope changing in the middle of a sprint, and how I guide my teams to handle them. 

Perhaps you’ve got different ideas for handling scope changes or you’ve faced other scope-changing scenarios. Let me know. Visit the Amazing Applications page on LinkedIn and leave a comment in the post for this episode.

That’s all for this episode and for the Amazing Apps podcast in 2020. Thanks for sticking with me or joining me as we switched over from the Scrum Dynamics show in June. Scrum and agile approaches to implementing business applications are still very much part of the show, but I hope you’ve learned something from all the episodes whether they’re about agile or not.

Let me know if you have a story to share about how your team built an amazing Power Platform or Dynamics 365 application. Whether you’re a Microsoft customer, an implementation partner or an ISV with an amazing story, I’d love to help you share it on this podcast.

Visit customery.com/guest and you can find out more about appearing on the next interview season coming up next year. I’ve already got some great interview recorded and others lined up. I’d love you to join me too.

Whether you’re celebrating Christmas and New Year in the middle of summer like me, I hope you and your family and teams stay safe and well, and keep sprinting.