In Defence of Agile/Scrum

Developers don’t much like “process”, and Agile has become a bit of an industry punching bag. Here’s why memes shouldn’t be your guide to good software practise.

The first thing to say is that agile does not equal scrum. They’re not the same things and they’re not synonyms. But many people criticising agile actually mean scrum, and when people talk about process or ritual or whatever, they often mean scrum.

I’m going to use the two interchangeably here, but I’ll particularly say Scrum, because that’s mostly what we’re talking about.

Full disclosure: I don’t have any certification or qualification in Scrum methodology. I’m sharing my own opinions and experiences in a number of organisations using Scrum – some better and some worse.

Scrum has become something of a punchline in software development, some management nonsense that consultants and pencil-necked middle managers want to push on us software gurus. See Silicon Valley, or Anthony Sistilli’s prodigious youtube shorts output.

There’s some truth to that. There is no shortage of software developers who want to smash in code and don’t want to be hassled with a bunch of process and meetings. I can certainly appreciate that.

Scrum isn’t intended to make software harder. It’s intended to acknowledge a simple fact: it’s not just about us. Software engineers are hired by businesses, AND given money by them, in order to solve problems and release features. And those businesses can and do have expectations and requirements. And that’s ok. They should.

But this relationship is fraught. Businesses often want to create assembly lines and sweatshops, dictating features on a whim and having them done within – ideally – minutes. But developers want to casually drop in for our three our day and smash in code that is close enough to the requirements. (Maybe that’s just me?)

More than anything, Scrum is intended to smooth this relationship. To provide a mechanism for both sides to get what they need and provide insight into the other side. This empowers developers, but also makes them accountable. It allows the business to set its priorities and get some idea of timelines.

The Agile Manifesto

Agile is a lot like socialism in that they both have a manifesto and the word means very different things to different people. Additionally, we have a lot of experience with it turning out very badly, while supporters insist that it just wasn’t done right.

In both agile and socialism there is merit to this. A real discussion of socialism, the difference between Leninism and Marxism, discussion of whether collectivist social policies are inherently socialist, etc, are thoroughly out of scope for the this post. (To be honest they might be more interesting, I just don’t want to start anything with Americans with twelve seconds of political education from neo-McCarthyist YouTube.)

The big difference (apart from the fact that one is a political ideology and one is a software methodology and they’re nothing alike) is that socialism as originally envisioned by Marx was more about the end-state than it ever was about implementation details. This left people like Lenin, Stalin and Mao to deal with the “how”, with the less than stellar results we have seen. Agile is the opposite. It’s all about the how, all about the implementation. So how do we still end up with bad agile? Why do we often end up with a bad implementation that has become a punchline in so much nerd-adjacent media.

So Why Doesn’t It Work?

There are three key reasons scrum doesn’t work in some teams or companies.

Organisational Dysfunction

Scrum doesn’t fix really fix anything. If your management is a bunch of incompetent apes flitting from one brainfart to another with no clear long-term vision, you’ll just get that… iteratively.

What scrum does do well is make such dysfunction obvious. We can never finish because the infrastructure keeps going down. We have trouble finishing because the requirements are so vague and change halfway through. If such things keep coming up, scrum highlights it. But it is then up to the business to fix it. Which isn’t always easy – or possible.

One common organisation dysfunction is that organisations don’t want to change, modernise or improve. They just want to tick a box. They saw a Business Review article or something on Agile and decided they have to do it. They’re not committed to genuine change or improvement and definitely not – gasp – allowing developers more autonomy and empowerment.

Product Owners – The Biggest Hurdle

Specific people have specific roles in Scrum and those people have to be committed to doing their job properly, or it doesn’t work. Of particular note in my experience is the Product Owner. The PO is the representative of the business.

They’re the one with the product knowledge, the subject matter expert, and the final say on how this thing should be built. A good PO is an asset. Unfortunately many POs are former Project Managers, and there are key mindset differences. PMs are typically focused heavily on deadlines, delivery, milestones and dictating specific tasks, while POs are focused on what is being built, and prioritisation. Explicit milestones are often being more iteratively delivered and negotiated in an agile environment. The product owner role also includes writing up the stories themselves.

A lack of detail in these stories is a killer. Vague stories make it impossible to estimate, let alone build. Stories that insist on specific implementations. Stories that change during development.

Developers Suck Too!

Which is not to say developers don’t also play our part. Disengaged, uninterested, and unsupportive developers make Scrum harder than it needs to be. It absolutely doesn’t have to be a morass of process and documentation. Well run sprints makes everyone’s life easier, even at the coalface. Stories someone else wrote give you clear expectations and documentations. You should be able to clarify anything with the PO.

But many devs reject Scrum because we find any form of process an imposition. The fact is – these processes hold us accountable. And they should.

To this end, constantly failing to finish or deliver work suggests an underlying issue. Constantly mis-estimating work suggests an underlying issue.

Assign yourself to stories. Move it along in the little board. That’s all that’s being asked.

Scrum Master – The Important Phantom Position

The last role that can cause issues is the Scrum Master. I’ve never been at an organisation where the scrum master was the cause of issues. So why do I say that? Because I’ve been in many organisations that don’t have a Scrum Master at all, which caused many issues. In many ways the Product Owner is the representative of the business, but the Scrum Master is the representative of the development team. They manage the process, facilitate meetings, etc. IMO this isn’t necessarily a full-time job for a small team, and a team or tech lead can take on this role fine. But someone should be organised and empowered to do it or the PO can and will run rough-shod over the team.

ScrumBut

We do scrum, but we change this thing. I’m not a purist, but the question then becomes what effect does this have on the principles of iterative development and constant feedback and improvement? If it harms those things, including the iteration cycle, maybe that’s not great?

ScrumBut is often a result of built-in organisational issues. If you have to wait three weeks to get legal sign off on deliverables… that’s a problem. If you have managers or the PO assigning stories to people because they’re the only ones who can do it anyway… that’s a problem.

ScrumBut is often a way of formally entrenching dysfunction, working it into the process. It’s worth thinking about the change you want to do and asking why you really want to do it? Is it papering over some underlying flaw in your processes, practices, access, or knowledge? If so, maybe you need to put the time into fixing those. Or at least acknowledge that fact.

Ownership – The Real Key

The most important part of Scrum to me is to keep a clear understanding of who “owns” what, in terms of who has the final say on any change or action. This is where conflicts arise. When people start taking over control of stuff they really shouldn’t be doing.

So here’s a short list of the “stuff” that people own.

The Sprint is owned by the dev team. No one should be moving stories around, moving it in and out, assigning people, changing estimates, except maybe the dev team. I mean, some of that definitely the dev team. And again, this stuff can be negotiated. I’ve had plenty of times when acting as scrum master at the request of the PO we’ve taken one five point story out of sprint because of a change of requirements, and put another in its place. No big deal. But we do that, not him.

This is critical, and one of the biggest failings I’ve seen around Scrum. Overly empowered Product Owners, or under empowered teams being overcommitted to work, or having their workload and priority dictated.

So what does the Product Owners actually own – beside “the product”. That’s important on its own. Owning the product means being the person with the vision for what should be built and how it should work. The product owner entirely and exclusively decides what the work is, what needs to be done. They dictate all of the requirements and acceptance criteria.

The Product Owner also decides the priority of these stories. Typically they put the backlog in a list of most important to least, and we grab the top x amount to work on in a sprint.

The PO also takes ownership of issues with the stories. If there’s not enough info in the story to do the job, or there are unknowns, that’s their job to find out and put in.

The dev team also owns estimates. That’s one of their key roles, because it informs the prioritisation. Oh, that’s only a 1? Let’s get it in this sprint then. What the dev team doesn’t and can’t do is negotiate on points. The points are the complexity we sincerely think it is. We should never pad, nor should we be convinced to lower points because that way it will fit in a sprint.

Something else important is that while the PO gets to decide the priority of the stories to go into sprint, once they’re in there it’s up to the team to prioritise things. We tend to go from top to bottom anyway, but no one should be dictating “do this first”.

The dev team also owns delivery, in two important ways. The sprint is delivered at the end of the sprint. All of the sprint work. Not “these stories need to be deployed by Wednesday, those on Thursday, this is due tomorrow”. If you needed it done by a specific time, you should have had it in the last sprint.

But we’re also responsible for failure to deliver. The single biggest issue I’ve seen in scrum is a consistent failure of the team to deliver the sprint. For whatever reason, and with whatever excuses. Teams routinely treat the sprint like it’s a “we’ll do our best to get this stuff done”, but it’s really supposed to be a commitment. Failure to deliver should be addressed in the retrospective, with sincere and honest

This is the most common area of serious dysfunction I’ve seen in Scrum teams. Instead of a sprint failure triggering serious introspection and drastic action it triggers… the next sprint. And the same again next time.

My Key Learnings from 10 years of Scrum

Retrospectives tend to turn into bitching sessions. That’s life. That’s the place for it. Try to inject some positivity where you can.

Note that the Scrum Master and Product Owner are often in some degree of conflict, if that role isn’t there, that conflict is with the entire dev team. This isn’t to say that PO and SM are enemies. They just represent different sides, and good people in both roles collaborate and negotiate very well.

The stories should be as small as possible. It should be the smallest possible deliverable that one person can do that provides a benefit. Large stories take longer to do, and have a much higher chance of getting bogged down, blocking the sprint. Many small stories is good.

Ensure everyone is committed to the process. People in key roles lacking that commitment – businesses unwilling to change, developers unwilling to attend meetings, etc, can break it for everyone. And yes. If you won’t do your job, you shouldn’t have your job.

Delivery of all work in the sprint is at the end of the sprint. Don’t let people bully early release of features.

Don’t skip standups, but keep them short. Avoid diversions into implementation or issues, or any sort of detail. 15 minutes is the oft-cited max for standups and even that is way too much for a small team. Under 5 is better and really all that’s necessary. One team I was on insisted that the speaker stand on one leg. They would also yell “RATBUS!” if someone went into details. No idea why, but as a term of art it works. Don’t hike to some other campus to do standups, arguably have the sprint board available but if you do for the love of god do not go through it story by story.

Standups tend to turn into status updates. Don’t fall into that pattern. The key way this happens is if product owners or other stakeholders are there, asking. Delivery of all features is the end of the sprint.

Don’t skip retros. Scrum is supposed to be iterative – which sprints give you, and incrementally improved – which the retros do. Don’t skip demos. They make your team accountable for being sure of the delivery of the product and ensure that stakeholders are aware of the features delivered.

Don’t skip backlog grooming. Keeping the backlog up to date keeps rough estimates accurate, removes uncertainty from stories, and allows developer input into approaches and feasibility.

If you’re skipping these things, again ask why. Is it “a waste of time”, or is it that there’s something here you either don’t value, or don’t want to be held accountable for? Why is that? Note the same applies to other parties, like stakeholders, who won’t come to the ONE meeting they’re expected to attend. Why is that?

Don’t let anyone tread on anyone else’s toes. Devs should not be putting things in sprint that the PO didn’t approve. And vice versa. POs shouldn’t be assigning stories to people. The CEO shouldn’t be coming to standup and explaining the company’s mission statement. Developers shouldn’t be making up requirements.

Enterprise Agile approaches and especially the Scaled Agile Framework – SAFe – are a deliberate attempt to undermine the empowering of developers and give that power back to managers, ticking the “We are doing the Agile” box without the inconvenience of making any actual changes to corporate culture, siloing, dysfunction or entrenched incompetence.

Remember that a sprint isn’t just a bundle of work tied together with string. It’s a firm and sincere commitment from the team to deliver the following. Where Scrum is most vulnerable is in areas when that level of commitment is abused or undermined. Ie, if someone keeps changing the sprint, what did I commit to?

In Conclusion

Software developers want to develop software, and we don’t really like process. But this isn’t a hobby project, it’s a business, and it’s perfectly reasonable for the business to need to be able to calculate timelines. It’s perfectly reasonable to hold developers accountable and constrain what they deliver.

Agile approaches like Scrum are a good compromise, giving some ownership to all parties, and providing a clear boundary for interaction.

People can do it badly, just like anything else. But understanding what the goal is, and how it intends to do so, goes a long way towards supporting Scrum. If you try it with an open mind and an understanding of that goal you may find it actually makes your life easier, not harder.

I wanted to say in the end that all of the above concerns are possible issues, possible friction, possible dysfunction. Vast number of people use Scrum without any of these issues. At its best it’s intended to be collaborative, not combative, and most places I’ve worked it has been an effective and collaborative approach with mutual respect and a perfectly natural give and take and reasonable compromises.