Requirements in agile development: how much or how little should we do?

Even-André: The title of your presentation is “User stories - the wonderful soup stone”. What does that mean?

GB: There are many local variations on the folktale about stone soup. The moral of the story can also be slightly different depending on how you interpret it. In the European version, a group of hungry travellers comes to a village. All they bring along is an empty pot – and a soup stone. The villagers are unwilling to share their food with the travellers, but are happy to add a little garnish to improve the soup. In the end both the travellers and the villagers can enjoy a tasty, filling soup. The moral here is that by contributing, collaborating, and sharing, everybody can enjoy a great result that nobody could have achieved alone.

This is how I see the original “story card” vision described by Kent Beck in his book about extreme programming. We need to sit together and collaborate directly with the customer, on site. Just like the soup stone, the story card is the means for creating a great result. It has little meaning on its own, but it helps us abide by the “We value Customer Collaboration over Contract Negotiation” credo from the Agile Manifesto.

The Scandinavian version of the folktale involves a lone traveller, soup made from a nail, and a grumpy woman. In the US, the travellers are soldiers. In these versions, the story holds more of an element of tricking the woman into contributing all the good stuff to the soup, but so that both parties are still very happy with the result.

So, the other analogy I take from stone soup (or nail soup) with regard to user stories is how we like to be fooled. You call something a user story – with neither a user nor a story – and everybody loves it because it sounds cool and you can say you are “agile”. By adding well-known elements from existing requirements techniques, such as user roles, user motivation, and acceptance test, the users of user stories think they have got something new. In truth it’s just old stuff in a very superficial version, and users still have to struggle with granularity. Epics and sub-stories are symptoms of the problem, not a solution. User stories don’t help you deal with dependencies – they can actually make things worse. To be honest that’s okay, because that was never the intention of user stories.

As soon as we introduce user stories to organisations with many teams, perhaps combined with distributed development and big, complex products, we fall into the old, familiar traps: put the user stories in a tool, so it is easier to share them! Make sure the product owner responsible for the user stories is located in a different part of the world than the team! Send the user stories to India, and let them code everything!

In short, the original vision of the story card gets lost and the result gets messy. But if everybody is happy, I don't mind, as long as I don't have to clean up the mess!

Even-André: Interesting viewpoint! So what do you see as the future of requirements management in more complex agile projects? Will there be a “standard” way of extending the user stories, or is it up to each organisation or project or team to make its own soup?

GB: If we look at history, we see that “standards” in software development are often short-lived. Structured Analysis & Design was a standard for a while, and Use Cases became standard – together with UML – for a while. User stories are a kind of agile standard, but I don't think they will last any better or longer than DFD or Use Cases. What is more persistent is the culture in the organisation and the values this culture is expressing. Here are two different scenarios that can help explain this phenomenon. I can imagine both of them being true in parallel, but in different organisations.

Scenario 1:

The organisation believes in individual contribution and rewards employees accordingly. Perhaps there are cross-functional teams established, but the teamwork suffers because of the focus on individuals and their careers. Cheap labour is a priority for management, and having many employees is a sign of success. Therefore, distributed teamwork is common. Tools and agile techniques like product backlog will be used to try to overcome the challenges of distributed work. These organisations will probably have templates for how to write user stories, and complicated tools to store and manage these user stories. The process will be driven by the selected tools. It is likely that requirement complexity is expressed in an overwhelming set of test cases in the name of  “agile”. It is important for the organisation to have a controlled and standardised process.

Scenario 2:

First of all, the organisation takes great pride in its product or service! Contributing to the product is the main motivator – together with genuine teamwork – for all team members. The teams are co-located. They don't care so much about the requirement technique they are using, as long as it gives them quick and usable feedback. Sometimes they draw storyboards, sometimes they create vivid narratives, sometimes they brainstorm user stories, and sometimes they act out personas and scenarios on stage or in a video. Sometimes they consolidate all the input in use cases, so they can handle more complex requirements in a product backlog. All this makes it easy to derive functional test and acceptance test scenarios directly from the use case scenarios. However, everything will depend on the product and release on which the teams are working, and on who the team members are. There will be no one-size-fits-all approach. Diversity, new ideas, and common sense will drive the process.

Even-André: The second scenario requires a lot of adaptation on the part of the team, so team members have to be very “fluent” in all types of techniques, or they will spend a lot of time reinventing the wheel or discussing methods. The purpose of having a “standard” process is to focus on what to do, and not how to do it. Do you think requirements are always so special and unique that we can't provide any reasonable standard?

GB: I would rather have teams “reinventing the wheel” and getting a result for which they feel ownership and know the content, rather than having them fill out templates and guessing what the different fields in the template mean. What I have seen as most successful are standards emerging from teams that copy good examples from each other. My experience is that top-down standards often slow development, and decrease the motivation for ownership and innovation. You need a structure in the organisation that supports teams in learning and copying good examples from each other. It can result in useful “standards”, and at the same time it can encourage the teams to try new techniques and get into a mode of continuous improvement.

That said, I admit that over the last 20 years I have developed my own “best standard” for consolidating requirements. In user story terminology it could be formulated like this: “How to define user stories with the right granularity - and organise them so that you can handle dependencies and manage your product backlog. In addition, how to group user stories in epics which support the same user goal so that usability and testability become an integrated part of the epic.” If I don't tell you I’m talking about use cases, would you still be interested?  User stories represent a lightweight technique that should stick to its original purpose: to serve as a trigger for oral communication between the team and the customer. The only products for which user stories can be used as requirements are products so simple that it really doesn’t matter what technique you use. On the other hand, if I have to choose between working with user stories in a team that is enthusiastic about it, and working with use cases in a team full of resistance, I would prefer to go with the enthusiasm. This is why I prefer diversity and freedom over standards: go with the energy.”

Even-André: In the second scenario you described, how do we satisfy “requirements” on documenting requirements for the future, or consistency between teams? Is it enough that the requirements are there during development, just like the soup, which is consumed when it is ready? Would that mean that we don't need to store the requirements?

GB: First of all, let’s remember that user stories are just “a promise for a future conversation”. If we want any kind of specification or documentation, we should use another technique.

Over the years I have taken a more pragmatic approach to the general question of documenting requirements for the future. It sounds like a good idea, but it is hardly ever done in practice. In the end, the system and the code comprise the only updated documentation. If you don't need the documentation for complying with FDA regulations or other standards, I would focus only on the documentation needed during development (yes – that means eating the soup while it is hot).

If you are dealing with life-critical systems, then it’s a world where old-fashioned analysis and documentation are important, and techniques like user stories bring limited value to the requirements process.

Even-André: Gertrud, thanks for sharing your insights about requirements with us. Can you describe a typical client of yours, and how you would support this client?

GB: I cannot think of any typical client – they all seem to be unique, though they often all work with the challenge of effective communication among the different parties and roles in the organisation. The statement “We need better requirements” is often interpreted as “better written requirements”. However, effective requirement communication doesn’t happen in writing – it takes place in the process which results in some writing. In an organisation, I look for business domain knowledge, usability/UX competence, design/programming knowledge, and test competence. If I’m lucky, these are all present, and I can get them together in the same room. So, the first principle I’m using is “Everybody, all together, from early on” – which is the mantra from our book about lean architecture (James Coplien and Gertrud Bjørnvig: Lean Architecture for Agile Software Development). When the vision and the overall requirements are understood by everyone, we can begin to dig into more detailed requirements and document the decisions we take along the way. In Scrum terms, this will happen during the “Wednesday meeting” - also once called the “grooming meeting” (until the term became politically incorrect). I want to avoid hands-off. In Scrum there are also examples of the product owner throwing requirements – the product backlog – over the wall to the development team. Signs of this being a problem include either endless sprint planning meetings or very low velocity in the development team.

Clients can also ask for support when things get too messy, and they need some help to clean up. User stories are popular as product backlog items in Scrum organisations. But user stories are at least as fragmentary as traditional numbered requirements. This becomes clear when you work from an ordered product backlog and get lost in dependencies. The ordering of a product backlog has a huge influence on the ROI, so it isn’t enough to be agile - you must also be lean. The cleaning help can be seen as “lean first aid” for an agile approach, without the necessary discipline. I’ll leave it to the reader to think about the difference between “agile” and “lean”, and how to get the best out of the combination of the two.