Part 2 of 4
Although applied now to projects, the kinds of management tool described in Part 1 arose originally in production, that is, factory environments. Scrum, however, uses ideas drawn from Product Development, a very different regime. Strangely, although projects, as defined by the professional bodies concerned with them, are much better fit for product development than for production, Scrum has nothing whatever to say about projects, as defined in Part 1.
Scrum: its origins
The name “Scrum” comes from a famous Harvard Business Review article of 1986, which tried to summarize for a largely American readership the ideas that Japanese companies (Fuji, Canon, Honda, NEC) had explored for faster, more flexible development of technological consumer products (copiers, cameras, cars, personal computers) in the preceding ten years or so. The metaphor is of a rugby team working together to move the ball down the field, without a fixed plan for doing so. That an actual scrum is a largely static and highly structured “set piece” activity, doesn't seem to matter.
Scrum as we know it in software and systems work was first described in an OOPSLA workshop paper of 1995. In that paper, Ken Schwaber wrote:
The stated, accepted philosophy for systems development is that the development process is a well understood approach that can be planned, estimated, and successfully completed. This has proven incorrect in practice. SCRUM[sic] assumes that the systems development process is an unpredictable, complicated process that can only be roughly described as an overall progression […] SCRUM is an enhancement of the commonly used iterative/incremental object-oriented development cycle.
—SCRUM Development Process, , K. in Business Object Design and Implementation: OOPSLA'95 Workshop Proceedings, Springer Verlag, 1997
Aside: Schwaber refers to the rugby scrum more–or–less correctly as “a tight formation of forwards who bind together in specific positions when a scrumdown[sic] is called”. I write as someone who was one of those forwards in my youth, as it happens, I was a tighthead prop. Anyway…
That was 1995, and the workshop was concerned with the expected development of universally applicable “business objects” to use as the foundation of pretty much all future enterprise line–of–business systems. And also, as I've heard anecdotally, to explore why the promise of rapid reliable, flexible systems development with OO languages had not appeared. It was a time when this sort of thing appeared to make sense, but came up wanting.
Scrum has, therefore, developed pretty much exactly contemporaneously with the rise of the web, and the abandonment of, firstly, large centrally managed line–of–business systems, and secondly the highly refined business object model as the foundation of applications in general. As it happens, 1995 is also when my career in IT began and I don't recall any iterative/incremental object–oriented development cycles being particularly widespread, at least not in software product development in the UK.
Scrum as of 1995 talks about (software) product development as being near chaotic, in the technical, non–linear dynamics sense, and claims that, by appeal to self–organizing systems:
The closer the development team operates to the edge of chaos, while still maintaining order, the more competitive and useful the resulting system will be
and
The primary difference between the defined (waterfall, spiral and iterative) and empirical (SCRUM) approach is that The SCRUM approach assumes that the analysis, design, and development processes in the Sprint phase are unpredictable.
—ibid
Indeed, Scrum, as described in that original paper, explicitly treats the Sprint as a mysterious and unfathomable thing, subject to external observation but not to understanding.
Nevertheless, this early description of Scrum still has a defined opening phase in which the scope of the next release of a product is chosen from a “comprehensive backlog” and the, cost and duration are estimated, using “standard function point estimating”. That kind of thinking still permeates the Scrum world. This initial phase also includes planning our modifications to the system architecture and high–level design, domain analysis, and design review. A bunch of Sprints then take place to develop the new functionality until the release is ready, when there is again a defined closing phase in which “integration, system test, user documentation, training material preparation, and marketing material preparation” take place. Some people still today talk about stabilization sprints before major releases. Looking back it is notable firstly how conservative this SCRUM method appears to be, as seen from 2019, and also to remember how controversial it actually was at the time.
Scrum has developed over time, and now is somewhat different from that early description, although the underlying stance towards development is that same. The normative form of Scrum is captured in the current state of The Scrum Guide. At the time of writing, that's the 2017 version. It isn't a big document, and it's worth reading. As some people say of the Bible: you'll be surprised at what isn't in it. However, short as the Guide is, it is still worth summarizing the content of contemporary Scrum here.
Descriptions of Scrum (including the Guide) have as a common feature that they tend to start at the beginning, as if we are approaching Scrum, or a Sprint, afresh. But it I find it more useful to begin in media res. This is how I'd describe the Scrum lifecycle:
Reflect on how the work has gone recently. Plan to try working differently for a bit, to see if that helps it go better.
Reflect on how the product or service is working for its users. Come up with some ideas to change or enhance it, to see if the result works better for them.
Given the above; decide what seems best to do next, at present; and how it seems best to do it, at present.
Work on the product or service in those terms, and no other and on nothing else, for a short timebox.
Take a little time every day to keep everyone in sync and see if you're on track.
Demonstrate what you've got, to production quality
Go to 1.
This is the usual sequence of Scrum events, but re-phased. We can see from this that Scrum is about learning. Learning what to build, learning how to do it. And Scrum is about change. Changing how you work, changing what you work on.
eXtreme Programming
Another, roughly contemporaneous to Scrum, approach to development which now looks surprisingly tame, as described, but was also hugely controversial at the time, is eXtreme Programming [XP]. It's worth remembering that the first XP project, in the late 1990s, was for payroll in manufacturing, literally cutting pay cheques for metal-bashers. XP was created by back-office programmers in medium-heavy industry. That's worth remembering, because in the early 2000s it became easy for folks in less glamorous sectors to assume, incorrectly, that XP's native habitat was the undisciplined first–generation .com startup and that it therefore was not for them. Since that time the collection of reasons why “XP is not fo us” has gown to encompass almost every industry sector and business model.
As described in the so–called White Book, XP, as Scrum does, talks about spending weeks, or even a few months, on exploration activities. From the 1st edition:
[…] you are done with exploration when the customer is confident that there is more than enough material on the story cards to make a good first release and the programmers are confident that the can't estimate any better without actually implementing the system.
—Extreme Programming Explained: embrace change Beck, K. Addison Wesley 2000
Something which could be a surprise to some re–reading it now. When XP is in flight, it mandates certain primary practices. From the 2nd edition, 2005, my gloss:
Sit Together—one workspace for the team; not cubicles, not random seats across an open-plan, not individual offices.
Whole Team—every capability needed to deliver is in the team.
Informative Workspace—displays, charts, “information radiators”, everywhere.
Energized Work—work as hard as you can for as long as you can work that hard and no harder and no longer.
Pair Programming—all production code is written by two people working together.
Stories—use descriptions of value to the user as the unit of planning an tracking
Weekly cycle—pretty much a one-week Scrum sprint, but focussed on delivery
Quarterly cycle—pretty much a 12-week Scrum sprint, but focussed on improvement
Slack—know what scope you will be content to drop if progress falls behind plan.
Ten-Minute Build—run the complete automated test suite on a full build in less than ten minutes.
Continuous Integration—integrate little and often; that is, several times a day.
Test-First Programming—changes to production code are justified only by a failing test.
Incremental Design—do design work every day, to best suit the current design to the current need.
Some of these practices have taken root in the industry at large: the idea of large suites of automated checks (often known as “tests”) running frequently, unsupervised, on a build server is an XP idea. Informative workspaces (full of “information radiators”) are commonplace. Whole (that is, multidisciplinary) teams are much more common than they were. Some practices are aspired to, Test-First (or even Test-Driven) Development, for example.
Others have found largely barren soil. It has remained surprisingly difficult to get a whole team sitting together in a space which belongs to them. Even if they do sit in close proximity the emphasis on face-to-face communication built into Sit Together has slipped away as messaging, “presence” and “chat/ops” have become more popular. Pair Programming continues to be very controversial and of limited popularity, although some teams have leapt ahead to mob programming.
On the other hand, a contemporary team might scoff at the idea of integrating only several times a day, rather than per hour. At having a full build and test cycle take as long as ten minutes.
The [User] Story practice has a particularly mixed legacy.
A potted history of software requirements: since the days of US DOD MIL-STD-498, and descendants such as the PMBOK, the industry has had baked in the assumption that requirements can, and should, be written down in an essentially complete, normative document. This has never worked particularly well (not even in the military), but to be fair, that standard does say, in its discussion of how to describe a “Computer Software Configuration Item” [CSCI] in a “Software Requirements Specification” [SRS], that
Each requirement shall […] be stated in such a way that an objective test can be defined for it. [emphasis added]
Not bad for 1994.
The standard also states that a Requirement for a CSCI should say what the security and privacy needs are, and other quality factors, including reliability, availability, and interestingly, flexibility “the ability to be easily adapted to changing requirements”. Also not bad for 1994. In fact, the list of characteristics of a CSCI Requirement is a pretty good checklist of things to think about. But, quite bad even for 1994, trying to write them all down in great detail far in advance of development is asking for trouble. Anyway, this is what I call the “the system shall…” style of requirements. And sometimes, the system shall… and it's really important that it definitely does and that everybody knows that from the start. Sometimes.
At about the time that the US military were developing the ideas that eventually ended up in MIL-STD-498, the Swedish telephone company, Ericsson, were developing ideas for how to specify the behaviour of the new generation of software-based switches for PSTN communications. Out of that, and other work, came the idea of the use case, described in full in 1992. There are things to note here about the use case as a requirements tool: firstly, it represented a huge step forward over the MIL-STD-498 style of thinking, as it puts the user and the value they get from interacting with the (proposed) system capabilities at the centre of thinking about requirements. The 498 document mentions users only briefly in passing. Secondly, use cases assume that we already know where the system boundary is, which is a very natural way to think if your world is telecoms networks. But, this means that in general a use case can easily drift into being more like a specification of a system than a statement of what users would want to get out of a system, a system to be specified.
Like the 498 style SRS, use case requirements tended to grow without bound. I recall seeing use case document templates which ran to dozens of pages—empty, without the substantive content yet added.
Then, XP develops the user story, also in the mid to late 1990s (it was a busy time, in methodological circles), defined in the 1st edition White Book glossary thus:
Story One thing the customer wants the system to do. Stories should be estimable[sic] at between one to five ideal programming week. Stories should be testable.
We will draw a discreet curtain around quite what an “ideal programming week” might be, but emphasise that it is a significant improvement over both SRS CSCI Requirements and use cases that user stories are, by definition, small. In two senses. XP aims for a release cadence measured in maybe a few days, probably a few weeks, very much not more than a few months; and the team should be able to complete, to production quality, several stories in that time. So, stories represent a relatively small amount of development effort. But stories also represent a small amount of scope. One thing, one. And a small one thing at that; written by hand, with a pen, on an index card. Some people develop remarkable skill at writing very small on the story cards—give them a broader pen.
Because stories must be small in the view of two different stakeholders in two different senses they can only be agreed through a negotiation, the XP “Planning Game”. This also is a big step forward over both 498 and use cases as generally practiced. Because they tended to be such large and fine grained documents the idea of going back to revise them is very unpopular.
Uncertainty
Just as Scrum does, XP emphasizes uncertainty as an unavoidable aspect of systems development:
uncertainty in what is wanted,
uncertainty in how to build it,
uncertainty in how long that will take and how much it will cost.
Unlike earlier approaches which sought to squeeze out uncertainty, in the style of manufacturing, Scrum and XP recognize that some level of uncertainty will always be present and might even constitute an advantage, if managed correctly, which is truly in the style of product development. Uncertainty is managed, not managed away.
Decision making about scarce resource allocation under uncertainty is the essence of economics, and XP is described in explicitly economic terms, also a big step forward. Since, I boldly assert, economic constraints are a key factor distinguishing engineering from applied science, and since the approaches claiming to be “software engineering” tend to ignore economics, I doubly boldly assert that XP is a good candidate for really being how we engineer software.
Enabling Assumptions
Developers can build, test and deploy changes very quickly, and safely.
It's worth noting that the first XP team were working in Smalltalk, then a popular language for “enterprise” development, which pretty much requires continuous integration if a team is to work together at all; which had a strong culture of automated testing quite early; and which allows incremental builds to happen very quickly. By contrast, the C++ systems that I was working on at that time could take tens of minutes to do incremental builds, and a full build was something you started on Friday lunchtime and hoped was finished by the time you got in on Monday morning. In my world, integration was a daunting task taken on only a few times a year.
Since then, a close approximation of the Smalltalk style, as close as the “curly bracket” languages will let you get anyway, has become normalized. This should not be a surprise, the change has been lead by companies will one–time Smalltalk developers in senior technical positions. With enough build servers and enough expertise in deploying frameworks and tooling, today most developers can work in an environment were incremental builds are quick enough not to worry about, full builds are very much intra–day, and large bodies of automated tests provide a lot of confidence that there have been no regressions.
The Business can make economic trade-offs between different capabilities and functions of a system.
Seems like an odd thing to require, perhaps, but the closest thing I've ever seen to a fist fight in an office arose from a XP team coach refusing to schedule a story into his team until and unless the product owner could put an estimate of financial value on it. This unpleasant episode revealed that the product management function had no way to even begin to estimate that. This is surprisingly common.
Affordances
And what are the affordances of Scrum and XP? Strangely, they afford making themselves work! Not by magic, you have to do the work, but if you start down the route of trying to make it so that you can get someone to write a user story and see it in production a few days, or weeks, later, it becomes easier to get them to write the next story. If you start down the route of learning what capability to build next based on feedback from real users, you will find it easier to get such feedback. If you start down the route of increasing internal and external technical quality in order to go faster, then you'll find that easier. And if you start down the route of changing how you work based on feedback from your recent progress…you get the idea. But you do have to put in the work to start down that route and stick to it.
Scrum optimized for learning, XP optimizes for being able to take advantage of your learning. XP weakens technical constraints on changing your mind quickly, cheaply, and without sacrificing quality, Scrum gives you a way to harvest reasons to change your mind for the benefit of your self, team, business, and users.
Both Scrum and XP were controversial at the time, particularly in respect of the very short duration from what we would now call “ideation” to deployment. Two, three, or four weeks duration from someone thinking up an idea for a feature or enhancement to putting it in the hands of users was considered infeasible, maybe even professionally irresponsible, by observers used to the stately pace of more traditional phases and gates models of development. In a contemporary “Lean Startup” kind of setting, that same delay might be considered at the outer limits of acceptability from the other direction.
By 2001 Scrum had lost its capital letters and gained a lot of the apparatus that now forms the default assumptions about how “agile” software development should work, along with the less challenging technical practices of XP. Note that the burndown chart bears a very strong resemblance to the depiction of progress made against time on the Adamiecki harmonogram mentioned in Part 1.
Scrum and XP dovetail together very nicely, but they leave a lot unsaid about aspects of development that matter a lot: portfolios of products and services, expertise outside the realm of programming (such as UX and UI design), long-term operations, working in heavily regulated environments. This is why I say, Scrum + XP < Agile. But that's with the benefit of a couple of decades seeing how they go.
Scrum (and XP, if anyone ever gets that far) remains controversial in some quarters. Partly this is because it is often done badly. The daily standup, for example, is meant to be an opportunity for a team to synchronize itself, but is far too often used as a status report to the project manager, which quite rightly feels like disempowering micromanagement. On the other hand, to someone used to the mulit-hour weekly status reporting sessions common in the pre-Agile world, it was a blessed relief.
And there are those who feel happiest being given a detailed spec and heading off for several days of solitary coding. OK, but that is incompatible with Agile work, which might be perfectly ok if learning is not a goal for that business. Put it that way and it seems a little odd. I once saw someone describe big infrastructure projects of the sort represented by Google's bigtable are a sort of anti-Scrum; a big coorinated development effort that just got done. Well, maybe. But if so, Google's wave was also an anti-Scrum: person centuries of effort sunk (as in “sunk cost”) into a product which, as it turned out, no-one wanted. See also Google+. This is not to knock Google, but recognise that their context might not be your context. if you're in the middle of deciding that Agile is not for you because you just know what you need to just do and plan to just do it, ask yourself if your pockets are as deep as Google's. If not, maybe their approach might not be for you.
In the next post we'll look at Kanban, and kanban.