Cycle time data from user stories can be used to measure predictability in Agile software development teams and quantify risks. Analysis of cycle time data from 2015 and 2016 for one team showed that: (1) Predictability improved after changing to 2-week sprints and adopting a scaled Agile process, with the likelihood of on-time completion increasing from 43% to 73%; (2) Software delivery risk decreased correspondingly; and (3) The team's aggregate predictability increased by a factor of about 1.7 between 2015 and 2016.
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Practical agile analytics: Measure predictability and quantify risk with cycle time
1. SagePath Technologies, LLC Copyright 2017
Practical Agile Analytics:
MEASURE PREDICTABILITY AND QUANTIFY RISK
WITH CYCLE TIME
1
2. One of the promises of Agile is that SW development
teams, and teams of teams, will be more predictable.
Ok, but...
Why does predictability matter?
How do I quantify it?
SagePath Technologies, LLC Copyright 2017 2
What’s in these slides
3. These slides focus on using user story cycle time data to measure the
predictability of Agile software development teams.
The same data that gives us our predictability measure is then used
to quantify the likelihood of user story completion.
Once we know the likelihood of user story completion, we can put a
number on the risk of missing software delivery commitments.
SagePath Technologies, LLC Copyright 2017 3
What’s in these slides
4. • Predictability in software development
• Why predictability matters
• Uncertainty and Agile team metrics
• Cycle time
• Cycle time data analysis
• In a nutshell: A few essentials to remember
• About the data visuals
SagePath Technologies, LLC Copyright 2017 4
Outline
6. When we say someone is predictable typically we mean:
We have a pretty good idea of what we are going to get and when.
We expect to see similar outcomes in the future.
Predictable people make promises they can keep:
They do what they said they were going to do.
They complete things when they said they would.
6
What do we mean by Predictability?
SagePath Technologies, LLC Copyright 2017
7. Predictable Agile teams:
Consistently deliver on sprint commitments
Typically deliver code that is working, tested, and maintainable
As a team of teams, consistently meet program objectives
There is high confidence this behavior will continue
SagePath Technologies, LLC Copyright 2017 7
Characteristics of Predictable Agile Teams
9. Are more teams needed? If so, how big should the teams be?
Is there a risk of missing delivery commitments?
Should outside vendors be considered?
What delivery timeframe should be communicated to customers so they can
effectively plan out their testing, maintenance, and upgrades?
What training is needed and when should that be scheduled?
SagePath Technologies, LLC Copyright 2017 9
Predictability and Software Deliveries
Knowing how much software is likely to be delivered in a
given timeframe helps answer key planning questions such as:
10. SagePath Technologies, LLC Copyright 2017 10
Predictability and Resource Allotment
The ability to
Accurately estimate delivery dates
Properly resource projects
And with confidence, make commitments to customers
Directly impacts funding and investment decisions
11. Predictability is a key element to achieving a successful
scaled Agile process.
• Effective and efficient collaboration between teams requires trust that the
other teams will complete their work when they said they would.
• Large complex SW projects and systems often have multiple, far-reaching
dependencies.
• Predictability is fundamental to effective dependency management.
SagePath Technologies, LLC Copyright 2017 11
Predictability and Scaling Agile
12. Poor predictability of software delivery teams leads to:
Missed delivery dates and missed revenue targets
Low confidence in delivery commitments and loss of customer trust
Erosion of trust between development teams and senior management
Increased schedule pressure and a corresponding drop in quality
Schedule delays and the associated financial consequences
Re-planning and priority shifts that disrupt development and exacerbate
project risk
SagePath Technologies, LLC Copyright 2017 12
Poor Predictability is Insidious
14. Software development is knowledge-based work that by its
nature always contains some level of uncertainty.
A few examples of uncertainties that can arise in a software project:
• Gaps in development team capabilities and skillsets
• The challenge of estimating work effort when innovation is required to get the job done –
How do you schedule a breakthrough?
• Ambiguity in requirements and unclear technical paths
• Test planning and execution – How much testing is enough?
SagePath Technologies, LLC Copyright 2017 14
Uncertainty Makes it Harder to be Predictable
15. Typical Agile team metrics include things such as story points, velocity,
capacity, and task hours.
These metrics can help teams understand how well they are doing with
respect to their past performance.
At the program level these metrics can be difficult to interpret.
It is often unclear how these metrics can be used to assess value delivery
and program level productivity.
15
Agile Team Metrics
SagePath Technologies, LLC Copyright 2017
16. One way to address this is to use a metric that
reflects actual work delivered at a macroscopic level.
Such a metric is Cycle Time.
16
Agile Team Metrics
SagePath Technologies, LLC Copyright 2017
Additional challenges with Agile team metrics can include:
- Unstable team velocities
- Inconsistent tracking of actual task hours
- Committing to user stories that are too big
- Story point estimation not being normalized across all teams
18. SagePath Technologies, LLC Copyright 2017 18
What is Cycle Time?
Cycle Time:
The amount of elapsed time it takes for a given work
activity to be completed.
19. It’s easy to measure
It’s easy to understand
It measures something real and important
It’s a difficult metric to “game”
SagePath Technologies, LLC Copyright 2017 19
Cycle Time as a Metric
20. SagePath Technologies, LLC Copyright 2017 20
Cycle Times in an Agile Process
There are a number of different cycle time measures that can be
applied to an Agile/scrum process. A few examples include:
• Backlog-to-Ready
• Ready-to-In Process
• Backlog-to-Deployed
In these slides our interest is in the predictability of SW teams so we
will look at the amount of time spent on story implementation.
This is referred to as the time In-Process.
21. In subsequent slides the term cycle time refers to:
The number of days between when a team pulls a user story into a sprint to begin
work on it and when the story is accepted by the Product Owner.
In the parlance of work flow:
This is the elapsed time between the date a user story goes into the In-Process state
and the date it goes into the Accepted state.
SagePath Technologies, LLC Copyright 2017 21
In-Process Cycle Time
22. It’s a direct measure of the time it takes to get something done.
It doesn’t require detailed information from individual
developers such as actual task hours.
It mitigates the variabilities of unstable velocities and non-
normalized story points.
It applies to both scrum and Kanban teams.
SagePath Technologies, LLC Copyright 2017 22
Why Use the In-Process Cycle Time?
24. • The data used for analysis is from an Agile team of teams.
• This team of teams uses an electronic tool to track workflow. Cycle time data is
easily extracted from this tool using an API.
• The cycle time data is for user stories started and completed in 2015 and user
stories started and completed in 2016.
• This team of teams made two significant process changes in January of 2016:
They moved from 3 week sprints to 2 week sprints
They adopted a scaled Agile process
SagePath Technologies, LLC Copyright 2017 24
About the Data
25. Some data scrubbing is often required before proceeding with
analysis. A few of the reasons for this include:
• Missing data points
• Data that needs reformatting. E.g. string data that should be numeric.
• Data that is suspect and may be in error
• Inconsistencies in the processes for gathering and input of some data points
The following user story categories were removed from analysis:
• User stories where the in-process date and the accepted date are the same.
• System test user stories.
• Stories flagged as obsolete.
SagePath Technologies, LLC Copyright 2017 25
Data Scrubbing
26. When it comes to data analysis, one of the first things a
person should do is plot the data and look at it.
The idea is to get a general sense of what we are dealing with and see if there
are any obvious trends or unusual features.
One good tool for doing this is a scatter plot.
Sometimes it is also useful to include a corresponding histogram to help gauge
the density of the data.
SagePath Technologies, LLC Copyright 2017 26
Looking at the Data
27. About the Chart:
• The chart on the left is a scatter plot of user stories
started and accepted in 2015.
• Each circle corresponds to a specific user story.
• The x-axis is the date a user story was accepted.
• The y-axis is user story cycle time in days.
• To the right of the scatter plot is a cycle time histogram
of the 2015 user stories.
What does this scatter plot tell us?
• The first thing to note is how spread out the data is in
along the y-axis (i.e. cycle times are all over the place).
• Since most of the teams are using scrum, the expectation
is that story acceptance would be clumped near or
below the sprint duration of 21 days.
• The histogram confirms what we are seeing. A significant
number of stories took 20 or more days to complete.
SagePath Technologies, LLC Copyright 2017 27
User Story Scatter Plot
28. About the Chart:
• The chart on the left is a scatter plot of user stories
started and accepted in 2016.
• Each circle corresponds to a specific user story.
• The x-axis is the date a user story was accepted.
• The y-axis is user story cycle time in days.
• To the right of the scatter plot is a cycle time histogram
of the 2016 user stories.
What does this scatter plot tell us?
• Contrary to the 2015 scatter plot, in this scatter plot the
user stories are more closely grouped along the lower
part of the chart.
• The histogram shows a strong peak between 10 and 15
days. This correlates well with the change to 2 week
sprints in 2016.
SagePath Technologies, LLC Copyright 2017 28
User Story Scatter Plot
29. About the Chart:
• In this chart the 2015 and 2016 scatter plots have been overlaid.
This plot is referred to as a “splatter plot†”
• The diameter of the marker circle for each user story is scaled
according to the size of the story it represents.
What does this plot tell us?
• The first thing to note is that the maximum story size is much
greater in the 2015 data than the 2016 data.
• The larger user stories have cycle times significantly longer than
one or two sprints.
• Somewhat unexpected: there are a number of user stories that are
small in size, but also have cycle times significantly longer than one
or two sprints.
SagePath Technologies, LLC Copyright 2017 29
Spatter Plot
†The earliest reference to the term “splatter plot” that I am aware of is by Dennis
Sweitzer in his presentation When Projects go Splat: Introducing Splatter Plots.
30. SagePath Technologies, LLC Copyright 2017 30
Quantifying Predictability
Predictability is the likelihood that any particular user story will
be completed within a given time frame such as one sprint.
If we can measure this likelihood, we can quantify predictability.
This likelihood can be determined from the cycle time data by
calculating the cumulative percentage of story completion over
time.
31. SagePath Technologies, LLC Copyright 2017 31
Measuring Predictability
About the Chart:
• This chart shows the cumulative percentage of 2015 stories that
were accepted over time.
• Note that the x-axis is in units of sprints rather than days.
• In the ideal, all stories that were committed to in a given sprint are
completed within that sprint.
• At the 1 sprint line, the larger the percentage of completed stories
the closer the teams are to the ideal.
Of Note:
• Only about 43% of the user stories were completed within 1 sprint.
• After 2 sprints the completion percentage is at 74%.
— This means that over one quarter of the user stories took longer than 2
sprints to complete
• Based on this data, the likelihood that any particular story would be
completed within one sprint is well short of 50-50.
2015
32. SagePath Technologies, LLC Copyright 2017 32
Measuring Predictability
About the Chart:
• This chart overlays the cumulative percentage of accepted user
stories from 2015 and 2016.
A look at the numbers:
• In 2016 the acceptance percentage within 1 sprint is 73%. This is a
significant improvement from the 43% seen in 2015.
• After 2 sprints the 2016 completion percentage is above 90%.
• Between 2015 and 2016 the likelihood that a user story will be
completed within 1 sprint increased from 43% to 73%.
• In 2016 user stories were 67% more likely to be completed within
1 sprint than in 2015. This means SW delivery risk decreased in
2016.
• Between 2015 and 2016 the aggregate team of teams
predictability increased by a factor of about 1.7.
Improvement in
Predictability
2016
2015
34. SagePath Technologies, LLC Copyright 2017 34
Predictability can be Measured and Quantified
By tracking cycle time we can quantify predictability.
In other words:
We can put a number on the likelihood that user story
commitments and program objectives will be met.
35. SagePath Technologies, LLC Copyright 2017 35
Quantifying Software Delivery Risk
Timely completion of user stories is a highly desirable objective.
Therefore, we can define SW delivery risk as:
Risk = 1 – (likelihood of on-time completion)
Once we quantify the likelihood of user story completion, we
can also quantify the corresponding project and program risk.
36. When you want to help teams understand their own work trends,
focus on team metrics such as velocity.
For forecasting elapsed time of software deliverables focus on work
delivered at a macroscopic level.
Forecast the likelihood of timely Agile team deliveries by looking at in-
process cycle time.
Cycle time allows us to quantify software project delivery risk.
SagePath Technologies, LLC Copyright 2017 36
Key Takeaways
38. The Data Plots:
• The data was analyzed using Python 3.6 and pandas from the Python Data Analysis Library .
• The data was plotted using the Python Matplotlib plotting library.
Background Images:
All background images are licensed under the Creative Commons Zero (CC0) license from the following sources:
unsplash.com : slides 17, 18, 21, 22, 26, 30, 34-37
pixabay.com : slides 1, 3, 5, 8, 13, 33
pexel.com : slides 23, 24
SagePath Technologies, LLC Copyright 2017 38
Data Visuals