Agile Ramblings

Home » Posts tagged 'estimation'

Tag Archives: estimation

Who is Estimating?

Part 3 of my Exploration of Estimation

In part 1, I posed the question of Why are you estimating! I hope that you thought about it and may have found an answer or two.

Part 2 explored What are we estimating. Usually we are asked to estimate effort (cost) which is then turned into a schedule (duration) which are two very different things.

The next question I’d to explore is Who is doing the estimating?

In my experience,  normally it is the technical people who will be doing the actually work (coding, sql, ux, etc) that are asked to provide an estimate. Developers are the best and most familiar example. And while at first glance this seems reasonable, when you start to dig into the entire workflow required to move an idea to reality in software, it begins to look a bit inadequate.

 

image

 

As we can see in the workflow, there are several phases of life that a feature/user story/use case goes from to be brought to life, only one of which is development. Often, there are many tasks across elaboration/analysis, development,  quality assurance, user acceptance and IT/Ops just to name a few common phases. Are developers performing all of these tasks? Do they take into account the effort required in all of the other disciplines when they do their estimates?

So why is it that developers are usually the only ones estimating effort?

 

image

Surely developers aren’t the only one working on your projects!! (Don’t get me wrong. I was a developer growing up, so I’ve definitely thought that the software development workflow revolved around me!) But this is a team sport, and without the business analysts, QA professionals, IT professionals and what ever other role is in your delivery workflow, we may have a difficult if not impossible time getting our software out the door!

 

image

 

So as a part of your estimation process, how do you understand the effort and/or time required to get work through the entire delivery process?  Do you take into account all of the phases of life, and hand-offs, and interactions in your delivery process? Do you get all of the people involved in the estimation process? In fact, I’ve seen (not often) teams involve a tester in the estimation efforts, but very infrequently do I see analysts, DBAs or IT/Ops folks involved in the process, even when they’re role is critical to the delivery of the product!

So who is involved in YOUR estimation process? Do you have all the people you should have involved?

And are you asking yourself this question in conjunction with the other two questions I’ve already asked you about? Hmmm….. I hope you’re all thinking about all of these questions! I hope to present a couple more questions in the next week and then start to tell you how I think we can start make some real changes in the way that we do all of this estimation stuff!!

What Are You Estimating

Part 2 of my Exploration of Estimation

In part 1, I posed the question of Why are you estimating! I hope that you thought about it and may have found an answer or two.

The next question I’d love for you to think about is What are you estimating?

I was just in a meeting that was discussing a RFP for a client. This is pretty common in the consulting industry where we see a Request For Proposal and in that, we need to define the scope of work that we can address, align our skills and services to deliver  that scope of work and then provide an estimated cost for that service. Seems kind of normal. But then we are also sometimes required to provide an estimated calendar duration for the project. So in essence, we are being asked to provide 2 estimates. We then present our guess (educated) at how much work needs to be done, and here is our guess (educated) about how long it will take the team to execute on that vision.

So when the developers were being asked to provide estimates to the work in the RFP, what do you think were they estimating?

I haven’t had a chance to ask them yet, but based on my experience (as both a developer and a person being asked to provide project duration estimate), the development team is probably estimating effort and not duration.

So who estimates duration then? How do we get duration from effort? And if we stacking a guess (how fast we deliver) on top of a guess (effort), are we getting a meaningful expected duration value?

If you go back to my rant on estimation (which I will reference often in this exploration series) you will see how most organizations, in my experience, do mental and mathematic gymnastics to come up with an expected duration value.

My belief is that we are not getting a useful duration value. But everyone treats it as useful, starts building elaborate project plans and organizational activities around this duration value, and then we are all shocked when we have to change request the project, drop scope, or scramble like crazy (and usually with questionable quality) in the latter half of the project when it becomes painfully obvious that we’re not going to make our original duration estimate. Either way, the initial setting of expectations based on the layer of several guesses seems to be a Bad Thing™.

So I’ll leave you with this final question, what are you estimating?

p.s. I will explore better ways soon! Just bear with me for a bit longer! Open-mouthed smile

Why Are You Estimating

Part 1 of my Estimation Series

A while I go, I wrote up a bit of a rant on a crazy estimation scenario that was presented to me by a colleague. If you didn’t get a chance to read it, you can find it here. We’ll call that rant Part .5 of this series of blog posts on estimation.

Since then, I’ve seen a bit of a swell on twitter of the #noestimates hashtag and a lot of discussion on topic. I was recently saw a tweet from Allan Shalloway that prompted a bit of a conversation:

image

(click imagine to see original tweet and conversation)

As you might have guessed, my thinking is more aligned with the #noestimates crowd and I’ll elaborate that over the coming months here, but this tweet started a conversation that I’ve had before.

Why do you estimate?

If I’m going to encourage you to stop estimating, it’s really important that we understand what I’m encouraging you to stop vs. what you might actually be doing.

As the conversation continued, Allan stated:

image

…which made wonder if we’re confusing prioritization with sizing, which is actually quite common.

So I won’t re-post the entire twitter conversation here, but I’ll start to share my thoughts on the why of estimate.

Intent

When I engage in an “estimating” exercise with a team, I very often try and restate the activity as a sizing exercise. For me, this starts us down the road of using the language that more accurately describes the end goal of our exercise. We want, to the best of our ability, describe how “big” we think this work item is. I’d like the team to use their past experience and compare the current work item (requirement, user story, etc.) against similar work items they’ve worked on before. This is a relative sizing technique that is used all the time by other agile methodologies such as Scrum or XP. You’ll often here the term “yesterday’s weather” in the same conversation about agile estimation techniques. The best predictor of today’s weather is yesterday’s. The team’s recent experience is the best predictor of its near-term current capabilities.

To me, the really important point of this exercise is sizing! Not scheduling. Not prioritizing. Those are completely different activities! The product owner/business representative on the  team will tell you, perhaps with size as a decision input, what is next and when they might hope for it to be done. I generally expect them to help the team constrain the story, but the delivery team are the ones that are responsible for determine the size of the story.

So in my mind, the intent of the “estimation” exercise is to allow the whole delivery team to determine the relative size of the story compared to work items they have done in the past.

Why?

So why are we asking delivery teams (Development, Ops, anyone who is creating value) to size their work items? This is where I think all of the estimation craziness and problems start.

We need to provide some sort of indication of the anticipated financial impact to the organization making the request of this work item. We also need to provide information that would allow us to set a delivery timeframe expectation. And the next level of this problem is to take 100s (or 1000s) of these work items and provide the same financial and delivery timeframe expectations. This is, for most teams, really really hard.

And I will state that it is my belief that size is not the primary factor in determining cost or delivery timeframe. Will a bigger story cost more? Usually, yes, but not always. Will a bigger story take longer? Usually, yes, but not always. What if to delivery a story, a team has no external dependencies? What if it does? What if the story is big, but a really well known problem? Unknown problem? What if the team is missing a team member when it finally pulls the story from the backlog? What if the team composition has improved?  There are so many things that can change the cost and expected delivery timeframe in addition to the size of the story that are very difficult to predict.

So in my mind, we are trying to size our work so that we can help business decision makers make the best decision for the organizations economic well being.

Final Thoughts

I’ll leave you with a couple homework questions. What is the intent of your team’s estimation exercises? Why is your team performing those exercises?

Part 2 will explore my approach to providing business decision makers with better information so that a team (business and delivery groups) can make better decisions on behalf of our organizations.

A Rant about Estimation – When Will We Stop Being Crazy

Warning: This is a bit of a rant. If at any point in this post, I seem like the crazy one, please tell me somehow. Comments, Twitter, LinkedIn. It would be really valuable to hear your thoughts. But I also encourage you to think about the situation, context and information I provided and decide for yourself whether the way we are acting is crazy. There are craziness checkpoints along the way! And if you think any of this is crazy as well, try to do something about it!


I had a work colleague recently ask a question on our internal mailing list. A summary of the question would be:

A company that is using ‘agile’ wants to know if their formula for converting story points into hours is sound because executives are doing short term (< 6 months) planning.

This is an example of how they break down their work. Seems reasonable and just like many other companies.

image

They also provided a formula. This is where the craziness starts which is startlingly common in our industry.

image

… and a sample of how the formula works.

image

Ok. At this point, all that is going through my head is “OMG”. Now comes the process by which they populate the formula.

  1. PMs provide the Epic Point estimate
  2. Developers decompose epics and provide point estimates for the user stories.
  3. Teams points (epic points or user story points?) per resource per sprint are tracked.

This is COMMON in my experience working with many organizations in our industry.

In all fairness to my colleague, he knows this is crazy. The developers in this organization know this is crazy. This is a typical scenario when executives and senior managers are trying to get information but using techniques that we have proven don’t work in our industry.

The good thing is, the way that they decompose work is fine and the formula will work as long as we don’t have have 0 velocity or 0 team members. Open-mouthed smile

The existence of the formula and the process by which we find numbers to put into it are where the craziness begins. First, a bit of information on “story points”.

Dave’s Craziness Meter

image

Story points are an abstraction agile teams use to REDUCE the perception of accuracy.  1 story point will require some small bit of effort to complete. 2 story points should require about twice as much effort as 1 story point. So that would be 2 times “some small bit of effort”. 5 story points require about 5 times “some small bit of effort”.  I hope I’m clear about how we are purposefully reducing the perception of accuracy. Why? Because it incredibly difficult (usually impossible) and time consuming to create accurate estimates for the effort required to do knowledge work using the time units desired by decision makers! 

I don’t even advocate using story points to my teams anymore. As numbers, it is too easy for people to plug them into a formula to convert them into something they are not intended to be converted into. I now generally only advocate sorting work by sizes like Small, Medium and Large.

Now let’s discuss the numbers that go into the formula from the example.

To start, a PM will make an point estimate in story points. In the example, it is 200 story points. I don’t know why a PM is doing an estimate on technical work.

image

Point estimates are bad. They imply accuracy where it normally does not exist. We have a point estimate of an abstraction that is trying to reduce the perception of accuracy. At least it is a single point estimate using an abstraction, so it could be OK except humans are optimistic estimators. Our interpretation of an estimate usually means that we expect the epic to fall in the middle of a normal distribution of epics and we have a 50/50 chance of falling under either side of the curve.

image

The problem with this is that the magnitude and likelihood of doing better than our estimate is the same as the likelihood and magnitude of doing worse than our estimate. And the other problem with this approach is it’s wrong. Using information from industry observations, we see that plotting the actual effort of our epics will form a log-normal distribution .

image

On a log normal distribution, the chance of doing better than our estimate is smaller than the chance of doing worse than our estimate. This means our epics usually fall behind the mode which means that they took more effort to complete than we expected. Simply put, when humans provide point estimates, we are usually wrong.

Let me state that again. When we estimate in knowledge work, we are usually wrong! And the formula above actually understands that!! The historical team buffer is a built-in admission that we need to add 20% to our estimates to make the numbers get closer to the actual values we observe!!! And I’ll bet that most epic actual values exceed the padded estimate!

Dave’s Craziness Meter

image

Ok, so point estimates are bad. And humans are bad estimators. That all came from the first point of the PM creating an epic point estimate. The rest should go more quickly. We’ve covered a lot of ground in that first exploration of the situation.

Next we have the developers decomposing the epics into stories and providing point estimates on those stories. All of the problems that we encountered in estimating above apply at this level as well.

The problem now is that it doesn’t work to simply sum up the points for the stories to get the points for the epic. If we look at the problems with point estimates, imagine doing that 20-100 times and adding the value together. We know that our our estimates are wrong (by at least 20% and probably more) and the impact of how wrong you are has a significant impact on effort. The only way to potentially add up the stories to see how much the epic will be is to estimate stories with an expected value that is the mean on a log-normal distribution and run a Monte Carlo simulation using all the stories in the epic. (That is an whole other blog post!)

I’m betting that a Monte Carlo simulation isn’t being used to turn the developer’s stories estimates into a meaningful epic estimate. Never mind the observed expansion (dark matter as David Anderson refers to it) of requirements that we see as we build the software and discover what we’ve missed. It’s always there, we just needed to build the system so that we could find it.

So our estimate is probably wrong. Significantly wrong. Period. Let’s move forward.

The next part of the formula is:

image

and the data in our example looks like this:

image

10 points is the average points per resource (I hate that word) per sprint. I’m assuming that they have standardized sprints for all teams in the organization. 2 is the number of team members (much better description) that will be working on this epic.

Do all developers produce 10 points per sprint, regardless of the task? Regardless of the skill level? Are we talking senior or junior developers? Is the business problem something they’ve done before, or something novel? Technology is know or new? Stat holidays? What if a developer is sick?

That 10pts could also be interpreted as developer velocity, and as we know, velocity can be highly variable for a team and is even more variable for individuals. In Scrum, we always talk about team velocity because it hides the variability of individual velocities that we know exists by averaging out the effort expenditure across multiple people over time. And we also know that velocity cannot be compared between teams! One team might have a velocity of 10 while another may have a velocity of 20, but the former team is more effective. The velocity number itself should not be interpreted as a standardized representation of effort that can be applied to any large scale planning activity (multiple project, multiple teams). It only works when you know the project and you know the team and it’s capabilities and past delivery history in specific contexts.

Dave’s Craziness Meter

image

So we have an epic estimate (crazy) divided by a developer velocity # multiplied by the number of team members (crazy). The next step is to multiply by the Historical Team Buffer otherwise known as padding the whole estimate.

image

The number from the formula is 1.2 in order to make the whole estimate 20% larger or 120% of the original estimate.

image

As I mentioned above, this buffer is a built-in indication that our estimates are wrong! We know we can’t deliver the epics as expected given our estimations, so we pad the estimate to make it closer to what we actually expect to do.

Dave’s Craziness Meter

image

So after all of that, we’ve arrived at a value of 12 sprints required to deliver this epic.

image

Phew!! Now we need to turn this into an expectation of effort expenditure or schedule.

image

Oddly enough, based on the question from our internal mailing list, we were not able to determine what the executives wanted. They either want to know how long it will take or how much it will cost. Unfortunately, I’d guess that they expect to get both from that  number. They expect that the project will be done in 960 hours of developer effort which will occur in 12 weeks of calendar time.

image

That is 12 sprints and 80 hours of effort per sprint. That seems to be 2 developers for 40 hours per week or 80 hours per week.

So, I can with a fair bit of confidence tell you that the work week is 40 hours, 2 man weeks is 80 hours, and if a sprint is 1 week long with 2 people, the project will spend 960 hours of effort in 12 weeks.

What I can’t tell you is if both developers will work 40 effective hours per week. You almost certainly will pay them 40 hours per week, but whether all of that effort is applied to the story is another question. Emails, meetings, slow days, coffee cooler conversations, code reviews, design sessions, non-project related tasks are just some of the kinds of things that eat into that 40 hour work week. Never mind sickness or other work absences. Most people I speak with use 6 hours as a number for planning how much time a developer can effectively spend on a story. I’ve seen organizations where a team member regularly, for a variety of necessary reasons, spends less than 2 hours per day on a project they are supposed to be full time on.

So that does two things to our calculation. We’ve determined that the story is now about 960 hours in effort. If the developer can’t work on it 80 hours per week, the number of iterations has to go up! If the developers are only effective 4 hours per day, that means we have to spend 24 iterations on the project to get it done. Or potentially, the developers can deliver it in 12 weeks, but have to work 40 hours of overtime per week, which means the costs go WAY up for the project as the organization has to pay overtime. Let’s not even try to incorporate the diminishing returns on overtime hours, especially in an environment where that overburdening is chronic.

So as an executive, I wanted a fairly certain number for how much effort a specific epic is going to take so that I can do appropriate planning for the next 6 months. And 960 hours and 12 weeks calendar time is something I’m going to hang my hat on. I mean, 960 is a pretty precise number, and the “confidence” that the process has given the organization in that number must be pretty good.

Dave’s Craziness Meter

image

And since we’re doing organizational planning and planning multiple epics for multiple teams in the six month period, I’m also making the assumption that all epics that are 200 points are the same. And I’m assuming that all story points are the same effort. I’d have to deduce that that any epic of 100 points would take exactly half as much effort or half as much time as any 200 point epic. Any 200 point epic is exactly the same as any other 200 point epic. There is a lot of confidence in our formula and approach.

Dave’s Craziness Meter

image

Ok. I’ve maxed out my craziness at this whole endeavor. My hope is that you think this is all a little crazy as well because if we start to accept that we’re acting a little crazy we can start to do something about it.

Final Thoughts

I’d like to clarify that this post is not intended to be critical of people who are using Agile estimation techniques and trying to fit inside of an organizational traditional project planning methodology. We are all trying to do the best that we can. But sometimes we get stuck in a way of thinking. I’m hoping that this blog post might just jolt some of you into thinking about these problems and deciding to try and do something about them.

The formula used is as an example in this post is just that. A formula. And a very common kind of formula as well. The values you input into the formula will work as long as you don’t have 0 velocity or 0 team members. (No one likes division by 0.) In Kanban we use a lot of formulas and a scientific approach to understanding our work and workflows.

It’s the drivers underlying how we want to use formulas that we really need to be critical of. The formulas and the data that we feed into them has to make sense and drive us towards creating valuable data points on which meaningful decisions can be made. If we use complicated formulas and suspect or fictitious data as parameters, that makes the output worse than garbage because it provides some sort of sense of accuracy and a false sense of confidence because we think “this must be right. Our formula is sound.”

As long as we, as an industry, support this behavior, we’re are contributing to the continuation of this problem.

There are solutions to this problem, but exploring those options will be explored in another blog post.

But in the mean time, I’d love to hear your feedback! Do you think I’m crazy or do you think our industry is crazy?