Chris Flaat's WebLog

Trying out pair programming

Software metrics primarily useful as negative indicators?

Dear Readers –

I was thinking about metrics, and it occurred to me that most of the metrics we commonly use in the industry are really good as negative indicators of quality, efficiency, testing, etc. but lousy positive indicators.  That is, most software metrics are really good at telling you when something is wrong with your project, but they don’t give you much assurance that the project is actually on the right track.  (I’m sure someone else has already thought of this, but I figured I’d pass on my random though nonetheless.)

Code coverage is a classic example.  Code coverage can be viewed as a measure of what you’re not testing.  I.e. if I have 60% code coverage then I know that 40% of my code is not being tested at all.  However, the 60% code coverage gives me no assurance that the testing of that 60% is actually any good, because there can of course be a vast combination of different code paths through any piece of complex code.

Bug numbers are another example.  A high rate of reported bugs is clearly a sign that something is amiss.  However, a low rate of reported bugs doesn’t necessarily mean you’re on track.  What if your QA team is off writing documents and not testing the product?  In that case a low rate of reported bugs is simply reflecting a lack of testing activity.  Similarly, a very low rate of bug fixing is often (though not always) a sign that something is amiss.  However, a high rate of bug fixing is not comforting – it may simply reflect that devs are rushing work in order to make the #’s look good.

Code complexity is another example.  High code complexity is generally a bad sign for your code’s maintainability (with the exception of some specific code patterns like a parsing or message handling function).  However, low code complexity doesn’t mean that your code is necessarily “good” in any other respect.  It could be utter trash, broken up into small functions.

Conformance of work estimates to actual time spent is another one.  If the actual time spent on a task is way longer (or shorter) than the estimate, that tells you that your estimation process is not very accurate.  However, if someone’s actual time spent on a task always closely jives with their estimates, that’s really not very comforting unless you’re absolutely, positively sure that no other aspect of their work has been compromised as a result (i.e. there’s no “distortion” as Robert D. Austin calls it in his excellent book Measuring and Managing Performance in Organizations, which I highly recommend to anyone interested in metrics).

By these comments I don’t mean to bash these metrics – they are very useful as a way of identifying potential problems and fixing them.  But they have to be viewed as specialized indicators, not numbers to be mindlessly met.  Most software metrics make great gauges but lousy controls.

Over & out!

Chris

People are not fungible resources

Hi all –

We are currently using both Scrum and more traditional project management on several efforts going on within our product unit, and I thought I’d share some learnings.

Something we’re running into is that getting people dedicated to one effort can be hard, depending on the management style of the relevant managers.  For some parts of the team, the relevant leads & managers dedicated people to one effort.  For other parts of the team, the leads & managers insisted on partial assignments, e.g. saying Boffo is 50% on team A and 50% on team B.

This is a basic point of difference between task-based project management (where people are assumed to be fungible resources) and people-based project management (where the team of people is first determined, and their work items second).

As you might guess, so far it is apparent that partial assignment of people does not work very well (both for efforts using Scrum and for traditional projects).  It seems to me that a time split will seldom be 50/50 – instead it will be 80/20 or 90/10, depending on which task is more urgent or more interesting.

On several projects where we aren’t using Scrum but where people are being loaned across teams in the division to do something, we heard again & again that partial assignments are causing problems because the people aren’t fully dedicated (timewise) to the tasks, and it’s very hard to schedule.

Likewise, on the efforts where we have used Scrum, having someone partially available doesn’t jive well – hard to build cohesion and have smooth coordination with people who aren’t always around & may have the tasks from your sprint on the back burner.

Moreover, it’s pretty obvious to most people that if you are working on 2 or more project simultaneously, there is a high price to be paid for context switches.

So, based on what we’ve seen so far, my advice to my readers is that you should avoid assigning people fractionally – instead, pick one task and throw them fully at it.  As has been said for a long time, people are not fungible resources.

Over & out!

Chris

Sprint 9 complete

Bridging the Gap, etc.

Greetings all –

My apologies for being behind on posting.  Things have been really crazy around here with VS2005 shutting down and work on the next major release ramping up.  I’ve been tasked to help out with some engineering improvement initiatives going on in the division and this has soaked up much of my available time.

During the last sprint (sprint 8), we put all the efforts of the MSBuild team on a Scrum team.  This had one downside, which was that the team got too large (10+ people).  I knew this would be an issue, but it turned out to be really big.  The meetings got too big and the goal was too unfocused, and people definitely didn’t like it (the sprint retrospective had a very negative cast to it).

During the current sprint (sprint 9), we finally split into two separate teams, one doing more internal-focused work and the other doing more forward-looking work (although the entire team still comes together for design decisions, etc.).  This has so far worked quite well (or so it seems).  It will be interesting to see what the team thinks at the sprint retrospective.

One of the more interesting phenomena is that during the last sprint a few team members were complaining about how they thought Scrum had too much overhead and they wanted to try something different.  One of the really ironic things about this is that they were using Scrum data to argue this, and they were using the Scrum retrospective as the window into their issues!  Whenever I start to drill into why people think we should use another process, they usually come back with an answer like “we have too many other distractions”, “so-and-so is too hard to work with”, etc. – generally things that have nothing to do with Scrum, but that Scrum has showed them.  Very bizarre.  Usually I’m pretty patient with this, but sometimes I want to pull my hair out and point out that if we weren’t using Scrum, the same issues would exist but there would simply be less visibility into them and less opportunity to fix them.

Another good one is that one or two of the people on the team will occasionally complain that they don’t like having a meeting every day, but these are often the same people who would otherwise complain that they didn’t have enough insight into what the rest of the team is doing.

Anyways, I’ve found that Scrum has been useful for us in “bridging the gap” between the end of one product cycle and the beginning of another.  When other teams were trying to figure out their plans, we simply continued to execute against our product backlog and kept doing sprints, and the team stayed engaged, we have avoided analysis paralysis, and things are rolling pretty well.  Right now I really feel like the avoidance of analysis paralysis is an enormous benefit, but a hidden one – you don’t notice the benefit unless you’ve lived through the problematic situation from past experience and realize the joy in not going through it.

That’s all for now!

Chris

Agile 2005

Hi all -

  Several people from MSBuild, the Developer Division, and Microsoft went to Agile 2005 last week in Denver.  It was very worthwhile.  I really enjoyed the keynotes and getting to meet a lot of others at MSFT doing agile methods as well as people from industry & academia.  I can now put a face with a lot of the agile gurus whose books I've read and that I've heard about.  The next Agile conference is July 2006 in Minneapolis.

  I would say that about 50% of the talks I went to were really good, 25% were interesting but not necessarily useful or applicable, and 25% weren't very good.

  For me, the top takeaway was the rise in prominence of automated acceptance testing (e.g. with Fit).  This is something that would be most useful inside MSFT, as we often have complex integration scenarios with other teams, and having a clear set of acceptance tests would do wonders to help clarify up front scenarios that are & aren't expected to work.  After all, if you can't provide a dozen testable examples of things that should work, you don't really understand what you're trying to do.

  Another obvious takeaway was the utility of computing velocity, which I’ve done retroactively for our past sprints.  I’ll probably blog separately about that.

  Another obvious takeaway is the trend toward automating everything that you find useful: dev & QA tests, metrics reporting, project status, etc.  If it’s automated, it will happen; if it’s not, it may not – this is just human nature.

  I’m somewhat annoyed that they have the Agile conference in the summer, as this is when I (and many others) already have travel plans.  Plus, we’re inside all day so it’s not like you really get to enjoy the weather anyway.  I would rather have it in the winter or fall.  But, not a huge deal.

Over & out!

Chris

Sprint 6 winding down

Well, we finally kicked off sprint 6 in June, and it finishes up next week.  Being summer, there have been a lot of vacations plus a Microsoft-wide forum that ate up a lot of time.  I expect that sprint 7 will have fewer randomizations, esp. since the core Whidbey work that the other part of our team is working on is winding down.  However, for several reasons we will still be a bit short on people during sprint 7, so we probably won't start any kind of dual-sprint approach until sprint 8 (which would presumably start in mid/late August).

During this sprint the team has spent a lot of time talking about improving estimates, and tracking the time spent.  Some people suggest that we track the time spent on each item, but of course some (myself included) think that's a bad and non-agile road to go down.  My personal view is that the things that slow us down tend not to be massive underestimation of work items (although we're far from perfect in that regard), but rather getting hit by other things.  We've tried to account for that in sprint 6 using a "risk reserve" as I've alluded to in prior postings.  That has worked fairly well in terms of setting aside sufficient time to deal with randomizations, although it's hard to get people to account for overhead and materialized risks.

Another issue that keeps coming up is that we keep saying we'll deploy our stuff every other week or so, but invariably it gets pushed to the end, which of course increases pressure in the last week.  Hopefully we can find a way out of this in future sprints.

I continue to see positive communication and team cohesion results from Scrum, which alone make me want to continue using it.  In addition I feel it's a great way of organizing work and getting shared focus on a set of goals.

The really interesting stuff will happen with sprint 8 when we try to do "mainline" product work using Scrum...

Over & out!

Chris

Pondering next steps

Hi all -

  With sprint 5 done, we are now in a quiet period, as we have balanced most of the resources on our project toward helping Whidbey get finished.  We will probably pull some resources back soon, however.

  I have been thinking about how to get the part of our team that isn't using Scrum onto the Scrum model.  It's hard, because this other part of the team is working in a reactive mode, fixing newly found bugs and other last-minute issues that arise.  This kind of thing is hard to model in a sprint backlog in any way other than sets of buffers.  So, I'm not sure what to do.  Maybe we will just wait until there's some "ordinary" work (that is, work that means forward progress on new capabilities) to be done, and include some of these reactive tasks via a buffer.

  Our entire feature team is too big to fit into one Scrum team, but would split up into two nicely-sized ones.  So I'm also thinking about how best to deal with this.  As far as scheduling, maybe we will stagger the teams off by two weeks so we don't have sprint endings and the associated planning meetings happening during the same time periods (yikes!).  Another issue is ensuring cross-team communication; we still want to operate as one feature team even though there is a split (but related) focus.  Maybe we'll have to follow the advice I read somewhere (Schwaber's book?) to have 1-2 pigs always visit the other team meetings as chickens and report back.

  If anyone has suggestions on dealing with two Scrum teams within a single conceptual "feature team", I'd be interested in hearing them.

Signing out!

Chris

Managing the Product Backlog

There was a question from Dave Froslie on what we use to store our Product Backlog.  We use an internal wiki page.  Wiki's are nice because they're easy to view, easy to edit (not only for the Product Owner but also for people with suggestions who can add them in a suggestion section at the bottom for the Product Owner to prioritize later), and you can easily see different versions.  You can also track modifications and hunt down people who mess with prioritizations.  :-)

For our Sprint Backlog we use an (ever more elaborate) Excel spreadsheet.

Take care,

Chris

Question about Scrum, waterfall, and more

Hi all –

A reader named Steve Henke writes with some questions:

First, you mentioned your concern about waterfall being used in the next major division release but with even more centralized control than before. Why do you expect more centralized control? Because of results from your team's Scrum efforts or other external factors?
 
Second, how do code reviews fit into your practices, if at all?

[…]

P.S. Have you blogged before about the most useful books on Scrum?

Let me first address the question about waterfall & centralized processes.  Most MSFT products are developed in a quasi-waterfall model where a bunch of planning is done (driven by PM’s), then a bunch of implementation milestones (driven by devs), and then a bunch of stabilization (driven by QA).  In reality we take a slightly more iterative approach by breaking the product up into milestones (but it’s certainly not “iterative” to the extent people use the term in the Agile community).  Of course, different groups can take a different approach (e.g. some internal MSFT teams exclusively use Scrum, some use XP, etc. – however most use a variation of the quasi-waterfall I described earlier).

I don’t know if there will be more centralized control in the next release, but it’s certainly a risk.  You never really know in a big organization.  However, I try to emphasize to some of the movers-n-shakers the desirability of letting each team make progress in its way (whether they will listen to me or not, I dare not guess).  To be fair, shipping a product that has thousands of people working on it introduces many challenges of scope, so there has to be a balance between centralized assessment of overall progress and local flexibility to make progress in the most locally efficient way.

Regarding code reviews, on my team they are mandatory for all checkins.  However, we do not prescribe the code review process very precisely; it is left up to the code reviewer’s discretion to decide how much time to spend analyzing someone else’s changes before signing off on them (or sending them back for changes).

As for useful books about Scrum, the one I recommend is Agile Project Management with Scrum by Ken Schwaber.

Over & out!

Chris

Scrum and development methods

I got a question from René Landgrebe about what development methods we use along with Scrum.  He writes:

"I wonder what developing method you use in combination with SCRUM. I
read about combining XP and SCRUM is very reasonable.

So, what combination do you use? XP, Test-driven development,
feature-driven... ???"

Scrum doesn't dictate much about your engineering practices, apart from encouraging a transparent, "debt-free" definition of "done".  "Done" in Scrum should generally mean that the code is implemented and checked in, bugs are fixed, tests are written, and any documentation which must accompany the software is written, etc.  But that's about all that Scrum says on the matter.

On our project that is using Scrum, we're using TDD (test-driven development) with unit tests.  We're not using XP or feature-driven development.  Team members occasionally do paired work in short spurts (say, 1-2 hours) but we haven't tried pair programming beyond that (one of our team members is planning to instigate a pilot of that, however).

One thing we've done which has proved useful is that we occasionally do whole-team offsites where we take the entire team (including program managers, QA, and dev) to a conference room in another building for a week (with a small skeleton crew left behind to field any urgent issues that arise).  This has two benefits: 1) close proximity in shared space so it's easy to ask questions of teammates, and 2) reduced randomizations from the larger division.

Chris

Sprint 5 complete

Greetings all –

            Well, sprint 5 ended and yesterday we had the sprint review.  Overall the sprint was successful, although we did have one work item bleed over (which we of course disclosed to the stakeholders at the sprint review).

We had small turnout from stakeholders & management.  We had one manager (my boss) and one stakeholder from within the division.  I’ve noticed an interesting phenomenon: now that we’re making good progress, the attention we get from management has been greatly reduced.  I suppose this is natural and encouraging, since it’s the inevitable plight of managers to spend much of their time on problem areas.

            In any case, morale on the team seems to be good and we feel pretty positive about how things are going.  Our VP & our managers are impressed with what we’ve done, which is a great position to be in.

            One thing that a number of team members have independently noted in the past couple weeks is that Scrum’s iterative nature is very good at highlighting problems and making it easier to fix, because the memory of the problem and its cause are fresh in your mind.  Everything we learned during sprint 5 and at the sprint retrospective will be fed into the planning for sprint 6.

            My biggest concern now is how to keep Scrum going once the division spins up the next major release.  My fear is that they’ll employ some kind of heavy-handed waterfall-oriented process, similar to what’s been done in the past but with even more central controls and constraints on how spec’ing, scheduling, and bug fixing are done.  If it’s done careless then it may not jive well with Scrum.  I’ll have to keep agitating on that front to ensure that whatever we do is iterative-friendly.

Over & out!

Chris

Too little sleep, too much caffeine

It seems like forever since I got eight hours of decent sleep.  I love my kids, but it sure would be nice if they would sleep past 6am!  My wife is on me for drinking too much pop -- she thinks 4 a day is too many (if only she knew it was more like 5 or 6!).  I figure Diet Coke and Diet Mountain Dew can't be that harmful -- if aspartame was toxic, I'd be long gone by now.

I feel like a tame version of the characters from "Fear and Loathing in Las Vegas", a caffeine-powered zombie ("I was halfway down the 5th floor corridor when the caffeine took hold...").  And there's the head cold too -- a stuffed-up head and on the verge of losing my voice.  Am I sick because I'm tired, or am I tired because I'm sick?  Or is it a vicious cycle, with one reinforcing the other?

Must focus...must pull together...must get back to work.

Farewell for now...

Chris

Trying to tame our estimates

Greetings all – 

It would seem that our estimates are slowly getting better with time.  When I compare the burndown chart for sprint 5 with the prior three charts, it’s quite good.  The first three look like a silhouette of the Olympic mountain range, whereas sprint 5 trends downward fairly nicely.  Sprint 5 is showing a fixed “drag” that is causing us to tick off less time than we had predicted (say, .75 days of work per actual day instead of 1.0), but this has led to a fairly steady sequence of getting behind, doing a cut, being caught up, slowly getting behind, and so forth. 

The next step, of course, is to better account for this “drag” and try to account for it in sprint 6.  What seems to be happening is not so much that the expected items are taking longer than expected, but that we are spending time on things that we didn’t account for in the sprint backlog.  This is right in line with what DeMarco & Lister say in chapter 9 of Waltzing with Bears (a book that I highly recommend), which is that most software project managers do a decent job of estimating the tasks that have to be done, but a lousy job of estimating that tasks that might have to be done. 

Much of the things that are causing us to tick off less than a day per person on sprint backlog tasks are “overhead” items that we never before tried to quantify: reading e-mail, answering questions from internal and external users, doing code reviews, attending meetings, etc.  We did have some blanket items for certain things (e.g. 2 days for unexpected high-priority bugs), but the list was incomplete. 

Going into prior product backlog selection meetings, we had only the most vague calculations of capacity and overhead, which was OK but not good enough, as we are seeing.  So I’m thinking that going into the next meeting I’ll work with the team to cook up a more comprehensive spreadsheet which accounts for both definite overhead (e.g. time in meetings) and possible overhead (e.g. build breaks that we have to respond to, unplanned absences, etc. – the kind of thing where some of them will occur but not all). 

The items which represent possible (but not certain) overhead are quite interesting.  I’m using some concepts from Waltzing with Bears to try to deal with this.  I’m not using their all-out risk diagrams, but simply planning to take possible items and multiply the cost when realized times the probability of occurrence, which should give a reasonable “risk mitigation reserve” on the schedule. 

I should also comment on my motives here.  So far there hasn’t been any attention to our estimates from higher-ups – if there was, I’d say we already stack up quite well against other teams in our product unit & division.  However, the people who seem to be most stressed out by our running behind is the team itself, and ensuring that they’re not feeling constantly “in the crosshairs” is very important.  Thus I see better estimates as a way to help avoid “behind schedule” situations (which are bad for everybody). 

More work has to be done on this, but I’m hopeful that our estimates can be much better for sprint 6.  I expect this subject to come up from team members during the sprint retrospective, as they’ve already been bringing up the issue of overhead that is important but not accounted for. 

Over & out!

Chris

Scrum and Productivity

Hi all –

A reader named David Wylie wrote and commented the following (I’m posting his question here with his permission):

I am new to scrum, and one of the selling points is significantly increase productivity. Have you experienced this, or is it just a different way of managing?

This is an interesting question.  I would say that Scrum can in fact improve productivity, but for complex reasons.  For one, Scrum gives the team a clear and shared focus during the sprint, and that puts pressure on leads and managers to reduce and eliminate distractions.  Another reason is that Scrum, applied properly, tends to increase team cohesion and cause the group to collectively produce more than it did before.

Carelessly applied, Scrum can reduce productivity if its artifacts are used as weapons and if management doesn’t internalize Scrum’s notion of removing features (as opposed to slipping or reducing quality) as the preferred “knob to turn” when things aren’t going as planned.  If leaders & managers can’t accept the idea that some sprint backlog items won’t get completed, that leaves the team with no real knobs left, other than working extra hours or being deceptive about their status – both of which will lead to serious problems.

Over & out!

Chris