Agile measurement
Anjali’s blog post on ‘Measurement versus engagement‘ made me think.
We’ve spent a great deal of effort over the last two years We’ve spent a great deal of effort over the last two years smashing painstakingly integrating Agile thinking into the strategy and design practices, but at the other end – the delivery end of the process – Agile is too often neglected soon after launch. painstakingly integrating Agile thinking into the strategy and design practices, but at the other end – the delivery end of the process – Agile is too often neglected soon after launch.
This shouldn’t be surprising.
It’s difficult, if not impossible, to keep all that innovationism going for extended periods of time. Often, the innovation team within a client organisation moves off onto new projects and hands the service to less risk-friendly teams, where people have KPIs set by line-managers who don’t know about very much about the project and almost certainly have no idea of probably much less understanding of the Customer Value.
Sometimes the fervent adoption of Agile methodologies that happened during the Vision and Service Definition phases turns out to be superficial and temporary. Who doesn’t like Agile when there’s tons of new code and excitement being demonstrated on a daily basis, and when you can get things done with less pain and far more rapidly than you ever did before? It’s definitely more difficult infinitely more challenging to adopt Agile as a culture, over the long term. “Iterative” and “emergent” are best intention ideas that make total sense during the Vision phase – but are VERY difficult to do on an ongoing basis, especially if the culture at large within the client business doesn’t get it, which is a big ask let’s face it!
But occasionally, even on innovation-geared projects, it’s the metrics what kills it.
Typically, there may have been a conversation early on in the project where you were asked to make some guesses about traffic and registered users. You may have been reassured that these would not be treated as promises, and you probably made the point that it’s very risky to make those kind of predictions with an innovative and emergent service like yours, I mean – how could you possibly predict these things, and they’re not the right things to measure anyway..?
Whatever. A metrics time-bomb has started ticking.
Agile gets rid of fixed scope/fixed budget which is great – but that means nothing if you replace them with some dumb fixed metrics. So, I’m arguing here not for less measurement or no measurement, but for an Agile approach to measurement – which, obviously, I absolutely understand the need for.
Agile measurement should focus on simplicity and Customer Value. It should make it easier to measure the things that matter to make the service better for its users – to improve the value exchange. Instead of asking what success looks like, ask what value looks like.
It follows then that Agile measurement should measure fewer things. There is a risk – especially with all the new tools and dashboards and diagnostics – of measuring far too much. Measurement is in danger of becoming an end in itself. A smaller set of metrics and diagnostics – “just enough”, in exactly the way as Agile methodologies treat other types of documentation – should help. Even trying to find the “one key metric” that rules them all – and is tied to the economics investment and supported by executive management – is suggested in this awesome paper I found online and have plundered ruthlessly for this blog post. That makes so much sense.
You should also try and measure “outcome” rather than “output”. It’s not about big numbers, it’s about having the biggest impact on Customer Value. A small community of incredibly engaged, high-value customers is much more valuable than 1 million people who registered to win a car. Obviously, you’ll have first needed to have defined upfront what Customer Value actually is – for you, your project and the business sponsoring it. If you’ve been working Agile from the outset, you should have a pretty good idea of your customer/end-user and their needs, and the value exchange that’ll get them returning hopelessly addicted. hopelessly addicted.
Maintaining a living, ongoing dialogue with users will certainly help – and is essential where the one key metric is for example a softie, like brand perception. But you should generally focus on a small set of engagement health indicators: like the number of visits per user per month, dwell time, participation, recommendations to friends, positive buzz across other social networks leading to sustained referrals. I’ll also add receiving offers of help from partners and people wanting to be involved, and good ideas from users – and the holy grail: people start using the site for purposes never envisaged. Remember, EBay realised there was an online market for cars when they found customers selling grown-up cars in the toy section.
Lastly, you should be able to review targets, metrics and diagnostics to optimise your ability to understand Customer Value as the service evolves. Within an emergent, iterative innovation-geared project, you may not even understand the true value and how to achieve it until much later.
What does anyone else think?
6 comments
Great post. Thanks.
I particularly liked the emphasis on ‘Less, but Better’ metrics. In a world where everything can measured the danger is that nothing is measured, or measurement paralysis sets in and nothing is agreed as a key metric. Or too much time is given to measurement over creativity. Sacrificing ‘everything’ for something meaningful, and agreeing this early on – & then sticking to it as much as possible – seems especially important here.
Any fool can list what might be measured. It takes slightly more nous to know (or to make a judgment call) on what should be measured.
There’s a super huge opportunity that exists for some kind of amalgamation of Hitwise/Phorm/Dunhumby and all the telecoms to track everything we do, everything we look at, everything we interact with and even everyone we speak with. This could then act as some kind of massive data overlord that tracks and targets ads at us to engage with interactive stuff and then measures how well that ‘engagement’ and ‘customer value’ rewards marketers with sales.
Until some weirdo builds that the best thing to do is talk to people lots to understand what they think about the stuff you’ve created. Quantify it with a questionnaire for the suits if needs be. Personally I think the latter is the only way forward.
Good post.
The problem comes in part, I think, from people treating the internet as an adjunct of direct marketing rather than a place to start dialogue, create experiences, co-create value, etc.
In addition, the focus on campaigns puts too much emphasis on short-term metrics, rather than the longer term relationships and goodwill generated by actually talking to customers on Twitter, sending them free samples, etc.
Shifting clients’ opinions isn’t easy, but I do sense a slight change in the attitude of even some of the bigger clients, as they start to understand that they need to adopt a more open, long-term approach about their use of the internet.
I like this a lot. In the vein of less but better, I think how actionable a metric can be is a good indicator of it’s quality, i.e. What are we going to do differently this week based on this data. So much of measurement in the Fortune 1000 is about looking back at past activities and finding a way to prove success by manipulating data. It’s time to make measurement a forward-looking endeavor.
its, not it’s. fast typing. damn.
I agree with the comment made earlier about measuring less, but really knowing what you are measuring. To me the only way this can be done is if measurement is thought of first. I find that loads of random data are often dug up and presented in an effort to prove a campagain was really sucessful. We can spin any numbers to do this, but meaningful metrics are the few that are built into a program before it starts. Understanding this can provide true insight.