I was reading an old conference program last week, looking for inspiration to write the program foreword for the excellent upcoming Superweek data event, when I came across the wisdom of my friend and former colleague Avinash Kaushik. Essentially he was stressing the need for analysts to be a part of the business, not simply be a data monkey:
” We, Analysts, feel we are in the business of providing data (well segmented custom reported calculated metricked data for sure!). But, still just data… The best data analysts are in the business of business. ” -Avinash
It sounds so simple (and it is), yet I continuously come up against smart people with responsibility for data who remain waaaay too focused on data points, rather than the business they support. A good example of this is the confusion that is stirred up when discussing KPIs…
What Is a KPI?
Its an obvious acronym everyone I meet in business is aware of, but the definition of what a key performance indicator actually is appears to get lost in translation. Here is my definition, taken from Advanced Web Metrics (3rd Ed):
In web analytics, a key performance indicator is a web metric that is essential for your business’s success. The emphasis here is on the word essential. If a 10% change—positive or negative—in a KPI doesn’t make you sit up and call someone to find out what happened, then it is not well defined. Good KPIs create expectations and drive action, and because of this they are a small subset of information from your reports.
So what’s the big deal?
The problem (as reading Avinash’s comments reminded me), is that “business’s success” gets forgotten. In fact, it is so regularly forgotten that I need to replace my emphasis. When running brain-storming KPI workshops with clients, I often end up being given a list of interesting data points—what I refer to as signals—for my opinion on. For example, data that the SEO manger should be aware of; information showing the performance of the latest ad campaign; percentage of visitors using the site-search box, and so forth. The result is I then have to stress that these are signals only. They are informative data points to the person responsible for creating/managing them, but nonetheless they are not KPIs for the business.
Here’s my schematic analytics data stack to illustrate my terminology:
Notes for my data stack diagram:
- Noise isn’t junk data – any spam should be filtered out and binned. Rather, noise is data that is simply not of interest (or relevant) right now. It may be of interest at a later date, or if your analysis produces a curve ball. So in effect noise and signal are interchangeable depending on your current focus. However they should always be separated out.
- Signal is your interesting i.e. informative and relevant data. It tells you something about what is happening and it is what analysts look at day-to-day i.e. short time-frames. For example, a spike in visitor or customer acquisitions; the number of file downloads per day; the performance of a specific marketing campaign; number of transactions, daily revenue etc. A signal is information—its what analysts build their hypothesis (story) on.
- KPIs—as I defined above—are your method to measure the performance of your long-term objectives, typically 12 months. Once determined they should not be modified, otherwise you lose the ability to make any long-term comparison (new KPIs can of course be added, or replace old ones). Naturally KPIs are data points as well. However they are special set, often a combination of metrics specific to your organisation i.e. not lifted direct from your reports. Hence I don’t refer to them as “data points” in the general sense.
Of course my data stack diagram is not to scale. Noise in reality is much, much broader and thinner, representing perhaps thousands of different metrics and dimension combinations. Signals will be fewer in number, perhaps representing a hundred metric/dimension combinations. KPIs should be very few—less than ten for any particular stakeholder. Less is more when it comes to KPIs—for example five could be very effective for a non-transactional site (remember segmentation of your KPIs will result in more than five KPIs to monitor!).
Re-emphasising The KPI Definition
Note my original definition above has not changed. However, I wanted to expand on the important parts:
1. Who are KPIs intended for?
The business, business, business! In other words NOT people with day-to-day responsibility for the performance of a campaign or the website. Rather, your stakeholders and leadership team. These people are not Google Analytics users and they should not need to use Google Analytics to understand what your KPIs are. That said, stakeholders are still likely to be analytical thinkers themselves.
Produce KPIs by job role
That is, once your stakeholder group has agreed on a common set of high-level KPIs for the business as a whole, each stakeholder has their own KPIs more specific to their role within the organisation. For example: Sales Stakeholder; Marketing & Communications Stakeholder; Product Stakeholder; Customer Retention Stakeholder, and so forth.
Make you KPIs hierarchical
Segment your KPIs depending on the managerial level of your audience. That is, KPI reports for the CMO will differ in segmentation detail form those who work within his/her team. Generally, the higher the managerial level the less segmentation required.
Avoid mixing signals with KPIs
Sure, your digital marketing team will need their own key metrics to measure their efforts. These will be fluid, in that they are likely to change regularly on short time-frames. They will be detailed, as by definition they will capture subtle changes. And in some cases the data points will confirm negative results i.e. rule out a certain hypothesis of visitor behaviour. These are important data points. They are the signals your digital team watch as their feedback mechanism for their day-to-day work. They are not KPIs.
2. What constitutes “essential” for a KPI?
As I state in my definition, a very good rule of thumb I use is a 10% change—positive or negative. For example, if your conversion rate (converting a prospect to a new lead or customer) changes by 10% it should make your stakeholder sit up and call someone to find out what happened. Would your stakeholder mind if you called them at 7pm to tell them of this change? If not, its a great KPI. If the thought of calling your boss in the evening to discuss KPI changes fills you with dread, then its certainly not a KPI.
BTW, calling your stakeholder at 7pm to give them the news that their KPI is dropping still makes you a hero. Spotting a problem early could save the company a lot of money…
3. Purpose: Whats the point?
The purpose of KPIs is to communicate the important events that are happing to senior management—those with vested interest in the outcome. Therefore by definition a KPIs purpose is to drive strategic change. That is, enable the business to make informed decisions—and that happens over long time frames (for example, you cannot change a product overnight). Nice to know data can be very interesting to the digital team and can even be the basis of a staff bonus system. However, avoid the common pit fall of thinking that interesting data is a KPI.
By maintaining focus on these three important points, you will gain traction with senior management and build your case for change. Form your point of view, that may be the business investing more in your team!
Things to consider when defining KPIs:
- Try to use monetary values where possible; everybody understands $$$.
- If not a monetary value, use a ratio, percentage, or average, rather than the raw number. This allows data to be presented in context.
- A KPI needs to be temporal, that is, time bound. This highlights change and its speed.
- A KPI drives business-critical actions. Many things are measurable, but that does not make them key to your organisation’s success.
Use KPIs to put your data into context. For example, saying “we had 10,000 visitors this week” provides a piece of data, but it is not a KPI because it has no context. How do you know whether this number is good or bad? A KPI based on this data could be “our visitor numbers are up 10 percent month on month.” This is a temporal indication that things are looking good over the time span of one month. In this example, the raw number should still be part of the KPI report, but it is not the KPI itself.
The vast majority of KPIs are ratios, percentages, or averages. However, sometimes a raw number can have a much greater impact. Consider the following examples:
- Our website lost 15 orders yesterday because our e-commerce server was down for 34 minutes.
- We lost $10,000 in potential revenue last week because our booking system does not work for visitors who use Firefox.
- We spent $36,000 last month on PPC keywords that did not convert.
Clearly, knowing whether any of these numbers are increasing or decreasing and what fraction of the total they represent is important, but the impact of these raw numbers is far greater at obtaining action and therefore should be the KPI in these examples.
The key point is that you should develop KPIs relevant to your particular business and your stakeholders. Any metric, percentage, ratio, or average that can help your organisation quickly understand visitor data and is in context and temporal should be considered a KPI.