Bright Matrices

As a student of analytics, I’ve learned that measuring success is never as simple as comparing two numbers. In fact, that’s exactly the kind of lazy analysis that can show false positives (or even worse, false negatives). To really do analytics justice, it’s vital to devote time to researching and understanding what you’re attempting to measure so that you can provide informative, relevant, and accurate findings.

A few months ago, a new metric for Twitter was created by Klout, a San Francisco-based company. Their “Klout score” measures your overall influence on Twitter: how much people listen to and respect your messages, and how likely they are to act upon and share those messages. Many third-party clients, such as Hootsuite and CoTweet, are now displaying the Klout score in their dashboards.

I personally think it’s great that Klout is gaining widespread acceptance. I feel it brings a true standard to the “quality” factor of social media analytics in contrast to the “quantity” factor of followers. However, I believe this score, like any other single metric, needs to be used carefully.

In this post, I’ll explain the mechanics of the Klout score, show an example of how similar Klout scores can be misleading when taken out of context, and explain why you can’t lean on new metrics to continue a pattern of “lazy analytics.”

The Klout Score

The Klout score is calculated from a broad range of data: your follower totals, how many accounts follow you back, how often you’re retweeted or mentioned, how frequently you tweet, etc. These data points are then boiled down into three values: your true reach, your amplification, and your network.

• True reach is “the size of your engaged audience,” or basically those accounts with whom you engage with regularly, have common followers, and share similar interests.
• Amplification “indicates how likely it is that your content will be acted upon,” in essence, the probability that you’ll get retweeted and how fast word with spread within your network.
• The third value, network, is “a measurement of the influence level of the people who interact with you.” It’s important to note the emphasis on interaction here: this value is composed of the ratio of your followers vs. how many accounts you follow, how many of those you follow also follow you back, as well as unique senders and retweeters.

So, does this mean we can just look at the scores of any number of accounts and make assumptions about relative influence by directly comparing their Klout scores? No!

Let’s use an example to illustrate why.

An Example of Relative Influence… or is it?

As of August 30, 2010, I have a Klout score of 48. According to Klout’s calculations, this ranks me about halfway on the overall scale of 0 to 100, but based on the total number of accounts measured, I’m more accurately placed in the 80^th percentile. Another Twitter account with a very similar Klout score is the CME Group, a Chicago-based financial firm billed as “the world’s most diverse financial marketplace.” Their current Klout score is 46; they’re also in the 80^th percentile, overall. The CME Group’s Twitter account currently has over 750,000 followers, whereas I currently have 375. Their true reach is calculated to be 280,000, while mine is 9 (not nine thousand or nine hundred, nine… one less than 10).

At face value, based on Klout score alone, it looks like I’m just as influential as the CME Group, but when you start taking into account factors like followers and true reach, this assumption falls apart.

What happened? First, it’s because we’re making assumptions. Second, we’re missing context.

Let’s start digging into the data. You’ll notice that our network scores are roughly similar (59 for the CME Group, 57 for me), though my amplification is a slightly higher (21 for the CME Group, 26 for me). That still doesn’t explain why we have similar Klout scores, so let’s examine the supporting data points.

• True reach: The CME Group has a massive amount of followers, but only follows back 0.003% of them; I follow back 20.3% of mine. They currently show 0% for both follower mentions and follower retweets, whereas I show 19% and 39%, respectively. This indicates that virtually no one seems to be talking to the CME Group or passing along their content. Note: While the 0% follower retweet looks like it’s a fluke, it may actually be a very small number (less than 0.1%) because of their high number of followers.
• Amplification: The CME Group shows 77 total retweets, a mention count of 10, and 69 unique retweeted messages. I have 37 total retweets, a mention count of 254, and 31 unique retweeted messages. Their inbound/outbound message ratio is 0.26; mine is 0.74. This indicates that my followers tend to talk about me more frequently and are more likely to respond to me when I tweet.
• Network: The CME Group’s follower to follow ratio is much higher than mine (172.71 vs. 3.16), their percentage of reciprocal followers is a bit lower (54% vs. 64%), and they have fewer unique folks who are mentioning them (10 vs. 71). They do have a higher number of unique retweeters, however (59 vs. 27). This indicates that my network is tighter and more engaged.

The key theme that emerges here is engagement. While my network and reach are much, much smaller in size and scope, I have a stronger connection with my audience. You also have to consider the function and audience of these accounts. The CME Group gathers and spreads time-sensitive financial information for consumption by their followers; they do some engagement, but are primarily focused on informing and sharing. I also share and retweet content (primarily on topics like social media and user experience), but I spend a larger percentage of my time connecting and talking directly with my followers. Not to mention, no one is trading stocks or making monetary decisions on my tweets (at least, not that I’m aware of). Finally, you need to consider the perspective: the CME Group offers a corporate experience vs. my individual one.

So, even though our Klout scores are very similar, you can’t say that I’m just as influential as the CME Group as a stand-alone statement. Our network, level of engagement, function, audience, and perspectives are significantly different enough to make this a false assumption. You could perhaps say we’re equally influential within our own unique networks, but not as a direct comparison. You have to consider the context.

Don’t Be Lazy With Analytics

As with website analytics, you should never use a single metric in a vacuum to make assumptions on the success or failure of what you’re measuring. Think for a moment about hits. Have you ever been asked to compare the number of hits on your website with those of another website? It’s a thoroughly unfair comparison. Even if the sites in question are similar in audience and purpose, you have many things to consider, such as: originality, readability, and freshness of content; strength and effectiveness of marketing and promotions; ease of use and overall usability; ranking for relevant terms in search engines. Each and every one of these factors, as well as a number of others I haven’t mentioned, all contribute to the success or failure of a website. Taking only one metric and directly comparing it to the same metric for a completely different website is not an apples-to-apples comparison, no matter what you may think. It’s essential to take everything in context and do thorough research of the data to determine relative success or failure when making comparisons.

It’s exactly the same with the Klout score.

You can see from my example how much happens behind the scenes in order to generate a Klout score. It’s a deeply interesting and unique metric in its own right, but, just like with website hits, it needs to be examined and valued in context with its supporting data points to form a complete picture of relative influence.

Don’t be lazy by looking at just one metric to measure your influence on Twitter. If you’re really interested in determining how well you’re doing or what you need to improve in order to make your presence in this channel more effective, do your homework: sift through the data, observe trends, and pay attention to the purpose and audience of your account. Klout allows you to refresh your score every six days, so take advantage of that feature and keep refreshing your score to see how your data shifts over time.