New Metrics Are No Excuse to Continue a Pattern of Lazy Analytics: An Example Using Klout
As a student of analytics, I’ve learned that measuring success is never as simple as comparing two numbers. In fact, that’s exactly the kind of lazy analysis that can show false positives (or even worse, false negatives). To really do analytics justice, it’s vital to devote time to researching and understanding what you’re attempting to measure so that you can provide informative, relevant, and accurate findings.
A few months ago, a new metric for Twitter was created by Klout, a San Francisco-based company. Their “Klout score” measures your overall influence on Twitter: how much people listen to and respect your messages, and how likely they are to act upon and share those messages. Many third-party clients, such as Hootsuite and CoTweet, are now displaying the Klout score in their dashboards.
I personally think it’s great that Klout is gaining widespread acceptance. I feel it brings a true standard to the “quality” factor of social media analytics in contrast to the “quantity” factor of followers. However, I believe this score, like any other single metric, needs to be used carefully.
In this post, I’ll explain the mechanics of the Klout score, show an example of how similar Klout scores can be misleading when taken out of context, and explain why you can’t lean on new metrics to continue a pattern of “lazy analytics.”
The Klout Score
The Klout score is calculated from a broad range of data: your follower totals, how many accounts follow you back, how often you’re retweeted or mentioned, how frequently you tweet, etc. These data points are then boiled down into three values: your true reach, your amplification, and your network.
- True reach is “the size of your engaged audience,” or basically those accounts with whom you engage with regularly, have common followers, and share similar interests.
- Amplification “indicates how likely it is that your content will be acted upon,” in essence, the probability that you’ll get retweeted and how fast word with spread within your network.
- The third value, network, is “a measurement of the influence level of the people who interact with you.” It’s important to note the emphasis on interaction here: this value is composed of the ratio of your followers vs. how many accounts you follow, how many of those you follow also follow you back, as well as unique senders and retweeters.
So, does this mean we can just look at the scores of any number of accounts and make assumptions about relative influence by directly comparing their Klout scores? No!
Let’s use an example to illustrate why.
An Example of Relative Influence… or is it?
As of August 30, 2010, I have a Klout score of 48. According to Klout’s calculations, this ranks me about halfway on the overall scale of 0 to 100, but based on the total number of accounts measured, I’m more accurately placed in the 80th percentile. Another Twitter account with a very similar Klout score is the CME Group, a Chicago-based financial firm billed as “the world’s most diverse financial marketplace.” Their current Klout score is 46; they’re also in the 80th percentile, overall. The CME Group’s Twitter account currently has over 750,000 followers, whereas I currently have 375. Their true reach is calculated to be 280,000, while mine is 9 (not nine thousand or nine hundred, nine… one less than 10).
At face value, based on Klout score alone, it looks like I’m just as influential as the CME Group, but when you start taking into account factors like followers and true reach, this assumption falls apart.
What happened? First, it’s because we’re making assumptions. Second, we’re missing context.
Let’s start digging into the data. You’ll notice that our network scores are roughly similar (59 for the CME Group, 57 for me), though my amplification is a slightly higher (21 for the CME Group, 26 for me). That still doesn’t explain why we have similar Klout scores, so let’s examine the supporting data points.
- True reach: The CME Group has a massive amount of followers, but only follows back 0.003% of them; I follow back 20.3% of mine. They currently show 0% for both follower mentions and follower retweets, whereas I show 19% and 39%, respectively. This indicates that virtually no one seems to be talking to the CME Group or passing along their content. Note: While the 0% follower retweet looks like it’s a fluke, it may actually be a very small number (less than 0.1%) because of their high number of followers.
- Amplification: The CME Group shows 77 total retweets, a mention count of 10, and 69 unique retweeted messages. I have 37 total retweets, a mention count of 254, and 31 unique retweeted messages. Their inbound/outbound message ratio is 0.26; mine is 0.74. This indicates that my followers tend to talk about me more frequently and are more likely to respond to me when I tweet.
- Network: The CME Group’s follower to follow ratio is much higher than mine (172.71 vs. 3.16), their percentage of reciprocal followers is a bit lower (54% vs. 64%), and they have fewer unique folks who are mentioning them (10 vs. 71). They do have a higher number of unique retweeters, however (59 vs. 27). This indicates that my network is tighter and more engaged.
The key theme that emerges here is engagement. While my network and reach are much, much smaller in size and scope, I have a stronger connection with my audience. You also have to consider the function and audience of these accounts. The CME Group gathers and spreads time-sensitive financial information for consumption by their followers; they do some engagement, but are primarily focused on informing and sharing. I also share and retweet content (primarily on topics like social media and user experience), but I spend a larger percentage of my time connecting and talking directly with my followers. Not to mention, no one is trading stocks or making monetary decisions on my tweets (at least, not that I’m aware of). Finally, you need to consider the perspective: the CME Group offers a corporate experience vs. my individual one.
So, even though our Klout scores are very similar, you can’t say that I’m just as influential as the CME Group as a stand-alone statement. Our network, level of engagement, function, audience, and perspectives are significantly different enough to make this a false assumption. You could perhaps say we’re equally influential within our own unique networks, but not as a direct comparison. You have to consider the context.
Don’t Be Lazy With Analytics
As with website analytics, you should never use a single metric in a vacuum to make assumptions on the success or failure of what you’re measuring. Think for a moment about hits. Have you ever been asked to compare the number of hits on your website with those of another website? It’s a thoroughly unfair comparison. Even if the sites in question are similar in audience and purpose, you have many things to consider, such as: originality, readability, and freshness of content; strength and effectiveness of marketing and promotions; ease of use and overall usability; ranking for relevant terms in search engines. Each and every one of these factors, as well as a number of others I haven’t mentioned, all contribute to the success or failure of a website. Taking only one metric and directly comparing it to the same metric for a completely different website is not an apples-to-apples comparison, no matter what you may think. It’s essential to take everything in context and do thorough research of the data to determine relative success or failure when making comparisons.
It’s exactly the same with the Klout score.
You can see from my example how much happens behind the scenes in order to generate a Klout score. It’s a deeply interesting and unique metric in its own right, but, just like with website hits, it needs to be examined and valued in context with its supporting data points to form a complete picture of relative influence.
Don’t be lazy by looking at just one metric to measure your influence on Twitter. If you’re really interested in determining how well you’re doing or what you need to improve in order to make your presence in this channel more effective, do your homework: sift through the data, observe trends, and pay attention to the purpose and audience of your account. Klout allows you to refresh your score every six days, so take advantage of that feature and keep refreshing your score to see how your data shifts over time.
6 commentsThe Mystery of the Silent Partner
Social media produces some interesting and downright curious behaviors. Over the past few months, I’ve noticed a rather compelling relationship that’s popped up in my social interactions. It tends to manifest in two similar fashions:
- Someone chooses to follow or friend you. They’re active within their own network; it’s not a spammer or automated account, but a real person, just like you. No matter what you post, write about, or discuss, they never talk to you, retweet you, or share any of your links. And yet, after weeks or months of time, they’re still a follower or friend of yours.
- You choose to follow or friend someone, and they then follow or friend you back. You like what the person posts, writes about, or discusses. You retweet them, comment on their activity, and try to engage with them. No matter what you do, however, they never respond (except, perhaps with invitations to be their friend on other social networks).
I call this relationship the “silent partner.”
For those watching at home, there are key subtleties to the “silent partner.” First, they are always connected to you by their own will. They chose to follow or friend you, not the other way around. And, for Twitter, it’s not always an auto-follow, either. Second, they’re not the New York Times or Mashable accounts; big names or key influencers in their respective fields who post and share content but don’t engage with specific individuals, even if you make an attempt to do so. They’re personal or corporate accounts who are otherwise talking and sharing with others, just not you.
What mystifies me about the “silent partner” is, well, their silence.
For me, I’m curious about the value you bring to their network. After all, they’re connected to you for a reason. You couldn’t possibly be that boring, chatty, or annoying to them, otherwise, they would have severed the connection long before now. And it’s not because the “silent partners” have massive numbers of followers or friends, where my posts would be small ripples in their community pool. Some of the accounts I’ve studied have very manageable numbers, and my chatter would be noticeable. Not to mention, I regularly engage with accounts with thousands or tens of thousands of connections; they’re talking back to me.
Based on these assumptions, it seems that neither the content you produce nor the size of the partners’ communities are clear, straightforward causes of their silence. So, what could it be?
For the “silent partners” who follow but never engage, my guess is that they pay attention to only a core piece of their community, using either Twitter lists or groups within Facebook, for example. Perhaps they desire the status of having a large number of friends or followers and focus only those who matter to them. Or, maybe they’re reluctant to let go of those they’ve attached themselves to and simply put up with you. In any case, they tend to fit the classical profile of the “lurker”: the person who simply reads and consumes without contributing back.
As for the “silent partners” with whom you try to engage with but never get a response back, that’s a trickier assumption. It could be the same thought as above, but they’re obviously ignoring you for some reason. Maybe what you say isn’t worth their time. Maybe you haven’t passed some sort of litmus test to be “worthy” of their engagement. Maybe they’re just watching to see what you’re all about.
Overall, the concept of the “silent partner” is one I want to examine further. I’m considering directly contacting some of mine to see if my assumptions are correct.
What about you, my dear followers? Do you have similar experiences? Or are you yourself the “silent partner” of your community? I’ve love to hear from you. That is, if you’re willing to break your silence…
No commentsAuto-Following and Mutual Follows: A Circle of Obligation
There have been a good amount of discussions recently that focus on influence vs. followers on Twitter, as well as how you decide whether or not to follow a specific account. I’ve also be reading posts and comments about perceived notions of “Twitter etiquette” with regards to following, specifically auto-following and mutual follows. I have some thoughts on these two specific interactions that I’d like to share, based on my personal observations and perspectives.
Auto-Following
“Auto-following” is, as you would expect, a process where a Twitter account starts following you automatically. This usually happens when you mention a specific word or phrase, or start following that feed (see the “mutual following” section below). There are two basic reasons I can see for why someone would choose to auto-follow: a bid to get more followers themselves or to monitor what’s being said about them, their business, or a specific topic in Twitter.
Personally, I think auto-following is highly inefficient. I once likened it to “shooting at a moving target, in high winds, blindfolded.” Nearly all of the auto-follows I’ve observed that come from keywords are totally off the mark. I once tweeted to ask for music recommendations, asking for “anything except country.” I immediately got followed by an account for a country musician. I never again tweeted anything about country, nor had I done so until that moment. Um, you’re doing it wrong!
Think about how many false positives and missed marks you could end up with if you decided to auto-follow in this manner for the purpose of monitoring. It’s much more effective to use some basic mention tools like Topsy, socialmention, Hootsuite, Google Alerts, Twitter’s search engine, or enterprise tools like WebTrends or Radian6. That way, you can check for instances where your brand is mentioned, in context, and reach out to specific accounts appropriately and intelligently.
Let’s go back a moment to the misguided exchange with the country musician. This could have been turned around had the musician engaged with me. They could have said something like, “Hey, I read that you don’t like country, but why not check out Song X from my new album?” Sure, it would have been a sales pitch, but that personal touch might have swayed me to at least give it a shot.
Mutual Following
The act of “mutual following” is simple: I follow you, you follow me. Most often, this is triggered automatically. What I don’t like about mutual following, particularly when dealing with individuals, agencies, or small organizations, is the expectations that come with it.
I don’t feel any obligation whatsoever to start following someone just because they started to follow me, and neither should you. I use Twitter as an information aggregator and professional networking tool. I purposely pick and choose which feeds to follow. I don’t have the time or the patience to wade through irrelevant or unrelated tweets simply to honor a “return the favor” agreement. That said, I also never expect anyone I follow to start following me in return. I’m quite certain that I’m not interesting to everyone, and that doesn’t bother me one bit.
You may notice some interesting behavior when accounts try to solicit mutual follows from you. They’ll start following you, then, when they don’t get the “expected” follow back, they drop you. Tools like Qwitter can send you updates on who unfollows you; most often, it’s the accounts that started following you randomly.
Now, I do see one solid use for mutual following from a customer service or issues management perspective: exchange of private, direct messages. For example, say you post a question or complaint to the Twitter account for your bank. They may ask you to follow them; when you do, they’ll follow you back. Once you’ve mutually followed each other, the bank can send you a direct message to discuss personal information about your account or give you contact information to get ahold of a representative. When the exchange is over, both parties can unfollow each other, if they choose.
Final Thoughts
Overall, I think both auto-following and mutual follows, with the exception of the customer service interaction I described above, are rather pointless and don’t contribute to the value that can be derived from using Twitter. You’ll get much more out of this channel by directly choosing the accounts you want to follow, using proven and effective tools to monitor comments and sentiment, and pursuing focused, helpful engagement.
Don’t contribute to the circle of obligation that surrounds these interactions, and never feel that you’re doing your followers or colleagues a disservice by avoiding them. Good relationships, whether in Twitter or in real life, should not be based on guilt or expectations.
1 commentNotes From Edward Tufte’s “Presenting Data and Information” Course
On July 28, 2009, I attended Edward Tufte’s “Presenting Data and Information” course in Philadelphia. Tufte is well respected for his expertise in data visualization and equally renowned for his complete disdain of PowerPoint as a communications tool.
I recently came across a page-and-a-half of handwritten notes I had taken during the lecture and wanted to share what I recorded. Below are elaborations on these notes.
- Begin with the content by asking the question: “how can something be explained?” Be guided by the task. Don’t choose the mode of presentation in advance: it’s not about pre-specifying the dataset or methodology.
- The character of relationships between elements is just as vital as the elements themselves. Provide “reasons to believe”.
- Use causality thinking: which properties effect and govern the cause.
- Annotate everything: annotation is the heart of explanation. Annotations reside in the background. They are subtle, but clear and help avoid optical clutter. Annotating “linking lines” adds credibility and texture to causal links.
- Use tables; don’t be afraid of them. People read huge tables all the time: think sports, weather, market data, etc. “Bring your presentation up to the level of the sports section.”
- Replace “chart junk” with evidence (but not selected evidence; avoid being a “cherry picker”).
- Supergraphics are interactive: they allow individuals to explore and find what’s important to them and encourage discussion among an audience. Everyone will look at a different section at the same time. Find a really good supergraphic to open your presentation and give it to the audience on handouts.
- There is no such thing as information overload: there is only failure of design. Add detail to improve clarity and content. Use more tables than graphics, especially for smaller numbers.
- PowerPoint is “a corrupt method of displaying information”. PowerPoint presentations set up a dominance relationship with extreme information denial (“a long and winding road”).
- Integrity, relevance, and interest are content properties that design will not correct.
- Find good reports and copy them (but have good taste); “stay out of the design business. Don’t get it original; get it right”.
- For intellectual models, avoid marketing speak and corporate pitches; you want “non-fiction credibility”.
- Use wall charts in project management. Make comparisons over space vs. a sequential series of slides (“stacked in time”).
- Letter codes, legends, keys, etc., are impediments to learning. They are not universal, but instead “one-off” creations that are good for only one instance. Get viewers out the decoding business: use direct labels.
- There two issues in information design: multivariate problem: the dimensions of data need to be communicated on a two-dimensional display (“flatlands”). Every interesting analytical problem is more than one to two dimensions or factors. Information resolution: A way to measure progress in communication and presentations. Think: what is the rate of information throughput in my presentation?
- Use the “brute force” method: build a model by getting a real object in the room.
- The segregation of information by modes of production is a conceptual error.
- Don’t insult or fail to respect your audience. Maintain intellectual integrity without patronizing. It’s not considered “dumbing down” to remove jargon for dispersal to a larger audience.
- Give users the freedom to consume the material how they want.
- For the opening screens (home pages) of websites, show off what people can learn there. People are good at scanning: they will scan, then scroll, then click; keep it flat and rich. 90% of every screen, excluding navigation, should be content.
- Sparklines reduce “recency bias” by showing changes over a larger span of time.
- When giving presentations, remember the following: Work on content. Practice. Show up early. Use handouts and Word documents (never PowerPoint slides). Leave more than one copy of technical reports. Define what the problem is, who cares, and what the solution is.
Here are several key quotes I jotted down during the lecture:
- “Do whatever it takes to display something.”
- “Document everything and tell people about it.”
- “The metaphor is the map.”
- “A feature that is buried is not a feature.”
- “Talent imitates, but genius steals.”
- “The person who heads up web design is a content expert.”
The course was more lecture than seminar, but Tufte kept me interested and engaged throughout the day. A bit prickly in person, he’s nevertheless an excellent presenter. I would highly recommend the course as a “professional development opportunity” (i.e.: training) for those interested in how to present content and data in pretty much any form, not just visual representations. Plus, you get all four of his books to add to your library, which is an excellent take-away… more so than a flash drive or fancy badge clip.
No commentsBored With Twitter? You Have No One To Blame But Yourself
Over the past few years, Twitter has become a serious method of communication for individuals, businesses, causes, and governments. Yet this channel still can’t shake the perception that it’s all about vapid observations on sandwiches and the activities of dimwitted celebrities. It’s sad, because Twitter has, in my opinion, moved beyond an “emerging channel” into an established communications platform that has much potential and plenty of uses. Not everyone is talking about fluff … there’s lots of good stuff out there.
If you think Twitter is boring or ephemeral, you have no one to blame but yourself. Why? Because it’s all in how you use it.
It’s quite possible you’re doing it wrong.
I’ve said over and over again that you are in control of your social networks. It’s your territory; you can do whatever you want. Twitter is all opt-in: if you want to get real-time updates on any number of personalities or topics, you just have to follow ones that interest you. Don’t like what you’re reading? Don’t follow those feeds anymore. Getting bored with the same chatter from the same folks? Shuffle the deck once in a while: go use Twitter’s search engine, check out real-time updates through Google, or look up hashtags to find feeds or topics that are of value to you, and follow them. Pull up the websites of your favorite hobbies, restaurants, musicians, etc. and see if they have a Twitter feed to follow. Try them out for a bit, and don’t feel bad about unfollowing them if it doesn’t work out.
What you’re reading on Twitter should be valuable to you. Don’t waste your time by following feeds you no longer read or find interesting. Don’t sit there and wonder why you’re not finding anything good by refusing to be adventurous and finding new sources of information. Go out there and get it done … and stop blaming Twitter for your boredom.
Update (8-10-10): While cleaning out my Twitter favorites this morning, I happened upon a saved tweet from Olivier Blanchard (@thebrandbuilder) that was the progenitor of this blog post. Therefore, I must give credit where it is due and cite his post for inspiration.
2 comments

