First, a look at the week that was.  Overall, the generic hashtags are being used consistently and in true Big Brother fashion, more hashtags are being used everyday, often introduced by CBS. Just this week I had to add, #BBTracker as well as a few running hashtags, #ZackAttack (a nickname for HG Zach), #Zankie (showmance between Zach and Frankie), #ZankieFallOut (the potential end of Zach and Frankie), and #EvictionPrediction. It’s a wonder why CBS continues to add hashtags that could possibly be steering people away from using their generic ones. What’s wrong with a simple #bb16?

Being elimination night, Thursday seems to be the peak show of the week with close to double the amount of tweets of the other two days. We will look at whether the ‘big night’ remains consistent in coming weeks in the interest of finding out whether context or type of show are more important for tweet numbers.

Total number of tweets containing the generic BB hashags for the week 6 – 12 July.

Taking a closer look at the Thursday show – conversation remained fairly consistent with no major spikes, just a lot of volume.

Total tweets by minute for the Thursday show.

HG Twitter Accounts

The graph below shows us which housemates’ twitters are getting the most mentions, and unsurprisingly at the top, is Joey, the first eliminated contestant voted out in a unanimous 13/13 vote. Two down the list is the most recently voted out contestant Paola, which we are guessing will probably be the most talked about by this time next week if there is a pattern. We saw this last year when HG Kaitlin was voted out early yet continued to be one of the most talked about HGs for the remainder of the series, largely thanks to her involvement in the racial-slur-scandal and somewhat thanks to her social media presence.

Number of times contestant Twitter accounts have been mentioned; 6 – 12 July.

Something new: Users by Timezone

Something we became interested in during last year’s BB broadcast was the difference in tweeters from one side of the US coast to the other, this year also considering the top 10 timezones joining the conversation with the generic hashtags:

Top 10 timezones using the generic BB hashtags on Twitter; 6 – 12 July.

Eastern time tweet by far the most of any of the timezones which fits with the documented distribution of the US population (47% live in Eastern Time). However, people in Mountain Time are tweeting more about Big Brother than expected with their distribution being only 5.4%, but publishing 9.45% of the total tweets regarding Big Brother. We’ve established that the Quito (Ecuador) timezone aligns with Chicago / Central time, so those users who say they’re in Quito, likely aren’t.

Until next time…

Let’s take a look back at some of the bigger stories of the World Cup from social media, as well as the prominence of the event in Europe.

One widely reported research result from the knockout stages of the World Cup was how Twitter users reacted to the penalty shootouts. Twitter’s own research department put out a graph of the Greece v Costa Rica match, which was widely picked up in the press.

In particular, Twitter noted that sometimes “silence tells the story”:

Parallels can be drawn here to other events. Particularly, we looked in the past at how different forms of television spark Twitter conversation, with reality television frequently seeing peaks in discussion during the show.

This contrasts with dramas such as Sherlock, which often see their peaks at the end, with a similar “anticipation” window during the show itself.

The US (and Australia) loves football

As we discussed previously, the World Cup has set viewing and streaming records in the United States.

It seems the presence of Americans in the Twitter conversation hasn’t been significantly hit by their team’s elimination. Germany v Brazil had the highest viewing figures of any World Cup semi-final in American television history, and was the highest ranked non-US game ever on ESPN/ESPN2.

A look at tweets on generic World Cup hashtags from July 10-14 show the US led the way in number of tweets. Brazil ranked second, with locals still interested through their team’s third-place playoff (and, of course, any tourists who had changed their timezone). London ranked third with finalists Argentina in fourth place:

In Australia, SBS also reported new streaming records for its World Cup coverage across mobile and online, with users showing a large preference for “live” coverage versus on-demand. SBS’ World Cup multi-stream service (below) won many plaudits, with the only negative being that sound issues persisted throughout the final.

Comic relief

As ever, beyond the discussion of the matches themselves, social media remains a hotbed for sarcasm and humour. FIFA president Sepp Blatter was a source of controversy throughout the tournament, and – sitting next to Vladamir Putin – remained a source of amusement (and marketing) in the final, as shown in this tweet by Betfair Australia:

Also prominent during the penalty shootout that decided the Netherlands v Argentina semi-final was a mistake from British commentator Peter Drury, who was featured on the television feed that went to range of countries including Australia.

Drury has never been one of the most popular commentators, and his mistake – being ready to proclaim the Netherlands victors in the semi-final – quickly spread around the internet. See the Drury penalty call below:

The view from Europe

We started this series of articles discussing the role of brands during the World Cup, and that was one of the themes in Europe as well. In many cities you were unable to move without noticing some form of localised World Cup branding, including the following example from Cyprus (which did not qualify).

Noticeable across Europe, though, were extensive World Cup decorations: from bars in basically every city, through to the large screens that inundated public squares, and – in the case of Amsterdam – a sea of orange which descended upon the city and sat above nearly every pathway in the Centrum.

And that’s the World Cup.

The authors do not work for, consult to, own shares in or receive funding from any company or organisation that would benefit from this article. They also have no relevant affiliations.

This article was originally published on The Conversation.
Read the original article.

Particularly interesting in the discussion of streaming figures is that such activity is able to be measured in “streaming minutes” or “data transferred” – much more specific metrics than traditional audience figures.

Accurate global TV ratings are still a way off, considering the official FIFA World Cup 2010 Audience Report came out almost a year after the tournament.

Twitter and ratings are undeniably connected, but the extent of the correlation often depends of the type of broadcast: whether it’s a live sporting event, soap opera finale or reality television show.

Indeed, a breakdown of the global tweets by timezone shows the dominance of US viewers in the Twitter conversation (including non-English hashtags):

Previous work by Nielsen has shown Sports, Reality TV and Comedy are genres where tweets have a causal relationship with viewer numbers, with Nielsen reporting that in 28% of sports programming measured, tweets had an impact on viewing numbers.

As we contended in last week’s article, it’s possible viewers tweet when bored, as well as when excited, during a game. But it is also possible that those not watching the game are also tweeting about it, so any correlation between ratings and tweets, for sporting events, needs a bit more research.

Match tweets

The US/World Cup love story continues in the graph below with the US vs Portugal match dominating match conversation for the week, and taking the lead in our “tweets by match” table.

ESPN also found that the:

USA vs Portugal contest on Sunday, June 22 is the most-viewed soccer match across all US television networks, averaging 18,220,000 viewers.

The top match in this graph has roughly a third more tweets than any match in the tournament so far, with top matches in previous weeks peaking at around 263,000.

It’s also possible from what we have discussed above that the Brazil vs Mexico match was bumped up by the large Mexican contingency located in the US timezones, as well as the enormous Brazil following on Twitter that we’ve seen in previous weeks.

The most talked-about event by far at the World Cup last week was the Luis Suarez bite.

The bite followed two other incidents of Suarez biting in the past, creating a storm of online conversation that can be seen in the visualisation of the most common words in tweets mentioning Suarez (note also the prominence of “Snickers”, a recurring example of brand impact on the World Cup):

The controversy around the bite mostly relates to whether Suarez intentionally bit the other player or just fell in an unfortunate position making it look like he bit him – as some Uruguayans have argued. Again, asking the question of whether video-technology should be more widely used in the game as recently discussed by Miguel Sicart.

Diego Maradona, the Argentinian whose hand-ball goal in the 1986 World Cup sparked much controversy said:

This is football, this is incidental contact […] They have no commonsense or a fan’s sensibility. Luisito, we are with you.

And a Reuters report notes:

The referee did not spot the incident during the match, but FIFA’s rules allow the use of video or “any other evidence” to punish players retrospectively.

Indeed FIFA did punish Suarez, announcing a nine-match ban on June 26 (the second spike visible in the graph below):

While the correlation is clear between real-time events and Twitter, the graph above quantifies just how vocal Twitter users have been around the Suarez incident, with the bite generating more than 3,500 tweets per minute at peak.

The lower volume for the announcement of the ban is also a signifier of the number of people watching the game(s) live and tweeting, versus those who use Twitter as a more general information source or discussion platform about the World Cup.

The authors do not work for, consult to, own shares in or receive funding from any company or organisation that would benefit from this article. They also have no relevant affiliations.

This article was originally published on The Conversation.
Read the original article.

Here, we look at the shift in conversation as the tournament begins to take shape – who is excited, bored or really winning on Twitter? – but first, a taste of what’s happening on social media in Brazil.

The opening ceremony and first match, Brazil vs Croatia, were huge successes on television and on social media. Brazilians, of course, probably talked about nothing else that day – but in Brazil, much of what was said was politicised.

FIFA was massively criticised for choosing a Belgian producer over Brazilians for the opening ceremony.

Francamente ! Na terra de Paulo Barros e Rosa Magalhães chamam um belga pra fazer uma bobagem de abertura destas!

— Leda Nagle (@Lnagle2) June 12, 2014

Translation: Honestly! In the land of Paulo Barros and Rosa Magalhães [two of the most successful Brazilian Carnival producers] they called a Belgian to do a silly opening like this!
– Leda Nagle, Brazilian journalist.

Another major disappointment was the disappearance on the official FIFA images of the moment that a paraplegic gave the initial kick-off using a mind-controlled exoskeleton built by the Brazilian scientist Miguel Nicolelis.

O genial exoesqueleto usado pelo rapaz que daria o primeiro chute infelizmente se perdeu na transmissão da abertura. Uma pena.

— Fernando Meirelles (@fmei7777) June 12, 2014

Translation: The exoskeleton worn by the guy that would do the kick-off unfortunately got lost in the opening broadcast. What a pity.
Source: Fernando Meirelles, Brazilian film-maker.

E teve exoesqueleto mesmo. Lamentável o desprezo total da transmissão ao que deveria ter o principal destaque. pic.twitter.com/PqDN9MUSEI

— Impedimento (@impedimento) June 12, 2014

Translation: And there was the exoskeleton indeed. Regrettable the complete disdain in the broadcast to something that should be in the spotlight.
Source: Impedimento, popular website dedicated to South American football and culture.

Still, others strongly criticised the crowd chants attacking the Brazilian president.

Parte do estádio grita: “Ê Dilma, vai tomar no c.. “. Outros gritam: “Ê Fifa, vai tomar no c…”

— Jamil Chade (@JamilChade) June 12, 2014

Translation: Part of the stadium shouts: hey, Dilma, f* off. Others shout: hey, Fifa, f* off.
Source: Jamil Chade, Brazilian journalist.

Updating the top matches

In our last article, we noted that Brazil vs Croatia was the most talked about match on its official hashtag (#BRAvsCRO), some distance ahead of England vs Italy, which was closely followed by Germany vs Portugal, Spain vs the Netherlands and Argentina vs Bosnia and Herzegovina. The updated chart, through the matches of June 21, looks as follows:

Of particular note here is that we have a new leader, in the Brazil vs Mexico match (an otherwise unspectacular 0-0 draw), with the Argentina vs Iran fixture (a 1-0 Argentina win, which Iran looked like winning at times) in second place.

The prominence of these two matches raises questions of whether people look to Twitter to fill in boring games, as well as to comment on exciting ones. The next three are familiar fixtures from the first week of matches.

Many of those at the bottom are the result of people using reversed hashtags in their tweets. Noticing this for the England vs Uruguay fixture, we also tracked the reverse hashtag specifically (#ENGvsURU), and recorded in excess of 27,000 tweets compared to 88,236 on the official hashtag (#URUvsENG).

So, while the official hashtags are performing as some form of marker, their success is not universal. One explanation for this is that while in Europe, the standard form is “Home Team vs Away Team”, for Americans the familiar format is “Away Team vs Home Team”, and so ordering hashtags for international audiences can be difficult.

What’s being shared?

Last time, we discussed how brands were dominating the conversation on official World Cup hashtags. This time, we’ll take a look at what is being shared on the match hashtags themselves.

As with last week’s data, we again see @worldsoccershop heavily represented, with their offer to give away free shirts if you retweet and a specific event happens (such as Ronaldo scoring in the Germany vs Portugal match) drawing a massive response.

Tellingly, the other tweets are largely dominated by US related content, the top two being ESPN responses (@Sportscenter being an ESPN-operated account) to the US’s victory over Ghana.

The first non-US tweet comes from the UK’s Sky Sports, and their @SkyFootball account, asking for responses on a penalty in the Brazil Game. Sky, interestingly, are not broadcasting the World Cup in the UK.

Other notables in the top 20 include celebrities such as Piers Morgan and Kobe Bryant, the US’s Comedy Channel (also not a World Cup broadcaster), asking Americans to “RT if you think WE WILL WIN”, and a quote from an unofficial Simpsons Quote Of The Day account, but really, @worldsoccershop was the huge winner.

The limitations of the 1%

As we discussed last time, the representativeness of Twitter research by those not subscribing to data providers such as GNIP is unclear with the World Cup, as Twitter traffic continually exceeds 1% of the total amount of tweets published at any particular time.

The flip-side of that limitation is we are able to graph the times at which conversation around the World Cup; through the team accounts, tournament hashtags, match hashtags and television hashtags we are tracking, exceeds that 1%, and by how far.

Of course, at any particular time, there are also many tweets relating to the World Cup which do not contain any of the previously mentioned identifiers:

The blue indicators in the graph above are the number of total tweets per second that exceeded 1% of total Twitter traffic.

Notable is that the World Cup is generating a smaller portion of the total Twitter traffic as it continues – which may not be much of a surprise – but also that while the opener generated the most prolonged period of >1% traffic, the matches on the morning of June 14 AEST (the matches of June 13 in Brazil) were the most prolific of the tournament on a per-second basis, with a particular peak during Spain’s demolition by the Netherlands.

It has yet to be seen how the next phase of the tournament will play out, and least of all what role Twitter will play; whether as a tool for excitement or boredom.

The authors do not work for, consult to, own shares in or receive funding from any company or organisation that would benefit from this article. They also have no relevant affiliations.

This article was originally published on The Conversation.
Read the original article.

by Darryl Woodford & Katie Prowd

The 2014 World Cup has already seen a significant volume of Twitter conversation across a number of (English language) keywords, including #joinin, #worldcup, #Brazil2014 and #worldcup2014, as well as the Twitter-marketed international hashtags:

• #Brasil2014 (Spanish)
• #Brésil2014 (French)
• #wm2014 (German)
• #Copa2014 (Portugese)
• #mm2014 (Finnish).

And unsurprisingly, riding this wave of hashtags are the brands that look to profit from the tournament – whether they’re official sponsors or not.

With the launch of a new interface designed to promote World Cup discussion, Twitter is actively encouraging users to flag support for their national team and to participate in World Cup discussion through Twitter.

On the opening day of the games Twitter presented a new layout, as well as a step-by-step process encouraging people to tweet their support for their team and change their profile image:

After clicking “Let’s go!” on the page above, users are escorted through a number of personalised set up pages; from selecting their national team and changing their profile picture:

… through to following favourite players, and even preparing a tweet using the #WorldCup hashtag and the account of your national team:

While these are obvious promotional tools, they have likely contributed to the increase in followers for many players, as well as the Twitter activity around the tournament in general.

While the BBC’s Gary Lineker on Tuesday described Brazil 2014 on air as “the first social media world cup”, South Africa 2010 also saw plenty of social media activity. However the impact of social media on traditional media coverage is particularly prominent in the UK at the moment.

Twitter has also been documenting the tournament through its blog and tweets from the TwitterData account. For researchers, replicating such analysis is difficult as World Cup-related tweets frequently exceed the limit of 1% of tweets that be freely accessed through the Twitter API. Despite this, there are a few notable stories from week one.

Brands seek to capitalise on World Cup audience

While it’s clear that the World Cup is a brand marketing exercise, the lead up to the tournament demonstrated how the brand is being appropriated for marketing purposes on social media, far beyond the official sponsors.

And while using the World Cup brand in traditional media may see offending companies hit with a lawsuit, using the social media hashtag appears to be a risk worth taking.

FIFA have not taken trademark infringement lightly either, officially releasing a warning in March stating that

The contribution of FIFA’s commercial affiliates is vital to the success of the 2014 FIFA World Cup and we therefore ask companies to refrain from attempts to free-ride on the huge public interest generated by the event.

Yet according to Alex Benady from PR Week:

FIFA, players, the media, the FA and other national associations, and of course brands with no contractual relationship with the World Cup, will all be working their social media networks for all they are worth.

Supporting this, the 20th most popular retweet in the week leading up to the World Cup using English keywords was the following from (unaffiliated) British company Fragrance Direct:

Other brands, sponsors and otherwise are also heavily represented in the most frequent retweets:

The top 25 tweets above contain many brands (including FIFA sponsors such as Adidas, Budweiser and EA Sports, as well as non-sponsors such as Goldman Sachs and Fragrance Direct), able to associate with the World Cup brand on social media on an equal basis.

While the brands may see this as merely interacting with a current event, for those at FIFA and for paying sponsors, this may well appear as ambush marketing.

Such trends extended into the first week of the tournament, with the top retweets over the first week notably also dominated by big brands and television networks:

Top 10 matches

With the first round underway, we can also see which matches (and teams) are receiving the most attention on Twitter:

This tells an interesting visual story of not only the top matches but also how the worldwide audience is using Twitter during the World Cup.

While the top match to date is (perhaps predictably) the opener of the tournament – Brazil vs Croatia – the presence of England vs Italy as the second may speak both to the audience participating in the hashtag conversation and the international interest in the game itself.

As the tournament continues, it will be interesting to correlate tweet volume with television audiences worldwide, as those figures become available, and to consider whether the teams with the most historic World Cup success, or FIFA Ranking, are those receiving the most attention this time around, both on Twitter and on television.

Other stories from around the web

Elsewhere on the web, analysis of both social media and statistical data around the world cup is gathering steam. Kimono Labs have launched what they claim to be the first open World Cup API, while the Regressing Blog on Deadspin features a round-up of the top prediction models on the web.

Also of interest this week is the CartoDB visualisation of Twitter activity around the World Cup opening match, and Twitter’s own visualisation of the increase in Neymar’s followers, part of their extensive coverage of the opener which also includes the Predictaroo.

We’ll be back after Round 2 with some more from the ground in Europe and Brazil, as well as the latest data from our Twitter Machines, and a look at how TV stations are using Twitter in the early stages.

This article was originally published on The Conversation. Read the original article.

A Quick Look Back

So, as we left things, we put out a few predictions for the week beginning January 20, as follows:

So, some hits and misses here.. The most concerning of these are Pretty Little Liars & Ravenswood, so let’s have a closer look at those, starting with Ravenswood. Essentially, I think this was an over-prediction influenced by the second half premiere, which rated at 181,200 tweets. We have now added an adjustment for these type of shows, that will highlight second half premieres in our predictor, with the option to manually exclude them where they rate significantly different. More concerning was “Pretty Little Liars”, which wasn’t a second half premiere but a regular show. Here, I think we had the reverse problem. Because there weren’t enough shows in the past two months (only 1 after the second half premiere), our prediction algorithm defaulted to include episodes from the previous series, where 300-400k tweets was the norm. Combining this with the 488,000 tweets from last week, this seemed a reasonable estimate, but the show actually nearly doubled that performance — it will be interesting to see how Pretty Little Liars fairs over the coming weeks.

The other errors of over 10% were both reality shows, and here I’m tempted to put the predictions on hold until we’ve established a measure of season context. Just as Big Brother follows a weekly cycle of Daily Shows, Nomination Shows and Eviction Shows, shows like American Idol, The Voice etc follow a somewhat standard season format between auditions, performances, eliminations and so forth. Modelling this is on my to-do list, but it’s after the establishment of the Twitter Excitement Index and indeed the modelling of regular season shows with their premieres, finales and other formats. Time and other priorities didn’t allow us to get predictions out for this week, and I won’t do so retrospectively, but we’ll get some out either late this week or early next.

Twitter Excitement Index

One thing we did finish last week was the establishment of our Twitter Excitement Index, which is a measure of the volatility of conversation on Twitter based on the principles of Brian Burke’s Excitement Index at Advanced NFL Stats. Over a few weeks Katie Prowd and I went through a few variations on this approach, looking at different measures to see which best captured the patterns in Twitter conversation, and I also owe thanks to Patrik Wikstrom for his help in tweaking our statistical approach. Essentially though, the theory here is that if you have two shows, they may both average 100 tweets per minute, but have very different engagement levels, as seen in this dummy data:

In the first graph, the activity is ‘spiky’, that is, every other minute people are prompted to tweet, while in the second there is an underlying level of 100 tweets, with minimal variation around that average, suggesting a constant stream of conversation but no particular moments which provoke users to tweet. These should be at opposite ends of the spectrum, and with our Twitter Excitement Index, the top graph would see a TEI of 9.9, while the second gets a TEI of 0.5. The scale here has been calculated to vary between 0 and 10 for presentation purposes, although in practice from our test data it would seem very unusual for a show to achieve a TEI of over 5.

The Australian Premieres

We are currently working on adjusting our metrics to work with Australian television, and this weeks premieres of The Block and My Kitchen Rules gave a good first test for this approach. But first, let’s cover the basics, and look at how these shows performed on Twitter:

As you can see here, My Kitchen Rules clearly won the night, both in terms of total tweets and audience peaks, from The Block in second place. The third line here represents The Biggest Loser, which wasn’t premiering but did air an episode in competition with the other two reality shows. The twitter audience for The Biggest Loser has fallen off a bit since its premiere, but this performance must be considered low by any standards. A similar picture was seen in the TV ratings, with My Kitchen Rules reaching 2.4m viewers, The Block 1.1m, and The Biggest Loser just 560k. However, it’s interesting to compare these to other recent reality ‘events’:

What is evident here is that Big Brother still owns the crown for Australian reality television, at least on Twitter, with both the Premiere and Finale easily outperforming the launch episode of My Kitchen Rules. For Channel 9, the performance of The Block suggests that the reality show success they achieved with Big Brother is not easily transferable to other shows, and for Channel 10 both the performance of Masterchef and Biggest Loser must be a cause of some concern… A similar pattern can be seen in the unique audience for Monday’s shows (n.b. the percentages add up to over 100% because of viewers who tweeted about multiple shows):

Finally, the launch of these shows also gave us an opportunity to test some of our new metrics, and after a slight hiccup caused by time zone confusion (the TEI saw the drop-off to QLD discussion as a good thing for the show, viewing NSW levels as large spikes), we ended up giving a slight win to MKR, although both ranked below the season launch of The Biggest Loser. The TEI does not take into account tweet volume, only measuring the volatility of conversation among the audience that *did* watch the show, and so should generally be read in combination with the audience size to properly understand the conversation around a show.

Mumbrella also covered some of this, and you can view their article here.

And so wraps another week in Telemetrics.. until next time!

Evaluating our predictions

Over on my own blog, we have been predicting tweet numbers for a selection of upcoming TV shows. It became immediately obvious is that our prediction methodology coped badly with Second half premieres. We had deliberately excluded season premieres and finales from our predictive data set, but it now appears clear that second half premieres (i.e. after a show goes on a winter hiatus, but within the same season) also receive a significant boost in tweets. With our new data (scraped from EpGuides & Wikipedia), we are now able to systematically identify these episodes, and plan to evaluate whether we can make a simple adjustment to account for the boost such shows receive. Overall, our average error for last week was 33.19%, however if we exclude the second half premieres that falls to 9.21%, which is within the realms of what we were expecting, and easily out-performs the simple average of the last 10 or 4 shows. Taking a simple average of the last 10 episodes would have recorded an overall error of 35%, falling to 25.5% once the second half premieres are excluded, a much larger figure than our 9.21%, although of course the sample sizes here are still small.

Filling out the last two months: Replacement Value & Tweets Above Replacement (TAR)

One significant issue we had been experiencing using our preferred “average weighted tweets over past two months” (of episodes) metric was that of small sample size. While I am reasonably confident that we can demonstrate the last 8 shows are a better estimator than the weighted average of all shows in our database, for shows we are not manually tracking there can be many holes in this data. A show which barely scratches the Nielsen top 5, and isn’t in our tracking system, may be missing data for 75% of the last two months shows, as seen here:

If we, as we have previously, simply don’t count shows for which we have don’t have tweet figures, we are heavily biasing the estimates. As you can see in our prediction data above, we predicted 10,245 tweets last week for “The Carrie Diaries”, which turned out to be a 12% over-estimate. A better solution here is to adapt a concept from sabermetrics called “replacement value” . Essentially, in baseball and other sports, replacement level is defined as the production you could expect from signing a “free agent”, that is – the next best player who is not under contract to any major league team. This allows, for example, to predict the impact of an injury, trade or contract expiry on a teams production, but also to measure other players against this level; for example through Wins Above Replacement. I plan to return to the latter concept, or in this case Tweets Above Replacement (TAR) in the future, as a means of measuring how well a show does in comparison to throwing a “replacement level” show on in that timeslot.

For our current purposes, and the Carrie Diaries example, we are more concerned with predicting how many tweets a particular episode may have got, despite the final statistics not being (freely) available. However, we do know what the bottom ranked show on Friday is, which – for example – was 3,700 tweets on 15th November, 1,600 tweets on 22nd November, and 6,600 tweets on 6 December. However, here too, we are slightly thrown by Nielsens method of reporting, as the Tonight Show was ranked fourth with 1,600 tweets, but ahead of BBC America’s “An Adventure in Space and Time” which ranked 5th with 9,200, based on number of impressions.

So, for now, let’s call the replacement value for Friday nights 5,000 (we’re currently calculating replacement values for each Day/Month combination). If we plug 5,000 in for those missing episodes, instead of ignoring them, our weighted tweets formula would have predicted 8552 tweets, an error of just 5.2% compared to the previous 12.2%.

We’re probably a few days away from properly integrating this into the prediction algorithm, but we do hold high hopes for this improvement.

Sometimes you can be too precise, or the problems of small samples

Another, perhaps short lived, tweak over the past 10 days was experimenting with weekly indexes rather than monthly. The idea here was that such a system would better account for weeks such as ‘sweeps’, when networks tend to put on their strongest programming, as well as for the effect of major events such as the Superbowl, Oscars, Golden Globes and so on. Particularly with the wresting shows, such a method would also account for the cycle around events such as Wrestlemania and Summerslam, which anecdotally appear to signal increased tweeting. For much of our older data, where we have 50-100 weekly shows weekly indexes appear to offer small incremental advantages in forward-predictions over the monthly indexes utilised in the above predictions.

However, when predicting for the future (where weekly indexes can be more volatile), and for the last few months of data when we often only have 25-50 shows in a week, the volatility of this index negated such advantages and actually showed a decrease in prediction accuracy. In particular, weeks 52 and 53 of 2013 and 1 of 2014 (i.e. over the Christmas break) saw such low weekly indexes that any show that had mild success during those weeks was suddenly forecast to have season-highs in their next episode, when in actuality the index was primarily impacted by a few extremely low days over the actual Christmas holiday (particularly when we exclude sporting events – i.e. the NBA and College Football bowls which dominate American broadcasting over the Christmas period).

It may be that eventually we can recover those incremental gains, but for current purposes we plan to return to the monthly indexes. This also means we can more easily predict shows for the current month, where the index has already been somewhat established, as opposed to ‘predicting’ the monthly index itself based on historical data and trends.

Some predictions for the coming week

So, with all that said, let’s head back to the spreadsheets and give our top 10 predictions for the coming week:

Twitter Excitement Index (TEI)

Finally, in the tradition of all good TV shows, we’ll leave you with a teaser for the coming week. Based on work by Brian Burke at Advanced NFL Stats, we have developed a measure we’re calling the “Twitter Excitement Index”, which essentially measures the peaks and troughs in Twitter conversation to establish a measure of excitement. That is, a show may be averaging 3000 tweets / minute, but have a time series graph which hovers slightly above and below the average over the course of an episode – in this case, it has attracted a large twitter audience but that audience doesn’t seem to be particularly provoked by the content of the show to tweet. By contrast, a show averaging 300 tweets / minute may be spiking continuously throughout the episode, showing that people were reacting to the events as they happened (as we have seen, for example, with Big Brother 15 in our previous work). By again stripping out those factors related to audience size, we are producing a metric which analyses the reaction of Twitter users to the content of the show. But, more on this next week..

Telemetrics, which I define as the application of Sabermetric principles to television ratings and social media engagement, has thus been the focus of my work over the past couple of months. We are beginning to make substantial progress in both the historical analysis, and future prediction, of Twitter engagement around television. At this stage, I should also mention that our progress here would not be possible without a team of research assistants & embedded student projects, which includes Katie Prowd, Portia Vann and Kate Guy, who have (respectively) put together the data and assisted in testing the metrics, taken on the work of developing dashboards to display our results, and investigated and designed infographics to better communicate results to a wide audience.

Our work to date shows that we are fairly accurately able to match the Nielsen SocialGuide capture technology, at least for shows which do not exceed the 1% Twitter API restrictions (as I have discussed previously in relation to Scandal). In terms of correlations between tweets and television ratings, we have observed a substantial variation across genre, format and country. In particular, early results suggest a significant difference between reality shows and more standard television fair such as sitcoms and drams. Additionally, ‘specials’ (which includes events such as the 2012 US Election debates and award shows, as well as potentially premieres and finales of series) do not appear comparable with standard episodes of series. Finally, for now, we have also noted substantial differences between similar formats in Australia as compared to the United States, which is particularly significant since much of the literature, and examples of best practice, come from the work of researchers and organisations such as Nielsen (and their SocialGuide subsidiary) which have focused on the US market. All of this means that drawing a fixed correlation between traditional television ratings and Twitter use does not seem a sensible approach.

Big Brother 15: CCI vs Nielsen Data

The first thing we had to do was verify our collection methodology. Luckily, we have long been collecting tweets around television, as part of work discussed previously here on Big Brother 15, which expanded to comparisons between Big Brother in Australia and the United States, and subsequently reality TV shows as a whole. With the start of the 2013 TV season, we added a range of new terms including popular sitcoms, dramas and sci-fi shows in order to broaden the number of exemplars available to us. However, returning to the Big Brother data made sense for verifying our methods, as both we and Nielsen SocialGuide recorded, and Nielsen had published on their website, the number of tweets, and unique users, for a large portion of the season, as visualised below:

From our perspective, that was pretty good. Tweets matched almost exactly, and the only major difference, on 23 August (22 August in the United States), was easily attributable, being the night in which the Head of Household competition continued after the show, thus resulting in us recording an oversampling of tweets (compared to Nielsen) which were attributable to the live feed, rather than the network broadcast. That said though, we were very happy with these numbers, and meant we could be confident our methodology was producing accurate results. Thus, excluding known data outages, we were happy to move forward with the data we had collected. It is worth noting here that while we were slightly different on unique users, it was by a relatively consistent amount. We’re still not sure of the cause of this; although it seems unlikely that we were counting just enough tweets from a group of repeat users to have slightly different terms but the same overall volumes, and so with Nielsen’s methodology essentially a black-box, the mystery will remain.

The weighted tweet index

The Weighted Tweet Index (WTI) was the first metric we created as part of the Telemetrics project. Essentially, the goal was to break raw volume numbers down into it’s constituent parts. Perhaps the best way to illustrate this is by way of an example.The 1750th ranked show (at the time) by pure volume, excluding specials and sporting events was an airing of the film “Space Jam” on Cartoon Network from Friday 6 July 2012. The data (in this case from Nielsen SocialGuide) records a total of 25,033 tweets attributable to the show, making it the 3rd ranked broadcast of the day. But what if that had aired on an average day, in an average month, and on an average network (taken in this case to be an average broadcast network – e.g. ABC, CBS, FOX, NBC)?

According to our current metrics, it turns out that a Cartoon Network show can expect to see about 16% of the twitter activity of an average broadcast network. Similarly, Friday’s can expect to receive around 52% of the Twitter activity of an average day (nothing new here for TV Execs, Friday is historically a dumping ground for non-performing shows), and July – being in the offseason for TV – receives 65% of an average month. Note that there’s a lot of refinement possible here: a kids movie during the school holidays (I assume) should probably be expected to do better than average, and we currently don’t account for the time of day nor what was being scheduled against it. But, for now, let’s stick with our numbers, so:

Weighted Tweets = (Old Tweets) / (Network Factor * Month Factor * Day Factor)

Therefore, new tweets = 25,033 / (0.16 * 0.52 * 0.65) = 25,033 / 0.0541 = 462,717 tweets if shown on an average broadcast network, on an average day, in an average month. With the greater degree of precision in the model (i.e. not rounding to 2 decimal places), our actual figure here is 466,805. As it turns out, once ranked by weighted tweets, Space Jam thus moves from being the 1,750th ranked show in this period to the 317th.

Predicting the Future

Much of the utility of this approach comes not necessarily from analyzing the past, though we’ll certainly do a lot of that over the coming weeks, but the ability to predict the future. Because our ‘Weighted Tweets’ figure has been stripped of much of it’s context, we can then take this figure (across all episodes that we have data for within a particular series), and apply to it the factors that will apply when it next airs. Here are two examples:

Here you can see one interface to our model. Once a show is selected at the top, historic data is pulled from the data store, and a number of metrics are calculated. Which of these metrics is the best estimator varies depending on the volume, source, and age of the data we hold. In this case, we have a number of recent episodes and so the weighted tweets for the last month were selected. We entered the day & month the show was airing, as well as the network, and (for now) set the growth factor as 1. Growth Factor is a variable that we are still experimenting with, although currently it is largely contained within the Month factor for current/future dates. Ultimately, we predicted 157,511 tweets, and when the Nielsen data became available, we saw the results:

Not too shabby. 7,000 on a total of 150,000 is a 4.80% error; for the first version of the model we’ll certainly take that. Looking at the last 10 shows, a pure average of the raw tweet totals over that time would have been 99,591 tweets, and for the last four shows 113,100 tweets, so our prediction certainly appears to be outperforming simple averages.

Both Teen Wolf & The Bachelor were essentially premieres (Teen Wolf being a second half premiere), and Wolf Watch appears to be a companion show to Teen Wolf, and so we are currently unable to predict any of those three, so let’s move on to WWE Monday Night Raw.

Here, again, we have a lot of data, with this being a weekly show, and so the monthly average was a sensible choice. Again we added in the new variables (Monday, Jan 2014 on USA), left the growth factor alone, and predicted 215,507 tweets, and the results were again quite pleasing:

We were off by 10,594 tweets, a 4.69% error, although this time the other way. It’s interesting to note that simply taking the average of the previous 10 RAW shows would have predicted 153,398 tweets, and an average of the last four shows would have predicted 165,275 tweets, so again we do seem to be on the right track.

Note here that we’re not necessarily predicting what it will get, but what it should expect to get. In part that’s because there is still a wide range of factors we don’t account for, but even as we add more refinements to the model over the coming months, there’s still one we can never really account for: content. Just as we saw in the Big Brother work I’ve published here previously (and more formal journal versions of that are still in the works), there are some things you cannot account for, such as the racism scandal that engulfed Big Brother 15 and saw a rapid increase in tweet volume.

However, in some ways knowing what a show should expect to get can be more useful, both in providing a barometer of success for networks, producers and social media strategists, and in providing industry and researchers with a list of shows which either exceed or fail to reach these levels, thus allowing an analysis of what may have contributed to the success or failure of a particular episode or series on social media.

More on our progress here in the coming weeks, but for now we’re pushing ahead with refinements to the above model, and new metrics galore!