A couple of presentations considered social media from a systemic perspective – here are Ben Light and Elija Cassidy on the practices of disconnection using social media:

And here’s our ‘very big data’ perspective on Twitter as a global social network:

Wilfred Wang explored Weibo, and highlighted the fact that even in centralised China regional differences still matter immensely in social media:

Jacinta Buchbach explored the implications of social media use for employers and employees:

Jean Burgess, Elija Cassidy, and Ben Light discussed the social media components of the Movember phenomenon:

While Ben Light tackled the elephant – er, the cat – in the room:

Finally, in a panel of the uses of social media for second-screen engagement with television, Darryl Woodford et al. introduced Telemetrics as a new set of methodologies and metrics for evaluating audience engagement:

And continued the discussion by considering the impact of existing follower/followee structures in the Australian Twittersphere for such engagement with TV content:

So much for 2014 – see you next year in Phoenix for IR16 (the call for papers has been released already)!

Given this popularity, and a significant overlap between the target market for Big Brother viewers and the social media platforms, it will be interesting to observe the extent to which social media activity (and perhaps, eventually, sentiment) acts as a predictor for votes on the show. In this blog, following the first round of nominations, first eviction and the first round of single nominations, we are going to look to the data from the last 2.5 weeks to try to test whether social media activity acts as a predictor of public votes.

So far, at least, it has been a mixed bag, but let’s start with the positive; the public vote for the ‘Perfect Pair’ dance competition, in which the winners were awarded $30,000, was held between the final two pairs – Lawson and Aisha & Dion and Jason. The public then voted for the pair with the best dance through JumpIn, but did they actually just vote for their favourite pair? If we use social media activity as a barometer, it seems that could be the case. Our data showed a tight race, which Lawson & Aisha just pipped, and indeed the public vote came back 51.8% in favour of Lawson & Aisha. Perhaps, if they had been up against, say, Travis and Cat – who were hardly mentioned this week – they would have won by even more:

Lawson also tells an interesting story in the overall polling; as seen in the chart below which highlights the running total for all housemates; largely anonymous until the dance-off and his decision to give Aisha the lions share of the prize money ($20,000) was rewarded in the social media volume.

Below is a running total of Twitter mentions for the pairs since launch night, however we will focus on the last week’s long-winded and highly debated eviction process for the time being. Nominees made up 5 of the six most talked about housemates on the night before the eviction process began, and the ones not being talked about were being carried by their partner based on the pairs table:

We can of course ask some other interesting questions from these charts: where were Skye and Lisa when they were ‘saved’? Were Jake and Gemma losers in the public vote due to anonymity, or hatred? What caused David and Sandra to be saved, when they were virtually anonymous through the first week, and only talked about subsequently in regard to David’s chauvinistic comments. Was it better for David to be hated, rather than not talked about at all? Related to this, there is the question of screen time and popularity inside the house, allowing us to address what went wrong for Gemma this week, given her achieved intent to secure airtime?

Up for eviction this week were Skye & Lisa, Jake & Gemma, Travis & Cat and David & Sandra. Ever since the Katie & Priya first week fiasco, Skye & Lisa have been by far the most talked about pair of the season and consequently were saved on Monday night as per our prediction based on the previous graph, with Skye & Lisa the most popular pair on the 22nd September. Interesting here, however, is that Gemma & Jake were the pair with the second most social media activity, and the most popular during the nomination period, indicating that the sentiment will also be a significant factor in creating further predictions.

While we have our own tool monitoring Big Brother discussion (http://bigbrother.thehypometer.com), Channel 9 (Mi9/JumpIn) have also launched a counter, the “Big Brother Radar”, which captures tweets and Facebook statuses by those who seek, deliberately, to be noticed by the radar using official C9 hashtags (e.g. #BBAUGemma). Our tool, by contrast, attempts to measure the underlying volume of discussion (and, by possible inference, interest) in the competitors as a whole, on social media.

Going forward, we hypothesise that those housemates who the public have no interest in will be those who struggle in a ‘vote to save’ format. That said, it’s probably not advisable to bet based on this information. It may be that the Radar format serves as a better prediction of those likely to be evicted (i.e. the effort to post with the correct hashtag is correlated to the effort to vote), it may be that sentiment proves highly significant, or indeed it may be that social media is not a good barometer of the BB voting public. Whichever of these proves to be the case however, the data is sure to be interesting.

Finally, it is worth noting that one of the problems of a lack of live feed – which we have ranted about previously – and indeed this year any live updates at all is that it allows producers to largely control the message; hence, social media reaction largely follows the amount of airtime given to contestants and the plot lines developed, much like a soap. By contrast in the USA, with 4 live camera views running 24 hours a day, users are able to create and share their own storylines about the housemates — generating ‘hype’ for the show which we do not see here. In Australian Big Brother we are told what to think, and we’ll leave it as an exercise for the reader how that reflects on wider society. Finally, we’ll leave you with a running total of the housemates mentions to date, where Skye continues to lead the way:

First, a look at the week that was.  Overall, the generic hashtags are being used consistently and in true Big Brother fashion, more hashtags are being used everyday, often introduced by CBS. Just this week I had to add, #BBTracker as well as a few running hashtags, #ZackAttack (a nickname for HG Zach), #Zankie (showmance between Zach and Frankie), #ZankieFallOut (the potential end of Zach and Frankie), and #EvictionPrediction. It’s a wonder why CBS continues to add hashtags that could possibly be steering people away from using their generic ones. What’s wrong with a simple #bb16?

Being elimination night, Thursday seems to be the peak show of the week with close to double the amount of tweets of the other two days. We will look at whether the ‘big night’ remains consistent in coming weeks in the interest of finding out whether context or type of show are more important for tweet numbers.

Total number of tweets containing the generic BB hashags for the week 6 – 12 July.

Taking a closer look at the Thursday show – conversation remained fairly consistent with no major spikes, just a lot of volume.

Total tweets by minute for the Thursday show.

HG Twitter Accounts

The graph below shows us which housemates’ twitters are getting the most mentions, and unsurprisingly at the top, is Joey, the first eliminated contestant voted out in a unanimous 13/13 vote. Two down the list is the most recently voted out contestant Paola, which we are guessing will probably be the most talked about by this time next week if there is a pattern. We saw this last year when HG Kaitlin was voted out early yet continued to be one of the most talked about HGs for the remainder of the series, largely thanks to her involvement in the racial-slur-scandal and somewhat thanks to her social media presence.

Number of times contestant Twitter accounts have been mentioned; 6 – 12 July.

Something new: Users by Timezone

Something we became interested in during last year’s BB broadcast was the difference in tweeters from one side of the US coast to the other, this year also considering the top 10 timezones joining the conversation with the generic hashtags:

Top 10 timezones using the generic BB hashtags on Twitter; 6 – 12 July.

Eastern time tweet by far the most of any of the timezones which fits with the documented distribution of the US population (47% live in Eastern Time). However, people in Mountain Time are tweeting more about Big Brother than expected with their distribution being only 5.4%, but publishing 9.45% of the total tweets regarding Big Brother. We’ve established that the Quito (Ecuador) timezone aligns with Chicago / Central time, so those users who say they’re in Quito, likely aren’t.

Until next time…

A Quick Look Back

So, as we left things, we put out a few predictions for the week beginning January 20, as follows:

So, some hits and misses here.. The most concerning of these are Pretty Little Liars & Ravenswood, so let’s have a closer look at those, starting with Ravenswood. Essentially, I think this was an over-prediction influenced by the second half premiere, which rated at 181,200 tweets. We have now added an adjustment for these type of shows, that will highlight second half premieres in our predictor, with the option to manually exclude them where they rate significantly different. More concerning was “Pretty Little Liars”, which wasn’t a second half premiere but a regular show. Here, I think we had the reverse problem. Because there weren’t enough shows in the past two months (only 1 after the second half premiere), our prediction algorithm defaulted to include episodes from the previous series, where 300-400k tweets was the norm. Combining this with the 488,000 tweets from last week, this seemed a reasonable estimate, but the show actually nearly doubled that performance — it will be interesting to see how Pretty Little Liars fairs over the coming weeks.

The other errors of over 10% were both reality shows, and here I’m tempted to put the predictions on hold until we’ve established a measure of season context. Just as Big Brother follows a weekly cycle of Daily Shows, Nomination Shows and Eviction Shows, shows like American Idol, The Voice etc follow a somewhat standard season format between auditions, performances, eliminations and so forth. Modelling this is on my to-do list, but it’s after the establishment of the Twitter Excitement Index and indeed the modelling of regular season shows with their premieres, finales and other formats. Time and other priorities didn’t allow us to get predictions out for this week, and I won’t do so retrospectively, but we’ll get some out either late this week or early next.

Twitter Excitement Index

One thing we did finish last week was the establishment of our Twitter Excitement Index, which is a measure of the volatility of conversation on Twitter based on the principles of Brian Burke’s Excitement Index at Advanced NFL Stats. Over a few weeks Katie Prowd and I went through a few variations on this approach, looking at different measures to see which best captured the patterns in Twitter conversation, and I also owe thanks to Patrik Wikstrom for his help in tweaking our statistical approach. Essentially though, the theory here is that if you have two shows, they may both average 100 tweets per minute, but have very different engagement levels, as seen in this dummy data:

In the first graph, the activity is ‘spiky’, that is, every other minute people are prompted to tweet, while in the second there is an underlying level of 100 tweets, with minimal variation around that average, suggesting a constant stream of conversation but no particular moments which provoke users to tweet. These should be at opposite ends of the spectrum, and with our Twitter Excitement Index, the top graph would see a TEI of 9.9, while the second gets a TEI of 0.5. The scale here has been calculated to vary between 0 and 10 for presentation purposes, although in practice from our test data it would seem very unusual for a show to achieve a TEI of over 5.

The Australian Premieres

We are currently working on adjusting our metrics to work with Australian television, and this weeks premieres of The Block and My Kitchen Rules gave a good first test for this approach. But first, let’s cover the basics, and look at how these shows performed on Twitter:

As you can see here, My Kitchen Rules clearly won the night, both in terms of total tweets and audience peaks, from The Block in second place. The third line here represents The Biggest Loser, which wasn’t premiering but did air an episode in competition with the other two reality shows. The twitter audience for The Biggest Loser has fallen off a bit since its premiere, but this performance must be considered low by any standards. A similar picture was seen in the TV ratings, with My Kitchen Rules reaching 2.4m viewers, The Block 1.1m, and The Biggest Loser just 560k. However, it’s interesting to compare these to other recent reality ‘events’:

What is evident here is that Big Brother still owns the crown for Australian reality television, at least on Twitter, with both the Premiere and Finale easily outperforming the launch episode of My Kitchen Rules. For Channel 9, the performance of The Block suggests that the reality show success they achieved with Big Brother is not easily transferable to other shows, and for Channel 10 both the performance of Masterchef and Biggest Loser must be a cause of some concern… A similar pattern can be seen in the unique audience for Monday’s shows (n.b. the percentages add up to over 100% because of viewers who tweeted about multiple shows):

Finally, the launch of these shows also gave us an opportunity to test some of our new metrics, and after a slight hiccup caused by time zone confusion (the TEI saw the drop-off to QLD discussion as a good thing for the show, viewing NSW levels as large spikes), we ended up giving a slight win to MKR, although both ranked below the season launch of The Biggest Loser. The TEI does not take into account tweet volume, only measuring the volatility of conversation among the audience that *did* watch the show, and so should generally be read in combination with the audience size to properly understand the conversation around a show.

Mumbrella also covered some of this, and you can view their article here.

And so wraps another week in Telemetrics.. until next time!

Evaluating our predictions

Over on my own blog, we have been predicting tweet numbers for a selection of upcoming TV shows. It became immediately obvious is that our prediction methodology coped badly with Second half premieres. We had deliberately excluded season premieres and finales from our predictive data set, but it now appears clear that second half premieres (i.e. after a show goes on a winter hiatus, but within the same season) also receive a significant boost in tweets. With our new data (scraped from EpGuides & Wikipedia), we are now able to systematically identify these episodes, and plan to evaluate whether we can make a simple adjustment to account for the boost such shows receive. Overall, our average error for last week was 33.19%, however if we exclude the second half premieres that falls to 9.21%, which is within the realms of what we were expecting, and easily out-performs the simple average of the last 10 or 4 shows. Taking a simple average of the last 10 episodes would have recorded an overall error of 35%, falling to 25.5% once the second half premieres are excluded, a much larger figure than our 9.21%, although of course the sample sizes here are still small.

Filling out the last two months: Replacement Value & Tweets Above Replacement (TAR)

One significant issue we had been experiencing using our preferred “average weighted tweets over past two months” (of episodes) metric was that of small sample size. While I am reasonably confident that we can demonstrate the last 8 shows are a better estimator than the weighted average of all shows in our database, for shows we are not manually tracking there can be many holes in this data. A show which barely scratches the Nielsen top 5, and isn’t in our tracking system, may be missing data for 75% of the last two months shows, as seen here:

If we, as we have previously, simply don’t count shows for which we have don’t have tweet figures, we are heavily biasing the estimates. As you can see in our prediction data above, we predicted 10,245 tweets last week for “The Carrie Diaries”, which turned out to be a 12% over-estimate. A better solution here is to adapt a concept from sabermetrics called “replacement value” . Essentially, in baseball and other sports, replacement level is defined as the production you could expect from signing a “free agent”, that is – the next best player who is not under contract to any major league team. This allows, for example, to predict the impact of an injury, trade or contract expiry on a teams production, but also to measure other players against this level; for example through Wins Above Replacement. I plan to return to the latter concept, or in this case Tweets Above Replacement (TAR) in the future, as a means of measuring how well a show does in comparison to throwing a “replacement level” show on in that timeslot.

For our current purposes, and the Carrie Diaries example, we are more concerned with predicting how many tweets a particular episode may have got, despite the final statistics not being (freely) available. However, we do know what the bottom ranked show on Friday is, which – for example – was 3,700 tweets on 15th November, 1,600 tweets on 22nd November, and 6,600 tweets on 6 December. However, here too, we are slightly thrown by Nielsens method of reporting, as the Tonight Show was ranked fourth with 1,600 tweets, but ahead of BBC America’s “An Adventure in Space and Time” which ranked 5th with 9,200, based on number of impressions.

So, for now, let’s call the replacement value for Friday nights 5,000 (we’re currently calculating replacement values for each Day/Month combination). If we plug 5,000 in for those missing episodes, instead of ignoring them, our weighted tweets formula would have predicted 8552 tweets, an error of just 5.2% compared to the previous 12.2%.

We’re probably a few days away from properly integrating this into the prediction algorithm, but we do hold high hopes for this improvement.

Sometimes you can be too precise, or the problems of small samples

Another, perhaps short lived, tweak over the past 10 days was experimenting with weekly indexes rather than monthly. The idea here was that such a system would better account for weeks such as ‘sweeps’, when networks tend to put on their strongest programming, as well as for the effect of major events such as the Superbowl, Oscars, Golden Globes and so on. Particularly with the wresting shows, such a method would also account for the cycle around events such as Wrestlemania and Summerslam, which anecdotally appear to signal increased tweeting. For much of our older data, where we have 50-100 weekly shows weekly indexes appear to offer small incremental advantages in forward-predictions over the monthly indexes utilised in the above predictions.

However, when predicting for the future (where weekly indexes can be more volatile), and for the last few months of data when we often only have 25-50 shows in a week, the volatility of this index negated such advantages and actually showed a decrease in prediction accuracy. In particular, weeks 52 and 53 of 2013 and 1 of 2014 (i.e. over the Christmas break) saw such low weekly indexes that any show that had mild success during those weeks was suddenly forecast to have season-highs in their next episode, when in actuality the index was primarily impacted by a few extremely low days over the actual Christmas holiday (particularly when we exclude sporting events – i.e. the NBA and College Football bowls which dominate American broadcasting over the Christmas period).

It may be that eventually we can recover those incremental gains, but for current purposes we plan to return to the monthly indexes. This also means we can more easily predict shows for the current month, where the index has already been somewhat established, as opposed to ‘predicting’ the monthly index itself based on historical data and trends.

Some predictions for the coming week

So, with all that said, let’s head back to the spreadsheets and give our top 10 predictions for the coming week:

Twitter Excitement Index (TEI)

Finally, in the tradition of all good TV shows, we’ll leave you with a teaser for the coming week. Based on work by Brian Burke at Advanced NFL Stats, we have developed a measure we’re calling the “Twitter Excitement Index”, which essentially measures the peaks and troughs in Twitter conversation to establish a measure of excitement. That is, a show may be averaging 3000 tweets / minute, but have a time series graph which hovers slightly above and below the average over the course of an episode – in this case, it has attracted a large twitter audience but that audience doesn’t seem to be particularly provoked by the content of the show to tweet. By contrast, a show averaging 300 tweets / minute may be spiking continuously throughout the episode, showing that people were reacting to the events as they happened (as we have seen, for example, with Big Brother 15 in our previous work). By again stripping out those factors related to audience size, we are producing a metric which analyses the reaction of Twitter users to the content of the show. But, more on this next week..

Telemetrics, which I define as the application of Sabermetric principles to television ratings and social media engagement, has thus been the focus of my work over the past couple of months. We are beginning to make substantial progress in both the historical analysis, and future prediction, of Twitter engagement around television. At this stage, I should also mention that our progress here would not be possible without a team of research assistants & embedded student projects, which includes Katie Prowd, Portia Vann and Kate Guy, who have (respectively) put together the data and assisted in testing the metrics, taken on the work of developing dashboards to display our results, and investigated and designed infographics to better communicate results to a wide audience.

Our work to date shows that we are fairly accurately able to match the Nielsen SocialGuide capture technology, at least for shows which do not exceed the 1% Twitter API restrictions (as I have discussed previously in relation to Scandal). In terms of correlations between tweets and television ratings, we have observed a substantial variation across genre, format and country. In particular, early results suggest a significant difference between reality shows and more standard television fair such as sitcoms and drams. Additionally, ‘specials’ (which includes events such as the 2012 US Election debates and award shows, as well as potentially premieres and finales of series) do not appear comparable with standard episodes of series. Finally, for now, we have also noted substantial differences between similar formats in Australia as compared to the United States, which is particularly significant since much of the literature, and examples of best practice, come from the work of researchers and organisations such as Nielsen (and their SocialGuide subsidiary) which have focused on the US market. All of this means that drawing a fixed correlation between traditional television ratings and Twitter use does not seem a sensible approach.

Big Brother 15: CCI vs Nielsen Data

The first thing we had to do was verify our collection methodology. Luckily, we have long been collecting tweets around television, as part of work discussed previously here on Big Brother 15, which expanded to comparisons between Big Brother in Australia and the United States, and subsequently reality TV shows as a whole. With the start of the 2013 TV season, we added a range of new terms including popular sitcoms, dramas and sci-fi shows in order to broaden the number of exemplars available to us. However, returning to the Big Brother data made sense for verifying our methods, as both we and Nielsen SocialGuide recorded, and Nielsen had published on their website, the number of tweets, and unique users, for a large portion of the season, as visualised below:

From our perspective, that was pretty good. Tweets matched almost exactly, and the only major difference, on 23 August (22 August in the United States), was easily attributable, being the night in which the Head of Household competition continued after the show, thus resulting in us recording an oversampling of tweets (compared to Nielsen) which were attributable to the live feed, rather than the network broadcast. That said though, we were very happy with these numbers, and meant we could be confident our methodology was producing accurate results. Thus, excluding known data outages, we were happy to move forward with the data we had collected. It is worth noting here that while we were slightly different on unique users, it was by a relatively consistent amount. We’re still not sure of the cause of this; although it seems unlikely that we were counting just enough tweets from a group of repeat users to have slightly different terms but the same overall volumes, and so with Nielsen’s methodology essentially a black-box, the mystery will remain.

The weighted tweet index

The Weighted Tweet Index (WTI) was the first metric we created as part of the Telemetrics project. Essentially, the goal was to break raw volume numbers down into it’s constituent parts. Perhaps the best way to illustrate this is by way of an example.The 1750th ranked show (at the time) by pure volume, excluding specials and sporting events was an airing of the film “Space Jam” on Cartoon Network from Friday 6 July 2012. The data (in this case from Nielsen SocialGuide) records a total of 25,033 tweets attributable to the show, making it the 3rd ranked broadcast of the day. But what if that had aired on an average day, in an average month, and on an average network (taken in this case to be an average broadcast network – e.g. ABC, CBS, FOX, NBC)?

According to our current metrics, it turns out that a Cartoon Network show can expect to see about 16% of the twitter activity of an average broadcast network. Similarly, Friday’s can expect to receive around 52% of the Twitter activity of an average day (nothing new here for TV Execs, Friday is historically a dumping ground for non-performing shows), and July – being in the offseason for TV – receives 65% of an average month. Note that there’s a lot of refinement possible here: a kids movie during the school holidays (I assume) should probably be expected to do better than average, and we currently don’t account for the time of day nor what was being scheduled against it. But, for now, let’s stick with our numbers, so:

Weighted Tweets = (Old Tweets) / (Network Factor * Month Factor * Day Factor)

Therefore, new tweets = 25,033 / (0.16 * 0.52 * 0.65) = 25,033 / 0.0541 = 462,717 tweets if shown on an average broadcast network, on an average day, in an average month. With the greater degree of precision in the model (i.e. not rounding to 2 decimal places), our actual figure here is 466,805. As it turns out, once ranked by weighted tweets, Space Jam thus moves from being the 1,750th ranked show in this period to the 317th.

Predicting the Future

Much of the utility of this approach comes not necessarily from analyzing the past, though we’ll certainly do a lot of that over the coming weeks, but the ability to predict the future. Because our ‘Weighted Tweets’ figure has been stripped of much of it’s context, we can then take this figure (across all episodes that we have data for within a particular series), and apply to it the factors that will apply when it next airs. Here are two examples:

Here you can see one interface to our model. Once a show is selected at the top, historic data is pulled from the data store, and a number of metrics are calculated. Which of these metrics is the best estimator varies depending on the volume, source, and age of the data we hold. In this case, we have a number of recent episodes and so the weighted tweets for the last month were selected. We entered the day & month the show was airing, as well as the network, and (for now) set the growth factor as 1. Growth Factor is a variable that we are still experimenting with, although currently it is largely contained within the Month factor for current/future dates. Ultimately, we predicted 157,511 tweets, and when the Nielsen data became available, we saw the results:

Not too shabby. 7,000 on a total of 150,000 is a 4.80% error; for the first version of the model we’ll certainly take that. Looking at the last 10 shows, a pure average of the raw tweet totals over that time would have been 99,591 tweets, and for the last four shows 113,100 tweets, so our prediction certainly appears to be outperforming simple averages.

Both Teen Wolf & The Bachelor were essentially premieres (Teen Wolf being a second half premiere), and Wolf Watch appears to be a companion show to Teen Wolf, and so we are currently unable to predict any of those three, so let’s move on to WWE Monday Night Raw.

Here, again, we have a lot of data, with this being a weekly show, and so the monthly average was a sensible choice. Again we added in the new variables (Monday, Jan 2014 on USA), left the growth factor alone, and predicted 215,507 tweets, and the results were again quite pleasing:

We were off by 10,594 tweets, a 4.69% error, although this time the other way. It’s interesting to note that simply taking the average of the previous 10 RAW shows would have predicted 153,398 tweets, and an average of the last four shows would have predicted 165,275 tweets, so again we do seem to be on the right track.

Note here that we’re not necessarily predicting what it will get, but what it should expect to get. In part that’s because there is still a wide range of factors we don’t account for, but even as we add more refinements to the model over the coming months, there’s still one we can never really account for: content. Just as we saw in the Big Brother work I’ve published here previously (and more formal journal versions of that are still in the works), there are some things you cannot account for, such as the racism scandal that engulfed Big Brother 15 and saw a rapid increase in tweet volume.

However, in some ways knowing what a show should expect to get can be more useful, both in providing a barometer of success for networks, producers and social media strategists, and in providing industry and researchers with a list of shows which either exceed or fail to reach these levels, thus allowing an analysis of what may have contributed to the success or failure of a particular episode or series on social media.

More on our progress here in the coming weeks, but for now we’re pushing ahead with refinements to the above model, and new metrics galore!