Almost since headline testing went live, customers have been asking for this, so we're quite happy to oblige. As much as anything, we see it as one more means to simplify complex decisions and enhance the role of editors as arbiters of engaging quality content. If a picture is worth a thousand words, then having the right picture is critical to telling any story to its maximum effect.

So what have we seen so far?

For one, editors don't seem to think of it as a testing feature, but rather a creative one. And that's great, because it's how we intended it. In some cases, we've seen instances of repetitive and sequential testing of headlines (moving toward optimization), followed by a similar process with images to find a match for the winner. Sometimes, it looks like an editor's brain (and judgement) playing itself out with the support of real-time data. (Except it’s not as creepy as that sounds).

As with headline testing, USA TODAY Sports has been one of our more creative customers in using the new feature. So we asked Tim Gardner, Assistant Managing Editor at USA TODAY Sports Media Group, what he thought about it:

"This tool allows us to perfect our presentation as quickly and effectively as possible. While headlines are one way to guide a reader into our coverage, Instant Image Testing takes that a step further. The elation of victory and the agony of defeat are easier to convey to our audience in an image, and this feature allows us to ensure we have the best combination of headline/image as possible in real time. It allows us to show off the best of the in-depth content that USA TODAY Sports is known for."

Thank you, Tim. And thanks to you and your team for allowing us to share these results:

By the way, Image and Headline Testing work in tandem. If you're using the VR platform, simply identify a story, enter the alternative headline or image of your choice and hit a test button. The platform does the rest. It runs a live A/B Test with your audience, and when it has enough data, it swaps full-time to the more effective option. Boom! Done. No CMS integration required.

If you like that idea, and want to see more, just let us know and we’ll gladly arrange it.

The post Introducing Instant Image Testing. Picking the Winners Just Got Easier appeared first on Visual Revenue.

It’s been a little more than two years since we launched our predictive analytics platform for editors, and it has been an amazing 24 months supporting their efforts to program content on homepages, section fronts and in social channels.

At about that same time, Andy Carvin of NPR was covering the Arab Spring and he provided a model for the future of newsgathering by tweeting at potential sources and confirming reports from over 9,000 kilometers away. Many, if not most journalists have now found their way onto Twitter and, like Andy, are using the conversation and its data to great success.

So what about this predictive data we use today? Is it also a model for the future and if so, where is it taking us? These past two years have seen it adopted in in hundreds of newsrooms globally, and we can use this experience to consider the next 24 months, for what may be an even bigger story of newsroom transformation. At Visual Revenue, we think it might look like this…

Among the many changes and trends we’ve seen in newsrooms around the world, what we cannot and must not overlook is human nature. It’s natural for every one of us to want to predict the best outcome — for any situation – especially an editor, who is literally plugged in to a position of regularly trying to find the best stories to promote. They make predictions every day, multiple times per day.

But the editor continues to live in this stormy place where predictions must happen faster and faster, with even more variables and more channels of data – some reliable and some not – pressing them to decide, to judge, and to predict. Do it instantly, or it’s too late! The luxury of time would be nice, but this job rewards the swift. There is no reward in suggesting the best front page for yesterday’s paper.

Thus, the emergence of data. What we’ve seen in two years is that when we arm these editors with the right data, they grow more confident. Instead of making 50 daily updates to their homepage, they’ll do 75. They take a more active hand in sculpting not just content, but the behavior of their audience, and they courageously resurface content that otherwise may have been deemed “old.” Editors with data enthusiastically do a job that comes naturally to them: exercising judgment of what their audience truly wants and what content they will truly adore.

When we arm knowledgeable human editors with more data, they don’t race to the lowest common denominator, but rather toward richer, better experiences. Good editors now embrace and devour the data to find out, almost like Goldilocks, when their promotion is too soft or when it’s too aggressive. Then it is their intuition, skills and instinct that get things to be just right. When we arm editors with data – particularly with more of the right social data – you will see them exploit social media and social sharing, not to the benefit of Twitter or the social channel, but for the purpose of driving people back to their property. They will discover, select and craft content that drives people not just to re-tweet it, because there is no direct value in that. They will use social channels as a discovery engine for an audience that will one day fall in love with their content.

The newsroom of tomorrow will be decidedly and enthusiastically even more data-driven. To the editors that lead this newsroom, they will not equate this embrace to surrender; it will not lead to lowest common denominator judgment. Data will drive decisions and accelerate accomplishment; it will help differentiate brands and journalists, and provide jet-engine propulsion to the exercise of good judgment on the part of editors.

This will not be because it can or it should, but because it must. Data as a complement to human editorial judgment is what will enable editorial teams to compete in the audience business. Advertisers have long since and wholeheartedly bought into data, and they are using it so aggressively through exchanges and programmatic buying that they are squeezing every last ounce from publishers. This imbalance must be addressed, and it can only happen with a combination of data-driven thinking and distinctive editorial voices.

Tomorrow’s newsrooms will be filled with clear measures of editorial success and, sometimes, failure, plus plain unambiguous directions on the steps to address them. What will not be so clear is the voice that defines each property’s response. That will lie in the hands of the editor –exactly where it should be.

The post The Newsroom of Tomorrow:
What Happens When Editors Have Data appeared first on Visual Revenue.

Advertisers have long since and wholeheartedly bought into data, and they are using it so aggressively through exchanges and programmatic buying that they are squeezing every last ounce from publishers.

This imbalance must be addressed, and it can only happen with a combination of data-driven thinking and distinctive editorial voices.

Dennis!

The post Data as a complement to human editorial judgment appeared first on Visual Revenue.

Today we announce the merger of our company with Outbrain, Inc.

Why, you might ask?

By merging the two companies, we combine the leader in online editorial decision support and optimization with the largest player in paid content recommendations. This will offer publishers an unmatched end-to-end solution for optimizing content on all of their pages across any channel, screen size and device, with both editorially-controlled and automated solutions.

We very much look forward to joining the Outbrain family, as together, we believe we can provide superior solutions that help media companies to best engage, delight and satisfy their audiences with great content. We believe this also will be one of their best long-term strategies to generate revenue. 

I am extremely proud of what we've been able to do on our own since Charlie, Alex and I launched the company July 5th 2010. Go no further than our Product or Customer page to get a sense of just how much we have achieved in less than 3 years. However, it would not have been possible without our awesome team and their families. Thanks Philipp, Varun, Gary, Vibeke, Jeff, Cody, Anita, Rich, Alana, Evan, Daniel, Ji, Josh, Rob, Paul, Matt, Bashar, Thomas, Joshua, Jeroen, Ganesh and Anthony – I LOVE YOU!

It would not be fair to post this without thanking our customers and especially Steve Lynas, Scott Cohen and everybody at the New York Daily News, who basically incubated us inside their newsroom. This also includes Jim Kennedy and our good friends at the Associated Press, in whose newsroom we sit today. Further to this, Betsy Morgan certainly helped us over the finish line – and I could continue. Thanks again to everybody who has played a role in our success. 

That said, we will be vigilant, like a dog with a bone, in assuring that our decision support platform for editors evolves from what it is today, to something truly great over the next few years. Our new scale simply allows us to do things we would otherwise have had to postpone. Stay tuned! – OR join us, as we expand the team!

Here's a set of stories from today's press you might find interesting, starting with a blog post from Yaron Galai, OutBrain's CEO, plus what the Wall Street Journal had to say.

As always, please do let me know your feedback. Should you have any questions about this announcement feel free to contact our account team — and you are always more than welcome email me directly.

Cheers :-)
@dennismortensen 

Dennis R. Mortensen 
CEO and Founder, Visual Revenue, Inc.

The post Outbrain acquires Visual Revenue appeared first on Visual Revenue.

There’s been much debate lately about many of the changing ways of the online publishing business. I like heated debate because it keeps us on our toes, and I like change because I believe change is the heart of our human existence. So let there be more of it. It keeps things interesting.

But the pace of change, while dizzying to some, creates for me something entirely different, yet just as bewildering. And that’s the degree and quantity of false assumptions (and therefore false conclusions) about what’s happening to media. Not at the fringes, where we will always see extremes. And not in the last five minutes, because there is no surer bet than that many of today’s darlings will turn out to be tomorrow’s dogs. But at the center of what media really is – and that’s content.

A most recent high-profile-hubbub concerns our new aggregation economy, and the merits of sitting on either side of the aggregation fence: creator or collector. Surely, that which is created originally and in-depth should provide and return greater value for a specialized audience than anything that’s skimmed off the top and sold to the masses – en masse, no less. These are different approaches; they’re different models with different returns for the business.

But they’re both centered upon the content delivered, the audience assembled, and the relationship between the two. Targeted or not, audiences have an expectation that must (or at least should) be met. That’s the critical point.

In a similar conversation we’ve witnessed for about two years now, the debate is not what’s being delivered to an audience, but on how that audience is getting there. Do they search? Are they social? Do they seek out content from the media properties of their choosing? Are homepages dead? Is every page a homepage?

All are nice questions. It’s among the most exciting and fastest changing topics as any, with new players entering and leaving the game at a sprinter’s pace.

What we’ve found, is that while the deck chairs are being rearranged a bit, the value and strength of a well defined media brand has never been stronger. More value (as measured in views and loyal audience) is being derived, particularly by news properties, from their home pages than from the quick hits and sexiness of search and social traffic that arrives through the side door. Thus, the real value is created by what you do with audience upon their arrival.

It’s a complex crunch of data that we use to prove that thesis. Important as it is, though, it should be viewed as a supporting point to a bigger truth that’s being missed. It’s being overlooked here, and also by the property that optimizes for advertising instead of content – and finds itself with a super-efficient page that no one is willing to look at. Or the algorithm that creates a great set of related links – that have no soul.

Media properties must have a soul. Don’t lose sight of that.

As the number of web properties expands toward infinity, this truth becomes ever more important. There will be increasingly endless avenues for people to find yours as well as a competitor’s, so every channel is important for bringing audience to your property. But the importance of every channel pales in comparison to audience experience: what happens when they arrive.

Are your visitors thrilled, delighted, satisfied, confounded or confused? Will they stay? Will they come back? Do they see a distinctive and consistent voice and is it one that attracts them? And how do you create that?

My company and my colleagues, data-driven as we are, see media as the most interesting and exciting of industries because it is so filled with emotion. It is human storytelling on so many different levels by so many different parties. Media is the industry that creates thrills, delights, surprises and satisfies most when it’s crafted by real people. It can be data-driven, but at its best, it’s more than something aggregated, optimized, algorithmic, or automatic.

It’s human. And it’s got soul. And that’s why we love it!

The post Audience & Algorithms:
To win, media properties must have a soul. appeared first on Visual Revenue.

We have most often espoused the importance of the homepage and other section front pages as vital drivers of audience and views for news properties. They remain, for many, the leading traffic channel, as our data has continued to show. Nevertheless, we know how important the social channel is to all publishers, particularly in exposing content to new audiences and taking advantage of the greater dynamics of the media world. It is not something to be ignored.

To date, though, what seemingly has been ignored is measuring the success or failure of the social channel for publishers. Most, if not all tools, have done little more than count clicks and tweets.

And to that, we say, "Introducing the Social Editorial Suite."

This expansion of our editorial support platform recognizes social as one of several vital and complementary channels for the publisher. And these new features give the editor — for the first time — something that does more than simply count clicks and tweets. We want to give them two things:

• a very clear look at how successful they are in the social channel, and
• some concise recommendations on what to do next in social.

We’ve built this to complement our existing platform with capabilities for social in the same way that we simplify editorial decision-making and drive success on homepages, section front and article pages. It’s our fervent belief that by capturing and consolidating their data in this way, we’ll make it easier to manage content.

And with that, editors can get on with what they do better than any algorithm ever will: serve up the content that their audience loves and craves to consume.

To our customers, we hope you like it. To everyone else, we’ll be glad to show you. Just drop us a line.

The post Announcing The Social Editorial Suite appeared first on Visual Revenue.

A Sporty Example

With Super Bowl XLVII (and its power outage) still fresh in our minds, one of our clients, USA Today Sports, used our platform to conduct the following headline test:

Headline A: "What Harbaugh regrets about Super Bowl" (3.06% CTR)
Headline B: "John Harbaugh explains Super Bowl tirade" (4.93% CTR)

Headline A (the original headline) got 3.06% CTR and headline B got 4.93% CTR, which are both strong CTRs. After only seven minutes of testing the two headlines, headline B had been declared winner with 99.93% certainty (explained later). Subsequently, the winning headline was served to 100% of the audience for over one hour. Finally, the editor even made the change permanent in their CMS. Note that by changing a few words only, a 61% lift had been achieved, which eventually resulted in tens of thousands more views for that article!

Four Challenges for Instant Headline Testing

Instant Headline Testing is essentially A/B testing for story headlines. However, there are four challenges when it comes to Instant Headline Testing that we need to take into account.

1. Headlines may be on the front page for only a couple of hours, so a headline test cannot take too long.
2. The number of readers varies greatly per front page.
3. The CTR of a headline depends on where it is positioned on the front page. For example, a headline positioned at the hero spot has a much higher CTR than one positioned at bottom.
4. Front pages are dynamic, so headlines can change position.

Our improved implementation overcomes these four challenges by using a Bayesian approach. Before I explain that, I’ll first discuss the frequentist approach to Instant Headline Testing.

Frequentist Approach to Instant Headline Testing

To conclude which headline is better than the other, we cannot just look at the highest CTR. We need to apply statistics in order make sure that the difference in CTR did not happen by chance. It may be the case that we cannot declare a headline as winner at all.

We apply statistics to the data that we have collected during the headline test. This data includes front page impressions (i.e., views) and clicks. The more data the better, right? Well, not quite. It’s important to realize is that the longer a headline test is running, the longer we are serving one headline of possibly lesser quality to 50% of the readers. This means that the corresponding article may lose out on value. In decision theory, the difference between the actual and potential article impressions is known as regret.

So, on the one hand, we want to collect as much data as possible in order to make a reliable conclusion, while on the other hand, we want to maximize article impressions, i.e., minimize regret. This raises two questions: (1) When can we stop a headline test? and (2) How do we know that one headline is better than the other?

Within statistics, the frequentist approach and the Bayesian approach are two well-known approaches when it comes to A/B testing. The frequentist approach provides a statistical test whether the CTRs of the two headlines are significantly different. That would answer our second question.

The frequentist approach doesn’t provide an easy answer to the first question because the statistical test assumes that the number of views are fixed before we start a headline test. Furthermore, we cannot run a headline test until we see a significant difference between CTRs as this would falsely increase the probability of obtaining a significant result, as Evan Mill explains on his blog. We would have to estimate how many views we would need in order to obtain a significant difference.

And this is where the four challenges come into play. Due to challenge 1, the headline test cannot take too long, say 20 minutes at most, which limits the number of views we can get. Because of challenge 2, the views per minute may be anything between 10 and 10,000, and we do need to have a tool that’s usable by all our front page editors. Challenge 3 determines the CTRs of the headlines as well. When the CTRs are closer together, we need more views in order to obtain a statistically significant difference. Including these three challenges when estimating the desired number of views is not straightforward. On top of that, when a headline changes position during a test (which is the fourth challenge), our estimate becomes completely invalid!

Let’s have a look at a Bayesian approach to Instant Headline Testing, which is one that I find much more straightforward and elegant.

Bayesian Approach to Instant Headline Testing

Whereas the frequentist approach assumes that the “true” CTRs remain the same, all that the Bayesian approach cares about is the data we have actually observed. So, we don’t need to worry about estimating the required number of views beforehand. Moreover, the Bayesian approach doesn’t mind that headlines change position while testing, so that overcomes challenge 4.

Below I’ll first explain how the Bayesian approach determines when to stop a headline test (which was our first question). The second question, determining whether one headline is better than the other is discussed in the next section.

During the headline test we keep track of number of views and number of clicks on headline A and B. Important here is the absolute difference between the number of clicks both headlines. When the absolute difference crosses the so-called Anscombe boundary, we stop the headline test.

The Anscombe boundary is defined in terms of (1) the number of views so far, (denoted by ) and (2) the number of views we will be serving the winning headline (denoted by ). Stated more formally, we stop when the following inequality is true:

where is the absolute difference between the number of clicks for both headlines. The shape of the Anscombe boundary is shown in the figure below. Here, we assumed that the front page gets 1,000 views per minute (VPM). So, after 20 minutes of testing, the front page has been viewed 20,000 times. The figure also shows the absolute difference for five simulated headline tests (CTR A=5%, CTR B=3%), which are denoted by gray lines.

The figure illustrates that the Anscombe boundary takes the trade-off between the number of views so far, and the number of views after conclusion into account. In a recent blog post, Aaron Goodman performed an interesting comparison between the frequentist, multiple testing, and Bayesian approaches. He demonstrated that the Bayesian approach is best at minimizing regret, or, in other words, maximizing article views.

Again, once the absolute difference passes the Anscombe boundary, we are ready to conclude which headline was better.

Certainty of Conclusion

It would nice to know with how much certainty we can conclude that one headline is better than the other.

The certainty associated with the CTRs of Headlines A and B can be modeled by a beta distribution. The beta distribution can be defined in terms of the number of views and CTR as follows:

The figure below shows how a beta distribution becomes more peaked as the number of views increases (while keeping the CTR at 20%). It illustrates that as we collect more evidence (i.e., views) our uncertainty about the CTR decreases.

The gray lines may give the impression that the peakedness increases linearly with the number of views, but please note that these lines represent the number of views on a log scale. Allow me to clarify this with the following image, which shows that the interval in which 95% of the probability density is located, i.e., the 95% credibility interval as it is called in Bayesian statistics, decreases exponentially with respect to the number of views.

The amount of overlap between A’s beta distributions and B’s beta distribution determines the certainty of our conclusion. We can estimate this certainty by generating a random value (i.e., drawing a random sample) from both beta distributions and note which value one is higher. If we repeat this, say, a million times, we can accurately estimate the probability that B is better than A. This probability is the certainty with which we can declare headline B as the true winner.

Instead of estimating the certainty, it can also be computed in closed form (see this post from John Cook’s blog), but the
stats.beta.pdf function from scipy doesn’t like very peaked beta distributions. One of the few exceptions that R is better than python! ;-)

Below are two figures that show how long it takes to conclude and the amount of certainty associated to that conclusion. In the first figure, the CTRs of both headlines are kept constant (CTR A=5%, CTR B=3%), and the views per minute (VPM) varies from 10 to 10,000. In the second figure, the VPM (=1000) and the CTR of headline A (=5%) are kept constant and the CTR of headline B varies from 0% to 10%.

In case of sufficient certainty, we declare the headline with the highest CTR as the winner of the headline test. After that, the editor may decide to continue to serve the winning headline to 100% of the readers.

The serving aspect is actually taken care of by our platform and doesn’t require any additional integration, but that’s material for a blog post that one of our fine front-end engineers should write!

Conclusions

Visual Revenue’s Instant Headline Testing tool enables editors to A/B test easily different headlines and to quickly see whether the quality has improved. The frequentist approach, which is often used for A/B tests, is unable to overcome the challenges that are associated with A/B testing headlines. The Bayesian approach, however, offers the flexibility that front pages require.

I have explained how the Bayesian approach, using the Anscombe boundary, determines when the stop a headline test. I also discussed how we can compute the certainty associated with concluding that one headline is better than the other.

If you would like to play with beta distributions and compute the associated certainty, Peak Conversion provides a nice graphical calculator.

If I still haven’t convinced you that the Bayesian approach is the way to go, then you may want to have a look at: John A. List, Sally Sadoff, and Mathis Wagner. “So you want to run an experiment, now what? Some Simple Rules of Thumb for Optimal Experimental Design.” NBER Working Paper No. 15701.

Alternatively, the frequentist approach provides methods that allow for setting up “checkpoints” where you may determine whether you want to stop a headline test. In other words, these methods offer correction for multiple testing.

Feel free to contact me if you would like to know more about Bayesian Instant Headline Testing, or if you’re interested in other data science projects at Visual Revenue in general. You can email me at: jeroen.janssens@visualrevenue.com or you can follow me on twitter at: @jeroenhjanssens.

The post TECH: Bayesian Instant Headline Testing appeared first on Visual Revenue.

Last night we thought it was time to review those figures again. Looking at aggregate data for January 2013 for sites with a sum total of some 15 billion views a month we see the following distribution:

It's an interesting but not surprising confirmation of the continued importance of the Homepage and the associated Section Front Pages. The world is evolving for sure, but there's still a very large impact on views that's driven by the traditional front doors. We included Direct Traffic into the study this time and you’ll see people like Alexis Madrigal from The Atlantic speak of this as “Dark Social” – in other words, you could choose to club Social and Direct together as one Social segment, which would then confirm a solid growth in Article Views attributed to Social over the last good year and a half. 

It's just one more snapshot we took from anonymized distribution data across Visual Revenue customer base. I hope you find it useful. 

Cheers :-)
@DennisMortensen

The post The Homepage continues to be the biggest traffic source for News Media appeared first on Visual Revenue.

Managing and monitoring performance for one media destination is hard enough but what if you are a publisher and responsible for multiple properties? Customers of ours like Amedia, Cox Media and NBC Universal have a host of individuals that manage the day-to-day running of the properties whether it be from the frontline or more from a holistic viewpoint. As I mentioned previously, we have the technical capability to now automatically track readers ‘cross network’ which solves one part of the puzzle for publishers managing multiple properties – now we take this to the next level.

Today I’m pleased to announce the launch of our Real-time Publisher Dashboard.

This dashboard provides a thirty thousand foot view for the publisher with an emphasis on less-is-more. Pål Nedregotten, VP Editorial Product at Amedia says that ‘Great insight often only comes from removing the less relevant data’ – and we couldn’t agree with him more. The Publisher  Dashboard displays a simple set of key metrics that provide a snapshot of the overall ‘health’ of each property. There are a number of views but here is the Front Page Performance dashboard  in action:

This allows centralized team to get a really good idea of which properties are performing well right now and which may need a helping hand. Other views include overall content performance, top performing content and social performance.

It is all too often that central teams at large publishers have access to far too much data and not enough insight, having an aggregate real-time view allows them to focus on their day-to-day without getting caught up in the details.

“Christmas came early this year :-)” – yes it did Pål, yes it did!

The post LAUNCH: Publisher Dashboard appeared first on Visual Revenue.

Bienvenue!  Here is the official press release:

Le Monde Signs on to use Visual Revenue Platform in Data Driven Newsroom

France’s Leading News Website Will Use Data to Increase Audience Engagement and Enhance Editorial Judgment

New York, NY — December 11, 2012 —   Visual Revenue, Inc., the leading provider of online publisher tools for real-time homepage editing, continues to expand its portfolio of online publishers with the announcement today that LeMonde, France’s leading news website and one its  most respected media properties, will join its rapidly expanding customer base.

The Visual Revenue Editorial Support Platform delivers real-time recommendations on what content to place where on their web pages and for how long.  It is an intuitive solution, designed specifically with Editors in mind, creating a collaborative environment where data enhances editorial judgment, ensuring content placement decisions can be made faster, more accurately and with greater levels of confidence.

“We chose Visual Revenue to be more connected with our readers and to understand exactly how the audience is engaging with the content on our front page in real-time,” says Jean-Christophe Potocki, Deputy-CEO LeMonde.fr. “Visual Revenue gives us the insight we need to help us take decisions on what content we should be placing in a given position.”

The Visual Revenue Platform will provide LeMonde with a specifically tailored solution based upon their tone and objectives.  This includes real-time recommendations for Homepage content placement, automated headline testing, and front page performance reports.

“We’re quite pleased to see a media property as respected as LeMonde joining our band,” says Dennis Mortensen, Founder and CEO of Visual Revenue, Inc. “This partnership underscores the amazing science behind our content recommendation engine and our understanding of the role that editorial judgment plays in delivering great journalism online. It is they who make the decisions, and they who are still kings of the data driven newsroom.”

About Visual Revenue

Visual Revenue, Inc. provides online editors and media organizations with tools that improve their performance.  Visual Revenue’s Front Page Decision Platform helps editors to better place content, providing real-time recommendations and predictive analytics to more than 250 global online publishers, including Comcast, The Atlantic, Dallas Morning News, NBC Universal, Cox Media Group, Le Monde, The Independent, News International and many more.”  The company was founded by former IndexTools (now Yahoo! Web Analytics) COO Dennis R. Mortensen and is headquartered in New York.  For more information, visit www.visualrevenue.com.

The post Bienvenue Le Monde! appeared first on Visual Revenue.