The Guardian Is Being Swamped With 'Dark Traffic' And No One Knows Where It's Coming From
Normally, web publishers can "see" where their readers come from. If you search for a story in Google, publishers can see that you landed on their site via a Google link, for instance. Generally, the biggest sources of incoming reader traffic are Facebook and Google.But recently, a massive chunk of The Guardian's readers have become invisible. On some stories, dark traffic is a more important source of readers than either Google or Facebook.
first identified "dark social" traffic back in 2012 to describe traffic coming messaging apps that had been stripped of referrer data because messaging and email use the secure "HTTPS" system rather than the open "HTTP" system used by web pages. Dark traffic is vexing because web publishers like to be able to offer their advertisers solid data on the quality of their readers. And editors like to know which types of readers are enthusiastic about their stories. Dark traffic threatens to undo all that.
The issue is a big deal for The Guardian because of its massive global footprint in the digital news business. Ten years ago, The Guardian was an also-ran as Britain's 11th most-read newspaper. More people read regional papers like Scotland's Daily Record and London's Evening Standard than the Guardian, which had only 383,000 readers. The country's news agenda was driven largely by The Times and The Sun, which circulated 3.4 million papers.
Those days are gone.The Guardian's massive website and mobile app now dominate news online, especially for left-leaning folks. It has 167 million visits a month from readers, according to Similar Web. Sources tell Business Insider it gets more than 100 million unique readers per month. The only UK media brands that stand in the Guardian's way right now are the BBC (which has the unfair advantage of receiving tax-funded special treatment from the government) and the Mail Online (which airs content The Guardian would never stoop to).
"Dark traffic is a thorny topic for us," Cordrey told Business Insider. "Overall, around 10 to 15% of mobile traffic to articles does not have a referrer. But of that, a significant proportion 'shadows' a known referrer - so we are currently looking at ways to make sensible attributions of dark traffic."
Here are some of The Guardian's charts illustrating the scale of invisible web readers. Cordrey has used the term "typed" to describe dark traffic visits because previously the company thought those readers were coming directly to the site after literally typing the URL "theguardian.com" into their web browsers. But the number of "typed" visits has become so massive that they are obviously coming from unknown links elsewhere on the web. People don't type in the URLs of individual stories to get to them.Note that among mobile readers, "typed" visits are the biggest group:
This next chart shows a surge in traffic after one story became a hit on Reddit. When you look at the mobile-only traffic, it turns out that most of the surge is dark traffic. Cordrey suspects that the traffic is coming from a mobile app in which the Reddit link became popular:
Drilling down a bit further, it looks as if the dark traffic is coming from Android users who click a link inside an app which opens the story inside a mobile Chrome browser:
Cordrey next looked at whether the dark traffic was coming from mobile Chrome (Android) or mobile Safari (iPhone) browsers. Android Chrome appears to be to blame:
So The Guardian is flying blind when it comes to millions of Android users.
There are a couple of alternative theories about the origin of dark social traffic: One, as mentioned previously, is the increasing use of web sites that use the "HTTPS" secure standard. This standard renders traffic anonymous in order to increase users' privacy. Since Edward Snowden's revelations about NSA surveillance, more sites have adopted HTTPS standards. Hacker News and Wikipedia, for instance, use HTTPS and all traffic coming from them now shows up as if it were "direct" traffic.The other is search. The search engine Duck Duck Go, which treats all users anonymously, has become more popular in the post-Snowden era. Its traffic arrives without referral data, and some are calling it "dark search."
"We recognise that we are likely to see more dark traffic, especially as people increasingly consume content through apps on mobile phones. On iOS this isn't such a problem, as the browser executes inside the app and the app can indicate this though setting the user agent string. But Android devices do not provide quite the same information," Cordrey says. "We have already started comparing notes with other sites to understand what they are doing to unravel the mysteries of dark traffic."There is of course one "winner" in all of this: If Google is retaining information on Android's dark traffic, then it would benefit Google if the search giant wanted to offer that information as a targeting mechanism to advertisers, especially if publishers were blind to the data. But as Google did not return our call, we cannot say that for sure.
- TTD approves Rs 2,938-crore budget for 2021-22
- Wholesale growth expected to continue for tractors, passenger vehicles and two-wheelers, says equity research firm Emkay
- Centre finalises One District One Focus Product to promote in cluster approach
- A Mumbai-based EV startup launched the cheapest electric car with a 200 km range
- HP's worldwide shipments grow 5.6% in fourth quarter of 2020, says IDC