Digital news organisations that rely on traffic from search engines are taking a hit as AI-driven search tools by OpenAI, Perplexity, and Google fundamentally change how users find information online.
A new study by content licensing platform TollBit has found that news sites and blogs receive 96 per cent less referral traffic from AI search engines than from traditional Google Search.
The findings of the study, reported by Forbes, also suggest that tech companies have ramped up the scraping of websites to collect data for training their AI models.
TollBit said that it analysed over 160 websites pertaining to national and local news, consumer tech, and shopping blogs between October and December last year. AI companies such as OpenAI, Perplexity, and Meta scraped websites two million times in the fourth quarter of 2024, the report said. Each web page was scraped about seven times on average, it added.
Developers use AI bots or web crawlers to go over websites and collect data for AI training purposes. However, website owners and publishers are generally unaware of AI companies accessing their content as the bots are not properly identified or disclosed.
For instance, TollBit said that Perplexity’s AI bots scraped a publisher’s website 500 times but sent only 10,000 referrals back to the site. The AI search startup has previously been accused of using unidentified web crawlers to scrape websites without adhering to the robots.txt standard.
Beyond the lack of transparency around AI scraping, publishers are reportedly bearing server costs that are being driven up by the rising bot traffic. “We are seeing an influx of bots that are hammering these sites every time a user asks a question…The amount of demand for publisher content is non-trivial,” TollBit CEO Toshit Panigrahi was quoted as saying.
Story continues below this ad
“AI does not read like humans do. Humans will click one link, they’ll click the second link and then they’ll move on…AI will read 10 to 20 links to get their answer,” Panigrahi added.
In the past, companies like OpenAI and Perplexity have claimed that their AI-powered search engines will direct more readers to websites and create new income streams for publishers. However, another report by global research firm Gartner had predicted that the rise of AI chatbots and virtual agents would lead to a 25 per cent decline in search traffic to websites by 2026.
More recently, US-based edtech company Chegg filed a lawsuit alleging that Google’s AI-generated summaries of search results had eroded its traffic and hurt its already diminishing revenue. Chegg claimed that its search traffic sank by 49 per cent in January year-over-year, up from an 8 per cent drop in Q2 2024, when Google rolled out AI Overviews.
Meanwhile, news publishers in several countries have filed copyright infringement lawsuits against OpenAI and Perplexity.
Story continues below this ad
In a lawsuit filed before the Delhi High Court last year, news agency ANI alleged that OpenAI used its copyright-protected material for AI training purposes. Earlier this year, a group of digital news publishers, which include The Indian Express, Hindustan Times, and NDTV, sought to join the ongoing legal proceedings against OpenAI over alleged misuse of copyright-protected content.
Other news publishers have entered into exclusive agreements with AI companies. For example, Associated Press, Axel Springer, and Financial Times have all struck content licensing deals with OpenAI, while Reuters and Axios have inked deals allowing Meta to use their content for its AI chatbot responses.