Picture: Enzozo/ShutterstockMore than four in ten of the top 100 news websites in the English language allow all AI web crawlers to scrape their content, Press Gazette analysis has found. CloseMost news websites block or permit web crawlers by editing a page on their URL named robots.txt, for example pressgazette.co.uk/robots.txt. The UK and US editions of the BBC website both had five crawlers blocked, and as did News UK-owned the-sun.com, thesun.co.uk and thetimes.co.uk. Three other IAC properties - People, Entertainment Weekly and Investopedia - block GPTBot but no other AI web crawlers. Several websites on the political right decline to block AI web crawlers, among them GB News, Newsmax, Zero Hedge, Breitbart and, despite other Murdoch-owned titles all blocking the bots, Fox News.

Source: Indian Express February 27, 2024 11:57 UTC