## AI Startup Anthropic Accused of Scraping Websites Despite 'Do Not Crawl' Rules

Two websites, **iFixit** and **Freelancer**, have publicly accused **Anthropic**, the AI startup behind the **Claude** large language models, of ignoring their **robots.txt protocol** and website policies to scrape their data for AI training. 

**Freelancer** CEO **Matt Barrie** reported that **Anthropic's ClaudeBot** generated **3.5 million visits** to their website in just four hours, significantly impacting their website's performance and revenue. **iFixit** CEO **Kyle Wiens** stated that Anthropic's bot hit their servers a million times in a single day, impacting their development resources. 

This isn't the first time AI companies have been accused of bypassing web scraping restrictions. **Wired** previously accused **Perplexity** of similar behavior, and **TollBit** confirmed that other AI firms, including **OpenAI**, were also ignoring robots.txt signals.

**Anthropic** has since acknowledged the issue and stated they are investigating the matter. **Freelancer** has blocked Anthropic's crawler completely, while **iFixit** blocked the specific bot after adding a line in their robots.txt file.  

This situation highlights the growing tension between AI companies and content creators. While AI firms rely on web data for training their models, content creators are concerned about copyright infringement, resource depletion, and revenue loss.  

**OpenAI** has already begun striking deals with publishers like **News Corp, Vox Media, the Financial Times, and Reddit** to access their content for AI model training. **iFixit** is also open to discussing licensing their content for commercial use.

This situation underscores the need for clear guidelines and agreements between AI companies and content creators to ensure fair use and prevent legal conflicts.

iFixit and Freelancer have accused Anthropic, the AI startup behind the Claude large language models, of ignoring their "do not crawl" robots.txt protocol and policy to scrape their websites' data.

AI Startup Anthropic Accused of Scraping Websites Despite 'Do Not Crawl' Rules

## Nuro's Self-Driving Delivery Comeback: AI Advancements Fuel Bay Area Expansion

Autonomous delivery startup **Nuro** is gearing up for a comeback, receiving approval from the California Department of Motor Vehicles to test its third-generation **R3 autonomous delivery vehicle** in four Bay Area cities: Mountain View, Palo Alto, Los Altos, and Menlo Park.

This marks a significant step forward for Nuro, which had faced setbacks and financial struggles in recent years. The company has restructured its team, focusing on **AI advancements** and putting vehicle manufacturing and commercial operations on hold.

**Nuro's R3** stands out as a driverless delivery vehicle, designed exclusively for goods. It operates on public roads but resembles a large sidewalk delivery robot, equipped with temperature-controlled storage units for food. The expansion into the Bay Area will be the third largest deployment of fully driverless vehicles in the United States, after **Waymo**.

**Nuro's R3** boasts improved capabilities compared to its predecessor, the **R2**.  It can operate at speeds up to 45 miles per hour, enabling **Level 4 autonomy** – capable of driving itself without human intervention in certain circumstances. This advancement stems from improvements in **AI**, with Nuro utilizing one to two large foundational models for tasks such as mapping, localization, perception, prediction, and planning.

Nuro's partnership with **Uber Eats** remains crucial, with the ride-hail and delivery giant set to utilize the **R3** for deliveries this fall. While scaled manufacturing is not on the immediate horizon, Nuro is exploring a path to market outside of autonomous delivery, indicating a broader vision for the future.

**Key takeaways:**

* **Nuro** is making a comeback in the autonomous delivery space, focusing on **AI advancements** and expanding its testing into the Bay Area.
* **Nuro's R3** offers significant improvements, including faster speeds and **Level 4 autonomy**.  
* The company is exploring a path to market outside of autonomous delivery, hinting at a broader vision for the future.

The California Department of Motor Vehicles this week granted Nuro approval to test its third-generation R3 autonomous delivery vehicle in four Bay Area

Stay up to date with our latest news

1 year ago Startups Articles Digested