> This page is for version v1 API.
> For other versions, use one of these documentation indexes:
> - v2 API (default): https://docs.cohere.com/v2/llms.txt
> - v1 API: https://docs.cohere.com/v1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.cohere.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.cohere.com/_mcp/server.

# Cohere Web Crawlers

> Cohere's policy on web crawlers and robots.txt for generative AI training.

We do not use Cohere bots or user agents for the purpose of crawling or scraping web content to
train generative AI foundation models at this time. Should Cohere make use of these tools to crawl
or scrape web content to train AI models in the future, Cohere will identify those bots and/or user
agents in the table below alongside information about how to disallow their access to your site.

| Bot Name | Description |
| -------- | ----------- |
| N/A      | N/A         |

## Robots.txt

It is Cohere's policy to require that crawlers be designed to respect `robots.txt`. Were we to
use a bot or user agent to crawl or scrape web content to train AI models in the future, web site
operators could follow the following guidance to block the Cohere bot or user agent from accessing
their website:

To block a Cohere bot or user agent from accessing your entire website, add the following to your
`robots.txt` file in the top-level directory for each sub-domain you wish to exclude.

**Example:**

```
User-agent: Coherebot
Disallow: /
```

Other methods, such as blocking IP addresses, may not work reliably or ensure persistent opt-out, as
they prevent the reading of a site's `robots.txt` file.

## Updates

If we make updates to this page, we will publish them in the [changelog](/changelog) on this site.
You can [subscribe to an RSS feed](https://docs.cohere.com/changelog.rss) to receive updates automatically.