> This page is for version v2 API (default).
> For other versions, use one of these documentation indexes:
> - v2 API (default): https://docs.cohere.com/v2/llms.txt
> - v1 API: https://docs.cohere.com/v1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.cohere.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.cohere.com/_mcp/server.

# Unlocking the Power of Multimodal Embeddings

> Multimodal embeddings convert text and images into embeddings for search and classification (API v2).

![embeddings.](/_fern-img/8b7edf88cdfe83a756d674bbb4bd35933670f0c7b6f7ca9e8c93d99af74877cf.webp)

> **This Guide Uses the Embed API.**
>
> Refer to the [API reference](/reference/embed) for more details.
>
> Image capabilities are only compatible with `v4.0` and `v3.0` models, but `v4.0` has features that `v3.0` does not have. Consult the embedding [documentation](https://docs.cohere.com/docs/cohere-embed) for more details.

In this guide, we show you how to use the embed endpoint to embed a series of images. This guide uses a simple dataset of graphs to illustrate how semantic search can be done over images with Cohere. To see an end-to-end example of retrieval, check out this [notebook](https://github.com/cohere-ai/cohere-developer-experience/blob/main/notebooks/Multimodal_Semantic_Search.ipynb).

## Introduction to Multimodal Embeddings

Information is often represented in multiple modalities. A document, for instance, may contain text, images, and graphs, while a product can be described through images, its title, and a written description. This combination of elements often leads to a comprehensive semantic understanding of the subject matter. Traditional embedding models have been limited to a single modality, and even multimodal embedding models often suffer from degradation in `text-to-text` or `text-to-image` retrieval tasks. `embed-v5.0-pro` and the `embed-v3.0` series of models, however, are fully multimodal, enabling them to embed both images and text effectively. We have achieved state-of-the-art performance without compromising text-to-text retrieval capabilities.

## How to use Multimodal Embeddings

### 1. Prepare your Image for Embeddings

**`PYTHON`**

```python PYTHON
# Import the necessary packages
import os
import base64


# Defining the function to convert an image to a base 64 Data URL
def image_to_base64_data_url(image_path):
    _, file_extension = os.path.splitext(image_path)
    file_type = file_extension[1:]

    with open(image_path, "rb") as f:
        enc_img = base64.b64encode(f.read()).decode("utf-8")
        enc_img = f"data:image/{file_type};base64,{enc_img}"
    return enc_img


image_path = "<YOUR IMAGE PATH>"
base64_url = image_to_base64_data_url(image_path)
```

### 2. Call the Embed Endpoint

**`PYTHON`**

```python PYTHON
# Import the necessary packages
import cohere

co = cohere.ClientV2(api_key="<YOUR API KEY>")

# format the input_object

image_input = {
    "content": [
        {"type": "image_url", "image_url": {"url": base64_url}}
    ]
}

co.embed(
    model="embed-v5.0-pro",
    inputs=[image_input],
    input_type="search_document",
    embedding_types=["float"],
)
```

**`cURL`**

```bash cURL
curl --request POST \
  --url https://api.cohere.ai/v2/embed \
  --header 'accept: application/json' \
  --header 'content-type: application/json' \
  --header "Authorization: bearer $CO_API_KEY" \
  --data '{
    "model": "embed-v5.0-pro",
    "inputs": [
      {
        "content": [
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD..."
            }
          }
        ]
      }
    ],
    "input_type": "search_document",
    "embedding_types": ["float"]
  }'
```

## Sample Output

Below is a sample of what the output would look like if you passed in a `jpeg` with original dimensions of `1080x1350` with a standard bit-depth of 24.

**`JSON`**

```json JSON
{
    "id": "d8f2b461-79a4-44ee-82e4-be601bbb07be",
    "embeddings": {
        "float_": [[-0.025604248, 0.0154418945, ...]],
        "int8": null,
        "uint8": null,
        "binary": null,
        "ubinary": null,
    },
    "texts": [],
    "meta": {
        "api_version": {"version": "2", "is_deprecated": null, "is_experimental": null},
        "billed_units": {
            "input_tokens": null,
            "output_tokens": null,
            "search_units": null,
            "classifications": null,
            "images": 1,
        },
        "tokens": null,
        "warnings": null,
    },
    "images": [{"width": 1080, "height": 1080, "format": "jpeg", "bit_depth": 24}],
    "response_type": "embeddings_by_type",
}
```