> This page is for version v1 API.
> For other versions, use one of these documentation indexes:
> - v2 API (default): https://docs.cohere.com/v2/llms.txt
> - v1 API: https://docs.cohere.com/v1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.cohere.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.cohere.com/_mcp/server.

# Unlocking the Power of Multimodal Embeddings

> Multimodal embeddings convert text and images into embeddings for search and classification.

![embeddings.](/_fern-img/8b7edf88cdfe83a756d674bbb4bd35933670f0c7b6f7ca9e8c93d99af74877cf.webp)

> **This Guide Uses the Embed API.**
>
> Refer to the [API reference](/reference/embed) for more details.
>
> Image capabilities are only compatible with `v4.0` and `v3.0` models, but `v4.0` has features that `v3.0` does not have. Consult the embedding [documentation](https://docs.cohere.com/docs/cohere-embed) for more details.

In this guide, we show you how to use the embed endpoint to embed a series of images. This guide uses a simple dataset of graphs to illustrate how semantic search can be done over images with Cohere. To see an end-to-end example of retrieval, check out this [notebook](https://github.com/cohere-ai/cohere-developer-experience/blob/main/notebooks/Multimodal_Semantic_Search.ipynb).

## Introduction to Multimodal Embeddings

Information is often represented in multiple modalities. A document, for instance, may contain text, images, and graphs, while a product can be described through images, its title, and a written description. This combination of elements often leads to a comprehensive semantic understanding of the subject matter. Traditional embedding models have been limited to a single modality, and even multimodal embedding models often suffer from degradation in `text-to-text` or `text-to-image` retrieval tasks. `embed-v5.0-pro` and the `embed-v3.0` series of models, however, are fully multimodal, enabling them to embed both images and text effectively. We have achieved state-of-the-art performance without compromising text-to-text retrieval capabilities.

## How to use Multimodal Embeddings

### 1. Prepare your Image for Embeddings

The Embed API takes in images with the following file formats: `png`, `jpeg`,`Webp`, and `gif`. The images must then be formatted as a Data URL.

**`PYTHON`**

```python PYTHON
# Import the necessary packages
import os
import base64


# Defining the function to convert an image to a base 64 Data URL
def image_to_base64_data_url(image_path):
    _, file_extension = os.path.splitext(image_path)
    file_type = file_extension[1:]

    with open(image_path, "rb") as f:
        enc_img = base64.b64encode(f.read()).decode("utf-8")
        enc_img = f"data:image/{file_type};base64,{enc_img}"
    return enc_img


image_path = "<YOUR IMAGE PATH>"
processed_image = image_to_base64_data_url(image_path)


# format the input_object
image_input = [
    {"content": [{"type": "image", "image": processed_image}]}
]

res = co.embed(
    model="embed-v5.0-pro",
    inputs=image_input,
    input_type="search_document",
    output_dimension=1024,
    embedding_types=["float"],
)
res.embeddings.float
```

### 2. Call the Embed Endpoint

**`PYTHON`**

```python PYTHON
# Import the necessary packages
import cohere

co = cohere.ClientV2(api_key="<YOUR API KEY>")

# format the input_object
image_input = [
    {"content": [{"type": "image", "image": processed_image}]}
]

co.embed(
    model="embed-v5.0-pro",
    inputs=image_input,
    input_type="search_document",
    embedding_types=["float"],
)
```

## Sample Output

Below is a sample of what the output would look like if you passed in a `jpeg` with original dimensions of `1080x1350` with a standard bit-depth of 24.

**`JSON`**

```json JSON
{
    'id': '0d9bb922-f15f-4b8b-9a2f-72577324528f', 
    'texts': [], 
    'images': [{'width': 1080, 'height': 1350, 'format': 'jpeg', 'bit_depth': 24}], 
    'embeddings': {'float': [[-0.035369873, 0.040740967, 0.008262634, -0.008766174, .....]]}, 
    'meta': {
    'api_version': {'version': '1'},
    'billed_units': {'images': 1}
    }, 
    'response_type': 'embeddings_by_type'
}
```