Open in GitHub

In this tutorial, we’ll explore reranking using Cohere’s Rerank model on Azure AI Foundry.

Reranking is a crucial technique used in information retrieval systems, particularly for large-scale search applications. The process involves taking an initial set of retrieved documents and reordering them based on how relevant they are to the user’s search query.

One of the most compelling aspects of reranking is its ease of implementation - despite providing substantial improvements to search results, Cohere’s Rerank models can be integrated into any existing search system with just a single line of code, regardless of whether it uses semantic or traditional keyword-based search approaches.

In this tutorial, we’ll cover:

  • Setting up the Cohere client
  • Retrieving documents
  • Reranking documents
  • Reranking semi structured data

We’ll use Cohere’s Embed model deployed on Azure to demonstrate these capabilities and help you understand how to effectively implement semantic search in your applications.

Setup

First, you will need to deploy the Rerank model on Azure via Azure AI Foundry. The deployment will create a serverless API with pay-as-you-go token based billing. You can find more information on how to deploy models in the Azure documentation.

In the example below, we are deploying the Rerank Multilingual v3 model.

Once the model is deployed, you can access it via Cohere’s Python SDK. Let’s now install the Cohere SDK and set up our client.

To create a client, you need to provide the API key and the model’s base URL for the Azure endpoint. You can get these information from the Azure AI Foundry platform where you deployed the model.

PYTHON
1# ! pip install cohere
2
3api_key_rerank = "AZURE_API_KEY_RERANK"
4endpoint_rerank = "AZURE_ENDPOINT_RERANK" # example: "https://cohere-rerank-v3-multilingual-xyz.eastus.models.ai.azure.com/"
PYTHON
1import cohere
2
3co = cohere.Client(
4 api_key=api_key_rerank,
5 base_url=endpoint_rerank,
6)

Retrieve documents

For this example, we’ll work with documents that have already been retrieved through an initial search stage (which could be semantic search, keyword matching, or another retrieval method).

Below is a list of nine documents representing the initial search results. Each document contains email data structured as a dictionary with two fields - Title and Content. This semi-structured format allows the Rerank endpoint to effectively process and reorder the results based on relevance.

PYTHON
1documents = [
2 {"Title":"Incorrect Password","Content":"Hello, I have been trying to access my account for the past hour and it keeps saying my password is incorrect. Can you please help me?"},
3 {"Title":"Confirmation Email Missed","Content":"Hi, I recently purchased a product from your website but I never received a confirmation email. Can you please look into this for me?"},
4 {"Title":"Questions about Return Policy","Content":"Hello, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective."},
5 {"Title":"Customer Support is Busy","Content":"Good morning, I have been trying to reach your customer support team for the past week but I keep getting a busy signal. Can you please help me?"},
6 {"Title":"Received Wrong Item","Content":"Hi, I have a question about my recent order. I received the wrong item and I need to return it."},
7 {"Title":"Customer Service is Unavailable","Content":"Hello, I have been trying to reach your customer support team for the past hour but I keep getting a busy signal. Can you please help me?"},
8 {"Title":"Return Policy for Defective Product","Content":"Hi, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective."},
9 {"Title":"Wrong Item Received","Content":"Good morning, I have a question about my recent order. I received the wrong item and I need to return it."},
10 {"Title":"Return Defective Product","Content":"Hello, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective."}
11]

Rerank documents

Adding a reranking component is simple with Cohere Rerank. It takes just one line of code to implement.

Calling the Rerank endpoint requires the following arguments:

  • documents: The list of documents, which we defined in the previous section
  • query: The user query; we’ll use ‘What emails have been about refunds?’ as an example
  • top_n: The number of documents we want to be returned, sorted from the most to the least relevant document

When passing documents that contain multiple fields like in this case, for best performance we recommend formatting them as YAML strings.

PYTHON
1import yaml
2
3yaml_docs = [yaml.dump(doc, sort_keys=False) for doc in documents]
4
5query = 'What emails have been about refunds?'
6
7results = co.rerank(
8 documents=yaml_docs,
9 query=query,
10 top_n=3
11)

Since we set top_n=3, the response will return the three documents most relevant to our query. Each result includes both the document’s original position (index) in our input list and a score indicating how well it matches the query.

Let’s examine the reranked results below.

PYTHON
1def return_results(results, documents):
2 for idx, result in enumerate(results.results):
3 print(f"Rank: {idx+1}")
4 print(f"Score: {result.relevance_score}")
5 print(f"Document: {documents[result.index]}\n")
6
7return_results(results, documents)
1Rank: 1
2Score: 8.547617e-05
3Document: {'Title': 'Return Defective Product', 'Content': 'Hello, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective.'}
4
5Rank: 2
6Score: 5.1442214e-05
7Document: {'Title': 'Questions about Return Policy', 'Content': 'Hello, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective.'}
8
9Rank: 3
10Score: 3.591301e-05
11Document: {'Title': 'Return Policy for Defective Product', 'Content': 'Hi, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective.'}

The search query was looking for emails about refunds. But none of the documents mention the word “refunds” specifically.

However, the Rerank model was able to retrieve the right documents. Some of the documents mentioned the word “return”, which has a very similar meaning to “refunds.”

Rerank semi structured data

The Rerank 3 model supports multi-aspect and semi-structured data like emails, invoices, JSON documents, code, and tables. By setting the rank fields, you can select which fields the model should consider for reranking.

In the following example, we’ll use an email data example. It is a semi-stuctured data that contains a number of fields – from, to, date, subject, and text.

The model will rerank based on order of the fields passed.

PYTHON
1# Define the documents
2emails = [
3 {"from": "hr@co1t.com", "to": "david@co1t.com", "date": "2024-06-24", "subject": "A Warm Welcome to Co1t!", "text": "We are delighted to welcome you to the team! As you embark on your journey with us, you'll find attached an agenda to guide you through your first week."},
4 {"from": "it@co1t.com", "to": "david@co1t.com", "date": "2024-06-24", "subject": "Setting Up Your IT Needs", "text": "Greetings! To ensure a seamless start, please refer to the attached comprehensive guide, which will assist you in setting up all your work accounts."},
5 {"from": "john@co1t.com", "to": "david@co1t.com", "date": "2024-06-24", "subject": "First Week Check-In", "text": "Hello! I hope you're settling in well. Let's connect briefly tomorrow to discuss how your first week has been going. Also, make sure to join us for a welcoming lunch this Thursday at noon—it's a great opportunity to get to know your colleagues!"}
6]
7
8yaml_emails = [yaml.dump(doc, sort_keys=False) for doc in emails]
PYTHON
1# Add the user query
2query = "Any email about check ins?"
3
4# Rerank the documents
5results = co.rerank(
6 query=query,
7 documents=yaml_emails,
8 top_n=2,
9)
10
11return_results(results, yaml_emails)
1Rank: 1
2Score: 0.13454832
3Document: from: john@co1t.com
4to: david@co1t.com
5date: '2024-06-24'
6subject: First Week Check-In
7text: "Hello! I hope you're settling in well. Let's connect briefly tomorrow to discuss\
8 \ how your first week has been going. Also, make sure to join us for a welcoming\
9 \ lunch this Thursday at noon\u2014it's a great opportunity to get to know your\
10 \ colleagues!"
11
12
13Rank: 2
14Score: 0.0010044164
15Document: from: it@co1t.com
16to: david@co1t.com
17date: '2024-06-24'
18subject: Setting Up Your IT Needs
19text: Greetings! To ensure a seamless start, please refer to the attached comprehensive
20 guide, which will assist you in setting up all your work accounts.

Summary

In this tutorial, we learned about:

  • How to set up the Cohere client to use the Rerank model deployed on Azure AI Foundry
  • How to retrieve documents
  • How to rerank documents
  • How to rerank semi structured data

In the next tutorial, we’ll learn how to build RAG applications by leveraging the models that we’ve looked at in the previous tutorials - Command, Embed, and Rerank.

Built with