Troubleshooting a Custom Model

In this post, we answer frequently asked questions about training models.

Troubleshooting Representation Models

Gathering enough data

While our Classify endpoint enables a user to build a classifier with just 5 examples per label, this classifier runs on our baseline model which has not been trained for specific use cases. Your dataset must contain at least 250 labelled examples to start training.

If you are unable to locate a relevant labelled dataset from online sources, we suggest trying to generate labelled examples using our Generate endpoint. Check out this sample preset of a user generating product feedback to train a product feedback classifier.

Data formatting best practices

Ensure your data is in a two-column csv. One column should be the sample text you'd like to classify or search, and the second column should be a label for the text. We recommend using a comma , as your delimiter.

Here are a few example lines from a dataset that could be used to train a model that classifies headlines as positive, negative, and neutral with our Classify endpoint:

headline,sentiment
The major construction companies of Finland are operating in Russia,neutral
$ESI on lows, down $1.50 to $2.50 BK a real possibility,negative
$SPY wouldn't be surprised to see a green close,positive

To pass data validation, ensure that:

• There are at least 5 examples for each label in your dataset
• Your dataset contains at least unique 250 examples in total (not 250 examples per label)
• Your data is encoded in UTF-8
• We will automatically deduplicate your dataset

Formatting errors for classification tasks

Cohere's Classify endpoint will return predictions for classes that sum up to 1. We currently do not support outputting classifications for multiple labels (known as multi-label classification). Each example text should be mapped to one label only.

Take this example below:

headline,topic
"How Robots Can Assist Students with Disabilities","technology,health"
"As Gas Prices Went Up, So Did the Hunt for Electric Vehicles","technology,economy"

Currently, we will process this data and train with two labels, technology,health and technology,economy instead of the desired three labels, technology, health, and economy.

In this case you will need to select one label for each headline.

Formatting for search tasks

At this time, if you are intending to train a representation model to use Cohere's Embed endpoint to perform a search task (not predicting a label), you will still need to assign a label to texts for representation training.

For example, if you are building a search engine for Hacker News posts and you want to either cluster similar posts or associate posts with a certain keyword, you would create a labelled dataset with the post titles mapped to the keyword. See a few sample lines labelled below:

post title,keyword
Advice for a new and inexperienced tech lead?,career
How do you deal with getting old and feeling lost?,personal
Best way to learn modern C++?,skills

If you are topic modelling and trying to find clusters, we recommend trying the baseline model. Check out our blog post on topic modelling Hacker News posts.

Troubleshooting auto evaluation metrics

When you are viewing auto evaluation metrics during or after your model has finished training, you may find that the F1, Recall, and Precision metrics are missing. This may occur if your dataset is extremely imbalanced (e.g. A binary dataset with 95% positive labels and 5% negative labels) and the trained model fails to predict one of the labels at all. This does not prevent you from using this trained model, it is simply a warning.

To resolve this warning, try adding more examples for labels with less data.

How long does it take to train a model?

Trainings are completed sequentially, and when you launch a training session, it is added to the end of a queue. Depending on the length of our training queue, training may take between 1 hour to a day to complete.

Using your trained model

To use your trained model in our API or SDKs, you must call it by its model UUID and not by its name. To get the model UUID, select the model in the playground and click Export Code. Select the library you are using and copy the code to call the model.

Restarting paused custom models

All trained models are paused after 24 hours of inactivity. To restart your model, select your model in the trained models panel and click on the Wake button, pictured below:

Troubleshooting failed training

Our engineers review every individual failed training and will attempt to rectify it without any action on your part. We reach out to individuals with failed custom models we cannot resolve manually.

Please reach out to [email protected] or post in our co:mmunity Discord if you have unanswered questions about training models.