Generating Feedback

Co.feedback allows users to provide feedback on responses created from Co.Generate. This feedback is used to improve the model. The endpoint can generate preference and performance feedback. This guide provides a starting point for using the endpoint.

You can see a live example of Co.feedback on our playground.

Or, you can read below to learn how to call the feedback endpoint with our API.

Generate Feedback

You can produce detailed feedback based on the annotator's acceptance of the generated response using the generate_feedback endpoint.

Parameters

The endpoint has a number of settings you can use to control the kind of output it generates:

• request_id (String, required): The request_id of the generation request to give feedback on.
• good_response (Boolean, required): Whether the response was good or not.
• model (String): The unique ID of the model.
• desired_response (String): The desired response. Used when an annotator edits a suggested response.
• flagged_response (Boolean): Whether the response was flagged or not.
• flagged_reason (String): The reason the response was flagged.
• prompt (String): The original prompt used to generate the response.
• annotator_id (String): The annotator's ID.

Example Requests

If the annotator accepts the suggested response, you could format a request like this:

generations = co.generate(f"Write me a polite email responding to the one below: {email}. Response:")
if user_accepted_suggestion:
    co.generate_feedback(request_id=generations[0].id, good_response=True)

If the annotator edits the suggested response, you could format a request like this:

generations = co.generate(f"Write me a polite email responding to the one below: {email}. Response:")
if user_edits_suggestion:
    co.generate_feedback(request_id=generations[0].id, good_response=False, desired_response=user_edited_suggestion)

Example Response

Generate Preference Feedback

Alternatively, you can generate feedback based on which response an annotator prefers with the generate_preference_feedback endpoint.

Parameters

• ratings (List[PreferenceRating], required): A list of PreferenceRating objects.
• model (String): The unique ID of the model.
• prompt (String): The original prompt used to generate the response.
• annotator_id (String): The annotator's ID.

Example Request

A user accepts a model's suggestion in an assisted writing setting, and prefers it to a second suggestion, a request might look like this:

generations = co.generate(f"Write me a polite email responding to the one below: {email}. Response:", num_generations=2)  
         if user_accepted_idx: # prompt user for which generation they prefer  
               ratings = \[]  
              if user_accepted_idx == 0:  
                   ratings.append(PreferenceRating(request_id=0, rating=1))  
                   ratings.append(PreferenceRating(request_id=1, rating=0))  
              else:  
                   ratings.append(PreferenceRating(request_id=0, rating=0))  
                   ratings.append(PreferenceRating(request_id=1, rating=1))  
 							co.generate_preference_feedback(ratings=ratings)