Generating Feedback
Co.feedback allows users to provide feedback on responses created from Co.Generate. This feedback is used to improve the model. The endpoint can generate preference and performance feedback. This guide provides a starting point for using the endpoint.
You can see a live example of Co.feedback on our playground.
Or, you can read below to learn how to call the feedback endpoint with our API.
Generate Feedback
You can produced detailed feedback based on the annotator's acceptance of the generated response using the generate_feedback
endpoint.
Parameters
The endpoint has a number of settings you can use to control the kind of output it generates:
request_id
(String, required): The request_id of the generation request to give feedback on.good_response
(Boolean, required): Whether the response was good or not.model
(String): The unique ID of the model.desired_response
(String): The desired response. Used when an annotator edits a suggested response.flagged_response
(Boolean): Whether the response was flagged or not.flagged_reason
(String): The reason the response was flagged.prompt
(String): The original prompt used to generate the response.annotator_id
(String): The annotator's ID.
Example Requests
If the annotator accepts the suggested response, you could format a request like this:
generations = co.generate(f"Write me a polite email responding to the one below: {email}. Response:")
if user_accepted_suggestion:
co.generate_feedback(request_id=generations[0].id, good_response=True)
If the annotator edits the suggested response, you could format a request like this:
generations = co.generate(f"Write me a polite email responding to the one below: {email}. Response:")
if user_edits_suggestion:
co.generate_feedback(request_id=generations[0].id, good_response=False, desired_response=user_edited_suggestion)
Example Response
Generate Preference Feedback
Alternatively, you can generate feedback based on which response an annotator prefers with the generate_preference_feedback
endpoint.
Parameters
ratings
(List[PreferenceRating], required): A list of PreferenceRating objects.model
(String): The unique ID of the model.prompt
(String): The original prompt used to generate the response.annotator_id
(String): The annotator's ID.
Example Request
A user accepts a model's suggestion in an assisted writing setting, and prefers it to a second suggestion, a request might look like this:
generations = co.generate(f"Write me a polite email responding to the one below: {email}. Response:", num_generations=2)
if user_accepted_idx: // prompt user for which generation they prefer
ratings = \[]
if user_accepted_idx == 0:
ratings.append(PreferenceRating(request_id=0, rating=1))
ratings.append(PreferenceRating(request_id=1, rating=0))
else:
ratings.append(PreferenceRating(request_id=0, rating=0))
ratings.append(PreferenceRating(request_id=1, rating=1))
co.generate_preference_feedback(ratings=ratings)
Updated 12 days ago