Order Examples
Does the order of in-context examples affect your task's output? If so, which ordering provides the best output?
LLMs can be sensitive to the order of examples in prompts1234. The script below uses instructor to test different example permutations and see how the output changes.
Implementation¶
from pydantic import BaseModel
import instructor
from openai import OpenAI
from itertools import permutations
client = instructor.from_openai(OpenAI())
class Example(BaseModel): # (1)!
input: str
output: str
class Response(BaseModel):
response: str
def inference(examples, query):
return client.chat.completions.create(
model="gpt-4o",
response_model=Response,
messages=[
{
"role": "user",
"content": f"{examples} {query}", # (2)!
}
],
).response
if __name__ == "__main__":
examples = [
Example(input="The movie was so good", output="positive"),
Example(input="The movie was somewhat good", output="negative"),
]
query = "The movie was okay"
permutations = list(permutations(examples))
results = [inference(permutation, query) for permutation in permutations]
print(permutations)
"""
[
(
Example(input='The movie was so good', output='positive'),
Example(input='The movie was somewhat good', output='negative'),
),
(
Example(input='The movie was somewhat good', output='negative'),
Example(input='The movie was so good', output='positive'),
),
]
"""
print(results)
#> ['negative', 'positive']
- This class can be customized to a specific task
- This prompt can be customized to a specific task
Info
For scenarios with a large number of examples, check out example selection techniques (KNN, Vote-K).
References¶
1: Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
2: Reordering Examples Helps during Priming-based Few-Shot Learning