A collection of meta-prompting techniques for evaluating and analyzing AI responses and solution paths.
A framework for critiquing and reflecting on the quality of responses, providing a score and indicating whether the response has fully solved the question or task.
Reflections: The critique and reflections on the sufficiency, superfluency, and general quality of the response.
Score: Score from 0-10 on the quality of the candidate response.
Found_solution: Whether the response has fully solved the question or task.
When evaluating responses, consider the following:
Provide thoughtful reflections on these aspects and any other relevant factors. Use the score to indicate the overall quality, and set found_solution to true only if the response fully addresses the question or completes the task.
reflections: "The response was clear and concise, addressing the main question effectively. However, it could have provided more context on edge cases."
score: 8
found_solution: true
Guidelines for analyzing solution paths to question-answering tasks.
Observations: Environmental information about the current situation that provides context for decision-making.
Thoughts: Reasoning about the current situation, analyzing what has been observed and planning next steps.
Actions: The steps taken to progress toward solving the task.
Search[entity]: Searches for the exact entity and returns relevant information if the entity exists. If not, returns suggestions for similar entities.
Lookup[keyword]: Returns the next relevant passage that contains the keyword. Used for finding specific information within retrieved content.
Finish[answer]: Returns the answer and finishes the task. Used when sufficient information has been gathered to provide a definitive response.
When analyzing a trajectory:
Guide the model through step-by-step reasoning:
Let's approach this step by step:
1. First, identify the key components
2. Then, analyze each component
3. Finally, synthesize the findings
Provide examples to establish the pattern:
Example 1: [input] -> [output]
Example 2: [input] -> [output]
Now apply this pattern to: [new input]
Generate multiple reasoning paths and select the most consistent answer.
Encourage self-critique:
Review your response and identify:
- Any potential errors or oversights
- Areas that could be explained more clearly
- Missing information that would strengthen the answer
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.
Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.
CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).