In this paper, we break down the self-correction process into two core components: mistake finding and output correction.

We propose a simple backtracking method to improve model outputs based on the location of logical errors. Backtracking reduces the computational cost

All models are given the same 3-shot prompts. We use three different prompting methods. Direct trace-level prompting involves using the whole trace as input

BIG-Bench Mistake consists of 2186 sets of CoTstyle traces. Each trace was generated by PaLM 2-L-Unicorn

Annotators can click on words to highlight the same word across the trace and the question text. Buttons on the right automatically become inactive

We use PaLM 2 L (Unicorn) to generate the traces used in BIG-Bench Mistake. All traces are generated at temperature = 0. We algorithmically append “Thought N:”

In this paper, we describe and release our dataset BIG-Bench Mistake for mistake-finding and propose a backtracking method to correct logical errors.

This paper explores few-shot in-context learning methods, which is typically used in realworld applications with API-based LLMs