BIG-Bench Mistake consists of 2186 sets of CoTstyle traces. Each trace was generated by PaLM 2-L-Unicorn

Annotators can click on words to highlight the same word across the trace and the question text. Buttons on the right automatically become inactive

We use PaLM 2 L (Unicorn) to generate the traces used in BIG-Bench Mistake. All traces are generated at temperature = 0. We algorithmically append “Thought N:”

This paper explores few-shot in-context learning methods, which is typically used in realworld applications with API-based LLMs