This work introduces SylloBio-NLI, a framework for evaluating LLMs' syllogistic reasoning in the biomedical domain.

New study tests AI’s ability to solve complex medical logic problems, revealing gaps in reasoning for biomedical applications.

A study on AI models’ ability to process logical arguments in medicine, revealing challenges with reasoning, accuracy, and handling complex tasks.

SylloBio-NLI evaluates AI reasoning in biomedicine using syllogistic tasks, leveraging human genome pathways for robust assessment of logical inference.

SylloBio-NLI uses syllogistic reasoning to evaluate AI models in biomedicine, revealing challenges and potential in LLMs for accurate inference.

Exploring the limitations of scaling syllogistic reasoning in biomedical NLI, this work highlights challenges related to knowledge base dependency

This appendix formalizes the process of generating syllogistic inference patterns for biomedical NLI.

Learn about two essential tasks in biomedical reasoning: testing logical validity and selecting the right premises to derive conclusions.

Explore the Reactome knowledgebase, detailing 11,226 protein-coding genes and 15,212 human reactions.

Overview of the experimental setup for evaluating LLMs on syllogistic tasks, including modules, parameters, and computational resources used.

Evaluation metrics for LLM performance include accuracy, F1-score, recall, reasoning accuracy, and faithfulness, ensuring meaningful and compliant responses.

Analysis of domain-specific LLM responses reveals issues like empty outputs, incorrect or repeated content, and misaligned Chain-of-Thought (CoT) reasoning.

LLMs demonstrate varying sensitivity to increasing distractors in reasoning tasks.

A. Formalization of the SylloBio-NLI Resource Generation Process

LLMs rely on contextual knowledge over domain-specific facts, maintaining accuracy even with synthetic gene names in reasoning tasks.