This work introduces SylloBio-NLI, a framework for evaluating LLMs' syllogistic reasoning in the biomedical domain.

New study tests AI’s ability to solve complex medical logic problems, revealing gaps in reasoning for biomedical applications.

A study on AI models’ ability to process logical arguments in medicine, revealing challenges with reasoning, accuracy, and handling complex tasks.

SylloBio-NLI evaluates AI reasoning in biomedicine using syllogistic tasks, leveraging human genome pathways for robust assessment of logical inference.

SylloBio-NLI uses syllogistic reasoning to evaluate AI models in biomedicine, revealing challenges and potential in LLMs for accurate inference.

Exploring the limitations of scaling syllogistic reasoning in biomedical NLI, this work highlights challenges related to knowledge base dependency

This appendix formalizes the process of generating syllogistic inference patterns for biomedical NLI.

Learn about two essential tasks in biomedical reasoning: testing logical validity and selecting the right premises to derive conclusions.

Learn how to access and utilize pretrained LLMs such as Mistral-7B and BioMistral for advanced natural language processing tasks.

Overview of the experimental setup for evaluating LLMs on syllogistic tasks, including modules, parameters, and computational resources used.

Evaluation metrics for LLM performance include accuracy, F1-score, recall, reasoning accuracy, and faithfulness, ensuring meaningful and compliant responses.

Explore zero-shot prompting techniques for logical reasoning tasks related to signaling and metabolic molecules in biological pathways.

Analysis of domain-specific LLM responses reveals issues like empty outputs, incorrect or repeated content, and misaligned Chain-of-Thought (CoT) reasoning.

Explore few-shot prompting techniques for logical reasoning tasks in biological pathways.

LLMs demonstrate varying sensitivity to increasing distractors in reasoning tasks.

LLMs rely on contextual knowledge over domain-specific facts, maintaining accuracy even with synthetic gene names in reasoning tasks.