SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas

arXiv:2505.14615

Anjiang Wei\(^{1*}\), Yuheng Wu\(^{1*}\), Yingjia Wan\(^{2}\), Tarun Suresh\(^{3}\), Huanmi Tan\(^{4}\), Zhanke Zhou\(^{1}\), Sanmi Koyejo\(^{1}\), Ke Wang\(^{5}\), Alex Aiken\(^{1}\)
* Equal contribution
\(^1\)Stanford University \(^2\)UCLA \(^3\)UIUC \(^4\)CMU \(^5\)Nanjing University

Abstract

We introduce SATBench, a benchmark for evaluating the logical reasoning capabilities of large language models (LLMs) through logical puzzles derived from Boolean satisfiability (SAT) problems. Unlike prior work that focuses on inference rule-based reasoning, which often involves deducing conclusions from a set of premises, our approach leverages the search-based nature of SAT problems, where the objective is to find a solution that fulfills a specified set of logical constraints. Each instance in SATBench is generated from a SAT formula, then translated into a story context and conditions using LLMs. The generation process is fully automated and allows for adjustable difficulty by varying the number of clauses. All 2100 puzzles are validated through both LLM-assisted and solver-based consistency checks, with human validation on a subset. Experimental results show that even the strongest model, o4-mini, achieves only 65.0% accuracy on hard UNSAT problems, close to the random baseline of 50%. SATBench exposes fundamental limitations in the search-based logical reasoning abilities of current LLMs and provides a scalable testbed for future research in logical reasoning.


Leaderboard

Model SAT UNSAT Overall Avg.
Easy Medium Hard Easy Medium Hard Easy Medium Hard
Random Baseline50.050.050.050.050.050.050.050.050.050.0
LLaMA3.1-8B57.960.048.930.414.817.544.137.433.238.2
DeepSeek-Distill-7B63.927.616.869.143.842.166.535.729.543.9
Qwen3-1.7B77.165.753.253.430.542.565.348.147.953.7
gpt-4o-mini82.182.490.742.312.913.262.247.652.053.9
LLaMA4-Scout84.376.766.452.024.337.568.150.552.056.9
LLaMA3.1-70B82.055.745.455.259.048.968.657.447.157.7
gpt-4o85.583.378.654.327.118.969.955.248.858.0
LLaMA3.3-70B90.789.075.739.527.130.065.158.152.958.7
DeepSeek-Distill-14B82.951.441.185.759.051.884.355.246.462.0
LLaMA4-Maverick80.286.286.176.825.717.978.556.052.062.1
Qwen3-4B84.178.178.680.731.922.182.455.050.462.6
Qwen3-8B82.776.767.581.634.832.182.155.749.862.6
DeepSeek-Distill-32B84.553.842.190.068.158.687.261.050.466.2
Qwen3-14B87.172.980.088.947.622.188.060.251.166.4
Qwen3-235B-Int890.083.383.286.146.219.688.064.851.468.1
Qwen-QwQ-32B92.575.759.384.151.946.488.363.852.968.3
Claude-3.7-Sonnet88.477.683.693.863.342.191.170.562.974.8
DeepSeek-V393.683.871.497.583.374.395.583.672.984.0
DeepSeek-R194.887.173.698.289.583.696.588.378.687.8
o4-mini97.096.791.198.288.165.097.692.478.089.3
Average84.173.266.772.946.439.378.559.853.063.8
Table 1: Model accuracy on SATBench using zero-shot prompting for satisfiability prediction. Difficulty levels are categorized as follows: Easy (4-19 clauses), Medium (20-30 clauses), and Hard (31-50 clauses). All open-source models are instruction-tuned.


Overview

SATBench Overview
Figure 1: Overview of the SATBench methodology. The generation pipeline begins with sampling Conjunctive Normal Form (CNF) formulas, followed by LLM-driven creation of story backgrounds and conditions. To ensure the logical puzzle's quality, both LLM-assisted and solver-based consistency validations are employed. The evaluation pipeline then examines the puzzle's prediction outcomes and checks its reasoning process.


Dataset

Metric Value
Number of Instances 2100
Average Number of Variables 36.0
Average Number of Clauses 20.6
Average Number of Words 546.2
Average Number of Sentences 55.2
Table 2: Dataset statistics for SATBench.


Benchmark Curation Pipeline Example

Example
Figure 2: Benchmark curation pipeline example. The process starts with sampling SAT formulas, followed by using a LLM to generate variable mappings and a story background. Clauses in the formula are then translated into narrative conditions. Consistency between the original formula and the generated puzzle is ensured through both LLM-based and solver-based validation.


Scaling Trend

Scaling trend on SATBench
Figure 3: Scaling trend on SATBench.


Difficulty Analysis

Difficulty Analysis
Figure 4: Difficulty Analysis. Impact of clause quantity on accuracy.


Reasoning Trace Evaluation

Model SAT UNSAT Overall
Trace
Pred. Trace Pred. Trace
Qwen-QwQ 75.552.360.752.452.4
Claude-3.7-Sonnet 83.247.466.461.154.2
DeepSeek-V3 82.965.785.071.168.4
o4-mini 94.774.683.674.174.4
DeepSeek-R1 85.273.890.382.178.0
Table 3: Accuracy in prediction and reasoning trace evaluation.


BibTeX

@article{wei2025satbench,
  title={SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas},
  author={Wei, Anjiang and Wu, Yuheng and Wan, Yingjia and Suresh, Tarun and Tan, Huanmi and Zhou, Zhanke and Koyejo, Sanmi and Wang, Ke and Aiken, Alex},
  journal={arXiv preprint arXiv:2505.14615},
  year={2025}
}