← All projects

Case study / Research / LLM evaluation

PrologEval

A benchmarking dashboard that pairs generated Prolog with execution tests and model-based evaluation.

PythonStreamlitSWI-PrologOpenRouterDocker
View repository ↗Run locally ↓Sources ↓

01 / Context

The problem

Plausible-looking code is not enough. I wanted to compare how models solve logic-programming tasks, how their programs behave when executed, and whether another model can judge them reliably.

02 / Ownership

My contribution

  • Developed the benchmarking framework and logic-programming evaluation metrics as part of collaborative research.
  • Connected code generation, execution and evaluation in a configurable experiment workflow.
  • Co-authored both studies; the journal article records equal contribution with Petros Kefalas. The conference paper also includes Ioanna Stamatopoulou.

03 / System design

Architecture

Architecture / workflow overview
  1. 01Configure

    Choose a task, generator models and generation settings in Streamlit.

  2. 02Generate

    Request Prolog through OpenRouter and collect time, token and code metrics.

  3. 03Execute

    Run queries with SWI-Prolog to inspect behavior independently of model judgments.

  4. 04Compare

    Review evaluator responses alongside execution results and export reports.

04 / Evidence

Results

Related publications
2
Public framework
Open source
Complementary checks
Execution + judgment

In the conference study, GPT-4.1 led the tested models, followed by Claude 3.7 Sonnet. The journal study found that harder tasks exposed performance differences and that model judges had low agreement. These are findings from those experiments, not a current model ranking.

05 / Boundaries

Limitations

  • A judge's verdict is not proof of correctness; execution-based checks remain essential.
  • Results apply to the tasks, model versions and settings studied.
  • The current repository includes both interactive and batch workflows. Its capabilities should not be read as the exact configuration of every published experiment.

06 / Explore the code

Run locally

The repository documents Docker setup and local use. Model requests require an OpenRouter API key; Prolog execution requires SWI-Prolog, included in the documented Docker setup.

Setup instructions

07 / References

Sources