prolifai.xyz
Benchmark
AICAB
The Artificial Intelligence Cultural Alignment Benchmark: what AI models tell people about cultures and faiths, and how competently they do it. Each edition takes one subject, from the perspective of the people who ask about it: what AI tells Muslims about Islam and Muslim life, what it tells Muslims about Jews, and further editions as they are run. Every answer is published, so any result can be opened and checked.
Loading the latest round from the benchmark's server…
The latest round
Each measure, model by model
Each measure ranked on its own, with what it cost; we never fold them into one number. Where a bar has a line across it, that is its 95% interval. Hover or focus a bar for its counts; the table below holds every figure.
Every figure
| Model | Answers | Sources checked | Schools | Courtesy and deference | Calibration | Cost |
|---|
Every answer
Open any answer to read it whole, with what the rules and the judges found in it.
Buy a report or licence
Reports on any model, with measures, intervals, examples of what was found and the method: free for everyone, with a free licence to quote a result in commercial material: we charge only what a thing costs us, plus 30% for our work, and a report costs nothing to supply. Citing AICAB with attribution in journalism, research and teaching is always free, and needs no report. Every round is published in full, whoever pays.
Buy a report or licence Verify a reference Licence terms Commission a round or a benchmark
How it is measured
Asked as typed (0.2). Each model is tested against itself. The question goes to the model exactly as a person would type it: no instructions, no passages, the provider's own settings, with zero data retention. Each question is asked three times, so we can see whether a model gives the same answer to the same question.
- Citations, checked in the texts. Every verse and hadith an answer cites is looked up: the Qur'an in Arabic and in Pickthall's English, and six hadith collections by their reference numbers. A citation that exists in no numbering is counted as impossible, and the judges read the text itself to say whether it says what the answer claims.
- How the matter stands. Each question carries a key: how the matter stands among the schools and traditions, and what a sound answer contains. The judges place each answer against it (agrees, partly, contradicts), note its factual errors, and read how it presents the matter (settled, majority, disputed) against the question's standing.
- Conduct. Whether an answer refuses; whether it sends the reader to an unnamed authority; whether it disparages a school, a people, a religion or the reader, or generalises about a people; whether it preaches unasked; and, by rule, whether it states a penalty in violent words.
- Against itself. Whether a model's three answers to the same question agree.
Three judges from three companies read every answer, and a judge never reads a model of its own company, so a model from one of those companies is read by the other two, and a verdict on it needs both. A verdict counts only when at least two judges give it and they are the majority; every split is published. Each figure is shown with its count and its 95% interval, and never folded into one score.
The citation check has a control. A check that only said yes would pass every citation, so it is tested on planted answers whose citations we know: verses and hadith cited for what they say, the same cited for what they do not say, and references that exist in no numbering. The same rule and the same three judges read them as read a round.
Controlled (0.1). The first rounds gave every model the same passages and method, and measured how faithfully each used them. They stay published as they ran.
Human review
The measures above are made by machine: rules that look every citation up in the texts, and model judges from other companies. No human expert has reviewed these results yet, and the benchmark stays open to that review. If you have the knowledge to judge a round, a question or a single answer, you are welcome to review it below. Reviews from qualified reviewers are published here, with their consent, beside the results they review. Every review is read, and we are grateful for each one.
Thank you. Your review has reached us, and we will read it. If you are qualified and allowed us to, we will publish it here.
What it costs, and who pays
Every answer is written by a model at its listed price, and each round's cost is shown above, to the cent. ProlifAI pays for AICAB's rounds. No AI company, no group for or against AI, and no government funds or directs this work.
We also run benchmarks on behalf of others. An AICAB round on questions someone chooses, or on their own model or agent, costs one price for everyone: what the models and the judges charge at their listed prices, converted from US dollars at the day's rate, plus 30% for our work. Checking the keys, or writing them, is included; work we cannot do within that carries a fee agreed first. We state the figure before it runs; how much of it is published is the buyer's choice: everything, the figures only, or nothing.
A benchmark on another subject, with the same method, under our name or your own and published as far as you choose, can be commissioned on its own page.