ScoreFluentHow we test our scoring and our questions
An AI score is only useful if it is consistent and close to what an examiner would give. This page explains how we choose the models behind ScoreFluent, how we test them before we change anything, how we set question difficulty, and how your feedback changes what we do. Where we have numbers, we give them, along with what they were measured against.
The right model for each job
No single AI model is best at everything. We use a different model for each task and choose each one by testing it on real answers.
| What's scored | Model | What it does |
|---|---|---|
| Writing (every exam) | Claude Sonnet 5.5 (Anthropic) | Marks each criterion of the exam’s rubric and writes your feedback. Spelling and form checks are rule-based, not AI. |
| Speaking: pronunciation, fluency, rhythm | Microsoft Azure Pronunciation Assessment | A dedicated speech-assessment model that scores sounds and timing. |
| Speaking: content | Claude Haiku 5.5 for the quick score, Claude Sonnet 5.5 for detailed feedback | Judges whether you actually answered the task. |
| Reading and listening | Fixed answer keys | Marked against the key. No AI judgement. |
| Checking new questions | A separate Claude model | Checks fill-in-the-blank items have one correct answer before they go live. |
| Live speaking practice | Deepgram (speech recognition) and Google Gemini (conversation) | Chosen for fast responses in real-time conversation. |
How we compare models before switching
When a new model comes out, we don't switch on a hunch. Every candidate goes through three checks:
- Replay real answers. We rescore recent student answers with both the current model and the candidate, and compare them criterion by criterion. Our last writing comparison (October 2026) used 164 answers, and our last speaking comparison used 261.
- A blind second opinion. A third, stronger model (Claude Opus) reviews both sets of marks and feedback without knowing which model wrote which. We treat it as a second opinion, not the final word.
- A fixed reference set. Every candidate is also scored on a frozen set of reference answers whose marks never change. The switch fails if exact agreement drops by more than 3 percentage points, if average error on any criterion rises by more than 0.15 of a band, or if average error on the overall score rises by more than 2.5 points.
Cost matters too, because cheaper scoring keeps practice affordable, but it never overrides these checks. In October 2026 we moved writing scoring to Claude Sonnet 5.5. It matched the previous model within one band on 99.4% of criterion scores across 159 answers, and the blind review preferred its feedback 80% of the time. The same month we kept an older model for generating one LanguageCert question type, because the newer model produced fewer valid questions.
Testing against real examiner marks
Where an exam board publishes examiner-marked answers, we test against them directly. For Cambridge C1 Advanced writing, we calibrated our scorer against 17 official examiner-marked scripts. Our criterion scores are now within one band of the examiner's mark 92.7% of the time, up from 75%.
The honest limit: published examiner-marked scripts are scarce, and 17 is a small sample. For most exams, our reference marks are set by us against the official rubric, not by the exam board's examiners. That's why every ScoreFluent score is an estimate. How ScoreFluent marks answers explains what that means for you.
How we set question difficulty
A practice question only helps if it feels like the real thing. If it's too easy you feel readier than you are, and if it's too hard you lose confidence.
- For PTE, we measured 110 official practice items, task type by task type. We measured length, reading grade level, sentence length, how much of the vocabulary is common English, and the spread of topics. From that we set a target range for every task type.
- The same ranges are used twice: once when a question is written, and again by an automatic check before it goes live. A question outside its range is rewritten and checked again. If it fails twice, it's retired.
- We never rewrite a question you've already answered, so your past scores stay comparable.
- Before any question goes live, we check that it has everything it needs: audio, images, model answers and explanations.
We started with PTE and are working through our existing questions task type by task type. New PTE questions must pass the check before they go live.
How your feedback changes scoring
When you tell us something looks wrong, through the feedback prompts or by telling Flo, our tutor, the report reaches our team directly. A person reads it, and a person decides what changes; nothing adjusts automatically. Some changes feedback has led to:
- A teacher showed us that flawless Read Aloud and Repeat Sentence answers were landing just short of the top score. We changed how the top band is scored for those tasks.
- Students told us some PTE questions felt longer or harder than the real exam. That started the difficulty calibration described above.
- An institute told us our Repeat Sentence items were too long. We now keep them to 5–10 words.
Use a score to decide what to practise next.
Start with a real task from the exam you are preparing for.