Skip to content
Reallang

How a band is decided, step by step

Sophie Lindqvist

Speaking examiner, university language center13 minPortland, OR

(How scoring works)

Two passes, three runs, and every quote checked against your own transcript before any of it reaches you. What the grader is allowed to say, and what it is made to leave blank.

Aug 17, 20269 minutes

Pass one finds evidence, not opinions

The first pass reads your transcript and pulls out items: a turn number, a quote, and whether it helps or hurts against one criterion. It does not assign anything. Its only job is to find the sentences that bear on the rubric.

Separating this from the judgement is the whole design. A single pass that reads a transcript and produces a number has no auditable step in it, the reasoning and the conclusion arrive together and neither can be checked against the other.

Every quote is verified on the server

Before you see anything, the server checks each quote is an exact substring of that turn of your transcript, after normalising whitespace and case. A quote that is not is retried once and then dropped.

This is the line that matters: the model does not get to paraphrase you into agreeing with it. If a claim about your answer cannot point at your words, it does not appear. The share that gets dropped is tracked, because a rising drop rate is the earliest sign a grader has started inventing.

Pass two bands, three times

The second pass sees only the verified evidence and the published descriptors, and assigns a band per criterion. It runs three times and the median is kept, same evidence, same rubric version, same model snapshot, with only the sampling varying.

The weighted total is then computed in code from the weights on the page. It is not asked for, because a model asked to do arithmetic over its own judgements will produce a plausible number rather than the right one.

What it refuses to do

A criterion with no evidence comes back as insufficient evidence rather than as a three, and the total is computed over what was scored rather than over what was not. A short session leaves more of it blank, which is correct.

Five things are pinned when a session starts, persona, scenario, rubric, grader and model snapshot, so any band can be reproduced exactly. And the interface will not draw a change between two sessions smaller than the measured noise, because a movement you cannot distinguish from re-rolling the dice is not progress.

Who this is about

Sophie Lindqvist

Speaking examiner, university language center

Sophie examines at a university language center and spends the rest of her week teaching, which shows: she is the examiner people wish they had got. She follows the format to the second and never helps, but her face is open and her 'why do you think that?' sounds like she wants to know. Start with her if the format scares you. She is the same test with the sharp edges filed down, the same silence after you stop, the same timing, the same questions, delivered by someone who looks pleased you are there. Once you can fill two minutes with her, move to James. She grew up in Portland, spent a year in Seoul, and keeps a list of the most surprising things candidates have told her about their hometowns.

How long the room runs
13 min
Rated by 0
Not yet rated
On this shelf
1 guide
Sophie Lindqvist, speaking examiner, university language center

Keep reading

Three more, and the personas who wrote them