How to evaluate a Mandarin transcription sample before signing

Why sample evaluation is worth half a day

For decision-grade recordings, the sample you see at signing usually sets the ceiling of what you will receive in production. Samples are also the easiest place for a vendor to do "showcase work" — they will send you something they have polished repeatedly, not their daily baseline. Spending half a day evaluating samples against an executable checklist is the highest-ROI due diligence you can do before signing.

"Reads smoothly" is the biggest trap

The easiest thing for a human reviewer to do with a misrecognized Chinese sentence is to make it read smoothly without making it correct. "Smooth" is rewarding to read but a negative signal for a decision-grade transcript — it means the reviewer chose readability over fidelity. When evaluating a sample, the first question is not whether it reads well, but whether it maps back to the audio at the moment.

Five risk axes for sample evaluation

Break sample evaluation into five risk axes, each with an executable judgment point. The five below cover the main risk surface of decision-grade transcription. Score each sample on each axis independently (a rough 1–5 is enough):

• Terminology error recognition — were industry terms, acronyms, and proper nouns identified and corrected, or smoothed over.
• Mixed-English handling — were English terms preserved as the speaker actually used them, and is the mixed-language formatting readable.
• Number and unit verification — were amounts, percentages, currencies, and dates checked back against the audio.
• Speaker attribution — in multi-speaker audio, is it clear who said what.
• Uncertainty markers — when audio or terminology is uncertain, are uncertain points marked explicitly, or filled in regardless.

Axis one: terminology error recognition vs. cover-up

In terminology-heavy recordings, ASR errors are rarely a single character off — they are often a completely unrelated word. A strong sample shows recognition judgment at every key terminology position, not just at positions where the original ASR read awkwardly.

Axis two: real handling of mixed English

Mixed-English samples are the most direct window into a vendor's bilingual judgment. The question is not just "was English kept" — it is whether English terms were accurately identified at the ASR stage (and not heard as a Chinese near-homophone), and whether the final formatting is readable for an analyst.

Axis three: number and speaker verification

Numbers and speakers are the two positions in a decision-grade transcript that absolutely cannot be wrong, as the speaker-and-number FAQ discusses. A recording that turns "1.32 billion yuan" into "3.2 billion yuan" pushes downstream commercial judgment off by an order of magnitude. A good sample shows visible cross-check work at both positions.

Axis four: how uncertainty is handled

When audio is uneven, speakers overlap, or terminology is unusual enough that a confident determination is impossible, decision-grade transcription should mark the position explicitly (for example "[?]", "[inaudible]", or "[TBC: XX]") rather than fill in something that looks plausible. Filling silently transfers the judgment responsibility to the reader.

Axis five: re-run the test on your own recording

The stability of an anonymized sample and the stability of real production work are often not the same thing. The single most valuable evaluation step is to submit a real internal recording of your own as a trial — same audio quality, same vocabulary, same context. A vendor scoring 95 on the curated sample and 70 on your real audio is the gap worth identifying.

How to compare vendors head-to-head

The hardest part of evaluating a single vendor is the lack of a baseline. Submit the same recording to two or three vendors for trial transcription and compare their handling at the same positions. Head-to-head comparison makes "this looks decent" suddenly look like a gap.

A few hard red flags

Any one of the following sample characteristics is reason to drop the vendor without spending more comparison time:

• Cannot provide the ASR draft alongside the human-reviewed final — review process is not transparent.
• All numbers in the sample match the ASR draft exactly — number positions likely were not verified.
• No uncertainty markers anywhere — the reviewer is filling silently to hide judgment gaps.
• Obvious mixed-English near-homophone errors left in place — bilingual judgment is insufficient.
• Refuses to provide a trial, or insists on a high trial fee — usually a lack of confidence in real-world quality.

Sample evaluation checklist

Distill the whole guide into a single scorecard. Score each sample on each item independently (a rough 1–5 is enough):

• Does the vendor provide ASR-draft / human-reviewed comparison?
• Are terminology errors in the final identified and corrected (not smoothed over)?
• Are mixed-English terms preserved as the speaker actually used them?
• Is there visible verification on numbers, units, and dates?
• Are speaker transitions clear and is Q&A structure restored?
• Are uncertain positions marked explicitly?
• Does the work on your own submitted trial match the quality of the curated sample?
• In head-to-head comparison, is the handling at the same positions more restrained and trustworthy than competitors?
• Are the vendor's responses to your term list / format preferences specific and executable?
• Overall trustworthiness — could someone who was not in the room continue working from this document?