Practice Time: an app that listens to a child read
A learning app where a four-year-old says a word out loud and an AI model decides whether they got close enough. The hard part was not the code — it was deciding what counts as correct.
This one started as a weekday routine on a fridge door and turned into the most interesting thing I’ve built at home, because it forced me to think hard about a question I don’t usually get asked at work: what should the model consider correct?
It’s a React app that runs a short practice session — phonics, letters, numbers, some Hindi. A child taps a microphone, says a word, and gets a star or another go. Under it are two multimodal calls to Gemini 2.5 Flash through Firebase AI Logic, one for audio and one for handwriting, and no backend of mine at all.
(It’s live, but everything is behind sign-in — a signed-out visitor gets the landing page and stops there. A child’s week isn’t something to leave lying around, and nothing below depends on seeing it.)
The loop
the speech loop from microphone capture through Gemini evaluation to spoken feedback
Nothing exotic. getUserMedia for the microphone, MediaRecorder to capture, a FileReader to get base64, and the blob goes into a generateContent call as inlineData alongside the prompt. The handwriting exercise is the identical shape with a canvas and a JPEG in place of the audio.
Two settings do the heavy lifting. temperature: 0, because this is a judgement and I want the same answer for the same audio. And responseMimeType: "application/json", because the alternative is parsing prose, and prose from a model that has decided to be encouraging is not parseable.
The prompt is the marking scheme
This is where it stopped being an engineering problem.
The first version asked, reasonably, whether the child had pronounced the word correctly. It was brutal. A four-year-old saying a Hindi phrase in a British-Indian accent, at speed, into a laptop microphone, is not going to satisfy a model’s idea of correct — and every “not quite” landed on a small person who had, in fact, tried.
The fix was to write down what I actually wanted marked:
Since they are 4 years old, accept heavily accented, partial, or imperfect toddler pronunciations as correct.
That single line is the product. Everything else is plumbing.
There’s a second line I’d defend just as hard — telling the model it is an audio analyser and must analyse the file. Without it, models sometimes decline politely: they explain they cannot hear audio, in the middle of a session where they demonstrably can. To a child that isn’t an error message, it’s the app not working. Naming the role kills it.
I’ve written a lot of enterprise prompts. This one taught me more than most, because the cost of a false negative was sitting next to me and could see my face.
Speech out, and the fallback
The app talks back. If an ElevenLabs key has been entered it uses that, because the voice is dramatically better for a child who is listening for a model to copy. If there’s no key, or the call fails, it falls through to the browser’s own speech synthesis, which is free, offline and adequate.
The key isn’t in the code. It’s typed in once and synced through Firestore, so the same key works on the tablet and the laptop without being pasted twice. That’s a convenience decision with a trade-off — it’s stored as a value in a document rather than in something built for secrets — and for a household app with one user I took it. In anything shared I wouldn’t.
The fallback is the part I’d keep in a work system. A paid, better-sounding dependency that degrades to a free, worse-sounding one is a much better shape than a paid dependency that degrades to silence. Silence is indistinguishable from broken.
What surprised me
The model is a better judge of a four-year-old than I am. Not because it hears better, but because it is consistent, and I am not. I get impatient at the fourth attempt at the same word; it doesn’t. At temperature: 0 it gives the same verdict to the same sound every time, and the fairness of that is a thing a child notices very quickly.
I did not expect to learn something about evaluation design from a phonics app for a pre-schooler. That’s most of why I’m writing it down.