Application · Developer Relations Engineer, Developer Advocate
I found two bugs in Sarvam’s code and fixed both before asking you for anything.
One is merged into sarvamai/skills. The other is open on the AI SDK provider, where streamText streams text to the screen and then hands back an empty string.
I found both while building kya bola, a benchmark I built that measures your speech API across all 64 languages in Project Vaani, district by district, rather than the 19 that are officially supported.
Nobody asked for any of it. That is roughly the point.
Every text-to-speech example in the skills repo raised a TypeError
Agents install these skills so they generate Sarvam calls with the right signatures. Which means a stale parameter name does not produce a typo. It produces working-looking code that fails on the first call, generated at scale, by machines that trust the file.
text-to-speech: use language_code for TTS convert and convert_stream
Agents install this skill so they generate Sarvam calls with the right signatures. All four text-to-speech examples on the page passed target_language_code. That parameter was renamed to language_code in sarvamai 0.1.29 for Python and 1.1.8 for JavaScript, both shipped on 3 August 2026. Every sample raised a TypeError on the first call, on both SDKs.
The rename is hard in both directions. 0.1.28 accepts only the old name, 0.1.29 and later only the new one, and the JSON body key changed with it. There is no alias, so one side always breaks. Rather than fork the skill per version, the callout names the SDK floor and the old parameter, which is enough for an agent to recover when it hits the error on a pinned project.
The AI SDK provider streams text correctly and then returns an empty string
This one is thirteen lines. It is also the more expensive bug, because it passes every smoke test and only shows up once somebody builds a real chat on top of it.
chat: emit text-start/text-end so streamText records the assistant message
const result = streamText({
model: sarvam("sarvam-105b"),
prompt: "Count 1 to 5, digits only.",
});
for await (const delta of result.textStream)
process.stdout.write(delta); // "\n12345"
await result.text // ""
await result.content // []
(await result.response).messages // []doStream emits text-delta without ever emitting text-start. The AI SDK keeps an id-keyed map of open text blocks and records a delta only if its id is already in that map. Nothing but text-start puts it there. So every delta is dropped and replaced with an error part, and the finished result never sees a character.
// ai/dist/index.mjs
if (part.type === "text-delta") {
const activeText = activeTextContent[part.id];
if (activeText == null) {
controller.enqueue({
part: { type: "error", error: `text part ${part.id} not found` },
});
return;
}
activeText.text += part.text;
}response.messages is empty, so an assistant turn cannot be appended to history. A useChat message carries no text part, so the bubble renders blank. doGenerate was already correct, because it builds content directly and never goes through the stream parts.
| before | after | |
|---|---|---|
| result.text | "" | "\n12345" |
| result.content | [] | ["text"] |
| response.messages | [] | [assistant: text] |
| fullStream error parts | 6 | 0 |
| useChat message parts | [] | [text] |
| tool calls, generateText | ok | unchanged |
India speaks 64 languages in this dataset. Speech APIs support 19.
Every speech vendor publishes one accuracy number per language. Hindi in Delhi and Hindi in Araria are not the same problem, and one number hides the difference. Underneath that there is a second gap: the Vaani transcribed corpus holds 64 languages, Sarvam's speech API documents 23, and the overlap is 19. Nobody has published what happens to the other 45.
"kya bola?" is what you ask when you did not catch it. Also, apparently, what most speech APIs are thinking once you leave the metros.
Mine, start to finish, across 29 commits. I wrote the shard-walking sampler, the district crosswalk, the Indic normalization rules, the three provider adapters, the scoring and the map. Both bugs above turned up while building it, which is the only reason I found them.
The support list barely predicts anything
I sent the 45 unsupported languages anyway, with language detection on, because that is what a developer building for those speakers would do. Rates above 100% are not a bug. Word error rate counts insertions, so a model returning more words than were spoken can exceed the length of the reference. It means the model is not declining. It is confidently producing fluent text in the wrong language.
| language | on the support list | word error rate |
|---|---|---|
| Tagin | no | 122.7% |
| Nyishi | no | 120.2% |
| Sumi | no | 116.2% |
| Kokborok | no | 107.9% |
| Santali | yes | 84.2% |
| Kashmiri | yes | 35.7% |
| Bundeli | no | 25.3% |
| Khariboli | no | 18.0% |
| Magahi | no | 16.8% |
Three systems, and the open one holds up
On the 19 languages all three support. Accent marks the winner in each row.
| language | IndicConformer | Saaras v3 | Saaras v4 |
|---|---|---|---|
| Hindi | 19.5% | 22.2% | 21.0% |
| Urdu | 36.5% | 48.8% | 47.8% |
| Manipuri | 21.4% | 26.7% | 26.7% |
| Tamil | 27.0% | 32.0% | 22.7% |
| Odia | 56.3% | 60.5% | 41.9% |
| Santali | 83.2% | 84.2% | 58.6% |
Why you should believe any of it
The Vaani team already benchmarked 21 ASR systems on their own Hindi evaluation set across 104 districts, in arXiv 2606.21408. The Hindi district map is theirs and repeating it would add nothing. This extends the same idea to the other 63 languages, makes it reproducible, and keeps it current as new models ship.
| metric | this harness | published | delta |
|---|---|---|---|
| approach 1 | 18.7% | 20.3% | −1.6pt |
| approach 2 | 15.9% | 16.9% | −1.0pt |
| district mean WER | 17.6% | 18.3% | −0.7pt |
| human v1 vs v2 WER | 9.4% | 10–15% (stated) | matches |
Three things in there that are useful to you
Regardless of what happens with a role.
- 01
Kashmiri measures 63.9% WER with standard normalization and 35.7% once Arabic short-vowel marks are stripped from both sides. Anyone benchmarking Sarvam's Kashmiri without that step is reporting a number roughly 28 points worse than reality.
- 02
For the 45 unsupported languages, auto-detect returns confident output in the wrong language rather than declining. Several score above 100% WER, which means insertions. A "we do not support this" signal would be more useful to a developer than fluent Hindi for a Tagin speaker.
- 03
The published support list barely predicts whether a language works. Magahi is not on it and scores 16.8%. Santali is on it and scores 84.2%. The supported and unsupported distributions overlap heavily.
Five ways I nearly published something false
Each of these gave me numbers that looked completely reasonable and were wrong. Three of the five made the story better, which is exactly why I would have kept them. Each is now pinned by a test.
- 01
Deleting the vowels
16.6% → 7.69%The Whisper text normalizer most evaluation code copies strips Unicode categories M, S and P together. In every Brahmic script the vowel signs and the virama are category M. Applied to Indic text it deletes the vowels from both sides, so error rates improve. A documented case takes Malayalam from 16.6% to 7.69% purely through this corruption. The score gets better because the comparison gets emptier.
- 02
Scoring the annotations
+18.4 pointsVaani transcripts carry transcriber markup: paired <noise> and <pause> tags in 56.3% of them, {english} glosses in 35.5%, [event] notes in 10.1%, truncation markers in 15.3%. Scored raw, that inflated word error rate by 18.4 points. The tags wrap real speech, so stripping tag and content would have emptied the reference for more than half the Garo clips. The braces are the opposite case: साइड {side} is one spoken word written twice, so the brace group has to go entirely.
- 03
The same bug wearing a disguise
9.4% → 32.9%The benchmark set writes <static noise>, with a space in the tag name. The pattern assumed one word, so "static noise" stayed in the reference as two words no model could ever say. That alone moved measured human-versus-human disagreement from 9.4% to 32.9%, which would have destroyed the one number every other number is judged against.
- 04
Counting an empty wallet as a bad model
a fake 91.3%A run against a second provider ate its quota after 11 clips. The other 70 came back empty and scored as total failures, producing a very tidy 91.3% error rate that described a billing status rather than a model. Refusals and infrastructure failures now go in different buckets: an API that rejects an unsupported language has genuinely failed the clip, an API out of credit never got to try, and the run aborts rather than quietly averaging it in.
- 05
Counting diacritic order as errors
63.9% → 35.7%Kashmiri looked broken at 63.9%, so I opened the transcripts. Reference چُھ, hypothesis چھُ. Same letters, same word, different order for an optional vowel mark that Unicode does not reorder. Kashmiri is 35.7%. I was one commit from publishing a claim about a vendor's Kashmiri support that was off by 28 points.
What the first ninety days look like
The posting says DevRel here is part engineering, part content, part community, and that feedback is one of the most important outputs rather than a side effect. This is that job, written as a plan.
- Weeks 1–2
Finish what I started
Land the open SDK PR and push the three verified fix branches still sitting on my machine: the generateObject schema conflict, the usage and reasoning field name mismatches, and streamObject. Then run the AI SDK integration suite against the provider on a schedule so the next one of these is caught by CI rather than by a developer in production.
- Weeks 3–6
A cookbook that actually runs
Every recipe with an Open in Colab notebook a stranger can run in ninety seconds, no local setup: a voice agent on LiveKit, multilingual RAG over Indic documents, doc digitisation end to end, and an ASR evaluation notebook built from the kya bola harness. Each one gets a test that runs it, so no recipe rots quietly after an SDK release.
- Weeks 3–6
Instrument the developer journey
The API error stream is the highest-signal feedback channel in the company and nobody reads it as content. Group the top failure modes by frequency, work out which are docs bugs and which are API bugs, and route them: docs bugs I fix, API bugs go to engineering with a reproduction attached. Publish the top ten as a Common Errors page, because the fix and the docs page are the same artifact.
- Weeks 7–12
Make the language coverage honest
Turn kya bola into a public, continuously updated scorecard that reruns on every model release, so the support list and the observed behaviour stop being two different things. Published with the human floor drawn on it and the failure modes named. A vendor that publishes its own weak districts is more trusted than one that publishes a single national number.
- Ongoing
Write the thing I needed
The fifteen-part course I wrote to learn this material exists because nothing explained what every number in a speech benchmark means and where it lies. That is the writing I would do here: not launch posts, but the reference a developer lands on at 1am when their WER looks wrong.
One more thing
This measures Sarvam's product and finds places it is beaten by a free MIT-licensed model. I would rather you see that before anyone else does. If something in it is wrong I want to fix it before it is public, and if any of it is useful, take it, regardless of what happens with a role.
I am applying for the Developer Advocate role. The fastest way to evaluate me is to read the merged PR and then the open one, in that order.
Applying to Developer Relations Engineer, Developer Advocate. Built on Project Vaani by ARTPARK and IISc, used under CC-BY-4.0. Benchmark methodology follows Pulikodan et al., Vaani Benchmark V1.0, arXiv 2606.21408. Boundary data is geoBoundaries under ODbL 1.0.