NEW YORK — What happened
Systems engineers released an extensive telemetry audit examining how major foundation models handle unstructured requests mapped to strict JSON-based API schemas. The dataset captured millions of production tool calls from financial and logistics deployments.
Why it matters
Agents are only as useful as their ability to reliably execute downstream actions. If a model generates a malformed payload or hallucinates an optional parameter in a financial transaction API, the financial and operational consequences can be severe.
Deep dive
The research isolated three primary failure modes: schema drift misinterpretation, forced parameter hallucination where models invent arguments not present in the OpenAPI definition, and cascading logic errors in multi-tool composition. Interestingly, larger parameter counts did not correlate with better schema adherence; models fine-tuned specifically on synthetic syntax trees performed best.
Report check
Public marketing materials suggest that native tool-use is virtually error-free. The audit verified that while syntax error rates have dropped below 2 percent, semantic parameter validity—meaning the arguments make logical sense within the user's intent—remains problematic.
Open questions
Can runtime verification layers catch semantic hallucinations before API execution, or must fine-tuning paradigms shift entirely?