Case study · From the Pluris libraryReceipt no. 0164
What sucked
Zapier's LLM-powered spec-generation system automatically produced OpenAPI specifications from API documentation, but its performance was opaque.
The team could not measure accuracy, quantify the impact of pipeline changes, or identify where to prioritize improvements.
What they built
Fractional AI built a comprehensive LLM evaluation framework for Zapier's spec-generation system, with seven distinct metrics measuring correctness and hallucination rates across endpoint paths and field properties. Running on Braintrust against curated, manually reviewed test data, it enabled iterative experimentation to tune prompts, context passing, and model selection across the four-step pipeline.
What changed
Endpoint hallucinations 26% to <1%
Hallucinated endpoint paths dropped from 26% to less than 1%.
Field type accuracy ~2x
Field type accuracy improved nearly 2x on the FieldTyper scorer metric.
~25% cost reduction per run
Prompt caching cut cost by roughly 25% (about $6) per run.
More fixes like this
- How Jewelbench Cut Jewelry CAD Modeling Time by 95%Ideas2IT Technologies · Manufacturing & Industrial
- How Superintelligent Interviewed 150 Staff for $500 With AIFractional AI · Technology & Software
- How Walmart Cut 18 Weeks From Fashion Design-to-ShelfHuman Machines Group LLC · Retail & E-Commerce
- How a Legal Tech Firm Won Back 7-Figure ARRBonsai Labs · Legal & Compliance