Nilforoshan: OpenAI's Decisions API cost 2x, scored worse
Hamed Nilforoshan ran OpenAI's new Decisions API against Jev on resume to job relevance for a service serving 2.5 million users, and says OpenAI came out 2x more expensive and 5 to 10% worse. vogel's own head to head found the opposite on speed.
Someone measured it
vogel put OpenAI's new Decisions API head to head with Jev and Clef
OpenAI put its Decisions API into public beta for all developers, and vogel immediately ran it against Jev and Clef, reporting that it is fast. OpenAI's own claim is decisions up to 10x faster than GPT-6 Luna through the Responses API.
OpenAI just released their Decisions API so I put it head to head with Jev & Clef
Matthew Berman ran Jev over 193,355 video hooks for $0
Matthew Berman had Jev classify 193,355 video hooks and says it cost him nothing. Each hook got three choices: what's in it for me, why do i care, what do i see first, and he reports the best combination breaks out 33.1% of the time against 9.1% for the worst. The write-up is a free PDF.
The case against
Nilforoshan says Decisions API cost 2x and scored 5 to 10% worse than Jev
Hamed Nilforoshan benchmarked the "Jev-killer" Decisions API against Jev on predicting how relevant a user query or resume is to a job description, scored 1 to 10, for a product he says serves 2.5 million users. His result: OpenAI is 2x more expensive and 5 to 10% worse.
OpenAI is 2x more expensive and 5-10% worse