Context
Jev is TypeSafe’s first public System One Model, designed for fast, structured decision-making inside software. Its positioning focuses on low latency, schema-constrained outputs, and calibrated probabilities for tasks such as classification, routing, and risk decisions.
Those promises are interesting for a monitoring product like MyMentions. Before using a new model in production, we wanted to verify whether Jev could reliably answer a practical question:
Is this post related to Glints as a job, recruitment, or career platform brand in Indonesia?
We compared Jev with the existing OpenAI-based model used in the application.
Methodology
We used a sample of 80 X posts mentioning “Glints” from a completed scrape. Both models received the exact same posts.
The classification task had two possible outputs:
related: the post is about Glints job listings, applications, hiring, careers, job-seeker support, or a user experience with the Glints platform in Indonesia.unrelated: “Glints” refers to light or sparkle, another topic or brand, or the post does not provide enough evidence of a connection to the Indonesian job platform.
The comparison used:
- OpenAI
gpt-4o-mini, using the application’s existing API pattern and structured JSON output. - Jev
jev-latest, using TypeSafe’sPOST <https://api.typesafe.ai/v1/systemone> endpoint. - The same brand, platform, handle, and post-text context for both providers.
- Five concurrent requests and up to two retries per request.
- Token usage returned by each provider to calculate cost.
For cost calculation, the benchmark used the documented Jev rate of $0.042 per million input tokens, with output tokens free. OpenAI was calculated using the configured benchmark rates of $0.15 per million input tokens and $0.60 per million output tokens.
The benchmark did not include independent human labels. Therefore, agreement is used as the primary comparison metric, but it should not be interpreted as formal accuracy.
Results
The models produced the same classification for 77 of 80 posts, giving an agreement rate of 96.25%.
There were only three disagreements. In every case, OpenAI classified the post as related, while Jev classified it as unrelated. All three posts contained wording such as “Glints in diamond eyes”, where “glints” referred to sparkle or light rather than the Glints job platform. A manual review of those three cases favored Jev’s classification.
Jev was 42.6% cheaper for this sample. The total savings were $0.00182588 across 80 posts. At the observed rate, 100 posts would cost approximately $0.00535 with OpenAI and $0.00307 with Jev.
Jev was also faster, with an average latency of 422 ms compared with 727 ms for OpenAI. This is approximately 1.72x faster on average. The P95 difference was smaller: 884 ms for Jev versus 1,006 ms for OpenAI.
An important detail is that Jev used more input tokens for this request format. Its lower cost came from its substantially lower input-token price and free output tokens, not from using fewer tokens.
Summary
For this specific classification workload, the initial benchmark supports two of Jev’s key claims: lower cost and lower average latency, while maintaining high agreement with the existing classifier.
The 96.25% agreement rate is encouraging, and the three observed disagreements were cases where Jev appeared better at distinguishing the Glints brand from the ordinary meaning of the word “glints”.
Documentation: