Can we really trust AI decisions? Inspect smarter and cheaper with 'jevals'

Imagine your trusted AI assistant is writing important emails or answering customer inquiries in real-time. How can you ensure the AI is making the right decisions and not making mistakes or outputting sensitive security information?

Until now, we had to use another smart AI (an LLM judge) to act as a “judge” and inspect every action one by one. It was like submitting every report to a judge and waiting for a verdict. However, this process is incredibly slow and expensive. Consequently, most companies barely manage to verify a small fraction of their total operations. Source: Openlayer releases jevals — a fraction-of-a-cent replacement for LLM judges

A new tool, jevals, has arrived to solve these concerns. Source: Show HN: jevals – replacing LLM judges with typed Jev decisions

Why does this matter?

As AI agents become more deeply embedded in our lives, the process of verifying that they are making the right decisions becomes increasingly important. It has evolved beyond simply using high-performance models into an “safety belt” to prevent those models from making mistakes.

Relying on spot-checking a tiny fraction of total tasks makes it difficult to catch all the potential catastrophic errors that could occur later. Because jevals is fast and cheap enough to verify every task, it allows systems to detect mistakes and protect users before an AI acts on them. Source: GitHub - openlayer-ai/jevals: Agent evals and guardrails as …

Understanding jevals: From ‘Judge’ to ‘Checklist’

Let’s rethink the traditional ‘LLM Judge’ approach. It is like sending a long-form essay to a judge detailing an AI’s behavior and asking them to “decide if this is correct.” Because the judge has to read and interpret the text, it is slow and costly. Source: Introducing jevals: typed decisions for agent evals and …

In contrast, the ‘Jev’ model used by jevals is a ‘System One’ model. Simply put, instead of reading long sentences and pondering, it answers predefined questions with a simple “Yes/No” or a numerical value. To use an analogy: instead of a human reading and grading long-form answers, it is like a machine grading an OMR sheet for a multiple-choice test in 0.5 seconds. Source: Jevals· independent Jev benchmark for models that decide

By ‘typing’ the answers and processing them all at once, it is much faster and dramatically cheaper. It processes up to nine inspection items in a single communication while incurring a very low cost of only $0.00006 per trace (unit of work). Source: Openlayer Releases jevals: AI Agent Evaluation for a Fraction …

Current Status: Fast, but occasionally confused

jevals is highly efficient. You can run more than 40 evaluation questions per AI task, and the total cost remains negligible. Source: jevals: An Evals Platform Built Around Typed Judges

However, not everything in the world is a multiple-choice question. The Jev model is not 100% perfect in situations requiring the understanding of complex and subtle conversational context. While it boasts high accuracy of over 96% for simple ‘Yes/No’ decisions, its accuracy can drop to the 80% range for problems where various rules are complexly intertwined. Source: We Tried Replacing Our LLM Judges With Jev - DEV Community

Interestingly, the tool provides a ‘probability’ score when it is unsure of its answer. It honestly states, “I am only about 60% confident in this judgment.” Developers can use this probability information to design smarter systems that only ask a more intelligent (and expensive) LLM when the initial answer is deemed ambiguous. Source: We Tried Replacing Our LLM Judges With Jev - DEV Community

What lies ahead?

MindTickleBytes AI Reporter’s Take

While the performance of AI agents is important, it is impressive to see that the method for verifying their decisions is changing from a “high-level judicial trial” to “efficient inspection.” Ultimately, as much as AI becomes smarter, the technology that quickly catches AI errors will make the AI era safer and more trustworthy.

References

1. [jevals- AI Agent Eval Library Python

   EveryDev.ai](https://www.everydev.ai/tools/jevals)

2. Jevals· independent Jev benchmark for models that decide

3. [jevals

   System One Models](https://systemonemodels.org/examples/tools/jevals/)

4. Show HN: jevals – replacing LLM judges with typed Jev decisions
5. GitHub - openlayer-ai/jevals: Agent evals and guardrails as …
6. Introducing jevals: typed decisions for agent evals and …
7. We Tried Replacing Our LLM Judges With Jev - DEV Community
8. jevals: An Evals Platform Built Around Typed Judges
9. Openlayer releases jevals — a fraction-of-a-cent replacement for LLM judges
10. Openlayer Releases jevals: AI Agent Evaluation for a Fraction …