TL;DR
AI launches often do more than introduce a technology. They establish what counts as progress, then demonstrate progress on those terms. You can check every result and still get caught in the narrative trap.
Five levels of judgment help you get out: understand the mechanism, test its limits, choose your own criteria, find the best real alternative, and decide whether the problem deserves your attention. For Jev, one alternative is a small local model with direct logit readout and normalization.
Technical fluency can make a launch’s chosen frame more persuasive. Someone who has read the paper and run the demo can become an excellent advocate without having evaluated the right comparison.
At work, a better alternative can make a dissenting judgment easier to hear. Show how it performs on the team’s actual problem; then you have a stronger basis for discussing what deserves attention.
There is a teaching problem here, too. Giving students the simple solution we arrived at is not the same as teaching them how to arrive at it. That distinction is central to our next AI Builders refresh.
Updated September 25, 2026, with the practical argument from my new video on Jev and AI hype (in Chinese): how to make a considered technical judgment useful to your team.
On September 15, TypeSafe introduced Jev, a model designed to return structured decisions and probabilities that software can use directly.
Think of a support ticket. Should it go to billing, technical support, or sales? A program needs a decision it can act on. It does not need a beautifully written answer that someone has to read first.
The launch offers a compelling account of what has been missing from AI automation. Chat models are powerful, but software needs fast, dependable decisions. Jev is designed around that need.
The need is real. The technical distinction is real. But notice how easily your question can change.
You started with: What does my work need?
Now you are asking: How do I use this impressive new thing?
Before you judged the technology, you may already have accepted its definition of “impressive.”
That is the narrative trap. Jev is a useful way to examine it, but the pattern is much older than this launch.
You can verify the facts and still accept the wrong frame
Let me choose the comparison, and I can find something I do better than Elon Musk. Then let me explain why that ability is underrated, why it will matter more in the future, and why everyone is measuring the wrong thing.
I can make myself look extraordinarily impressive without saying anything false.
The problem is the leap from a particular advantage to a much larger conclusion. Why should my chosen dimension decide whom you pay attention to? What happened to the dimensions I left out?
A successful technology launch often performs the same move. It tells you where the industry’s real bottleneck lies, why existing approaches fall short, and what the future requires. Within that account, the new product occupies a particularly attractive position.
Three claims have quietly been bundled together:
It wins on the selected dimension.
That dimension matters to your task.
It is the best way for you to accomplish that task.
Proving the first claim does not prove the other two.
The frame also shapes the future you imagine. ChatGPT is changing white-collar work; surely physical labor is next.
We take the shape of one breakthrough and project it onto a different technical problem, without checking whether the conditions that made the first breakthrough possible exist there.
In my conversation with neuroscientist and Tsinghua University professor Liu Jia, he challenged our intuition about what is hard. The language and explicit reasoning we regard as our highest abilities are comparatively tractable for today’s AI. In his account, the far greater challenge lies in the largely unconscious intelligence behind perception and bodily control—what he described as System 1. Sensing, prediction, and action have to work together in real time. Success at reasoning does not mean we have solved that problem, or that it is a smaller problem to finish next.
Columbia University professor Changxi Zheng reached a complementary conclusion from his work in physical simulation and robotics. He sees ChatGPT as a historically exceptional breakthrough, not a template every domain is destined to repeat. He is skeptical that one model will suddenly make a wide range of physical tasks ready for real production. Instead, he expects a long period of progress through specialization: solving particular tasks reliably, fast enough, and at a cost that makes them worth automating.
Neither argument says embodied AI has no future. They challenge the assumption that its future must follow the shape of language AI’s past. Different bottlenecks may require different breakthroughs. Progress can be substantial without producing the all-purpose moment people are waiting for.
The leap is from “It happened in language” to “It must happen in robotics.” Then “Will this route work?” quietly becomes “How soon?”
Jev invites another extrapolation: AI built for conversation will give way to AI built for machines. But software has been returning values for other software to use for decades. A function returning a number, a classifier selecting a category, and an API returning structured data all serve that purpose. Agentic systems already combine learned judgments with ordinary program execution.
The question is where a different model improves that arrangement. Does it produce better decisions, reduce a meaningful delay, or make a workflow more dependable? Those are useful claims to investigate. Calling it “machine-native” does not establish that the need was previously unmet, or that meeting it requires a new paradigm.
The same thing happens to our view of alternatives. We compare which new service is faster or which framework is more complete. We stop asking whether a different kind of solution would work.
You can read the paper, reproduce the benchmark, and identify exaggerations in the marketing—and still remain inside the launch’s frame.
You have answered its question more carefully. You have not yet decided whether it asked the right question.
Five levels of judgment, using Jev
The first two levels examine what the technology can deliver. The next three examine the terms on which you are evaluating it.
1. Can I make it work?
Read the documentation. Call the API. Send a support ticket and receive a category with probabilities. The demo works.
You now know how to use the product.
But the task, the implementation, and the definition of success may all have come from its examples. You have learned to demonstrate it. You have not yet evaluated it.
2. Where does it stop working?
Replace the tidy example with real tickets. What if information is missing? What if a request spans two departments? When should the system defer to a person?
Look at other people’s experiments. Check whether their results reproduce.
This is substantial progress. Yet even if every reported result holds, you still need to ask why those tasks and comparisons were chosen.
TypeSafe’s own launch discussion says:
The relatively shorter input paints our model in an advantageous light.
That is a candid qualification. It also illustrates the distinction: an honest demonstration is not necessarily a representative one.
3. What should I measure?
If a classification triggers an action in my system, I want to examine at least four things:
Quality: Is the decision correct, and what happens when it is wrong?
Latency: Is this step actually holding up the work?
Consistency: Could an irrelevant change flip the decision?
Calibration: Among decisions assigned 90% confidence, are roughly 90% correct?
The task determines the weights. A live interaction might depend on tens of milliseconds. A nightly batch job might care much more about detecting and recovering from errors.
A latency improvement does not, by itself, establish that latency was the problem worth solving.
Consistency needs the same scrutiny. In Jev’s published insurance example, one decision’s probability ranges from roughly 0.43 to 0.53 across runs with irrelevant identifiers changed. That can be a smaller fluctuation than competing models produce and still cross a program’s 0.5 decision threshold.
“More stable than the alternatives” and “stable enough for this action” are different standards.
This is where engineering judgment starts to matter. Levels three and four require more than diligent reading. You need to understand what makes a system usable—and enough about the mechanism to recognize options that were absent from the comparison.
4. What is the best real alternative?
This is the level people most often skip, and the one most likely to change the conclusion.
Ask: If I do not use this technology, what is the best way to solve the same problem?
Imagine someone telling you that you should choose a partner who cooks well. After all, you need to eat every day. You might immediately begin comparing people’s cooking.
But eating well need not depend on your partner cooking. You could hire someone to prepare meals or eat out. If those options work for you, how much weight should cooking carry in your choice of partner?
Eating matters. It does not follow that a partner’s cooking ability must matter equally. Between those propositions sits an unasked question: What else could meet this need?
Now return to Jev. Your program needs a good decision quickly. Does that require a new cloud service?
An existing open model already computes scores for candidate tokens. For a decision among defined options, you can map the options to answer tokens, read their logits, and normalize the scores—without generating an answer.
SemIf, formerly OpenJev, provides an implementation of that approach. It is an independent project, not a reproduction of Jev’s undisclosed model or training. The important point is that another candidate has entered the comparison.
Now the question is no longer simply which service is better at returning decisions. It is how your system should obtain the decision it needs.
Direct readout does not automatically provide calibrated probabilities. Test it on the same workload, using the same quality, latency, consistency, and calibration requirements. But if it meets those requirements, a local model may already be enough.
And local execution has architectural value. A system making frequent decisions can avoid another network round trip and another dependency on an external service being available.
Jev may offer quality, calibration, or operational convenience that this route cannot match. That would be a reason to use it. The reason has to survive comparison with the best real alternative, not just the alternatives selected for the launch.
5. Does this problem deserve my attention now?
Even after finding a better alternative, one question remains.
Was this decision step a meaningful constraint on your work in the first place?
If your existing system already meets its requirements, evaluating Jev, building a local substitute, and optimizing either one may simply be work that the launch added to your calendar.
You can do that work extremely well without advancing the thing you originally wanted to accomplish.
This is why I do not think most people need to chase Jev. For someone whose problem is already adequately solved, a new implementation is not automatically a new priority.
Nor must you spend days completing all five levels before you are allowed to stop. Once you can see that a release does not change an important constraint, you can move on. If you intend to adopt it or make a detailed technical claim about it, do the corresponding work.
You can understand a technology, build your own version, and decide not to use it.
Why the story spreads—and why challenging it is hard
Some of a narrative’s most persuasive advocates are the people who have carefully completed the first two levels.
They have more than enthusiasm. They have technical details to defend it. That makes them convincing, without necessarily helping them notice what the comparison left out.
The stories reaching us have already passed a selection test
New technologies compete for attention every day. Most never become a topic of conversation. The ones we hear about often fit what people already hope for, worry about, and want to believe about the future.
A successful launch story therefore has more than a technical proposition. It is easy to grasp, satisfying to repeat, and offers the pleasure of seeing what comes next before everyone else.
Return to the comparison with Elon Musk. Choose an advantage, then make that advantage sound like the thing the future will depend on, and you have a compelling story.
Jev returns decision probabilities directly; the comparison models generate structured results. That comparison can reveal the advantage of avoiding generation. It does not include every reasonable alternative, such as using a local model without generating answer text either. But as the story travels, the conditions can fall away. What remains is a much larger claim: previous models were wrong for programmatic decisions; now someone has finally built the right thing.
Winning a comparison becomes the beginning of a new era.
Doing the homework can make you a better advocate
Many people in the audience will never read the original. A commentator who reads the paper, studies the documentation, and runs an experiment has already done considerably more work.
The work is real. The mistake is treating it as the end of the evaluation.
The commentator can explain the mechanism and identify a few limitations. That can produce a strong sense of having understood a significance other people have missed. Yet without level three’s choice of criteria or level four’s search for alternatives, a clearer explanation may only give the launch’s chosen frame more authority.
Add some extrapolation about the future and the result is highly shareable: enough technical detail to establish expertise, attached to a future people want to hear about. The audience feels informed. The commentator gains attention.
You can feel ahead of the crowd while becoming unusually good at telling the future the launch wants people to believe in.
No deliberate deception is required. Confidence from having done the homework, reinforced by attention, can deepen commitment to the story. It becomes easy to mistake “people love this explanation” for “the judgment is being vindicated.” Attention does not answer the questions at levels three and four.
A considered rejection can sound like ignorance
Someone who has done no research can say, “This isn’t worth following.” Someone who has examined the mechanism, compared alternatives, and considered the actual need can reach the same conclusion.
The verdict sounds identical. The reasoning is not.
A person who stopped at the first two levels may mistake the second judgment for the first: you must not have read the paper, run the experiment, or understood what makes the technology important.
The questions about criteria and alternatives go unanswered. Instead, the person asking them is dismissed as uninformed. “Is this the right comparison?” becomes “You don’t even understand the technology.”
I have encountered this while discussing MCP. In an earlier account of a hostile response, I described a viewer saying he wanted to put me on his “SB list”—an insult meaning, roughly, an idiot list. I had discussed the development, debugging, and distribution tradeoffs between calling scripts directly and introducing a protocol. The response used terminology and objections to a simplified analogy to dismiss the argument wholesale.
What makes that kind of response difficult is its apparent expertise. Technical vocabulary and confidence are visible; the distinction between explaining a technology and evaluating when to adopt it is less obvious.
Being attacked does not make a judgment correct. The useful question is: Did the response address the tradeoff, or merely establish familiarity with the technology?
This is why challenging a narrative is so much work. Its advocates can make an existing story more compelling. The person questioning it has to establish that they understand the story, show what it omitted, and explain why a more appropriate comparison changes the conclusion.
One side can offer “the next revolution is here.” The other may have to teach a method of judgment—and still be mistaken for someone who has fallen behind.
A bare recommendation to skip a technology is therefore not enough. In the earlier articles below, the tasks, alternatives, and reasons for choosing matter as much as the verdicts. They let readers distinguish a dismissal made before understanding from a choice made after it.
The same trap returns under different names
My co-founder Yan Wang has been examining these choices in our community for years. A few of his earlier articles help make the five levels less abstract.
The short quotations below are translated from his Chinese originals.
MCP: a common interface does not settle the data flow
In his October 8, 2025 article, Yan criticized routing everything through a model’s context:
For situations such as GUIs that conventional programs can already handle, forcing all information through an LLM’s context window instead introduces uncertainty, reduces task success, diminishes the LLM’s effectiveness, and increases inference costs.
His example was flight search. A tool returns structured results. Ordinary software can render them as an interface. Asking the model to receive all that data and generate or repeat the interface introduces model work where no model judgment was needed.
An earlier article from March 2025 examined another cost: calls that once fit into a script become communication between clients and servers. Debugging can now require tracing multiple processes, logs, and formats.
These are specific engineering critiques. They ask who should process which information, and where execution should happen.
The better alternative may be straightforward: send information to the model when it needs to interpret it; let code perform deterministic operations directly.
Skills make this distinction especially useful. An agent can load instructions, consult files, and execute scripts as needed. It can still connect to tools through MCP. That does not mean every intermediate result must travel through its context.
RAG: relevant fragments can lose the relationships you need
In an April 16, 2025 experiment, Yan and a collaborator worked with three years of group-chat history.
Chunking, embedding, and retrieval could help answer who had said something. It was much less useful for explaining how the group’s understanding of AI collaboration had evolved.
Yan put the failure this way:
It wasn’t a lack of capability. The model simply didn’t have the complete picture from which to form a judgment.
A change in someone’s views, or an argument that eventually reached a resolution, lives in relationships across time. The most semantically similar fragments may omit precisely those relationships.
They instead supplied the complete history where it fit within the context window, then iterated on reflection, verification, and synthesis.
For larger collections, we often start with documents that preserve their structure, search, and routing: let the agent choose what to open and what to read next. If a question requires several searches, cross-checks, or calculations, the agent can organize those steps.
The choice follows from what the question requires the model to understand. “I have private data” is not enough to settle the architecture.
Agent frameworks: fewer lines of code can mean more inherited assumptions
On March 5, 2025, Yan wrote:
These overly abstract frameworks replace the builder’s intuition with the framework architect’s assumptions. That is not an effective solution; it is part of the problem.
He examined several different assumptions: agents coordinating through messages, workflows represented as graphs, and code serving as the medium for tool use.
These choices determine how a problem is decomposed and how state moves through a system. They are not merely convenient API wrappers.
When your task fits the assumptions, the components can save time. When it does not, the few lines you avoided writing can become several layers you must trace through to change the behavior.
That is why his advice was to understand the model, tools, and execution loop before committing to a framework. The question was what you were inheriting along with the convenience.
OpenClaw: automatic memory is not the same as dependable knowledge
In his February 15, 2026 analysis, Yan described the difficulty of inspecting automatic memory updates:
You see the result, but it is hard to see the cause: which entries changed this time, why one was deleted, or why two unrelated things were merged.
The concern was the memory-maintenance process, not whether the source code was open.
An agent can wake periodically, consolidate notes, and remove material without requiring the user to manage every step. That can produce the appealing feeling that it understands you better over time.
It can also compress away an important research conclusion or merge unrelated matters. A shared memory pool can carry one project’s preferences into another.
Yan kept the useful idea of accumulating memory, reused OpenCode’s execution capabilities, and made the records easier to control: separate project and personal memory, explicit files, and Git history.
The criterion changed from “Does it remember more?” to “Can I inspect, preserve, isolate, and recover what matters?”
Fine-tuning: sometimes the answer is to adopt it
In a February 15, 2025 article, Yan was trying to make a model write in his voice.
Broad style instructions, comparisons to other writers, and a few examples were not reliably capturing the fine distinctions he kept correcting.
He described the task as getting AI to internalize corrections that were “scattered, small, yet precise.”
Here, fine-tuning had a plausible job. The missing ingredient was not another document containing facts. It was a way for many recurring behavioral corrections to influence the output consistently.
That justified testing the approach. It also explained the work required: preparing examples, checking generalization, and watching for changes to behavior he wanted to retain.
The same method that tells you when to leave a technology alone should tell you when to use it.
Knowing what to skip leaves room to notice what matters
Permanent skepticism is not judgment. It can miss an opportunity just as thoroughly as hype can manufacture one.
In January 2025, when we introduced our Agentic AI course, I wrote:
We estimate that Agentic AI will create a 10–18-month window of opportunity.
The change we cared about was the ability to use tools, make multiple decisions, and improve the result through feedback. AI could execute a step, inspect what happened, and decide what to do next, rather than waiting for a person to arrange every move.
That is why advising people not to start with an agent framework was compatible with investing heavily in agentic AI. Adding “agent” to a product name does not establish that capability. A specialist chatbot and a system that can keep a task moving are different things.
Then, on March 21, 2025, sixteen days after his critique of frameworks, Yan published “The Underestimated Claude Code”.
While attention centered on Claude 3.7, he saw the command-line tool as a reusable building block: other applications could draw on its ability to use tools, execute, and correct mistakes. They would not all have to build that loop themselves.
Six months later, Claude Code was attracting much broader attention. The small-looking interface concealed a much larger architectural role.
On March 27, 2025, he also wrote about document-driven development: delivering a document was as important as delivering code. The practical problem was an agent forgetting prior decisions and undoing earlier work.
By 2026, discussions of harness engineering were making the role of documents, tools, constraints, and feedback much more visible.
Following the underlying problem—how to make AI complete useful work reliably—lets understanding accumulate. Following names makes each launch feel like a new field you must begin learning from scratch.
Why a carefully chosen solution can look too basic
This creates a difficult teaching problem.
Remove the investigation and show only the final approach: don’t add MCP here; give the model the complete document; a model and a tool loop are sufficient for this task.
It looks simple.
But “this is the only approach I know” and “I compared the alternatives and this is all we need” can produce almost identical code.
The difference is hidden in the options that were rejected.
School can reinforce the wrong instinct. An exam gives you the problem. You are rewarded for solving it, and solving a harder problem usually signals greater ability.
It is easy to carry “harder means better” into work.
In real work, choosing the problem is part of the work. When a launch defines a difficult, fashionable problem for you, the old habit can take over: you become very good at solving a question whose importance you never established.
Building something complex takes skill. So does discovering which complexity you can leave out.
The second kind is harder to demonstrate. Add three components and everyone can see them. Investigate carefully enough to remove two, and the investigation disappears from the final diagram.
We have received repeated feedback that AI Builders feels too basic. An experienced engineer may supply the missing comparisons from years of failed attempts. Someone who has not encountered those problems sees only an obvious-looking answer.
That helps explain a contrast we have seen: engineers at OpenAI and Anthropic find depth in material that a computer science student can experience as elementary.
But it is our job to make that depth learnable. We cannot require students to arrive with the experiences that make the course make sense.
Explaining an answer simply and teaching someone to reach it are different accomplishments.
If a student leaves remembering only that the teacher said not to learn a certain tool, the next launch still leaves them waiting for the teacher’s verdict.
We have changed whom they trust. We have not given them their own judgment.
How to make a better judgment count at work
There is a practical consequence of that teaching problem. Imagine returning to a team that is excited about Jev and saying, “My instructor says we don’t need it.” Even if the conclusion is right, you have given your colleagues little reason to trust it. They cannot tell whether you have worked through the comparison or are repeating someone else’s verdict.
Finding the alternative is the hard part. You need enough experience to recognize which other approaches could solve the problem, and enough skill to put one to the test. Yan brings that depth of implementation experience to our discussions. It cannot be passed on as a list of technologies to avoid.
A more useful contribution starts with the problem your colleagues want to solve. If they need fast decisions inside a workflow, agree on what a good decision means and build a small comparison around that requirement. Let the team inspect the errors, latency, consistency, and confidence estimates. If a local approach meets the requirements with fewer dependencies, you now have something concrete to propose. If it fails, you have learned something your objection alone could not have revealed.
The alternative tests your judgment as well as theirs. An apparently obvious substitute may turn out to miss an important part of the job. You only discover that when you try to make it work.
This also changes the conversation. Your colleagues can see that you understand the attraction of the new technology and can help deliver what they wanted from it. You are offering a way forward. When you then ask whether this part of the system deserves more effort, that question comes from someone who has helped establish what is already sufficient.
You do not need to begin by persuading everyone that the whole narrative is wrong. Start with a useful comparison they can examine. Competence makes your reasoning easier to trust; making the reasoning visible gives them a way to challenge it. That is how a private judgment can become a better decision for the team.
And that is another reason our teaching has to include the work behind the conclusion. A student should leave able to make the comparison—and explain what it changes.
Put the missing decisions back into the course
That is a central part of the AI Builders refresh we are planning for November and December 2026, with the learning path rebuilt around what people will need in 2027.
Adding fashionable names would be easy. It would also accept the very premise we are trying to challenge: more novelty and complexity must mean more depth.
What needs to be added is the experience of making the choice.
Take Jev or a comparable release. First understand it well enough to build a working version. See why people find it compelling. Then open up the comparison: why those metrics, why those opponents, and what constraints have we accepted unnecessarily?
Build the alternative. Test both approaches on the same task.
Then change the task. Increase the data, tighten the latency requirement, or introduce a failure the original solution cannot handle. Can you tell when the simple answer stops being enough? Do you know what to add next?
For a Jev-style lesson, the ambition is concrete: learn to build a capable local decision system, then understand why you might not need to pursue that particular technology further.
You should have both abilities: the ability to make it work and the ability to explain why you would—or would not—use it.
Our August update to Module 3 already previews this approach. A text-correction exercise sounds small. Establishing a baseline, examining failures, revising instructions, and comparing results forces a series of consequential judgments: did this change help, and why?
The year-end refresh will also draw on what we see in enterprise training, changing model capabilities, and students’ actual needs. Some automations that once deserved a custom implementation no longer do. Other tasks have become possible and worth attempting.
Assignments, lesson comments, and project discussions should make the decisions visible: what you expected, what happened, what you changed, and what justified the change. Practical instructions must help students carry those judgments into their own work.
We want to spare people pointless trial and error. We must not spare them the experiences necessary to understand a choice.
A new technology can offer a new solution. It should not get to decide what is worth your time.
A course should help you make that decision yourself.
I co-founded Superlinear Academy with Yan Wang. This essay grew out of our technical discussions and students’ feedback on how we teach. Read the Chinese version.
“Best real alternative” is the key test here. In Jev Social we learned that route accuracy alone is a bad benchmark—a fast, correct route can still fail in browser execution or return weak evidence.