last Tuesday at 10:23 PM
$40m in funding, 2 years in stealth.
Performs on-par with SemIf which was built in a couple days and apparently uses raw Qwen, with no fine-tuning. SemIf runs in your freaking browser. Oh and Jev is twice as expensive?
Is it surprising that Jev consistently thinks it's Qwen?
I'm almost convinced that Jev is a scam. Take Qwen, fine tune it a little, tell investors it cost $10m, spend $1m on advertising, profit.
yesterday at 11:09 AM
I think itās much more likely that these benchmarks are not good, in that they do not explore much of the space that jev was (likely) trained to cover. The doom demo is a good example ā Iād like to see a wide variety of things like that included in any benchmark, not just āemail classificationā or what have you.
Think of it this way: thereās some time needed to optimize / design an architecture, and the world gets that for free when itās described. As to the rest of the last two years spent, is it more likely a former oAI lead spent them fucking around, or adding as many RL environments as possible to its model that is supposed to be a generalized classifier?
Right now my prior is that jev is probably better than these rando weekend models, whether or not we know how to test and demonstrate that in a benchmark. Itās also super cheap, so I donāt think thereās a strong reason not to try out building with it first, then walk down the ladder to an open model if you need to for some reason.
yesterday at 1:40 PM
So if benchmark says Jev sucks, it's benchmark's fault. And yet Typesafe doesn't share anything about their internal benchmark that apparently proves how amazing Jev is.
Funny enough, when I tell people that Jev cannot play tic-tac-toe I hear a similar argument - it's not what Jev was built for. Jev has this elusive use case that noone can describe, so when Jev fails, it's just because it wasn't built for it. Convenient.
Don't you find it suspicious that Jev cannot play tic-tac-toe or checkers, but can play Doom? Don't you find it suspicious that nothing of the Doom demo was shared: no harness, no control loop, no state encoding, no prompts - nothing.
And the explanation is obvious - Jev isn't playing doom. The harness is. They essentially built a Doom bot, dumbed down it's control loop for Jev, and gave reins to Jev. Look mom, Jev is playing Doom!
The state says "you're pointing at the cacodemon" or "you're not pointing at the cacodemon" and Jev has to decide whether to press fire. Frontier intelligence!
You can build a harness where a coin flip is playing Doom.
yesterday at 6:24 PM
Oh Iām all for replicating. Suspicious is a big word, and to my mind fairly useless state of mind.
I just went ahead and built some stuff with Jev to get a feel for it, including a small chat harness ā in this case the harness sends out like 40 parallel API calls to get a probability distribution against the 800 or so tokens in that call, and then combines up the likely ones and runs it through another narrowing process. With that in place, Jev can talk. Although itās not very talkative, but it definitely can respond to queries.
I tried it out for some computer use usecases, and it has potential to be very fast there ā it had enough comprehension to do the selecting and tool calling and pass back control to the harness at the right times.
So, upshot - useful and interesting tool. I didnāt benchmark it against any of the open jev clones because a) itās cheap, b) Iām not using it for anything major right now and c) like I said above, Iād be surprised if that team just spent two years wasting time on a weekend project.
Iāll double down and say that if this arch turns out to be genuinely useful, (and I think it could be), then when we get good broad benchmarks, this release of Jev will benchmark higher against the weekend clones than it does now.
yesterday at 11:27 AM
[dead]
last Tuesday at 11:55 PM
Isn't the entire deal with jev that it is fast? I'd be interested to know how the energy cost of the Qwen-based model compares with Jev. Of course, Jev is currently locked up so we don't know... "Trust me bro Jev is revolutionary and amazing, pay more money for our inferior product which costs more to run, and of which you need to access by sending us the data"
yesterday at 8:11 PM
[flagged]
yesterday at 12:38 AM
You can spot vibecoded websites by how they include the prompt or commit-style comments into the literal interface, instead of communicating it via visual context (or simply excluding it)
> Sort by any column; values the run could not produce always sort last. Hover a cost for how it was priced, a latency for the endpoint. Names link to each project.
A designer would never write this, but an LLM just inserts it by making it small grey text next to the interface, just like it does with inane code comments.
yesterday at 6:55 AM
EYEBROWS EVERYWHERE
There are lots of tells in every vibe coded design.
yesterday at 3:26 AM
And why is a designer needed for every website in the world?
yesterday at 10:28 AM
I (not GP) have no issues with CSS part of the design (just as I am fine with bootstrap websites). But if you can't be arsed to write 4 lines of text describing it in human language instead of the claude vomit, you probably didn't put much effort to begin with.
yesterday at 7:00 AM
They're not needed. And there are bad designers everywhere.
And I know HN people think marketing is a bad word.
But marketeers know that design is part of the messaging. And nowadays, having clear AI tells in your design shouts "I made this in an afternoon, so don't take this seriously"
First impressions are everything in a world where attention spans are shortened every year.
Act accordingly.
yesterday at 12:57 AM
It's helpful metadata, what's wrong with it? Dashboards at work do this.
yesterday at 2:04 AM
One personās helpful metadata is another personās noise. Itās much better for a door to visually indicate that it should be pushed open than to put up a sign there.
However, doing the former requires a level of empathy with humans that LLMs rarely have.
Human brains have caloric demands. It is possible for humans to process enormous amounts of unrelated facts to make a decision, but itās tiring. Itās much better to not do that, especially just to get some basic information.
To anthropomorphize a bit, an LLM might find it charming and interesting to read someoneās life story as a preamble before their taco recipe. Humans by and large find that annoying, not because we canāt understand the biography but because processing that information is not free.
So itās probably possible to design using an LLM. You would probably have to be intentional about it.
yesterday at 2:43 AM
Well said
Much has been discussed about AI coding but there's been a lack of discussion on the design/UX side as that's also a step being rushed by via prompting + "looks pretty enough".
We're entering an era of hyper dense UIs with plenty of excess and quick workarounds.
yesterday at 1:39 AM
If you don't have any kind of agent instructions saying "don't copy code into prose", you'll inevitably get text that reflects some previous state of the system instead of its current state.
yesterday at 1:04 AM
It's fine in isolation to have help text for complex interfaces needing explanation. Better yet contextual hints.
But there's about 10 other examples explaining intention of the coding rather than immediately useful information to the user, it's all over this one site in small grey text. And I guarantee you nobody is reading them carefully. Just like how nobody likes reading a 15 line LLM code comment over a simple function.
Most of it could be better solved with more thoughtful design or deleted. The link explanation is particularly egregious.
last Tuesday at 9:37 PM
Good project but this one also exists https://huggingface.co/spaces/multimodalart/jev-decision-ind... and the results do not seem to add up and also model sets are different... still needs time to mature likely
yesterday at 11:27 AM
[dead]
yesterday at 1:22 AM
I was trying to figure out what exactly the tests here are. I guess I found some of the questions (here: https://github.com/fstandhartinger/jevbench/blob/main/datase...)
e.g.,
"instructions": "Which intent does the user's message express?",
"labels":["set_alarm", "play_music", "weather", "send_message", "turn_off_lights"],
"state": "Play some Taylor Swift.",
"expected": "play_music"yesterday at 11:25 AM
[dead]
last Tuesday at 8:42 PM
jev ceo on why he eschewed benchmarking: https://www.latent.space/i/216783460/privacy-benchmarking-an...
last Tuesday at 10:06 PM
Many SaaS vendors forbid benchmarking, I find it crazy that such anti-competitive terms are standard across the industry but they are. Generally the goal of such terms is to "control the narrative" around the product, regardless of the truth of performance being better or worse than competitors.
yesterday at 7:53 AM
They released some examples of what their workflow evals are like. I'm sure you could reverse engineer a benchmark from that
last Tuesday at 9:38 PM
Interesting; was curious how this didn't fall into trouble with ToS. Apparently the "no benchmarks" clause was intended for "limited preview" audiences and didn't get removed at launch on accident.
yesterday at 9:36 AM
But also no big deal because "Iām extremely anti-public benchmarks."
last Tuesday at 10:26 PM
I mean, that's a great reason to ignore JEV entirely.
"Trust, but verify" isn't just a catchy cliche. It's the only way to operate where models and code are fast to market.
yesterday at 11:24 AM
[flagged]
yesterday at 9:47 AM
Can you please add our model to benchmark? https://gambler-relay-us-west1.leo-fish.ts.net/demo
yesterday at 11:32 AM
Happy to ā please open an issue at https://github.com/fstandhartinger/jevbench/issues with an endpoint or runnable code, the model and licence, and whether it was trained on the public items. Every entrant runs through the same harness, including the sealed set. I definitely prefer open models I can run locally, because sending our private held-out set of tasks to an external API produces some headaches on ur side and basically means we have to rotate the test set a lot, to avoid contaimination. We can call API hosted models though, we'll just flag the leaderboard entries appropriately.
yesterday at 10:12 PM
For something that was announced a week ago, and now there are ~80 model competing is crazy!
yesterday at 6:13 AM
How do you handle task distribution and prevent the benchmark from favoring models that are tuned specifically to these 534 questions?
yesterday at 11:23 AM
[dead]
yesterday at 12:00 AM
yesterday at 2:08 AM
It is strange that you put the BGE reranker in the list but not BART which is an actual zero shot classifier.
yesterday at 11:26 AM
Good call... BART-large-MNLI is the classic zero-shot baseline and it should be there. Adding it to the next run.
yesterday at 6:25 AM
534 English decisions in one full run sounds substantial.
last Tuesday at 9:44 PM
https://is-it-ai-slop.app.mintapis.com/ is a fun tool. Is the source or methodology for that in the github repo? I couldn't find it immediately.
We've been experimenting with Jev for classifying email, some thoughts here: https://housecat.com/blog/classifying-email
Flagging AI written email is a much requested feature too.
yesterday at 12:26 AM
Keysmashing my keyboard resulted in 86% confidence that the text was AI written. I don't think this is a particularly good classifier, I've never seen an LLM output "kad jfkhasljkdhf laksjhdf".
Edit: If that's not realistic enough for you, the text "Hello world! My name is GravitasIsOverrated and I like coding and cooking. This text is 100% genuine, and not AI generated at all." results in 85% confidence that it's AI generated.
More broadly, I don't know why this would work. Qwen/Jev/whatever doesn't magically have the ability to discern AI-authored text from non-AI-authored text, and will increasingly get worse at it as the hallmarks of AI-written text change.
yesterday at 11:29 AM
Thanks! Yeah, the code is open: https://github.com/fstandhartinger/who-is-right. Email classifying is definitely a good Jev usecase, I agree
yesterday at 12:28 AM
I asked free chatgpt to give me some essays that will fool a slop detector and they all fooled this slop detector. Its best guess was "6% slop probability 87% confidence answered in 0.5 s for $0.000027" and yet it was 100% slop.
I am very skeptical slop detectors will ever work.
yesterday at 2:26 PM
Lmao this just proves Jev is being shilled for money. Nobody would speak like this for fun.
yesterday at 2:51 AM
the slop detector giving 86% confidence on a keysmash is the real finding here, not the leaderboard score. confident and wrong is worse than an LLM that just hedges.
yesterday at 6:32 PM
[dead]
yesterday at 7:05 AM
[flagged]
yesterday at 7:58 PM
[flagged]
last Tuesday at 11:26 PM
[flagged]
last Tuesday at 1:01 PM
yesterday at 3:44 AM
[flagged]
yesterday at 3:24 AM
[dead]