A Faster Alternative to Jq
116 points - today at 7:12 AM
SourceI appreciate performance as much as the next person; but I see this endless battle to measure things in ns/us/ms as performative.
Sure there are 0.000001% edge cases where that MIGHT be the next big bottleneck.
I see the same thing repeated in various front end tooling too. They all claim to be _much_ faster than their counterpart.
9/10 whatever tooling you are using now will be perfectly fine. Example; I use grep a lot in an ad hoc manner on really large files I switch to rg. But that is only in the handful of cases.
lemagedurage
today at 10:13 AM
True. I feel like the main way a tool could differentiate from jq is having more intuitive syntax and many real world examples to show off the syntax.
montroser
today at 9:26 AM
Then this is for the handful of cases for you. When it matters it matters.
> I see the same thing repeated in various front end tooling too. They all claim to be _much_ faster than their counterpart.
>
> 9/10 whatever tooling you are using now will be perfectly fine
Are you working in frontend? On non-trivial webapps? Because this is entirely wrong in my experience. Performance issues are the #1 complaint of everyone on the frontend team. Be that in compiling, testing or (to a lesser extend) the actual app.
Fair, but agentic tooling can benefit quite a lot from this
Opencode, ClaudeCode, etc, feel slow. Whatever make them faster is a win :)
httpsterio
today at 10:00 AM
The 2ms it takes to run jq versus the 0.2ms to run an alternative is not why your coding agent feels slow.
It's not running jq locally that's causing that
enricozb
today at 10:18 AM
I am excited for some alternative syntax to jq's. I haven't given much thought to how I'd write a new JSON query syntax if I were writing things from scratch, but I personally never found the jq syntax intuitive. Perhaps I haven't given it enough effort to learn properly.
I wonder so often about many new CLI tools whose primary selling point is their speed over other tools. Yet I personally have not encountered any case where a tool like jq feels incredibly slow, and I would feel the urge to find something else.
What do people do all day that existing tools are no longer enough? Or is it that kind of "my new terminal opens 107ms faster now, and I don't notice it, but I simply feel better because I know"?
I process TB-size ndjson files. I want to use jq to do some simple transformations between stages of the processing pipeline (e.g. rename a field), but it so slow that I write a single-use node or rust script instead.
This isn't for you then
> The query language is deliberately less expressive than jq's. jsongrep is a search tool, not a transformation tool-- it finds values but doesn't compute new ones. There are no filters, no arithmetic, no string interpolation.
Mind me asking what sorts of TB json files you work with? Seems excessively immense.
This reminds me of someone who wrote a regex tool that matches by compiling regexes (at runtime of the tool) via LLVM to native code.
You could probably do something similar for a faster jq.
Now I'm really curious. What field are you in that ndjson files of that size are common?
I'm sure there are reasons against switching to something more efficient–we've all been there–I'm just surprised.
> Now I'm really curious. What field are you in that ndjson files of that size are common?
I'm not OP,but structured JSON logs can easily result in humongous ndjson files, even with a modest fleet of servers over a not-very-long period of time.
So what's the use case for keeping them in that format rather than something more easily indexed and queryable?
I'd probably just shove it all into Postgres, but even a multi terabyte SQLite database seems more reasonable.
Replying here because the other comment is too deeply nested to reply.
Even if it's once off, some people handle a lot of once-offs, that's exactly where you need good CLI tooling to support it.
Sure jq isn't exactly super slow, but I also have avoided it in pipelines where I just need faster throughput.
rg was insanely useful in a project I once got where they had about 5GB of source files, a lot of them auto-generated. And you needed to find stuff in there. People were using Notepad++ and waiting minutes for a query to find something in the haystack. rg returned results in seconds.
You make some good points. I've worked in support before, so I shouldn't have discounted how frequent "once-offs" can be.
The use case could be e.g. exactly processing an old trove of logs into something more easily indexed and queryable, and you might want to use jq as part of that processing pipeline
Fair, but for a once-off thing performance isn't usually a major factor.
The comment I was replying to implied this was something more regular.
EDIT: why is this being downvoted? I didn't think I was rude. The person I responded to made a good point, I was just clarifying that it wasn't quite the situation I was asking about.
bigDinosaur
today at 9:06 AM
Certain people/businesses deal with one-off things every day. Even for something truly one-off, if one tool is too slow it might still be the difference between being able to do it once or not at all.
adastra22
today at 9:06 AM
At scale, low performance can very easily mean "longer than the lifetime of the universe to execute." The question isn't how quickly something will get done, but whether it can be done at all.
swiftcoder
today at 9:39 AM
Deal with really big log files, mostly.
If you work at a hyperscaler, service log volume borders on the insane, and while there is a whole pile of tooling around logs, often there's no real substitute for pulling a couple of terabytes locally and going to town on them.
InfinityByTen
today at 8:18 AM
You don't know something is slow until you encounter a use case where the speed becomes noticeable. Then you see the slowness across the board. If you can notice that a command hasn't completed and you are able to fully process a thought about it, it's slow(er than your mind, ergo slow!).
Usually, a perceptive user/technical mind is able to tweak their usage of the tools around their limitations, but if you can find a tool that doesn't have those limitations, it feels far more superior.
The only place where ripgrep hasn't seeped into my workflow for example, is after the pipe and that's just out of (bad?) habit. So much so, sometimes I'll do this foolishly rg "<term>" | grep <second filter>; then proceed to do a metaphoric facepalm on my mind. Let's see if jg can make me go jg <term> | jq <transformation> :)
password4321
today at 9:34 AM
Optimization = good
Prioritizing SEO-ing speed over supporting the same features/syntax (especially without an immediately prominent disclosure of these deficiencies) = marketing bullshit
A faster jq except it can't do what jq does... maybe I can use this as a pre-filter when necessary.
Speed is a quality in itself. We are so bugged down by slow stuff that we often ignore that and don’t actively search for another.
But every now and then a well-optimised tool/page comes along with instant feedback and is a real pleasure to use.
I think some people are more affected by that than others.
Obligatory https://m.xkcd.com/1205
Imustaskforhelp
today at 9:47 AM
I am not sure if it was simon or pg who might've quoted this but I remembered a quote about that a 2 magnitude order in speed (quantity) is a huge qualititative change in it of itself.
hrmtst93837
today at 9:46 AM
[dead]
hrmtst93837
today at 8:08 AM
[dead]
Having used `jq` and `yq` (which followed from the former, in spirit), I have never had to complain about performance of the _latter_ which an order of magnitude (or several) _slower_ than the former. So if there's something faster than `jq`, it's laudable that the author of the faster tool accomplished such a goal, but in the broader context I'd say the performance benefit would be required by a niche slice of the userbase. People who analyse JSON-formatted logs, perhaps? Then again, newline-delimited JSON reigns supreme in that particular kind of scenario, making the point of a faster `jq` moot again.
However, as someone who always loved faster software and being an optimisation nerd, hat's off!
> Having used `jq` and `yq`
If you don't mind me asking, which yq? There's a Go variant and a Python pass-through variant, the latter also including xq and tomlq.
Integrating with server software, the performance is nice to have, as you can have say 100 kRPS requests coming in that need some jq-like logic. For CLI tool, like you said, the performance of any of them is ok, for most of the cases.
Nice.
Some bits of the site are hard to read "takes a query and a JSON input" query is in white and the background of the site is very light which makes it hard to read.
When initially opening the page it had broken colors in light mode. For anyone else encountering it: switch to dark mode and then back to light mode to fix it.
CodeCompost
today at 9:02 AM
I suspect the website is vibe-coded, like the tool itself.
I think this has just been fixed. A bit of dark mode was leaking into light in the css.
I still saw the same bug just now (Firefox on macOS).
Fine in Firefox on Android.
Note that the scales of the charts are all different, which makes them hard to compare.
Also, there are lots of charts without comparison so the numbers mean nothing...
White text with light background, yeah.
keysersoze33
today at 7:54 AM
I had the same problem (brave browser)
vladvasiliu
today at 7:52 AM
Looks fine to me on Edge/Windows.
youngtaff
today at 7:57 AM
Broken on iOS Safari too
silverwind
today at 10:17 AM
Effort would be better investigated making `jq` itself faster.
onedognight
today at 10:03 AM
Having the equivalent jq expression in these examples might help to compare expressiveness, and it might help me see if jq could “just” use a DFA when a (sub)query admits one. grep, ripgrep, etc change algorithms based on the query and that makes the speed improvements automatic.
I highly recommend anyone to look at jq's VM implementation some time, it's kind of mind-blowing how it works under the hood: https://github.com/jqlang/jq/blob/master/src/execute.c
It does some kind of stack forking which is what allows its funky syntax
functional_dev
today at 10:19 AM
The backtracking implementation in jq is really the secret sauce for how it handles those complex filters without getting bogged down
vbezhenar
today at 10:12 AM
Looks like naive implementation of homemade bytecode interpreter. What's so mind blowing about that? Maybe I missed something.
First of all, congratulations! Nice tool!
Second, some comments on the presentation: the horizontal violin graphs are nice, but all tools have the same colours, and so it's just hard to even spot where jsongrep is. I'd recommend grouping by tool and colour coding it. Besides, jq itself isn't in the graphs at all (but the title of the post made me think it would be!).
Last, xLarge is a 190MiB file. I was surprised by that. It seems too low for xLarge. I daily check 400MiB json documents, and sometimes GiB ones.
forgive me my rant, but when I see "just install it with cargo" I immediately lose interest. How many GB do I have to install just to test a little tool? sorry, not gonna do that
From their README [0]:
> Jq is a powerful tool, but its imperative filter syntax can be verbose for common path-matching tasks. jsongrep is declarative: you describe the shape of the paths you want, and the engine finds them.
IMO, this isn't a common use case. The comparison here is essentially like Java vs Python. Jq is perfectly fine for quick peeking. If you actually need better performance, there are always faster ways to parse JSON than using a CLI.
[0]: https://github.com/micahkepe/jsongrep
I learned a number of data processing cli tools: jq, mlr, htmlq, xsv, yq, etc; to name a few. Not to the level of completing advent of code or anything, but good enough for my day to day usage. It was never ending with the amount of formats I needed to extract data from, and the different syntax's. All that changed when I found nushell though, its replaced all of these tools for me. One syntax for everything, breath of fresh air!
igorramazanov
today at 10:21 AM
Same! Nushell replaced almost all of them
Had to spend some efforts to set up completions, also there some small rough edges around commands discoverability, but anyway, much better than the previous oh-my-zsh setup
Ideally, wish it also had a flag to enforce users to write type annotations + compiling scripts as static binaries + a TUI library, and then I'd seriously consider it for writing small apps, but I like and appreciate it in the current state already
Same here, nushell is awesome! It helped me to automate so many more things than I did with any other shell. The syntax is so much more intuitive and coherent, which really helps a lot for someone who always forgot how to write ifs or loops in bash ^^
keysersoze33
today at 7:57 AM
I was a bit skeptical at first, but after reading more into jsongrep, it's actually very good. Only did a very quick test just now, and after stumbling over slightly different syntax to jq, am actually quite impressed. Give it a try
What were your syntax stumbling blocks? I must be honest I've used jq enough but can never remember the syntax. It's one of the worst things about jq IMO (not the speed, even though I'm a fan of speedups). There's something ungrokkable about that syntax for me.
steelbrain
today at 8:32 AM
Surprised to see that there's no official binaries for arm64 darwin. Meaning macOS users will have to run it through the Rosetta 2 translation layer.
QuantumNomad_
today at 8:37 AM
I’d install it via cargo anyway and that would build it for arm64.
If the arm64 version was on homebrew (didn’t check if it is but assume not because it’s not mentioned on the page), I’d install it from there rather than from cargo.
I don’t really manually install binaries from GitHub, but it’s nice that the author provides binaries for several platforms for people that do like to install it that way.
You can use cargo-binstall to retrieve Github binary releases if there are any.
baszalmstra
today at 8:38 AM
Really? That is your response? This is an high quality article from someone who spend a lot of time implementing a cool tool and also sharing the intricate inner workings of it. And your response is, "eh there are no official binaries for my platform". Give them some credit! Be a little more constructive!
His response at least fits the discussion and is relevant to the tool, not generic hollier-than-thou scolding.
To address the concern, anyway, I'm sure it would soon be available in brew as an arm binary.
Speed is good! Not a big fan of the syntax though.
furryrain
today at 8:37 AM
If it's easier to use than jq, they should sell the tool on that.
adastra22
today at 9:03 AM
The fastest alternative to jq is to not use JSON.
quotemstr
today at 8:47 AM
Reminder you can also get DuckDB to slurp the JSON natively and give you a much more expressive query model than anything jq-like.