yesterday at 5:48 AM
Having used D1 in production - it's not very reliable. There was a comment on HN from one of D1 devs [1] saying that D1 is a sql wrapper on top of durable objects that mimicks sqlite distributed like nature.
> I'll let you in on a sort of dirty secret: It's almost always better to use Durable Objects storage, rather than D1. Even if you only want a single global database, it's better to implement that as a singleton Durable Object, than by using D1. Because that's all D1 itself actually is: a singleton Durable Object that exposes an API to its SQLite database. It's just a wrapper.
[1] https://news.ycombinator.com/item?id=48611834
yesterday at 11:19 AM
D1 also gives read replication and point in time recovery, but my biggest issue with it is you can’t create D1 dbs dynamically like you can with durable objects so things like tenancy are harder
today at 1:00 AM
[flagged]
yesterday at 8:12 AM
Completely. D1 flaky as hell!
yesterday at 10:22 PM
Aren't Durable Objects like... proprietary? Tied to a specific cloud platform?
yesterday at 8:52 AM
Its funny, when I was reading part time parliament paper (paxos) and while learning zig and odin language created a pure paxos library and then the first application for that library was to build a multi-master sqlite. The two approaches I took are very different in case of zig I build zaxonlite (https://github.com/insanai/zaxonlite) and have a benchmark on it that includes rqlite. It uses WAL replication as strategy. The second attempt I tried in odin is sqlodin (https://github.com/insanai/sqlodin), in this the system does not use WAL but run sql query on each node, as result subset of sql query in sqlite are not supported.
yesterday at 12:42 PM
Wow, a 155 page book for a relatively new project is interesting - what was your workflow for that? It looks mostly generated - is it used in its entirety as context to a top-level agent when working on new features, or are you directing it to just take relevant subsections? Why the PDF output (it does look nice)?
today at 1:08 PM
I do not use book for a context, book is the outcome of the development. I am still old school I feel any product should have good documentation and should be available offline, also personally I like physical things so many of them will print and read. I follow a rigorous process I used to follow since 1998, without AI, which is write something like PEP or RFC before starting work on things. So if you look at the whole project it has design records and each contains details. I read through the docs, even though I use AI from time to time. I used to write a lot of restructuredtext (RST) docs. So I wanted to move the whole pipeline to rust, I used to use latex for it earlier but then wanted to move to typst. So for it wrote another system which helps me with website, html, pdf. You can take a look if you like the project is https://github.com/insanai/guidedog, sibuna also I ported from typst pipeline to guidedog which uses rst markup. If you want to look at development part I especially use dsc (short for discussions) so if you type `guidedog dsc --help` to see the details.
yesterday at 8:46 PM
I feel like there should be a comparison table listing all of these engines and how they compare on their external dependencies, whether a single leader/any node can write, their shape (embeddable library vs server), etc.
IMHO support for transactions is a pretty important dividing line. Some engines support full SQLite transactions while others - like rqlite - don't officially support BEGIN/COMMIT [0], which matters if you rely on multi-statement transactions.
A friend/colleague has been building a pure-Python client for dqlite [1] (DB-API and SQLAlchemy), which may interest folks here. dqlite has full support for transactions, and like rqlite doesn't have external dependencies (ie no S3/Consul).
No knock on rqlite or any other engine - dqlite has warts too [2].
[0] https://rqlite.io/docs/faq/#does-rqlite-support-transactions
[1] https://github.com/letsdiscodev/python-dqlite-dbapi
[2] https://github.com/canonical/dqlite/issues/882 opened by my friend, https://github.com/canonical/dqlite/issues/908 , https://github.com/canonical/dqlite/issues/912 opened by me
yesterday at 10:44 AM
As someone who loved sqlite and the simplicity, the truth is that with agents I am finding it is extremely easy to manage postgres and it is realyl nice having all the SQL features of postgres -- sqlite is quite limiting in a lot of SQL. So if you are considering remote sqlite and all that, feels a better time than ever to just use a database server. Of course, sqlite is amazing for local tools etc.
yesterday at 9:08 PM
The problem is "remote" SQLite. Just run a "single" beefier box with in-process SQLite and Litestream backups to object storage and never have to think about RDBMS again. I'd wager there are 90%+ applications that would be better served, more performant, and cheaper with this design.
I'm able to handle around 200k/s on a sub $100 box. https://github.com/pairshaped/gleam_sqlite_benchmarks/blob/m...
yesterday at 3:56 PM
How are agents helping you? What was the trouble with Postgres before?
yesterday at 10:25 PM
My guess is that coding agents expose deep documentation and advanced patterns better or least faster that wading through everything.
yesterday at 5:02 PM
I've had the exact same experience. In the before-time I often reflected on how my love of sqlite was borderline irrational for what it is. Now, in a few projects I've let the LLM do postgres, and it's so nice having the pain if a db server mostly just handled. Still choose sqlite for anything embedded or without a hosted server.
yesterday at 6:52 PM
It’ll be a fine day when phones can use an embedded Postgres
yesterday at 8:22 PM
I am the same way with MSSQL. I used to avoid it like the plague because of how the permissions/logins/everything worked. Now I can throw a T-SQL script at it in a few seconds to make it behave.
yesterday at 10:40 AM
The comparisons between various projects, including between rqlite and Litestream, were interesting. (rqlite uses a consensus algorithm between nodes to keep them synced and allow changing the primary node, while Litestream just syncs itself to S3 so another node could take over with the data from there.)
I fully expected the article to get into D1 shortly and mention how that's about the same relationship between Cloudflare Durable Objects (which D1 is based on) and celld, an open source reimplementation of Cloudflare Durable Objects which writes to S3. But then the article treats D1 with a very sudden snarky disinterest, and then the final section feels very awkwardly defensive of rqlite and defeatist.
yesterday at 8:17 AM
If you don't get any joy out of writing about your personal experiences in your own voice, why bother writing about them at all?
yesterday at 10:08 AM
Kinda grating to have the story of your project told in default Claudese, to be honest.
yesterday at 10:52 AM
I get zero LLM vibes from this blog post.
yesterday at 11:56 AM
It's loaded with it to my eyes. (Pangram also agrees and flags the whole post as AI generated.)
# 2014: rqlite
The need was concrete.
# Before 2014, and alongside it
Two things deserve mention for honesty’s sake.
[...] and the main difference is of one of packaging:
# What the twelve years show
Three things, I think.
The core was right. [...] Nobody has found a reason to stop doing it.
The design space has three corners, and they appeared in order.
yesterday at 2:06 PM
Yeah it felt like claudeslop to me too — and such a strange choice. Maybe dude dictated it and said Claude to write it up. Save some time. Even so.
yesterday at 5:57 PM
If you talk to Claude all day every day like some of us are forced to, it's plain as day.
yesterday at 1:40 PM
I envy you. For better or worse, Claude is the author I've read the most in this year. There's a handful of human authors I know well enough so that I would recognize a text that was written by them. With LLMs, it's much easier than with humans, since the current generation of LLMs seems utterly incapable of changing their style.
yesterday at 11:43 AM
There are a bunch but in my opinion not conclusive.
yesterday at 5:50 AM
Is there a project that does all of these together?
- embeddable like sqlite
- clustered like dqlite
- has a standalone node like rqlite
- can be backed up to S3 like litesteeam
- optionally AP like cr-sqlite alongside CP
yesterday at 6:48 AM
https://github.com/punnerud/mpedb is not there yet, focused on sync between sqlite3, Postgresql and Oracle. Shown that concurrent writes to sqlite3 is possible at the speed of Postgresql (My own project)
yesterday at 9:48 AM
This sounds a lot like Apache Ignite 2.x, though it does not have SQLite's level of polish.
yesterday at 12:14 PM
It's never going to be embeddable, no?
yesterday at 12:33 PM
It is a set of Java dependencies so a full data-bearing Ignite node will happily be an in-process part of your Java monolithic app or microservice. It has likewise bindings for .Net and some other languages but not so seamless as in case of JVM.
yesterday at 4:08 AM
I have never examined this in any detail before. It looks far easier than the mass majority of clustered database architectures.
yesterday at 5:21 AM
using it for control layer pretty happy so far
yesterday at 8:32 AM
can you query the local rqlite, or do you always have to go to the leader?
yesterday at 3:01 PM
Last time I looked into it, you CAN bypass the rqlite raft layer, and hit the local sqlite directly, but you are not guaranteed consistency. Depending on your use case, that may be okay.
yesterday at 10:14 AM
docs - Queries, by default, are also serviced by the cluster Leader.
it is great, fun, easy.
yesterday at 10:14 AM
How long have you been using it? I've always been disappointed with similar systems, etcd including.
yesterday at 10:50 PM
It's basically holding tenant and operator master registry that app nodes pull down resulting in very light load on rqlite. The architecture is resilient to control layer being down (just can't provision new tenants).
yesterday at 6:47 AM
I maintain one of the client libraries for rqlite - I’ve found it an excellent concept really well implemented. One of my favourite bits is it gives you back json making Dapper-like mapping super easy in .NET and I suspect other languages.
yesterday at 8:31 AM
I've been looking for a reason to use rqlite, but so far sqlite + litestream have been safe enough (some dataloss acceptable in extreme cases)
My understanding is that rqlite gives you "stored in more than place" guarantees, which is nice if you need it. Much slower than a 1 node db though.
yesterday at 2:06 PM
yesterday at 7:28 AM
We use rqlite in production, it's the database we chose for (the control plane of) an open source observability solution we're building (morio.it).
Just came here to say two things:
- rqlite is amazing and warmly recommend - Philip is an exceptionally dedicated, kind, and helpful maintainer
5 stars, no notes. Thank you Philip, I'm eternally grateful for all your great work.
yesterday at 5:09 AM
Was a fun read until the end where the author made damn sure that history records who drew the blueprint.
yesterday at 7:58 AM
Seems fair to me. The VC-backed SQLite services have been shouting about their services as if they came up with the idea, and this only balances that out.
He's right that histories start too late, and I'd like to see more posts like this that trace the history back further than the last 5 years. Like for databases, a lot can go back 20+ years and it's good to read how the idea came about (independently from many people), how it developed over time and why.
yesterday at 6:31 AM
"Honesty" is what crossed the threshold for me. It's pretty weird to spend 12 years on something and not bother to write the thousand word history yourself.
yesterday at 4:33 AM
I know that latest rails suggests SQLite as one of the production options. I don’t get how it’d work with concurrent access though. Time to look into it.
yesterday at 9:16 PM
I'm using it w/ a Rust server and I can get up to 80k writes per second, 200k reads per second. You won't get that with other databases without paying a lot and doing a lot of special architecture.
yesterday at 9:41 PM
That sounds like a lot. Is the rust process serving parallel requests?
yesterday at 10:01 PM
Rechecked my benchmarks. 193,000 writes per second (rust), each write holds the lock for about 5 µs.
Full report here: https://github.com/pairshaped/gleam_sqlite_benchmarks/blob/m...
Obviously real world queries will vary, but I tried to be representative of the type of queries / updates I am sending.
yesterday at 4:40 AM
Wal or wal2 can help you with that.
yesterday at 5:55 AM
Can use one sqlite per tenant or look into the new multi writer lib on github.
yesterday at 12:55 PM
Because sqlite is extremely fast. Most projects won't hit its limits
yesterday at 9:58 AM
So, has this been Jepsen-tested yet?
Because if it hasn't, it's likely there are problems lurking.
yesterday at 12:56 PM
https://philipotoole.com/testing-rqlite-read-consistency/
yesterday at 4:02 PM
“Their tests showed that rqlite maintains linearizability of record insertion and modification, provided strong read consistency is enabled.”
The rqlite docs however tell you to never use Strong mode and that Linearizable is sufficient. https://rqlite.io/docs/api/read-consistency/#strong
yesterday at 6:30 PM
This smells a lot like the situation MongoDB had in its early years that made it lose a lot of trust in the community.
yesterday at 10:02 AM
[flagged]
yesterday at 4:26 AM
There is also hctree from sqlite.
yesterday at 3:17 AM
Very timely for me. I run dqlite in a small cluster and I'm looking to move off it to something more robust. What's recommended in this regime?
yesterday at 4:08 AM
What kind of issues did you run into?
yesterday at 4:11 AM
Loads of '500 database locked' errors. https://github.com/canonical/dqlite/issues/908
yesterday at 8:29 PM
Have you tried https://github.com/canonical/dqlite/pull/909 ?
today at 3:44 AM
Yes, as I noted in the issue. dqlite is the kind of "load bearing" project that would benefit from formal verification: https://ubuntu.com/blog/hunting-a-16-year-old-sqlite-bug-wit...
today at 12:26 PM
Ah yes, sorry - saw your name there too but didn’t read closely enough.
Are you still going to use dqlite after this experience, or has it shaken your confidence in it? Also what’s been your experience with microk8s itself - I hadn’t heard of it before
yesterday at 5:56 PM
Really disappointing to see that someone can't be bothered to tell the story of their own project in their own words.
yesterday at 4:03 AM
Like the idea of distributed SQLite, but do we have benchmark? What use cases it has advantage?
yesterday at 4:10 PM
I'd love to see a distributed sqlite solution using something like OmniPaxos[1] being used for consensus instead of raft
[1]: https://github.com/haraldng/omnipaxos
yesterday at 3:11 AM
today at 3:57 AM
[flagged]
yesterday at 2:29 PM
[flagged]
yesterday at 1:56 PM
[flagged]
yesterday at 12:30 PM
[flagged]
yesterday at 5:24 AM
[flagged]
today at 5:43 AM
[dead]
yesterday at 7:38 AM
[flagged]
yesterday at 3:38 PM
[flagged]
yesterday at 3:15 PM
[flagged]
today at 12:30 AM
[dead]