I was excited at first and thought this was SQL atop S3 et al. I've jerryrigged a solution to this using SQLite with a customized VFS backend, and would suggest that as an alternative to this particular project. You get the benefit of ACID transactions across multiple tables and a distributed backend.
Edit: actually not sure if you can use :memory: mode since Litestream uses the WAL (IIRC), so maybe a ramfs instead
The website is a bit fancy but the readme seems to pretty straightforwardly explain why you might want this. It seems to me like a nice little (13k loc) project that doesn't fit my needs but might come in handy for someone else?
You generally can't use object storage directly for this stuff; if you have a high volume of writes, it's incredibly slow (and expensive) to write them individually to s3. Similarly, on the read side you want to be able to cache data on local disk & memory to reduce query latency and cost.
You can check out Neon.tech who makes an OS Postgres-on-s3 and DuckDB who makes an embedded DB with transaction support that can operate over S3
Essentially every standalone or embedded key-value storage solution treats the KV store and its operation like a database, from what I can tell -- which is sensible because that's what they are! But people use object stores exactly because they don't operate like traditional databases.
Now there are problems with object stores (they are very coarse grained and have high per-object overhead, necessitating some design that can reconcile the round hole and the square peg) -- but this is just the reality of what people are working with. If there is some other key-value store server/implementation you know of, one that performs and offers APIs like an actual database (e.g. multi writer, range scans, atomic writes) but with unlimited storage, no provisioning, and it's got over 10+ different widespread implementations across every major compute and cloud provider -- I'm interested in what that project is.
I'm confused here, because Google says it's a db bundled with the application, but that's not really what I get from the landing page.
What problem does it solve?
Looks like it has a pretty similar structure under the hood, but DuckDB would get you more powerful queries.
FYI duckdb directly supports writes (and transactions) so you don’t necessarily even need the separate store step.
I don’t really know rust either but I’m currently writing some bindings to expose Rust libraries to NodeJS and not having too much trouble.
For rust -> c++ I googled one time and found this tool which Mozilla seems to use to call Rust from C++ in their web browser, maybe it would “just work”: https://github.com/mozilla/cbindgen?tab=readme-ov-file
Try 12.
> NOTE
> Snapshot isolation and transactions are planned but not yet implemented.
“ Snapshot isolation: SlateDB supports snapshot isolation, which allows readers and writers to see a consistent view of the database. Transactions: Transactional writes are supported.“
I don’t know if this was intended to be intentional funny, but there is a little ambiguity in the expression “great cost”, typically great cost means very expensive.
Very cool and useful shim otherwise :)
Always that way around, ESL's a minefield!
(Native BrE, SW UK.)
At a great cost is exactly at a very high cost, not "at a good price".
Any actual sources for your claim except for being a "native" speaker.
What makes SlateDB not qualify for this definition? It seems to qualify for me.
> Small databases can be stored on a file system, while large databases are hosted on computer clusters or cloud storage. The design of databases spans formal techniques and practical considerations, including data modeling, efficient data representation and storage, query languages, security and privacy of sensitive data, and distributed computing issues, including supporting concurrent access and fault tolerance.
SlateDB delegates all of this to the object storage behind it. (I don't mean it in a disparaging way at all, just a fact)
Rust is needed to use SlateDB at the moment
Sounds like they just cancel each other out. Not sure what advantage embedding will yield here
I've built something similar for low-cost storage of infrequently accessed data, but it uses our DBMS (MySQL) for the WAL (+ cache of hot reads), so you get proper durability guarantees.
The other cool trick to use is to use Bε-trees (a relatively recent innovation from Microsoft Research) for the object storage compaction to minimise the number of write operations needed when flushing the WAL.
https://docs.rs/slatedb/latest/slatedb/config/struct.WriteOp...