If all you want to do is queues and whatnot, then sure, you don't need an in-memory KV store.
The point of an in-memory KV store is to do stuff that needs the performance characteristics of RAM. You obviously can't get the performance characteristics of RAM over a network connection. This is like, a tautology.
Why Postgres wouldn’t run on the same server as the app? It’s actually pretty common.
Also, never realized redis has native support for a Bloom Filter.
Acting as shared memory for an inherently single-CPU language like JS is one I can think of. However, I don't use Redis, so you'd be better placed to drive the discussion forward with examples.
IPC, I suppose?
My Sidekiq background job system runs entirely on top of Redis. Structures like Sorted Sets become the basis for indexes. Lists provide extremely fast queue behavior and Hashes map easily to persistent objects. Databases, traditionally, have not performed well when used as queues.
Those are the big 3 structures necessary to implement anything: trees, lists and maps.
I do see the point of Redis if you have multiple hosts, but I was unsure why someone would use it on just one host.
That's perfectly fine. You can compare Redis to other specialized tools just like you can compare it with Postgres and SQLite.
If you only have one long lived process or good global variable control, then it is much less appealing in the single-server scenario. Similarly, if you require access from multiple hosts, it becomes a less obvious choice (especially if you already have a database in the loop). And redis is also overkill is you’re using it only as a cache.
As in performance improvement - cache should never be considered a datastore, e.g. you can pull the plug and nothing else happens (aside losing performance). It'd be a lot more beneficial all the processes to have a local cache, themselves. The latter is at least 4 orders of magnitude faster than redis. Now you may like some partitioning, too.
What's the overhead of postgres vs redis, when run locally? Why do you think postgres isn't run locally?
There's nothing special about postgres. It's just a program that runs in another process, just like redis. For local connections, it uses fast pipes, to reduce latency, and you get access to some faster data bulk transfer methods. I've used it in this way on many occasions.
Except for temporary tables which are wiped after each session.
Postgres has a shared memory cache, which can be set the same as redis, so your operations will all happen with in memory, with some background stuff putting it onto disk for you in the case your computer shuts off. Storage won't be involved.
BUT, postgres still has ~6x the latency [1], even when run from memory.
[1] https://medium.com/redis-with-raphael-de-lio/can-postgres-re....
And having a locking system fluctuate in latency between milliseconds and seconds would cause all sorts of headaches.
> And having a locking system fluctuate in latency between milliseconds and seconds would cause all sorts of headaches.
With the frequency that a locking system is likely to be used, it’s highly unlikely that those pages would ever get purged from the buffer pool.
a) Writes will be happening on a slow, highly contentious disk.
b) You have no guarantees that the data will be in-memory.
Both of which make it a poor solution for use cases such as locking.
That's fine. If what you want is a single-node deployment combined with in-memory purism them SQLite has your back.
If you have to parse and plan queries every time PostgreSQL is obviously much slower than Redis. It is much more interesting to see what happens if prepared statements are used.
https://github.com/raphaeldelio/redis-postgres-cache-benchma...
A bit more work yes that could be simplified, but fully supported if you control the stack.
Highly recommended for running tests, especially in CI/CD pipelines. Doing this simple change can speed up DB-heavy tests 30-50%.
Is that relevant though? Some benchmarks on the web show Postgres outperforming Redis in reads as well as low-volume writes (less than 1k key-value pairs), and Redis only beating Postgres for high volume key-value writes.
I wonder what type of use cases have high volume writes.
Maybe queuing, locking and pub/sub ?
But for lots of cases, especially internal business tools, we can scale up a single instance for a long time, and this in-memory caching makes things super fast.
There’s a library, django-cachalot [1], that handles cache invalidation automatically any time a write happens to a table. That’s a rather blunt way to handle cache invalidation, but it’s wonderful because it will give you a boost for free with virtually no effort on your part, and if your internal business app has infrequent updates it basically runs entirely in RAM, and falls back to regular database queries if the data isn’t in the cache.
And it's far more likely you are continuously upgrading your process than Redis.
Ideally every significant part should live in its own universe, only interacting with other parts via well-defined interfaces. Sadly, it's either more expensive (even as Unix processes, to say nothing of Windows), slower (Erlang / Elixir), or both (microservices).
Or do you compare using separate processes to having multiple threads in a single process?
As far as isolation goes, if you are worried about the properties of reading and writing data to the disk then I simply don't know what to tell you. Isolation from what?
Perfomance-wise, I do not know of a nice portable way of flushing changes to disk securely that does not block (like, e.g. fsync does).
If you own the whole system and can tune whole kernel and userspace to run single application, sure, why overengineer. Otherwise software faults (bugs, crash due to memory overcommit, oom killer etc.) take down single process, and that can be less disruptive than full stop/start.
I think a good rule of thumb is:
* If you need it from a single process and it doesn't need to survive process restarts, just use basic in-process data structures.
* If you need it from multiple processes or it needs to survive process restarts, but not system restarts, use redis.
* If it needs persistence across system restarts, use a real DB.
It does... and it's a blocking operation (Bad). It's used when storing data to a file to allow restarts w/o losing the said data.
Redis helps you get to the point (need cache on separate machines) where you might want to consider moving to postgres, or not because you don't feel it warrants the investment of programmer hours at that point.
on edit: got downvoted for ..? I guess what I said was so supremely stupid I deserved it. I do admit it was perhaps slightly obvious, but not so obvious that it warranted downvoting
How could that very clear statement be misunderstood as to be well I am building a bunch of stuff I might need to tomorrow, it is the exact opposite of that!
I guess it's my fault, nobody on HN seems to understand a single thing I say even though the words seem reasonably clear to me.
But yeah redis is not an answer for most single machine questions.
That said, anitirez renaming it to lredis or reldis would be epic and one of my favorite moves of all time
I want a “redis” with something akin to the Consul client - which is a sidecar that participates in the Raft cluster and keeps up to date, cheaper lookups for all of the processes running on the same box.
The few bit of data we needed to invalidate on infrequent writes went into consul, and the rest went into the dumbest (as in boring, not foolish) memcached cluster you can imagine.
But as you say there was the network overhead, and what would be lovely is a 2 tier KV store that cached recent reads locally and distributed cache invalidation over Raft. Consistent hashing for the systems of record, broadcast data on put or delete so the local cache stays coherent.
I wonder if something like Cockroach DB might even work for small clusters.
I feel that the virtualize and distribute everything to hell and back-trend might actually be about to break, there are signs, and G knows it's about time. The amounts wasted on cloud providers for apps that would run everything just fine on a single server, the effort wasted configuring their offerings, surreal.
I have no idea how fast Redis can get, but it is entirely possible for an RDBMS to execute a query in well under a millisecond. I have instrumentation proving it. If everything is on the same machine, I would wager that IPC would ultimately be the bottleneck for both cases.
I don't think this is a realistic scenario at all.
If you need a KV store, you want it to be the fastest by keeping it in-memory, you want it to run on each node, and you don't care how much it cost to scale vertically, then you do not run a separate process on the same node. You just keep a plain old dictionary data structure. You do not waste cycles deserializing and serializing queries.
You only adopt something like Redis when you need far more than that, namely you need to have more than one process access shared data. That's at the core of all Redis usecases.
https://redis.io/docs/latest/develop/interact/search-and-que...
After the significant success of the first conversion, we've been working to convert all the rest of our apps.
And no, host language data structures aren't useful because they aren't in shared memory between all the worker processes, and even if we found a module that implemented them in shared memory, we like to be able to preserve the sessions across a host restart, and then we'd need a process to save the data structures to disk and load them back, and by the time we did that we'd have just reinvented redis.
Also session data is often Blobs which db's don't process as efficiently as columnar data.
You might if you want the KV to persist between app restarts (for warm starts.)
in that case just use a regulator hashmap - it has nanoseconds performance compared to the sub-millis.
The entire point of Redis that it trades durability and convenience for performance.
When the processed message rows are deleted, the pages sitting on the disk are not deleted until vacuum is run. If you run an online vacuum it doesn’t delete the page from disk, and a full vacuum (that will free up the disk space) locks the database while it completes the vacuum (which if you’re now dealing with high throughput, is not great).
One approach to address bloat without dealing with vacuum is to setup a timestamp-based partition on your queue table. This way you just drop old partitions and it frees the disk space without needing vacuum.
- agreed, autovacuum is definitely something to keep a close watch on – but as long as disk I/Os are fast enough (i.e. not EBS), our queue tables could fill the space freed for PgSQL but not the OS (the non full VACUUM mentioned by my GP), so in the end, assuming the queue did not grow infinitely, actual disk usage stabilized, and we never had to VACUUM FULL outside of buggy (on our side) situations;
- high-throughput is relative: we ran in the range of hundreds to thousands of tasks per second without any issue nor any particular customization, so sure, PgSQL will not hold the load if you are Google, but I'd bet it should satisfy the needs of at least up to the last centile of companies.
- I like your idea about timestamp-defined partitions, I will definitely keep this one in mind if the need arises.
Now on the pros:
- being able to store & manipulate basically all the data handled by our stack from a single source sounds like a minute detail, but actually helps a lot when we have to debug an issue;
- being able to use virtually any language and still be able to access all the persistent data from our stack from a single source revealed itself to be an unexpected advantage to develop a quick'n dirty e.g. python/nim/racket script to explore data/debug/generate stats/etc.;
- our PgSQL machines ran much better “by themselves” than we feared at first, and finally were far from being the main point of contention on day-to-day ops – in the end we just altered a couple of settings and just... let them be;
- PgSQL provides a lot of tools to inspect it live, which provides additional help to identify bottlenecks in (i) your server setup and (ii) your stack;
- PgSQL support of JSON/JSONB is actually very good, and very helpful to store that hard-to-normal-form-ize last part of your data.
> ”PgSQL provides a lot of tools to inspect it live”
Any particular tools you like for this?
One of the Planetscale guys did a podcast and said at GitHub every MySQL instance is doing like 50k+ connections while if you need more than 100 connections in Postgres you already need PgBouncer.
You can have as much connections as you want, but you'll have to trade it for having lower work mem numbers, which hurts performance. Traditional advice is to keep it below 500 per PostgreSQL instance (I'd say physical host).
I've ran dozens of micro services handling thousands of requests per second with a total connection limit of around 200 of which most was still unused - all without any server-side pooler.
> it doesn't matter if they are all active
It does, if the connection is inactive (doesn't hold an open transaction) you should close it or return it to the pool.
Between transactions? Yes, absolutely
In fact, many libraries do it automatically.
For example, SQLAlchemy doc explicitly says [0]:
> After the commit, the Connection object associated with that transaction is closed, causing its underlying DBAPI connection to be released back to the connection pool associated with the Engine to which the Session is bound.
I expect other reasonably sane libs for working with transactional databases do the same.
So, if you are doing pooling correctly, you can only run out of available connections if you want to have a lot of long running transactions.
So, why would you want every of your 50k frontends keep an open transaction simultaneously?
[0] https://docs.sqlalchemy.org/en/20/orm/session_basics.html#co...
Of course, the better design is to write a nonblocking worker that can run async requests on a single connection, and not need a giant pool of blocking workers, but that is a major architecture plan that can't be added late in a project that started as blocking worker pools. MySQL has always fit well with those large blocking worker pools. Postgres less so.
From the perspective of keeping the number of open connections low it doesn't really matter if you close it or return to the pool, because in either case the connection becomes available to other clients.
If you are saying that a connection pool can be shared between processes without pgbouncer, that is news to me.
> Parent comment is talking about one connection per process with 50k processes.
It is actually not clear what parent comment was talking about. I don't know what exactly did they mean by "front ends".
It's just an architectural decision to spawn a process per connection that Postgres made long time ago.
It's a tradeoff like most decisions.
Back in the days MySQL had huge issues with scaling to multi-core systems (maybe they fixed it now, I haven't used MySQL for a long time) while Postgres never had this problem.
When designing an application around Postgres you just have to take into account that:
1. opening a new connection is rather expensive, so you should have a connection pool
2. opening too many connections is problematic, so you should keep the size of your pool rather small and return connections back into the pool ASAP
That's it.
It's not that hard in practice.
Postgres uses an old design model which predates threads; I have no idea if they have made any progress in updating their design. In the past I have heard the core devs talk about how difficult it would be to do this.
Oracle Database moved to a hybrid process/thread model at some point, this is the superior solution ( I have no idea if it was done well or not, but from standpoint of how to achieve better leverage of CPU vs IO, this is the way ).
If the PG devs had enough time/money, I am sure they would move towards a hybrid model where the focus would be on processor affinity with IO being all tied to events.
Unrelated to mysql - I do consider using redis in any capacity a blunder (it's likely ok for nodejs users, I suppose)
[0]: http://dimitrik.free.fr/MySQL_Connect_2013/MySQL_Perf-Connec...
A counterpoint is that the barrier to using redis is tiny and in exchange you have a very high performing system that has extensive library support and takes load from your primary database.
If you want to cache API responses, for example, could you do that in Postgres? Sure. And you could support things like TTLs with cron jobs sweeping stale cache values. Or you can just use redis.
Advisory locks are cool and useful. They can be a little problematic you want something like PGBouncer and you’re stuck between session advisory locks and transaction interleaving.
Having separate systems has downsides (network calls, availability, domain knowledge), but the tradeoffs with Redis of all things are pretty low.
(I've not read the article: the title alone seems like an orange vs apple sales)
Obviously this can be fine, but for plenty of Redis use cases it really isn't equivalent with that limit.
There is no namespace or table for that. Meaning that the single common locking space will be shared by all the users/apps of your database.
The key has to be an int with a quite low limit (like 30k) if my memory is good. So you can't use the key to use for namespacing.
Redis has many niches where it is more suitable than a database. In particular, the low latency and simple concurrency model make it a perfect choice for building shared token buckets, for example. For simple operations that need to be executed many times at low latency, SQL is usually not the best choice. And sometimes, you really just need K/V and don't want to deal with schema migrations, DBA chores, debugging auto vacuum...
>Advisory locks allow you to leverage the same locking engine PostgreSQL uses internally for your own application-defined purposes.
That is like telling me that I can get across the river in a nuclear submarine instead of walking over the footbridge. PostgreSQL locks cause me plenty of headaches already without my application logic being dependent on them.
- Want a KVstore? Do you know autovacuum? Do you know the connections pool limit? Do you want throughput or safety?
- Want a queue? is it sequential? Is it throttled? Is it fanout? Is it split by topic?
- Want a pubsub? do you care about more than once? do you care about losing messages? do you care about replaying messages?
- Want a lock? Do you know the connections pool limit (again)? Do you know about statement_timeout?
Yes, you can solve almost all issues listed above, but it's not that trivial.
There are lots of these, several of which are commercial ventures so there's definite interest behind it:
https://worker.graphile.org/ (Node.js)
https://riverqueue.com/ (Go)
https://github.com/acaloiaro/neoq (Go)
https://github.com/contribsys/faktory (Go)
https://github.com/sorentwo/oban (Elixir)
https://github.com/procrastinate-org/procrastinate (Python)These days with m.2 disks and database based cacheing (solid cache) and first class support in Rails, and postgres able to handle the job queues and things like GoodJob the choice isn’t so clear cut to “just use redis”.
I won’t be replacing sidekiq in our current production app any time soon but any new projects would get some serious consideration.
But that’s not what I thought the point of the article was. I think the main argument from the article can be summed up with this line:
> PostgreSQL has a lot more capabilities than you may expect when you approach it from the perspective of just another SQL database or some mysterious entity that lives behind your ORM.
To me, this is the key line. If you’re only using your database from behind an ORM (any database, IMO), you’re probably missing functionality. And if you need to add another service (like Redis), you may be better served by using the database you already have setup, rather than adding another dependency.
Redis is pretty awesome. So is rabbitmq. When you’re a < 100 person startup, probably stick with postgres. Over 100 and I would seriously start considering adding more specialized platform tools.
Caching, often the most important application of Redis: oops.
Updates in Postgres are notoriously more expensive than inserts (generate garbage, require vacuuming); durability guarantees, not important for caching, make writes significantly slower.
Automatic expiration is very convenient and fool-proof.
- https://github.com/janbjorge/PGQueuer
disclaimer: Im the auther.
Keep the number of moving parts as low as possible.
Sure it's possible, but I prefer sticking to more widespread use of postgres.