OrioleDB beta7: Benchmarks
135 points by oliverrice 4 days ago | 14 comments
  • akorotkov 4 days ago |
    Some notable benchmarks from the OrioleDB beta7 release:

    * 5.5x Faster at 500 Warehouses: In TPC-C benchmarks with 500 warehouses, OrioleDB outperformed PostgreSQL's default heap tables by 5.5 times. This highlights significant gains in workloads that stress shared memory cache bottlenecks.

    * 2.7x Faster at 1000 Warehouses: Even when the data doesn't fit into the OS memory cache (at 1000 warehouses), OrioleDB was 2.7 times faster. Its index-organized tables improve data locality, reducing disk I/O and boosting performance.

    Try it yourself:

    Clone the OrioleDB repository from GitHub and follow the build instructions, or use their Docker image. https://github.com/orioledb/orioledb#installation Alternatively, run OrioleDB on Supabase. Read the blog post for more details. https://supabase.com/blog/orioledb-launch

    Run your own workloads or existing benchmarks like go-tpc or HammerDB to see the performance differences firsthand. We Would love to hear about others' experiences with OrioleDB, especially in production-like environments or with different workloads.

  • znpy 4 days ago |
    > Our roadmap includes support for multi-master configurations, enhancing availability and fault tolerance. This will allow for read and write operations across multiple nodes, improving performance and resilience.

    Fu--ing finally!

    • oliverrice 4 days ago |
      what's your use-case for multi-master that would not be supported by something like regional routed for read-replicas with high availability? i.e. do you have a specific need for low global write latency, or is it motivated by something else?
      • znpy 3 days ago |
        > do you have a specific need for low global write latency, or is it motivated by something else?

        It's motivated by liking of reliable systems that keep working when a node fails.

        There are a bunch of non-webscale companies that pick mysql over postgresql because they can use either percona xtradb cluster or galera cluster.

        Having an open source multi-master solution would mean that postgresql could finally be used over there as well.

        • oliverrice 3 days ago |
          yeah, fair shout `columnar index` for hybrid analytical workloads and `multi-master` roadmap items haven't been put in an order on the schedule yet so maybe multi-master makes sense to do first. Both align with webscale requirements though. Columnar would move into the AlloyDB space and multi-master would be more like PlanetScale (although not quite the same audience in the latter case)
        • akorotkov 3 days ago |
          Thank you, your feedback is appreciated.

          The important design issue about building active-active multi-master on the base of raft protocol is about being able to apply changes locally without immediately putting them into a log (without sacrificing durability). MySQL implements a binlog, which is separate from a log, to ensure durability. OrioleDB implements copy-on-write checkpoints and row-level WAL. That gives us a chance to implement MM and durability using a single log.

  • merqurio 3 days ago |
    All my kudos for the OrioleDB team, they have been working for years with the Postgres core devs to get the extensibility patches they need for their storage extensions merged back.

    It’s not just about building their extension but actually making Postgres better for everyone. I would have loved that big corps would have taken this approach, as it opens the doors to others to add features for different use cases and making postgres more of a DBMS framework

    • kiwicopple 3 days ago |
      > I would have loved that big corps would have taken this approach, as it opens the doors to others to add features for different use cases

      EDB and Cybertech definitely made a great start with Zheap[0] although the initiative stalled for whatever reason

      Hopefully the community can support this effort to improve the Table AM API - it would be beneficial even beyond oriole. As you point out, a Pluggable Storage system in Postgres would open up a few new use cases

      [0] https://wiki.postgresql.org/wiki/Zheap

  • vvern 3 days ago |
    Can people please, for the love of god, stop running tpcc with think time disabled. When run in this way it is not the TPC-C benchmark, and is not "simulating real database workloads that is considered a modern standard in database applications." TPC-C generally has an open-loop traffic arrival rate that scales with the size of the data and is lightly contended. When run without think time, it becomes closed loop, and generally dominated by the contention that was not supposed to be dominant.

    This instance is less bad than some in that it's at least comparing the same sort of database and doing it using the same driver -- so it is at least an apples to apples measurement of something.

    Still, please, as a community we need to stop getting rid of the think time and quoting the output as tpmC or as a standard benchmark.

    See https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-c... for the spec.

    • pashkinelfe 3 days ago |
      I'll explore performance with this parameter. Thanks for the advice!
    • akorotkov 2 days ago |
      Thank you for your feedback! We tried to enable think time with go-tpc, thanks to @pashkinelfe. That leaves us with 1 tpmC per connection, growing linearly up to ~300 connections for both heap and OrioleDB. So, in order to experience a storage bottleneck, we would need dozens of thousands of connections. Given PostgreSQL runs a process per connection, that would be more of a stress test for the Linux kernel. Additionally, PostgreSQL requires N^2 of memory depending on the number of connections, and it becomes sensible at this scale. All of that could be resolved by migrating PostgreSQL to co-routines and resolving memory requirement issues. However, this is currently out of scope for us. Could you recommend another benchmark to reveal storage bottlenecks, given that TPC-B and YCSB are too trivial?
      • akorotkov 2 days ago |
        > Additionally, PostgreSQL requires N^2 of memory depending on the number of connections,

        For sure, not all the PostgreSQL memory is N^2. AFAIR, just a couple of components, including deadlock decoding, require a quadratic amount of memory. Normally, they are insignificant but growing fast if you are rising max_connections.

      • vvern 2 days ago |
        One approach to mitigate the connection problems for tpcc would be to utilize a connection pooler like pgbouncer or yandex/odyssey. It’s certainly more complexity.

        Another suite to look at is sysbench. It’s very flexible, for better or for worse, but it can allow you to create an interesting mix of queries at different scale factors. For something like this where you’re going head to head with Postgres, having more dimensions with more benchmarks isn’t going to hurt. Ideally you’ll see a nice win across the board and get an understanding of the shape of differences.

  • benpacker 2 days ago |
    Very excited about this and it’s future!