PostgresBench is a new open-source benchmark framework that compares the performance of various managed PostgreSQL services in a transparent way, allowing anyone to verify the results.
Imagine you are choosing a database from a cloud provider for an important service. Every provider claims, “Our service is the fastest.” Yet, when you actually test them, the results are all over the place. Why do these differences exist? Perhaps it is because the test environments differ, or the measurement methodology lacks transparency.
Recently, a “transparent report card” has emerged to quench this thirst—one that anyone can trust and utilize. It is called ‘PostgresBench’.
Why is this important?
The database is the heart of a service. If the heart beats slowly, the entire service feels sluggish. Developers and companies pay for ‘managed PostgreSQL’ (where you rent a server pre-configured with PostgreSQL), but it is not easy to judge how well they will actually perform for your specific use case.
PostgresBench provides objective criteria for these vague questions. Because all testing methods, scripts, and result data are published, anyone can repeat the tests under identical conditions to verify performance personally Source: PostgresBench: A Reproducible Benchmark for Postgres Services. In other words, rather than just relying on vendor advertising, “trustworthy comparison” that we can verify ourselves has become possible.
Making it easy to understand
To understand PostgresBench, think of a ‘standardized exam.’ In an exam, every student is given the same test paper and their skills are measured over a set amount of time. That is how you compare scores fairly.
PostgresBench is the same. This tool uses an industry-standard tool called ‘pgbench’ to conduct tests as if handing out a common exam paper Source: PostgresBench — A Reproducible Benchmark for Postgres Services; Source: PostgreSQL: Documentation: 18: pgbench. This exam paper includes ‘TPC-B-like tasks’—complex processing patterns frequently used in real-world practice, such as data insertion, deletion, and modification Source: PostgresBench: A Reproducible Benchmark for Postgres Services; Source: PostgresBench: Reproducible Benchmark for Managed Postgres.
In short, PostgresBench acts as a fair referee that provides the same ‘playing field with equal difficulty’ to database ‘athletes’ to measure who handles the work faster and more reliably Source: PostgresBench: Open Benchmark for Postgres Services.
Current Status
PostgresBench included the following famous services in its first test cohort:
- Postgres by ClickHouse
- AWS RDS
- AWS Aurora
- Crunchy Bridge
- Neon
It evaluated performance on these services using two data sizes: 100GB and 500GB Source: PostgresBench: Reproducible Benchmark for Managed Postgres; Source: PostgresBench: Open Benchmark for Postgres Services. Furthermore, it measured throughput, latency, and stability by applying continuous load for 10 minutes in real-world environments, such as 256 clients connecting simultaneously and 16 active threads Source: PostgresBench: A Reproducible Benchmark for Postgres Services.
What lies ahead?
PostgresBench is highly likely to become the new ‘standard’ for database performance comparison. Much like ‘ClickBench’, which has already established itself as a transparent methodology in the field of analytical databases, PostgresBench will likely be utilized as a core metric for selecting PostgreSQL services Source: PostgresBench: A Reproducible Benchmark for Postgres Services.
Users will no longer rely solely on vendor marketing copy but will be able to verify and choose the optimal database for their business scenarios based on published scripts and configuration values Source: PostgresBench: Reproducible Benchmark for Managed Postgres.
MindTickleBytes AI Reporter’s Perspective
Databases are the foundation of technology, yet performance measurement has often been a ‘black box.’ Some vendors test only under conditions highly favorable to them. The ‘perfect transparency’ that PostgresBench aims for holds meaning beyond simple benchmarking. Disclosing technical truths demonstrates confidence in the service and, above all, gives users like us the power to make smarter choices about better technology. Isn’t this the healthy way for technology to advance?
References
- PostgresBench: A Reproducible Benchmark for Postgres Services - https://clickhouse.com/blog/postgresbench
- PostgresBench — A Reproducible Benchmark for Postgres Services - https://postgresbench.clickhouse.com/
- PostgresBench: A Reproducible Benchmark for Postgres Services - https://github.com/ClickHouse/PostgresBench/
- PostgresBench: Reproducible Benchmark for Managed Postgres - https://www.devdigest.org/articles/postgresbench-reproducible-benchmark-for-managed-postgres
- PostgresBench: A Reproducible Benchmark for Postgres Services - https://vuink.com/post/pyvpxubhfr-d-dpbz/blog/postgresbench
- PostgresBench: A Reproducible Benchmark for Postgres Services - https://hb.int2inf.com/s/item/5yvx36P6dyEyUz8CE2Hur5-postgresbench-benchmark-for-managed-postgres-services
- PostgreSQL: Documentation: 18: pgbench - https://www.postgresql.org/docs/current/pgbench.html
- PostgresBench: A Reproducible Benchmark for Postgres Services - https://www.weaving.news/news/019ee692-e7e3-7289-8bf4-5a0b6f53ed74
- PostgresBench: Open Benchmark for Postgres Services - https://www.linkedin.com/posts/clickhouseinc_postgresbench-a-reproducible-benchmark-for-activity-7445500419889377280-Cpm6
- PostgresBench: A Reproducible Benchmark for Postgres Services - https://hb.int2inf.com/en/s/item/5yvx36P6dyEyUz8CE2Hur5-postgresbench-benchmark-for-managed-postgres-services
- To change the design of databases
- To transparently compare the performance of managed PostgreSQL services
- To check for security vulnerabilities in databases
- sysbench
- pgbench
- ClickBench
- It is based on private test results
- It publishes all results, configurations, and scripts so anyone can verify them
- It was created solely to promote a specific company's service