DATABASE SYSTEMS BENCHMARKING

András Gábor, gabora@ inf.unideb.hu
University of Debrecen, Institute of Informatics, Department of Information Technology

Abstract

Database systems play a fundamental role in information technology ever since the early days. Different models were in use before the relational database model emerged and took the leading role. Today all business and most non-business applications use RDBMS software products, which became just as important as the hardware or the operating system.
The available RDBMS software products vary in many ways. They differ in prices, performance, manageability, scalability, functional capabilities, etc. It is hard to make a good comparison when we have to decide on one for a project. We need a good database performance measurement result, in other words a benchmark result. We can either design and conduct our own or rely on some public benchmark results.
In this paper I will give an overview of the traditional benchmark approaches that are available and most often used. I will briefly present the models used in these tests and the measures they report. I will also point out the drawbacks of these well-known public results, such as the TPC benchmarks that may be misleading at times. Here the main reason is that benchmarking is done by the vendors themselves and the results not only represent the capabilities of the software but the special DBA and tuning knowledge of the team performing the tests.
These traditional benchmarks measure the relational capabilities of the databases. But most RDBMS vendors are now implementing new functionalities in their software too. Object relational and other SQL:1999 extensions are examples of these. The available benchmarks don’t measure the benefits of their usage in any ways and application developers don’t use new features possibly because they don’t know much about them. For instance the nested relational model is more than ten years old and it has been implemented in database systems since 1997.
There is a need for benchmarking the new features, because these can save development time, increase maintainability and evolvability and sometimes performance as well. I will propose benchmarks that measure some of the new functionalities in RDBMS (ORDBMS) products.