MySQL On ARM

openGauss on ARM

2022-08-23T00:00:00+00:00

openGauss is an open-source relational database built for internet-scale and optimized for ARM architecture (retaining compatibility with x86). DB kernel is derived from PostgreSQL so PgSQL users will find a lot of things familiar but a series of optimizations are added to make it faster and support distributed setup (cluster ecosystem). It supports a row-based, column-based, and in-memory storage engine with full ACID compliance. The ecosystem and contribution from the community continue to grow.

Given the DB is optimized for ARM, it sparked my interest and so I decided to evaluate it.

Key Features

openGauss was derived from PostgreSQL 9.2 but post that many features were added to openGauss to make it more enterprise-ready.

Multi-threaded (vs multi-process): One of the most discussed features of PgSQL is the need for multi-threading. Fortunately, openguass has been ported to use a multi-threaded model. Each process in PgSQL maps directly to a thread.
Thread Pool: PgSQL still doesn’t have a thread pool and needs an additional component like pgpool or so. openGauss has an inherent thread pool allowing it to scale and handle the multiple active short-lived connections effectively.
Incremental Checkpoint: Given the time it takes for the time-based checkpoint this feature comes as a savior with continuous checkpointing (more on lines of MySQL).
Doublewrite: To avoid half-cooked pages (again on lines of MySQL).
Optimization of Global Counter: Most global counters in opengauss use thread local storage copy that is then aggregated to the main counter.
NUMA scalability: openguass has been designed/optimized/tuned to ensure it scales well with multiple NUMA nodes and increases scalability and that is quite evident with broader spread use of thread local storage, multiple threads, threadpool, etc… allowing it to scale better with multi-numa node machines.

While a lot of these features are done a lot more is being done especially around IO.

Setup

Machine Configuration:
- ARM: 96 vCPU (4 NUMA) ARM Kunpeng 920 CPU @ 2.6 Ghz
Workload (using sysbench):
- CPU bound workload
- pgbench: select, update workload
- sysbench: read-only, read-write, write-only workload (pattern: uniform, zipfian)
Other configuration details here
- Shared Buffer: 80GB
- Data: 32 GB (pgbench)
- Data: 75 GB (sysbench)
Storage: SATA SSD
- sequential read/write IOPS: 65K/44+K (8K blocks)
- random read/write IOPS: 51+K/38+K (8K blocks)
openGauss Version: 3.0.1 [compile from source]
Scalability: 1-1024 threads: 21/42/84 threads for server and 3/6/12 threads for sysbench (core-binding).

Benchmark

Let’s first explore the benchmark using some standard suits like pgbench, and sysbench. We will then discuss more specific configuration, NUMA scalability, threadpool, how it performs compared to pgsql, etc…

pgbench

Observations:

pgbench read-only workload continues to scale well with increasing scalability but does hit a contention after a certain point that indeed suggests a scope of further improvement.
pgbench update workload continues to scale well with increasing scalability.

drop in the read-only workload of pgbench could be attributed to the memory allocation routine.
+ 3.40% 6236 worker gaussdb [.] GenericMemoryAllocator::AllocSetAlloc<true, false, false>

sysbench

Observations:

sysbench read/update workload continues to scale linearly before hitting a threshold, post that point the performance is almost flat (better than going down due to increased contention).

numa scalability

opengauss scales well with the increasing scalability. Let’s now explore how it performs with increasing NUMA nodes.

Observations:

opengauss is well optimized for NUMA nodes. With increasing NUMA for different variants of workload, it continues to scale well. Update workload could be further improved but like other databases, with 4 NUMA, performance doesn’t regress.

effect of threadpool

threadpool is best suited for the environment where there are a lot of short-lived connections. Let’s see the effect of threadpool.

Observations:

threadpool is ideally expected to improve performance and it does in opengauss case too but only for the pgbench-ro workload. For all other workloads including sysbench ro workload, it failed to show the improvement. Infact, for update workload, a serious regression is observed with threadpool.

pgsql vs opengauss

Opengauss is inherited from PgSQL so it would be interesting to see how these both perform. We tried our best to match the configuration using the latest release (PgSQL 14.5/OpenGauss 13.0.1) of both the databases and on comparable servers (24 ARM cores, 48 GB of memory, and similar IO volumes).

Observations:

Read-only workload performance looks comparable.
Update workload performance seems to show a significant difference despite of the multiple enhancement in opengauss. (Note: opengauss is operated with threadpool and incremental checkpoint turned off so things are comparable).

Conclusion

Based on the overall evaluation, numa optimization, features, etc.. opengauss seems promising. Given it is relatively new, there is ample scope for improvement.

If you have more questions/queries do let me know. Will try to answer them.

MariaDB on openEuler-ARM stack - Multi-NUMA study

2022-07-19T00:00:00+00:00

MariaDB on the openEuler-ARM stack has shown promising results during the initial evaluation. (If you have missed that blog click here). Taking the assessment further we decided to evaluate the setup in a multi-numa environment since it represents enterprise deployment. The aim is to find out if things scale like other OS-ARM stacks and if the bottleneck continues to remain the same or if something else pops up on the openEuler-ARM stack.

Setup

Machine Configuration
- ARM: 96 vCPU (4 NUMA) ARM Kunpeng 920 CPU @ 2.6 Ghz (openEuler - ARM stack)
Workload:
- sysbench: point-select, read-only, update-index, update-non-index
Other configuration details here[ (io-capacity/max=10K/18K)]
- data-size: 74GB, buffer-pool: 80GB (all-in-memory)
- redo-log-size: 20 GB
- storage: sequential read/write IOPS: 33+K/16+K/random read/write IOPS: 28+K/14+K
MariaDB Version: 10.10 (#88b2235 in progress)
- OS Version (mostly with default settings)
- openEuler: 20.03 LTS-SP2

Evaluation

read-only workload

Observations:

Just like other OS-ARM stacks, the read-only workload continues to scale almost linearly with the openEuler-ARM stack with increasing NUMA nodes.
Infact, due to the lower contention of some mutexes (Lock_table_cache), the performance of the read-only workload on the openEuler-ARM stack is better than other stacks.

write workload

Observations:

Again, we continue to observe the same pattern as other OS-ARM stacks. Write workload fails to scale on multi-numa nodes.
Top-Contention points continue to remain the same.

openEuler-ARM stack

| wait/synch/rwlock/innodb/log_latch                        | 19169202.9823 |  199868944 |
| wait/synch/rwlock/innodb/trx_rseg_latch                   |   785002.5091 |  123518641 |
| wait/synch/mutex/innodb/buf_pool_mutex                    |   638941.9645 |   56602817 |
| wait/synch/cond/mysys/COND_timer                          |   402259.4476 |       1532 |
| wait/synch/cond/aria/SERVICE_THREAD_CONTROL::COND_control |   389933.2353 |         13 |
| wait/synch/rwlock/innodb/fil_space_latch                  |   325025.1544 |    4433912 |
| wait/synch/rwlock/sql/MDL_lock::rwlock                    |   311635.8508 |  236808760 |
| wait/synch/sxlock/innodb/index_tree_rw_lock               |    46259.4764 |  134161482 |
| wait/synch/mutex/sql/LOCK_table_cache                     |    21153.7182 |   77818000 |
| wait/synch/mutex/sql/THD::LOCK_thd_data                   |    11876.1819 |  178734356 |

centOS-ARM stack

| wait/synch/rwlock/innodb/log_latch                        | 16410690.9523 |  215740728 |
| wait/synch/cond/mysys/COND_timer                          |   834065.5579 |       3066 |
| wait/synch/rwlock/innodb/trx_rseg_latch                   |   833276.8134 |  131173975 |
| wait/synch/cond/aria/SERVICE_THREAD_CONTROL::COND_control |   811668.6162 |         27 |
| wait/synch/rwlock/sql/MDL_lock::rwlock                    |   640257.8943 |  258275876 |
| wait/synch/mutex/innodb/buf_pool_mutex                    |   587507.9794 |   59236236 |
| wait/synch/rwlock/innodb/fil_space_latch                  |   341941.7540 |    4669815 |
| wait/synch/sxlock/innodb/index_tree_rw_lock               |    64058.7856 |  145183127 |
| wait/synch/mutex/sql/LOCK_table_cache                     |    38422.9402 |   84390443 |
| wait/synch/mutex/sql/THD::LOCK_thd_data                   |    11220.5507 |  196408431 |

the openEuler-ARM stack continues to perform on par with other stacks and to check that out that we did some comparative analysis against the centos-ARM stack (using ramdisk since both servers don’t have comparable IO systems).

Evaluation using thread pool

OS-Level Scheduler has a significant role to play with thread handling so we also decided to evaluate the effect of the thread pool.

Observations:

As expected, enabling thread pool has +ve effect on performance just like other OS-Hardware stacks.

Conclusion

Based on the evaluation we can comfortably say that MariaDB continues to perform on par/better on the openEuler-ARM stack (when compared to proven stacks). Given the said fact it could be worth considering supporting the openEuler-ARM stack.

If you have more questions/queries do let me know. Will try to answer them.

Evaluating MariaDB on openEuler-ARM stack

2022-07-06T00:00:00+00:00

openEuler is an opensource linux based operating system with a customized scheduler, io, libraries, etc.. It is optimized for ARM64 architecture and so it could be interesting to evaluate different enterprise software running on the openEuler-ARM stack. As part of this study, let’s explore how MariaDB that already has packages on ARM for different operating systems performs on openEuler.

Setup

Given that openEuler is optimized for ARM let’s try to evaluate it against ubuntu-arm stack (proven configuration).

Machine Configuration (from huaweicloud)
- openEuler-on-arm: 24 vCPU - 48 GB (Kunpeng 920 2.6 Ghz)
- ubuntu-on-arm: 24 vCPU - 48 GB (Kunpeng 920 2.6 Ghz)
Workload:
- sysbench: point-select, read-only, update-index, update-non-index
Other configuration details here
- data-size: 35GB, buffer-pool: 40GB (all-in-memory)/20 GB (part of data in memory)
- redo-log-size: 10 GB
- storage: EVS volume with 22K IOPS (read/write mixed).
MariaDB Version: 10.8 3 (GA).
OS Version (mostly with default settings)
- Ubuntu: 18.04.3 LTS
- openEuler: 20.03 LTS-SP2

Benchmarking

Read Only Workload

Let’s start with read-only workload. sysbench offers 2 read-only workloads: point-select, and read-only.

Observations:

MariaDB on openEuler continues to score for all kinds of scenarios (uniform, zipfian) with improvement crossing 25% in some cases.
Performance Profiling pointed out the following facts
- openEuler is better at handling contention (reported lesser contention).
- memcpy seems to be optimized in openEuler given it doesn’t show up in top-5 perf hot function list.
- performance schema profiling too reconfirmed a significant reduction in LOCK_table_cache contention.

Now let’s test the io-bound use-case for read-only workload

Observations:

Depending on workload either the performance of openEuler is on-par for lesser contention workload viz. uniform or better for higher contention workload viz. zipfian.

Write Workload

Let’s start with write workload. sysbench offers 2 write workloads: update-index, update-non-index

Observations:

With write workload, openeuler has mixed performance. Especially, with uniform workload openeuler continue to lag but start scoring with zipfian use-cases.
To understand this lag better we tried multiple studies and eventually found out that flushing of pages is not consistent with openeuler there-by increasing the checkpoint age and in-turn causing the lag.

The extended run shows a clear picture. openEuler continues to perform better than ubuntu during an initial run but then starts losing steam due to inconsistent/below par page flushing.
Performance profiling reveals a significant increase in contention of rseg->latch and fil_space_latch with openeuler (both occupying top slot vs ubuntu).

openeuler

| wait/synch/rwlock/innodb/trx_rseg_latch                   | 18733914.8115 |  176880804 |
| wait/synch/rwlock/innodb/fil_space_latch                  |  6036822.3603 |    6801714 |
| wait/synch/cond/mysys/COND_timer                          |  1857132.6029 |       6896 |
| wait/synch/cond/aria/SERVICE_THREAD_CONTROL::COND_control |  1834108.5978 |         61 |
| wait/synch/rwlock/sql/MDL_lock::rwlock                    |   293917.5930 |  344366933 |
| wait/synch/mutex/innodb/flush_list_mutex                  |   181595.7034 |  403167359 |
| wait/synch/sxlock/innodb/index_tree_rw_lock               |   101674.7901 |  194147135 |
| wait/synch/rwlock/innodb/log_latch                        |    58528.4128 |  302800939 |
| wait/synch/mutex/sql/LOCK_table_cache                     |    43197.0327 |  110726733 |
| wait/synch/mutex/innodb/buf_pool_mutex                    |    26246.6964 |   72853477 |

ubuntu

| wait/synch/cond/mysys/COND_timer                          | 1889638.3321 |       7310 |
| wait/synch/cond/aria/SERVICE_THREAD_CONTROL::COND_control | 1866600.4146 |         62 |
| wait/synch/rwlock/innodb/trx_rseg_latch                   |  880059.8838 |  432965780 |
| wait/synch/mutex/innodb/flush_list_mutex                  |  740073.9749 |  666248407 |
| wait/synch/rwlock/sql/MDL_lock::rwlock                    |  461974.0141 |  659530098 |
| wait/synch/rwlock/innodb/fil_space_latch                  |  411518.1018 |   19912041 |
| wait/synch/rwlock/innodb/log_latch                        |  276165.8275 |  607470128 |
| wait/synch/mutex/sql/LOCK_table_cache                     |  141710.7974 |  188402516 |
| wait/synch/mutex/innodb/buf_pool_mutex                    |  126368.0575 |  177463579 |
| wait/synch/sxlock/innodb/index_tree_rw_lock               |   43622.2066 |  403046978 |

Now let’s test the io-bound use-case for write workload

Observations:

Performance is quite similar to cpu bound use case.

Conclusion

From the study so far, openEuler on ARM looks promising. It has scored well on the read-only front and has shown mixed results for write workload. The issue of contention needs to be looked at to help understand why the said latches are prominent on openEuler and possible IO issues.

If you have more questions/queries do let me know. Will try to answer them.

Stabilize your MariaDB performance using this simple step

2022-03-07T00:00:00+00:00

While benchmarking MariaDB Server, I often observe a spike in performance at the start that eventually drops by 10-15% and then continues to remain constant at the said level. Concern is not the drawn down but the fluctuation that users may observe especially while running longer workload. Fortunately, found a way to resolve this but still investigating why it is so.

Setup

Machine Configuration:
- ARM: 64 vCPU (2 NUMA) ARM Kunpeng 920 CPU @ 2.6 Ghz
- x86: 64 vCPU (2 NUMA) Intel(R) Xeon(R) Gold 6151 CPU @ 3.00GHz
Workload:
- sysbench update-index uniform
Other configuration details here (+ thread_handling=pool-of-threads).
- Shared Buffer: 80GB
- Data: 74 GB
- Redo-Log: 20 GB
Storage: NVME SSD
- sequential read/write IOPS: 190+K/125+K
- random read/write IOPS: 180+K/65+K
MariaDB Version: 10.8 3 (work-in-progress. Wanted to use redo-log optimization).
Scalability: 1-1024 threads

Benchmarking

For benchmarking, we execute sysbench update-index (uniform) workload for 60 mins and monitor the throughput every second.

Observations:

ARM starts with approximately ~ 150K throughput but within a short period of time throughput drops down to 125K and then continues to remain stable at the new level there-by leading to a draw-down of 15% from the recent high.
x86 continues to show a lot of jitter in the performance.

I started doubting the furious flushing, filling of the redo log, etc… None of these indicators explained the draw-down.
Accidentally, I tried to clear the OS cache and things started to improve. Note: “command used to purge os cache: echo 3 > /proc/sys/vm/drop_caches”

Observations:

So clearing the OS cache, at a regular interval (in this case every 100 seconds) has a +ve effect on throughput in ARM case but it fails to have a +ve effect in the x86 case.
Let’s reconfirm this observation with a different configuration machines.

Observations:

With these different configuration machines (24 vCPU, 48 GB, 22K IOPS), regularly clearing the OS cache helps in reducing jitter for both ARM and x86.
Just to rule possible ambiguity, vm.swapiness is set to 1 on all benchmarked machines.

Conclusion

So clearing OS cache seems to have a +ve effect on performance. It helps reduce jitter. But it is not a blanket advice. As we saw above, it could have different effect based on the machine and configuration. It could be something worth trying in your environment to explore if that helps. Also, we are yet to trace why/what cached data clearance is helping reducing jitter.

If you have more questions/queries do let me know. Will try to answer them.

Profiling new version of MariaDB

2022-02-24T00:00:00+00:00

MariaDB is continuously evolving and in order to make it more scalable lot of age old, data constructs are being upgraded/revamped to the new age scalable constructs. This series of changes has helped it scale better than most of the open-source databases available. If you have been profiling MariaDB for quite some time now then it is important to ensure that you upgrade/widen your profiling scope to cover these new hot spots.

Mutex vs Latch

There are a lot of use-cases where the flow doesn’t need a mutex (exclusive access) but what it really needs is a latch (multiple readers, single writer) kinds of access. MariaDB has started identifying such use-cases and has ported them to use latches (vs original mutexes). So the traditional tracking mechanism for mutex hotspot needs to be widened.

Most of the users may be using following settings to track MariaDB performance bottleneck

performance-schema=1
performance-schema-instrument=’wait/synch/mutex/innodb/%=ON’

Since the said construct only captures mutex and doesn’t capture locks/latches profiling, information obtain using performance_schema.events_waits_summary_global_by_event_name will not present a complete picture.

Let’s understand how the hotspot has changed from 10.5 to 10.6 to 10.8

10.5.16

| wait/synch/mutex/innodb/log_sys_mutex         | 10993543.1228 |   73893323 |
| wait/synch/mutex/innodb/lock_mutex            |   409445.6364 |   72562319 |
| wait/synch/mutex/innodb/redo_rseg_mutex       |   282889.2468 |   60315800 |
| wait/synch/mutex/sql/LOCK_table_cache         |    87841.0034 |   30053316 |
| wait/synch/mutex/innodb/buf_pool_mutex        |    31164.3056 |   24091341 |
| wait/synch/mutex/innodb/log_flush_order_mutex |    30715.0940 |    6204400 |

10.6.8

| wait/synch/mutex/innodb/log_sys_mutex         | 8196724.5317 |   94451895 |
| wait/synch/mutex/sql/LOCK_table_cache         |   53218.7635 |   38989014 |
| wait/synch/mutex/innodb/log_flush_order_mutex |   22115.2731 |    5461808 |
| wait/synch/mutex/innodb/buf_pool_mutex        |   19749.9868 |   24328777 |
| wait/synch/mutex/threadpool/group_mutex       |    9871.5299 |   73233384 |
| wait/synch/mutex/sql/THD::LOCK_thd_data       |    6153.8019 |   91115395 |

log_sys_mutex contention continued.
lock_mutex, redo_rseg_mutex disappeared. Does that mean contention has been completely resolved in 10.6?

10.8.3

| wait/synch/mutex/innodb/buf_pool_mutex                 | 48952.9588 |   20786569 |
| wait/synch/mutex/sql/LOCK_table_cache                  | 34947.7470 |   43650073 |
| wait/synch/mutex/threadpool/group_mutex                | 14281.8536 |   84727091 |
| wait/synch/mutex/innodb/fil_system_mutex               |  7047.8210 |    5240336 |
| wait/synch/mutex/sql/THD::LOCK_thd_data                |  6963.7040 |  103323350 |
| wait/synch/mutex/innodb/flush_list_mutex               |  6182.9708 |    4925339 |

log_sys_mutex contention also disappeared.

If you are used to the old way of mutex profiling then you should expand the scope and now start tracking rw-locks.

10.6.8

Lock_mutex is now ported to latch and so contention is not visible with mutexes tracing.

| wait/synch/rwlock/innodb/lock_latch                       |    11823.5208 |   92213015 |

As part of this transition profiling of some important mutexes (converted to latches) has been removed. One such latch is redo_rseg_mutex which is now converted to latch but is not performance profiled. I tried local changes to cover it and found that it still represents significant contention. MDEV-27935

| wait/synch/rwlock/innodb/trx_rseg_latch_key                 | 342616.7699 |   98834024 |

10.8.3

Same way contention for log_mutex is not visible with 10.8.3 as log-mutex is now ported to log_latch.

| wait/synch/rwlock/innodb/log_latch                        |  2993779.4089 |  103110276 |

Let’s see how all these optimization has helped.

Improvement has significantly improved the performance of MariaDB.

Conclusion

More such mutexes are lined up to get ported to latches that will helps scale the MariaDB further. Also, it opens up an opportunities to explore NUMA optimized distributed latch.

If you have more questions/queries do let me know. Will try to answer them.

Benchmarking MariaDB on ARM with TPCC

2022-02-22T00:00:00+00:00

Majority of the users use cases are covered with sysbench variants of workload but there are users who have use-cases that could be best represented with TPCC or for that matter they would like to compare 2 databases using TPCC as a base standard. To help fill this gap I decided to evaluate TPCC using MariaDB on ARM.

Setup

Machine Configuration:
- ARM: 128 vCPU (4 NUMA) ARM Kunpeng 920 CPU @ 2.6 Ghz
- x86: 64 vCPU (2 NUMA) Intel(R) Xeon(R) Gold 6151 CPU @ 3.00GHz
Workload (TPCC using sysbench) : –tables=10 –scale=100
- https://github.com/Percona-Lab/sysbench-tpcc
Other configuration details here (+ thread_handling=pool-of-threads).
- Shared Buffer: 120GB
- Data: 95 GB
- Redo-Log: 30 GB
Storage: NVME SSD
- sequential read/write IOPS: 190+K/125+K
- random read/write IOPS: 180+K/65+K
MariaDB Version:
- 10.8.3 (work-in-progress. Wanted to use redo-log optimization).
Scalability: 1-1024 threads

Benchmarking

For ARM, we benchmarked TPCC with 4 different (1/2/4) NUMA configurations.

We also compared 2 NUMA ARM and 2 NUMA x86 configurations to check if the ARM is performing on par with what the user would expect or used to. Both configurations have the same number of vCPU with different frequencies (2.6 (arm) vs 3 (x86)).

Observations:

MariaDB on ARM continues to perform in line with the expectation even with TPCC workload.
MariaDB on ARM continues to be on par or even beat x86 performance, especially with higher scalability (despite of difference in frequency and ~50% lesser TCO).
For higher scalability, with 4 NUMA nodes, throughput continue to drop well below 50% (with respect to 2 NUMA node). Let’s quickly understand what is causing it.

perf profiling with 2 NUMA

+   17.66%         12557  mysqld   [kernel.kallsyms]    [k] queued_spin_lock_slowpath
+    8.70%          6544  mysqld   mariadbd             [.] l_find
+    5.56%         78524  mysqld   [kernel.kallsyms]    [k] finish_task_switch
+    3.34%          2887  mysqld   mariadbd             [.] MYSQLparse
+    2.89%          2090  mysqld   mariadbd             [.] rec_get_offsets_func
+    2.55%          1899  mysqld   [kernel.kallsyms]    [k] __wake_up_common_lock

| wait/synch/rwlock/innodb/lock_latch                       |  1497129.9573 |   73915212 |
| wait/synch/rwlock/innodb/log_latch                        |   802585.5836 |   79094893 |
| wait/synch/cond/mysys/COND_timer                          |   217272.8043 |        786 |
| wait/synch/cond/threadpool/timer_cond                     |   216049.2886 |        432 |
| wait/synch/cond/aria/SERVICE_THREAD_CONTROL::COND_control |   209020.7979 |          7 |
| wait/synch/sxlock/innodb/index_tree_rw_lock               |   118856.8702 |  118740596 |

perf profiling with 4 NUMA

+   58.35%         31357  mysqld   [kernel.kallsyms]    [k] queued_spin_lock_slowpath
+    3.40%          1883  mysqld   mariadbd             [.] l_find
+    1.62%        146440  mysqld   [kernel.kallsyms]    [k] finish_task_switch
+    1.46%          1062  mysqld   mariadbd             [.] rec_get_offsets_func
+    1.45%           883  mysqld   mariadbd             [.] buf_page_get_low

| wait/synch/sxlock/innodb/index_tree_rw_lock               |  6927204.8781 |   52455490 |
| wait/synch/rwlock/innodb/lock_latch                       |  6760422.3653 |   28812428 |
| wait/synch/rwlock/innodb/log_latch                        |  2650246.1917 |   32740980 |
| wait/synch/rwlock/sql/MDL_lock::rwlock                    |   221800.4496 |   40688789 |
| wait/synch/cond/mysys/COND_timer                          |   188957.6817 |        673 |

Contention for lock_latch (in previous versions lock_sys.mutex) and log_latch (in previous version log_sys.mutex) has increased (inline with expectation as revealed with sysbench workloads too) but it is suprising to see index_tree_rw_lock contention increasing with 4 NUMA nodes. This contention now occupies top slot and has grown multi-fold from 6th to 1st position. Surely it deemed for the further investigation cum optimization. (I tried some quick approaches like cacheline and spin-mutex. Nothing helped. Need to look at the access pattern).

Conclusion

TPCC workload scale quite well with MariaDB on ARM. Growing NUMA nodes continue to pose challenges but those could be addressed as part of the wider/generic optimization. Sysbench access workload mostly shows up log_sys and lock contention but with TPCC we started seeing a different contention too.

If you have more questions/queries do let me know. Will try to answer them.

MariaDB threadpool and NUMA scalability

2021-12-24T00:00:00+00:00

MariaDB/MySQL default uses one thread per connection. This approach is generally good if the connection is active for a longer time. If the connections are short-lived then the cost of creating a connection could overshadow the cost of running the query. Also, with increasing scalability, OS-scheduling introduces more jitter. In this case, threadpool could act as a good alternative.

In this article, we will explore the effect of the thread pool with growing NUMA scalability.

Threadpool

MariaDB support dynamic threads pooling allowing it to scale up/down the number of thread groups as needed. Each thread group has 2 threads: listener and worker. Worker thread count could further increase if the stall is detected. You can read more details here but let’s try to understand important parameters that we plan to use/experiment with.

Configuration:

thread_handling=pool-of-threads (to enable thread pool).
thread_pool_size=X (number of threads group that determines how many statements execute in parallel).
thread_pool_idle_timeout=1 (number of seconds after which idle thread should exit if there is no work).


show status like 'Threadpool%';
+-------------------------+-------+
| Variable_name           | Value |
+-------------------------+-------+
| Threadpool_idle_threads | 201   |
| Threadpool_threads      | 202   |
+-------------------------+-------+
2 rows in set (0.001 sec)

Setup

Machine Configuration:
- ARM: 128 vCPU (4 NUMA/2 Socket) ARM Kunpeng 920 CPU @ 2.6 Ghz
Workload (using sysbench):
- CPU bound workload: Pattern: uniform, zipfian
Other configuration details here (+ skip-log-bin).
- Buffer Pool: 80GB
- Data: 70GB
- Redo-log: 20 GB
- Configuration details: click here [+ skip-log-bin]
Storage: NvME SSD
- sequential read/write IOPS: 190+K/125+K
- random read/write IOPS: 180+K/65+K
MariaDB Version: 10.6.5 (tagged GA)
- Scalability: 512-1024 threads
  - 28/56/112 threads for server
  - 4/8/16 threads for sysbench

Benchmarking

Without Thread Pool (default)

Observations

For read-only workload, with increasing scalability (in form of NUMA) throughput continues to grow linearly.
For write workload, with increasing scalability (in form of NUMA) throughput drops. (why? Check this arcticle)

Now let’s enable threadpool and see if this effect could be reversed or limited as threadpool could help eliminate the OS-scheduling jitter.

With Thread Pool (thread_handling=pool-of-threads)

Observations

read-only workload:
- Enabling threadpool helps improve the performance in a range of 10-20% for the majority of the scenarios except for 4 NUMA point-select use-case.
- For 4 NUMA point select, there is a regression in performance (30-40%) with thread pool enabled.
write workload:
- Enabling threadpool helps improve the write performance especially with the high contention use-case. Infact, in some cases, improvement is up to 100% that is 2x performance.
- For the non-contention use-case, the performance of 4 NUMA threadpool is either on par or marginally less in some cases.

So we have mixed results with the majority of the cases seeing good benefit with the use of thread pool. The main cause of concern is the 4 NUMA point-select that have shown serious regression. Performance Profiling revealed that the issue is related to memory allocation in lock-free-hash due to the limited number of threads with higher throughput (1 million).

+ 62.41% 550335 mysqld mariadbd [.] lf_alloc_new

Threadpool default configuration boots 112 threads (based on a number of cores allotted to the server) threads. Limited threads serving 1024 scalability introduce the NUMA allocation challenge.

This prompted us to increase the base number of thread groups from default to 1024+

With Thread Pool (thread_handling=pool-of-threads, thread_pool_size=1500, thread_pool_idle_timeout=1)

Setting thread_pool_size > 1024 is pretty clear but then why we need to set thread_pool_idle_timeout. This will ensure that if the thread is not serving any query or workload for more than 1 sec it is released and doesn’t hog the resources. Especially important when we try workload < 1024 threads (like 512 use-cases). (default of thread_pool_idle_timeout=60 sec which is too high for short workloads).

Note: legend tpool* refer to the above configuration

Observations

read-only workload:
- So it helped 4 NUMA point-select use-case but made other scenarios worse or comparable to the w/o threadpool variant.
write workload:
- So additional fine-tuning doesn’t help.

That suggests depending on the workload user should tune the thread-pool parameters but enabling threadpool for system-wide usage seems to be a good idea including NUMA scalability.

Conclusion

Enabling thread pool continue to help in most of the scenarios (with a significant gain observed in high contention scenario) but increasing NUMA scalability seems to pose a challenge that needs some more attention to help resolve the LF_HASH allocation issue.

If you have more questions/queries do let me know. Will try to answer them.

GCC vs Clang - with MariaDB on ARM

2021-10-01T00:00:00+00:00

It is a well-known fact that a good compiler can emit an optimal code thereby allowing software to produce better throughput. Clang compiler popularity continues to grow and since I am working mostly on performance issues I am often asked if I have tried MariaDB-on-Arm with clang compiled binaries. Finally, I got some time and decided to try it out.

Setup

ARM instance: 24 vCPU Kunpeng 920 (2.6 Ghz), 48 GB of memory
MariaDB-Server: 10.6 trunk (#76972163)
Compiler:
- clang-11.1
- gcc-10.1
- Was also checking the ARM C/C++ compiler but seems like there is licensed version only (please let me knows if there is a free-to-use for opensource community license available).
OS: Ubuntu 18.04
Workload: sysbench point-select, read-only, update-index, update-non-index. CPU bound, pattern=uniform(uni)/zipfian(zip).
Scalability: 128, 256 (+ 512) threads
optimization flags: default (O2), O3, Ofast

Benchmarking

read-only workload

The difference is quite marginal in most cases so a normal graph will not be able to capture it clearly so we will use some kind of heat map.

Notes: green cell represents the highest throughput for the said test case. Like for ps-uni (point-select uniform 128 threads), the highest throughput is seen with clang-o3 and lowest with clang-ofast

Observations

In most cases, clang O2/O3 has performed better than gcc.
Ofast fails to perform for both gcc or clang.
default (gcc-O2) tend to perform worse/average for most cases. (For no scenario, it is able to perform best).

read-write workload

Observations

For read-write workload, the picture is quite different. clang-O2 continues to perform worse. clang-O3 average and clang-Ofast fill the missing lower throughput gaps left by clang-O2. In short, clang continues to perform worse for the read-write workload.
On other hand, gcc continue to score well for read-write workload with O3 and Ofast giving the best performance.
It is interesting to note that only for update-non-index (zipfian) 256 use-case, clang-Ofast score quite well.

increasing scalability/contention further

Based on overall observation it sounds like with a lot of contention and higher throughput clang continues to perform but with normal contention and lower throughput gcc continues to perform. Let’s try to re-confirm this with some more experiments.

Observations

Again, we continue to see mixed results but with increasing scalability it sounds like gcc with -Ofast tends to score.
If we try to consider all the 3 heat-maps and try to score them (-1 for red count and +1 for green count) and add it columnwise then clang -O3 and gcc -Ofast seems to be on-par.

Let’s try to run a full benchmark with all thread scalabilities (1-256) with the short-listed modes (gcc-Ofast, clang-O3)

As we could see for read-only workload clang-O3 continue to score marginally but with read-write workload gcc Ofast continue to score with some visible differences.

Conclusion

Based on the said study, there is no clear winner. Depending on use-case at times clang score and at times gcc score. Of-course the optimization mode could be O3 with clang and Ofast with gcc. Also, make a note that -Ofast (optimize very aggressively to the point of breaking standard compliance).

If you have more questions/queries do let me know. Will try to answer them.

Adaptive Purge in MariaDB

2021-09-21T00:00:00+00:00

If you are using MariaDB for some time now you may have heard about adaptive flushing. “Adaptive” refers to a behavior where-in the algorithm auto-tunes itself based on certain parameters. In the new-generation world, it is called an “AI-based algorithm”. The same concept is now being applied to purge. Purge is a critical and resource-consuming operation so scheduling of purge along with user workload needs to be balanced. This is what exactly adaptive purge would do.

A quick note on purge

MySQL/MariaDB doesn’t immediately delete the data. Data is marked for delete and updated data is placed in the mainline tree. Old data is copied over to UNDO logs. This is needed to support multi-version concurrency control (further details are beyond the scope of this blog so you can always read details here).

These old copies needs to be cleaned/removed/purged once there is no active transactions referring to it. This is done by the purge operations in the background. There are dedicated purge thread(s) meant to do this operation. Configurable using innodb_purge_threads (default=4/max=32).

If purge is being done aggressively along with the active user workload, the latter will start seeing effects like jitter, reduced performance, etc.. Let’s see how adaptive purge could help solve these issues.

Benchmarking purge with user workload

MariaDB-Server:
- MariaDB Server 10.6.4 (GA; without adaptive purge)
- MariaDB Server 10.6.5 (trunk work-in-progress; with adaptive purge folded)
Machine Configuration:
- ARM: 24 vCPU (2 NUMA) ARM Kunpeng 920 CPU @ 2.6 Ghz
- X86: 24 vCPU (2 NUMA) Intel(R) Xeon(R) Gold 6151 CPU @ 3.00GHz
- Disk: read/write 350 MB/sec
Workload :
- sysbench: test: oltp-update-index, scalability: 128, pattern: uniform
- 5 rounds 300 seconds each is executed with a 20-sec gap between each round.
- In parallel, history length is tracked.
Server configuration: here (+ innodb_purge_threads=4 (default)).
- Data: 35GB

Benchmark (without adaptive purge):

With purge-threads=4

With purge-threads=8

Observations

In both cases (arm and x86) user workloads continues to show a lot of jitters with tps occasionally touching 0. history-length is kept in check that suggests an aggressive purge is taking place.
Given the workload is user-workload the jitter with tps touching 0 is least expected and that needs to be improved. Investigative analysis showed that aggressive background purge continues to generate a lot of redo that puts pressure on redo log filling it fast and once the threshold is crossed furious flushing starts causing jitter with user workload.
Here an important decision needs to be made. If we have limited resources, active workload (like user workload) needs to get priority in using it and passive workload (like purge) could be delayed or throttled to a level so that the mainline performance is not affected.

Adaptive Purge (part of MariaDB-10.6.5)

The purge system is multi-threaded with N threads progressing in parallel. If we can limit these threads (based on the redo log fill factor) we can limit the pressure, purge is adding on the redo-log allowing the user/active workload to make effective use of redo-log. On the flip side, there will be an increase in history length and in turn larger undo-logs (that can eventually truncate with mariadb supporting undo log truncate).

Let’s try to devise an algorithm using this rule and see if that helps improve active workload.

Phase-1:
- adaptive purge continues to monitor the redo-log size. If the redo-log-size < lower watermark nothing is done.
Phase-2:
- Once the redo log size crosses the lower watermark, the adaptive purge will kick in.
- As part of the logic, the range between lower and higher watermark is divided into N buckets (N = innodb_purge_threads). As and when redo log continues to cross the bucket, purge threads continue to reduce/increase by 1.
- Say, innodb_purge_threads=4. This will create 4 buckets. When the redo log is within the range of 1st bucket then the purge system will operate with 4 purge threads. When the redo log grows and enters the range of the 2nd bucket, the purge system will operate with 3 purge threads and so on. With increasing redo-log size, purge is throttled and with reducing size aggressive purge is re-enabled.
Phase-3:
- Once the redo-log crosses the higher watermark (by then purge system is already operating with 1 purge thread) an additional delay of 10 ms is introduced to avoid scheduling purge back to back.

Note: lower and higher watermarks are pre-configured to 20/80 (currently not user-configurable. not sure if really needed). The number of buckets/ranges depends upon the number of purge threads. The ranges are created using arithmetic progression if possible otherwise, an average distribution is used.

Benchmark (with adaptive purge):

With purge-threads=4

With purge-threads=8

Observations

As expected, with reduced pressure on redo log, jitter in user/active workload considerably reduced. Importantly, overall throughput improved from an average of 10K tps to 55K tps (ARM). This improvement is due to reduced pressure on log_sys mutex (related to redo-log).
As a flip-side, history length increases but when there is an idle time it continues to fall steeply allowing it to reach considerable range without affecting the mainline performance.
Also, with changing redo-log ranges, the purge system continues to become aggressive (purge threads increasing) or conservative (purge thread decreasing).

Caveats

If your workload and/or configuration are such that even without purge the active workload continues to put pressure on redo-log, in such cases furious flushing will take place but this algorithm will not solve the said issue since it is not originating from the purge.
With the faster disk (say NVMe SSD), we observed that flushing is done quite fast thereby keeping a check on redo-log size and so adaptive purge may not kick in (depending on size of redo-log).

Conclusion

Adaptive purge not only helps reduce jitter in performance but also helps improve overall throughput there-by improving the responsiveness of the server (including latency).

If you have more questions/queries do let me know. Will try to answer them.

Tune your MariaDB IO workload using this simple step

2021-07-14T00:00:00+00:00

Tuning IO workloads is often challenging given it involves optimal usage of available IO bandwidth. MariaDB has multiple options to control this but often users tend to ignore the simpler options and tend to play around with complex or wrong options. In this article, we will take a step-by-step approach and see if we can tune an IO workload.

A quick note about flushing

Before we get into details let’s first refresh our understanding of the flushing in MariaDB as it exists today (10.6.3 GA).

During boot time all the pages are added to the free list.
Workload then starts demanding the page. If the page is not present then the page is allocated from the free list and moved into the LRU list followed by IO from disk to the allocated page.
If workload happens to change the page then said page is added to flush list that is sorted based on changes done (lsn).
The page-cleaner thread continues to flush the page to disk at regular intervals as per the configured parameters. This ensures that checkpoint age is kept under the threshold to avoid furious flushing. Flushed pages are marked clean (not freed). They continue to hold the data and workload can pin them for further use (without need for IO). This also means flushing doesn’t increase the free-page list length.
Whenever workload needs a page it tries to get it from a free list. If the free list is exhausted LRU algorithm kicks in. A page is selected for replacement. If the said page is clean then it could be used immediately (without the need for IO) else the page first needs to be flushed to the disk. This is termed as LRU flush and different from the normal flush (note: normal flush doesn’t generate free page but LRU flush will generate free page).
Now the LRU flush algorithm has an option to flush a page whenever needed or flush in batches (like N pages) at times. Also, it is quite possible that out of these N pages, some are already clean so there is no IO overhead to replace such pages (only evict and reload the new page).

With that basic understanding let’s now see if we can tune the IO workload.

Setup

Machine Configuration:
- ARM: 64 vCPU (2 NUMA) ARM Kunpeng 920 CPU @ 2.6 Ghz
- X86: 64 vCPU (2 NUMA) Intel(R) Xeon(R) Gold 6151 CPU @ 3.00GHz
Workload (using sysbench):
- IO-bound (only 50% of the data in buffer pool)
- Pattern: uniform
Other configuration details here (+ skip-log-bin).
- Buffer Pool: 35GB
- Data: 70GB
- Redo-log: 20 GB
Storage: NvME SSD (for both machines)
- sequential read/write IOPS: 190+K/125+K
- random read/write IOPS: 180+K/65+K
MariaDB Version: 10.6.3 (tagged GA)
Scalability: 512 threads

Iteration-1

Given IO involved, the most obvious setting that the majority of the user tends to tune is innodb_io_capacity and innodb_io_capacity_max. A general recommendation suggests setting these values to something lower like 1K/2K for a high-speed disk to avoid writing a copy of the page multiple times there-by making old copies stale in no time and in turn reducing the SSD endurance.
The flip side of these really low values is clearly visible on performance.

Once the adaptive flushing kicks it is unable to keepup with the rate at which pages are being modified causing performance to drop close to 0 (actual values are around 100-500 range) as dirty pages hit the threshold.

Iteration-2

Learning from experience we now decided to increase the innodb_io_capacity/innodb_io_capacity_max to 12K/24K. Why 12K? We have to start somewhere and some of the common cloud volumes offer speed in the range of 250-350 MB/sec that is around 16K-22K.

There is an improvement but still, the performance is not consistent. In fact, for a production user, a system like this is more dangerous due to a wide range of performance fluctuations. The drop in the above case is coming from dirty page hitting threshold (innodb_max_dirty_pages_pct=90).

Iteration-3

Given we have a really fast disk let’s allocate the maximum possible capacity. Please make a note that innodb_io_capacity represents IOPS which includes read and write operations and some of them could be sequential too. Given this fact, we set innodb_io_capacity/innodb_io_capacity_max to 120K/120K allowing adaptive flushing to figure out best possible usage.

So despite giving the maximum possible IO bandwidth, jitter in performance continues. In fact, the server did consume considerable IO bandwidth with the average hitting around 2-2.5 GB/sec (for ARM), 1-1.5 GB/sec (for x86).

Understanding what is going wrong

By increasing the innodb_io_capacity, we were providing more bandwidth to the flushing algorithm to ensure that the redo log threshold is not hit but we forgot the fact that with IO workload, LRU flushing has a big role to play.
Also, the innodb_io_capacity limit is meant for normal flushing but LRU flushing doesn’t respect the said limit. Instead, there are different parameters to control how many pages LRU flushing will flush. Tunning these limits will ensure that there are enough free pages available beforehand to load a new page.
One may wonder LRU flushing is even enabled in the above scenarios but we still see the jitter in performance. Let’s understand an important parameter innodb_lru_flush_size. When the LRU flushing algorithm needs to flush it will flush innodb_lru_flush_size pages in a single invocation. The default value of this parameter is 32 pages. Even though the running transaction needs only 1 page, more are freed so that other threads don’t need to wait. Unfortunately, just 31 extra pages for 512 active threads is a big mismatch, and this kind of turns into each thread invoking a cycle of LRU flush. LRU flush cycle involves flushing a page to disk in-turn involvement of doublewrite buffer too and more mutex contention. This increases the latency of the LRU flush with jitter in performance.

Playing around with innodb_lru_flush_size

Let’s experiment with different innodb_lru_flush_size starting with 512 (1 multiple), 1024 (2 multiples), 2048 (4 multiples), 4096 (8 multiples), etc… This way LRU flush will ensure that there is at least 1 free page per thread and also help reduce the LRU latency with fewer double-write buffer invocations.

Also, let’s restore the value of innodb_io_capacity and innodb_io_capacity_max back to 12K/24K as even with these values REDO log was kept in check (without causing a flush storm of furious flushing).

It is evident from the graph above that the optimal size for innodb_lru_flush_size is 2048 for ARM (increasing it beyond that doesn’t help) and 1024 for x86 (2048 has slightly lesser tps).
Also, it is interesting to note that despite giving the same resources for ARM and x86, ARM IO-bound workload performs way better than x86 with a significant margin (71K (ARM) vs 36K (x86)).

Does this increases the overall IO

Overall io continues to remain at the around same level (capped at around 2.5 GB/sec for ARM and 1.5 GB/sec for x86).
So with the said tuning, we ensured the right things are made available at the right time that helps improve the overall performance.

Flipside

With an increase in innodb_lru_flush_size from the default of 32 to 2048, the latency of the thread that invokes LRU flush would be slightly higher during the transaction that causes LRU flush to invoke. Assuming all threads has equal chances of invoking LRU at regular interval the latency should amortize as we could see from the graph above.

What about innodb_lru_scan_depth

Definition of innodb_lru_scan_depth as per the documentation.
Specifies how far down the buffer pool least-recently-used (LRU) list the cleaning thread should look for dirty pages to flush.

But the said condition should be read with innodb_lru_flush_size as it exists in the code.

n->flushed + n->evicted [total-pages freed by this batch] < max [batch limit (innodb_lru_flush_size)] &&
UT_LIST_GET_LEN(buf_pool.free) [free pages but the value is dynamically changing with consumption active in background] < free_limit [innodb_lru_scan_depth]

So this means:

If innodb_lru_flush_size pages are freed then the batch will end OR
If free-pages > innodb_lru_scan_depth batch will end.

But tuning innodb_lru_scan_depth is difficult given it is linked to dynamically changing free-list length. Say user set innodb_lru_scan_depth = 1024. So once 1024 pages are freed batch should end but it is quite possible that after freeing 900 pages 200 get pined and the free count drops down to 700 and the batch continues to work further (increasing latency of the invoking thread).

So it is advisable to keep innodb_lru_scan_depth > innodb_lru_flush_size so that each batch will free up at least innodb_lru_flush_size and will end. Dynamically changing buf_pool.free limit makes it difficult to set a good value for innodb_lru_scan_depth. Also, I see the documentation or interpretation as a misnomer now. Maybe originally, variable semantics was inline but as per the existing condition, it is better to tune innodb_lru_flush_size.

Conclusion

While tuning of an IO workload it is important to keep a watch on all facets of IO viz. normal flushing (meant for redo log), LRU flushing to ensure enough free pages are available, etc.. From the experiment, it is quite evident that setting innodb_lru_flush_size to 2/4 multiples (of scalability) helps in improving performance with less jitter (without increasing IO).

If you have more questions/queries do let me know. Will try to answer them.