spa_sync() iterates over all the dirty dnodes and processes each of them by calling dnode_sync(). If there are many dirty dnodes (e.g. because we created or removed a lot of files), the single thread of spa_sync() calling dnode_sync() can become a bottleneck. Additionally, if many dnodes are dirtied concurrently in open context (e.g. due to concurrent file creation), the os_lock will experience lock contention via dnode_setdirty().
The solution is to track dirty dnodes on a multilist_t, and for spa_sync() to use separate threads to process each of the sublists in the multilist.
On the concurrent file creation microbenchmark, the performance improvement from dnode_setdirty() is up to 7%. Additionally, the wall clock time spent in spa_sync() is reduced to 15%-40% of the single-threaded case. In terms of cost/reward, once the other bottlenecks are addressed, fixing this bug will provide a medium-large performance gain and require a medium amount of effort to implement.
Updated by Electric Monk about 5 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
commit 94c2d0eb22e9624151ee84a7edbf7178e1bf4087 Author: Matthew Ahrens <firstname.lastname@example.org> Date: 2017-03-16T12:09:34.000Z 7968 multi-threaded spa_sync() Reviewed by: Pavel Zakharov <email@example.com> Reviewed by: Brad Lewis <firstname.lastname@example.org> Reviewed by: Saso Kiselkov <email@example.com> Reviewed by: Brian Behlendorf <firstname.lastname@example.org> Approved by: Dan McDonald <email@example.com>