Days of a mirror admin

How to change the l2arc_write_max dynamically

2011-12-07T10:34:00.000+09:00

The ZFS sets the value of the l2arc_write_max to the each cache device of the L2ARC only when you add it to a pool of the ZFS or restart the OS. If you want to change the value dynamically, you should follow the following steps with the mdb.

Start the mdb with the root privilege.
# mdb -kw
Print the list of the addresses of the actual write max for each device.
> L2ARC_dev_list::walk list | ::print -a l2arc_dev_t l2ad_write
30074670a00 l2ad_write = 0x800000
300ebae8818 l2ad_write = 0x800000
3014a4269b0 l2ad_write = 0x800000
30164ae3be0 l2ad_write = 0x800000
Set a proper value to these addresses.
> 0x30074670a00/Z c00000
0x30074670a00: 0x800000 = 0xc00000
> 0x300ebae8818/Z c00000
0x300ebae8818: 0x800000 = 0xc00000
> 3014a4269b0/Z c00000
0x3014a4269b0: 0x800000 = 0xc00000
> 30164ae3be0/Z c00000
0x30164ae3be0: 0x800000 = 0xc00000

Changing the value dynamically may cause unexpected behavior. I don't recomend it though I have never experienced any problem.

How the L2ARC works

2011-12-06T17:51:00.000+09:00

The L2ARC of the ZFS is quite different from an ordinary L2 cache of a CPU. The following figure shows the structure of the caches of ZFS. The L2ARC caches only blocks in the ARC read from HDDs. It doesn't cache dirty blocks in the ARC written to HDDs. There is no data transfer between L2ARC and HDDs. When the ZFS doesn't cache a requested block in the ARC but the L2ARC, it copy the block to the ARC and respond it.

The L2ARC caches blocks almost evicted from the ARC. ZFS periodically scans a limited range from the tail of a LRU in the ARC. It gathers blocks not cached yet in the L2ARC until the total size of them reaches the size defined by l2arc_write_max. The default is 8MiB. The scan range equals the l2arc_write_max times the l2arc_headroom. The default of the l2arc_headroom is 2. A higher value makes the total of the gathered blocks near to the l2arc_write_max. The following figure based on the comment in the arc.c shows this process.

The L2ARC forms one or more ring buffers. ZFS copies a hit block in the L2ARC to the ARC, but it doesn't move this to the head of the L2ARC. It evicts old blocks from the L2ARC to make room to copy new blocks. If the L2ARC has multiple SSDs, each of them forms a separate ring buffer. The ZFS uses them in the round robin manner.

There are some parameters to control the cycle of the scanning. The ZFS chooses one of two cycles by the total size of the gathered blocks. If the total size is equal to or smaller than the half of the l2arc_write_max, it chooses the minutes of the l2arc_feed_secs. The default is 1. Otherwise it chooses the milliseconds of the l2arc_feed_min_ms. The default is 200. The cycle becomes slow if the L2ARC caches enough blocks and the total size of gathered blocks is small. Otherwise the cycle becomes fast.

I investigated the behavior of the L2ARC on our server with the dtrace script shown below and found a few tips. A higher value of the l2arc_headroom increases the speed of filling the cache. A higher value of the l2arc_write_max also increases the speed if the cache isn't enough filled. Otherwise it slows down the speed because of the slow scanning cycle.

Higher l2arc_write_max is considered harmful

2011-12-04T00:26:00.001+09:00

This post is completely wrong. It shows how to tune the L2ARC of the ZFS as follows.

Change the Record Size to a much lower value than 128 KB.
Change the l2arc_write_max to a higher value.
Set the l2arc_noprefetch=0.

All of them causes bad effects.

You must not set the record size of the ZFS except you use the ZFS for DBMSs. The ZFS automatically chooses a proper record size of each file. Setting the record size spoils it.

You shouldn't set the l2arc_write_max a higher value to boost filling the L2ARC. It causes an opposite effect. The ZFS uses this parameter to decide the cycle of the filling. It chooses 1 sec cycle if the size of data to fill the L2ARC is equal to or less than the half of this parameter. Otherwise it chooses 200 ms cycle. A higher value lets the ZFS choose 1 sec cycle and slows down the filling.

If you want to boost it, you should set the l2arc_headroom a higher value. A proper value depends on the workloads of your ZFS. The default value 2 is too small in many cases. Higher value allows ZFS to gather data up to the l2arc_write_max and choose 200 ms cycle.

You shouldn't set the l2arc_noprefetch 0. If you set 0, the ZFS tends to store unused prefetched data to the L2ARC. This data unnecessarily evicts cached blocks from the L2ARC and lowers the efficiency of the L2ARC.

That post shows the design of the L2ARC is based on the characteristics of SSDs in the old days. This is actually true. The L2ARC expects rather low IOPS and avoid writing much data. You can't however improve the performance of the ZFS by adjusting the parameters of the L2ARC to nowadays incredible SSDs. The L2ARC can't work effectively on such messed parameters by design.