Project

General

Profile

Bug #6914

kernel virtual memory fragmentation leads to hang

Added by Matthew Ahrens about 5 years ago. Updated almost 5 years ago.

Status:
Closed
Priority:
Normal
Category:
-
Start date:
2016-04-14
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

This change allows the kernel to use more virtual address space. This will allow us to devote 1.5x physmem for the zio arena, and an additional 1.5x physmem for the kernel heap.

We saw a hang when unable to find any 128K contiguous memory segments. Looking at the core file we see many threads in stacks similar to this:

> ffffff68c9c87c00::findstack -v
stack pointer for thread ffffff68c9c87c00: ffffff02cd63d8b0
[ ffffff02cd63d8b0 _resume_from_idle+0xf4() ]
  ffffff02cd63d8e0 swtch+0x141()
  ffffff02cd63d920 cv_wait+0x70(ffffff6009b1b01e, ffffff6009b1b020)
  ffffff02cd63da50 vmem_xalloc+0x640(ffffff6009b1b000, 20000, 1000, 0, 0, 0, 0, 
  ffffff0200000004)
  ffffff02cd63dac0 vmem_alloc+0x135(ffffff6009b1b000, 20000, 4)
  ffffff02cd63db60 segkmem_xalloc+0x171(ffffff6009b1b000, 0, 20000, 4, 0, 
  fffffffffb885fe0, fffffffffbcefa10)
  ffffff02cd63dbc0 segkmem_alloc_vn+0x4a(ffffff6009b1b000, 20000, 4, 
  fffffffffbcefa10)
  ffffff02cd63dbf0 segkmem_zio_alloc+0x20(ffffff6009b1b000, 20000, 4)
  ffffff02cd63dd20 vmem_xalloc+0x5b1(ffffff6009b1c000, 20000, 1000, 0, 0, 0, 0, 
  4)
  ffffff02cd63dd90 vmem_alloc+0x135(ffffff6009b1c000, 20000, 4)
  ffffff02cd63de20 kmem_slab_create+0x8d(ffffff605fd37008, 4)
  ffffff02cd63de80 kmem_slab_alloc+0x11e(ffffff605fd37008, 4)
  ffffff02cd63dee0 kmem_cache_alloc+0x233(ffffff605fd37008, 4)
  ffffff02cd63df10 zio_data_buf_alloc+0x5b(20000)
  ffffff02cd63df70 arc_get_data_buf+0x92(ffffff6265a70588, 20000, 
  ffffff901fd796f8)
  ffffff02cd63dfb0 arc_buf_alloc_impl+0x9c(ffffff6265a70588, ffffff6d233ab0b8)
  ffffff02cd63e0a0 arc_read+0x8df(ffffff9fff25bbf8, ffffff60c0b62000, 
  ffffff610727c300, fffffffff79ab9b0, ffffff6d233ab0b8, 0, ffffff0200000080, 
  ffffff02cd63e0fc, ffffff02cd63e0d0)
  ffffff02cd63e140 dbuf_read_impl+0x18f(ffffff6d233ab0b8, ffffff9fff25bbf8, a)
  ffffff02cd63e1c0 dbuf_read+0xeb(ffffff6d233ab0b8, ffffff9fff25bbf8, a)
  ffffff02cd63e220 dmu_tx_check_ioerr+0x76(ffffff9fff25bbf8, ffffff620e9a36c8, 0
  , 66a6)
  ffffff02cd63e350 dmu_tx_count_write+0x38d(ffffff62d2cf5e88, cd4cce00, 1000)
  ffffff02cd63e3a0 dmu_tx_hold_write+0x5a(ffffff6cd375e880, 1a, cd4cce00, 1000)
  ffffff02cd63e5b0 zfs_write+0x4a1(ffffff60eb3408c0, ffffff02cd63e850, 10, 
  ffffff63801f3b08, ffffff02cd63e880)
  ffffff02cd63e630 fop_write+0x5b(ffffff60eb3408c0, ffffff02cd63e850, 10, 
  ffffff63801f3b08, ffffff02cd63e880)
  ffffff02cd63e900 rfs3_write+0x503(ffffff02cd63eaa0, ffffff02cd63e960, 
  ffffff6211b84ec0, ffffff02cd63eca0, ffffff63801f3b08, 0)
  ffffff02cd63ec20 common_dispatch+0x479(ffffff02cd63eca0, ffffff6d16894b40, 2, 
  4, fffffffff824ba36, ffffffffc01ae060)
  ffffff02cd63ec40 rfs_dispatch+0x2d(ffffff02cd63eca0, ffffff6d16894b40)
  ffffff02cd63ed20 svc_getreq+0x1c1(ffffff6d16894b40, ffffff634278f120)
  ffffff02cd63ed90 svc_run+0x146(ffffff61060daa38)
  ffffff02cd63edd0 svc_do_run+0x8e(1)
  ffffff02cd63eec0 nfssys+0xf1(e, fb230fbc)
  ffffff02cd63ef10 _sys_sysenter_post_swapgs+0x149()

These threads are all waiting for a 128k segment to become from the zfs_file_data arena which has space but no 128k segments:

> ffffff6009b1b000::vmem
ADDR             NAME                        INUSE        TOTAL   SUCCEED  FAIL
ffffff6009b1b000 zfs_file_data          316049563648 397276086272 293445790 
    0
> ffffff6009b1b000::print vmem_t vm_freemap
vm_freemap = 0x1e000

Also available in: Atom PDF