Bug #1618

zfs causing system to hang in vmem_xalloc()

Added by Bryan Leaman over 2 years ago. Updated almost 2 years ago.

Status:Closed Start date:2011-10-09
Priority:High Due date:
Assignee:George Wilson % Done:

100%

Category:zfs - Zettabyte File System Spent time: -
Target version:-
Difficulty:Hard Tags:needs-triage

Description

During the duplication of a zpool to another local zpool using 'zfs send | zfs receive', the system often hangs several hours into the operation. When it's hung, I can no longer ssh into the box although it is pingable. Existing terminal sessions can still run some commands (iostat, vmstat, etc.) but others will just hang such as "mdb -k". The only way to recover is to reboot.

I generated a core dump using kmdb, and initial analysis seems to point to a hang in vmem_xalloc(). The zfs send and receive processes as well as the mdb process that I executed were all hanging in vmem_xalloc(). The thread stacks and symptoms are very similar to the following issues:

http://wesunsolve.net/bugid/id/6989594
http://wesunsolve.net/bugid/id/6942439 (was "fixed" in snv_148 but apparently not for all scenarios)
http://wesunsolve.net/bugid/id/6983481

One of the filesystems in the zpool is a zvol with 8k volblocksize and 150GB allocated space. Usually the system hangs just after this filesystem is duplicated and it moves on to the next one.

I can provide a core dump for analysis. In the meantime I'm trying the workaround in 6983481 of increasing segziosize. System is an HP DL360 G4p with 6GB RAM running oi_151a 64-bit kernel.

History

Updated by Bryan Leaman over 2 years ago

More details:

# echo '::walk thread | ::findstack -v' | mdb {unix,vmcore}.0 | grep vmem_xalloc
  ffffff000c6062c0 vmem_xalloc+0x635(ffffff024aa56000, 20000, 1000, 0, 0, 0, 0,
  ffffff000c606580 vmem_xalloc+0x546(ffffff024aa58000, 20000, 1000, 0, 0, 0, 0,
  ffffff000c344130 vmem_xalloc+0x635(ffffff024aa56000, 20000, 1000, 0, 0, 0, 0,
  ffffff000c3443f0 vmem_xalloc+0x546(ffffff024aa58000, 20000, 1000, 0, 0, 0, 0,
  ffffff000cb22e30 vmem_xalloc+0x635(ffffff024aa56000, 20000, 1000, 0, 0, 0, 0,
  ffffff000cb230f0 vmem_xalloc+0x546(ffffff024aa58000, 20000, 1000, 0, 0, 0, 0,

# mdb {unix,vmcore}.0
> ::pgrep zfs
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R   2131   1060   2130   1054      0 0x4a004000 ffffff0260536038 zfs
R   2130   1060   2130   1054      0 0x4a004000 ffffff0258559088 zfs

> ffffff0260536038::walk thread | ::threadlist -v
            ADDR             PROC              LWP CLS PRI            WCHAN
ffffff025e5c7500 ffffff0260536038 ffffff025f383c00   1  59 ffffff024aa5601e
  PC: _resume_from_idle+0xf1    CMD: zfs receive -vFd data1
  stack pointer for thread ffffff025e5c7500: ffffff000c343f90
  [ ffffff000c343f90 _resume_from_idle+0xf1() ]
    swtch+0x145()
    cv_wait+0x61()
    vmem_xalloc+0x635()
    vmem_alloc+0x161()
    segkmem_xalloc+0x90()
    segkmem_alloc_vn+0xcd()
    segkmem_zio_alloc+0x24()
    vmem_xalloc+0x546()
    vmem_alloc+0x161()
    kmem_slab_create+0x81()
    kmem_slab_alloc+0x5b()
    kmem_cache_alloc+0x1fa()
    zio_data_buf_alloc+0x2c()
    arc_get_data_buf+0x18b()
    arc_buf_alloc+0xa2()
    dbuf_noread+0xa6()
    dmu_buf_will_fill+0x1f()
    dmu_write+0xef()
    restore_write+0xee()
    dmu_recv_stream+0x7ab()
    zfs_ioc_recv+0x42d()
    zfsdev_ioctl+0x15e()
    cdev_ioctl+0x45()
    spec_ioctl+0x5a()
    fop_ioctl+0x7b()
    ioctl+0x18e()
    _sys_sysenter_post_swapgs+0x149()

> ::pgrep mdb
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R   2727   2019   2727   2010      0 0x4a004000 ffffff025f43d098 mdb

> ffffff025f43d098::walk thread | ::threadlist -v
            ADDR             PROC              LWP CLS PRI            WCHAN
ffffff025549cc00 ffffff025f43d098 ffffff025af020c0   1  60 ffffff024aa5601e
  PC: _resume_from_idle+0xf1    CMD: mdb -k
  stack pointer for thread ffffff025549cc00: ffffff000cb22c90
  [ ffffff000cb22c90 _resume_from_idle+0xf1() ]
    swtch+0x145()
    cv_wait+0x61()
    vmem_xalloc+0x635()
    vmem_alloc+0x161()
    segkmem_xalloc+0x90()
    segkmem_alloc_vn+0xcd()
    segkmem_zio_alloc+0x24()
    vmem_xalloc+0x546()
    vmem_alloc+0x161()
    kmem_slab_create+0x81()
    kmem_slab_alloc+0x5b()
    kmem_cache_alloc+0x1fa()
    zio_data_buf_alloc+0x2c()
    arc_get_data_buf+0x18b()
    arc_buf_alloc+0xa2()
    arc_read_nolock+0x12f()
    arc_read+0x79()
    dsl_read+0x33()
    dbuf_read_impl+0x17e()
    dbuf_read+0xfd()
    dmu_buf_hold_array_by_dnode+0x1c9()
    dmu_buf_hold_array+0x6e()
    dmu_read_uio+0x4d()
    zfs_read+0x255()
    fop_read+0x6b()
    vn_rdwr+0x17f()
    gexec+0x146()
    exec_common+0x4e8()
    exece+0x1f()
    _sys_sysenter_post_swapgs+0x149()

> ::pgrep utmpd
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R    523      1    523    523      0 0x42000000 ffffff025e45d098 utmpd

> ffffff025e45d098::walk thread | ::threadlist -v
            ADDR             PROC              LWP CLS PRI            WCHAN
ffffff0254495480 ffffff025e45d098 ffffff024fed9800   1  60 ffffff024aa5601e
  PC: _resume_from_idle+0xf1    CMD: /usr/lib/utmpd
  stack pointer for thread ffffff0254495480: ffffff000c606120
  [ ffffff000c606120 _resume_from_idle+0xf1() ]
    swtch+0x145()
    cv_wait+0x61()
    vmem_xalloc+0x635()
    vmem_alloc+0x161()
    segkmem_xalloc+0x90()
    segkmem_alloc_vn+0xcd()
    segkmem_zio_alloc+0x24()
    vmem_xalloc+0x546()
    vmem_alloc+0x161()
    kmem_slab_create+0x81()
    kmem_slab_alloc+0x5b()
    kmem_cache_alloc+0x1fa()
    zio_data_buf_alloc+0x2c()
    arc_get_data_buf+0x18b()
    arc_read_nolock+0x409()
    arc_read+0x79()
    dsl_read+0x33()
    dbuf_read_impl+0x17e()
    dbuf_read+0xfd()
    dmu_buf_hold_array_by_dnode+0x1c9()
    dmu_buf_hold_array+0x6e()
    dmu_read_uio+0x4d()
    zfs_read+0x255()
    fop_read+0x6b()
    pread+0x22c()
    pread32+0x24()
    _sys_sysenter_post_swapgs+0x149()

Updated by Bryan Leaman over 2 years ago

Increasing segziosize yields the same panic as reported in 6983481. From my latest core dump:

> ::status
debugging crash dump vmcore.1 (64-bit) from oi151dev
operating system: 5.11 oi_151a (i86pc)
image uuid: 5ec234f3-eeb2-ec51-9908-f077dc971ee1
panic message: hati_pte_map: flags & HAT_LOAD_REMAP
dump content: kernel pages only

> ::panicinfo
             cpu                3
          thread ffffff000d226c40
         message hati_pte_map: flags & HAT_LOAD_REMAP
             rdi fffffffffb95c468
             rsi ffffff000d225770
             rdx                0
             rcx                0
              r8 ffffff000d2256c0
              r9 ffffff000d2257b0
             rax                0
             rbx                0
             rbp ffffff000d2257a0
             r10 ffffff024aac1240
             r10 ffffff024aac1240
             r11 ffffff000d225680
             r12         743a6363
             r13 ffffff024ad892b8
             r14         dfff3573
             r15                1
          fsbase                0
          gsbase ffffff0250dc9580
              ds               4b
              es               4b
              fs                0
              gs              1c3
          trapno                0
             err                0
             rip fffffffffb86c870
              cs               30
          rflags              246
             rsp ffffff000d2256b8
              ss               38
          gdt_hi                0
          gdt_lo         200001ef
          idt_hi                0
          idt_lo         10000fff
             ldt                0
            task               70
             cr0         8005003b
             cr2         fed32fe0
             cr3          3c00000
             cr4              6f8

> $c
vpanic()
hati_pte_map+0x416(ffffff024ad892b8, 0, ffffff0003a3f7e0, 743a6363, 1, 0)
hati_load_common+0x15d(ffffff024aab0e78, ffffff0206a00000, ffffff0003a3f7e0, 427
, 1, 0)
hat_memload+0x81(ffffff024aab0e78, ffffff0206a00000, ffffff0003a3f7e0, 27, 1)
segkmem_xalloc+0x13b(ffffff024aa56000, 0, 20000, 4, 0, fffffffffb898408)
segkmem_alloc_vn+0xcd(ffffff024aa56000, 20000, 4, fffffffffbceeb00)
segkmem_zio_alloc+0x24(ffffff024aa56000, 20000, 4)
vmem_xalloc+0x546(ffffff024aa58000, 20000, 1000, 0, 0, 0)
vmem_alloc+0x161(ffffff024aa58000, 20000, 4)
kmem_slab_create+0x81(ffffff024e6ee348, 4)
kmem_slab_alloc+0x5b(ffffff024e6ee348, 4)
kmem_cache_alloc+0x1fa(ffffff024e6ee348, 4)
zio_data_buf_alloc+0x2c(20000)
arc_get_data_buf+0x18b(ffffff025b4fd430)
arc_buf_alloc+0xa2(ffffff024e69d080, 20000, 0, 0)
arc_read_nolock+0x12f(0, ffffff024e69d080, ffffff0255009180, 0, 0, 6)
arc_read+0x79(0, ffffff024e69d080, ffffff0255009180, ffffff025b4168b0, 0, 0)
dsl_read+0x33(0, ffffff024e69d080, ffffff0255009180, ffffff025b4168b0, 0, 0)
traverse_prefetcher+0x115(ffffff024e69d080, 0, ffffff0255009180,
ffffff025b4168b0, ffffff000d226180, ffffff02511a2a00)
traverse_visitbp+0x149(ffffff000d226b20, ffffff02511a2a00, ffffff025b4168b0,
ffffff0255009180, ffffff000d226180)
traverse_visitbp+0x4dd(ffffff000d226b20, ffffff02511a2a00, ffffff027562eb70,
ffffff0268ac9000, ffffff000d226260)

Updated by Bryan Leaman over 2 years ago

After additional testing and several more hangs, I've noticed some things that may be helpful. Each time the system hangs, the blocked threads all seem to be waiting to allocate virtual memory for zio data buffers and in every crash dump the WCHAN address is always ffffff024aa5601e:

> ffffff024aa5601e::wchaninfo -v
ADDR             TYPE NWAITERS   THREAD           PROC
ffffff024aa5601e cond       14:  ffffff02904b5860 fbe
                                 ffffff02d1e37b00 ctfconvert
                                 ffffff02d12f07c0 fbe
                                 ffffff02558860e0 ctfconvert
                                 ffffff02698b8bc0 fbe
                                 ffffff02d2902020 ctfconvert
                                 ffffff0261eaa020 locale
                                 ffffff02559b9160 login
                                 ffffff028ff60860 fbe
                                 ffffff02cf755060 fbe
                                 ffffff028c7873e0 fbe
                                 ffffff02d17ff7c0 fbe
                                 ffffff025588cba0 fbe
                                 ffffff02d28e90e0 fbe

I'm starting to think this is vmem fragmentation, but that's not supposed to be an issue with 64-bit kernels. ::kmem_slabs shows very high waste in the kmem magazines:

> ::kmem_slabs
                                    Partial              Unused
Cache Name                   Slabs    Slabs   Buffers   Buffers  Waste
------------------------- -------- -------- --------- --------- ------
kmem_magazine_1                116      116     29116     28144  96.7%
kmem_magazine_3                 80       80     10000      9381  93.8%
kmem_magazine_7                140      140      8680      8295  95.6%
kmem_magazine_15               109      109      3379      3127  92.5%
kmem_magazine_31                72       72      1080       950  88.0%
kmem_magazine_47                 7        7        70        63  90.0%
kmem_magazine_63                16       16       112        92  82.1%
kmem_magazine_95                12       12        60        47  78.3%
kmem_magazine_143               20       14        60        25  41.7%

In every case, the arc is not completly full and arc_no_grow=0. So I'm not sure where the memory shortage is. Any suggestions would be appreciated.

p                         =      1047 MB
c                         =      3827 MB
c_min                     =       630 MB
c_max                     =      5043 MB
size                      =      3555 MB
arc_no_grow               =         0
arc_meta_used             =      1178 MB
arc_meta_limit            =      1260 MB
arc_meta_max              =      1297 MB

The system is not out of memory:

> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     557030              2175   36%
ZFS File Data              611404              2388   39%
Anon                        41988               164    3%
Exec and libs                1663                 6    0%
Page cache                  11396                44    1%
Free (cachelist)            15065                58    1%
Free (freelist)            314750              1229   20%

Total                     1553296              6067
Physical                  1553295              6067

I'm currently building a debug kernel to see if that identifies any issues. Otherwise I am running out of ideas. The latest hang occurred several hours after the zfs send | zfs recv of a small 5GB snapshot with a 128k recordsize. Until then, the machine was stable for most of the day doing illumos builds.

Updated by Bryan Leaman about 2 years ago

More details as requested by :

> ::kmastat
cache                        buf    buf    buf    memory     alloc alloc
name                        size in use  total    in use   succeed  fail
------------------------- ------ ------ ------ ---------- --------- -----
kmem_magazine_1               16    917  29116     475136B   4599628     0
kmem_magazine_3               32    474  10000     327680B    222114     0
kmem_magazine_7               64    325   8680     573440B   2088140     0
kmem_magazine_15             128    239   3379     446464B    836697     0
kmem_magazine_31             256    111   1080     294912B    604528     0
kmem_magazine_47             384      0     70      28672B     11708     0
kmem_magazine_63             512      8    112      65536B      1202     0
kmem_magazine_95             768      8     60      49152B       947     0
kmem_magazine_143           1152     26     60      81920B      1997     0
kmem_slab_cache               72 405525 748715   55758848B  10199107     0
kmem_bufctl_cache             24 1164367 1883927   46206976B  11254579     0
kmem_bufctl_audit_cache      192      0      0          0B         0     0
kmem_va_4096                4096 114368 127520  522321920B    223572     0
kmem_va_8192                8192  12848  17504  143392768B    263560     0
kmem_va_12288              12288    209    280    3670016B      8860     0
kmem_va_16384              16384  57339  68400 1120665600B    129429     0
kmem_va_20480              20480    148    156    3407872B      5781     0
kmem_va_24576              24576     70     75    1966080B      2475     0
kmem_va_28672              28672     58     68    2228224B      4014     0
kmem_va_32768              32768     91    124    4063232B      2318     0
kmem_alloc_8                   8 209571 236913    1929216B  22702948     0
kmem_alloc_16                 16  44484  44929     733184B   1666110     0
kmem_alloc_24                 24  89434 103206    2531328B   4151821     0
kmem_alloc_32                 32  17734  21000     688128B  44116998     0
kmem_alloc_40                 40  16037  25900    1060864B 392790898     0
kmem_alloc_48                 48 145207 178865    8826880B   2040065     0
kmem_alloc_56                 56  58654  72988    4210688B   1324221     0
kmem_alloc_64                 64  60959 100626    6647808B  31835181     0
kmem_alloc_80                 80 115716 147050   12046336B  20581335     0
kmem_alloc_96                 96  19781  31611    3158016B   1979223     0
kmem_alloc_112               112  23878  30870    3612672B   1404296     0
kmem_alloc_128               128  69688  71579    9457664B   8819142     0
kmem_alloc_160               160   2805   5900     966656B 109602103     0
kmem_alloc_192               192  18955  32320    6619136B  14670871     0
kmem_alloc_224               224    887   4437    1069056B  23685733     0
kmem_alloc_256               256    453   2025     552960B 174021196     0
kmem_alloc_320               320   7185   7344    2506752B   2505081     0
kmem_alloc_384               384    480    560     229376B     38493     0
kmem_alloc_448               448    107    160      81920B    202766     0
kmem_alloc_512               512    476    528     270336B    675589     0
kmem_alloc_640               640    314    360     245760B    203526     0
kmem_alloc_768               768     74     95      77824B      3550     0
kmem_alloc_896               896   5308   8478    7716864B     72981     0
kmem_alloc_1152             1152    190    266     311296B   2984754     0
kmem_alloc_1344             1344   2616   2634    3596288B      5192     0
kmem_alloc_1600             1600    106    135     221184B     87795     0
kmem_alloc_2048             2048    477    552    1130496B  78483415     0
kmem_alloc_2688             2688    146    174     475136B    135988     0
kmem_alloc_4096             4096    205    219     897024B    490695     0
kmem_alloc_8192             8192    125    131    1073152B    263653     0
kmem_alloc_12288           12288     28     34     417792B      1626     0
kmem_alloc_16384           16384     62     67    1097728B      1110     0
kmem_alloc_24576           24576     48     52    1277952B       743     0
kmem_alloc_32768           32768     74     80    2621440B     17730     0
kmem_alloc_40960           40960     15     21     860160B       136     0
kmem_alloc_49152           49152      8     13     638976B        73     0
kmem_alloc_57344           57344      8     14     802816B        77     0
kmem_alloc_65536           65536     11     15     983040B        68     0
kmem_alloc_73728           73728      4      9     663552B        70     0
kmem_alloc_81920           81920      1      6     491520B        57     0
kmem_alloc_90112           90112      2      4     360448B         9     0
kmem_alloc_98304           98304      3      6     589824B        16     0
kmem_alloc_106496         106496      5      7     745472B         9     0
kmem_alloc_114688         114688      5      5     573440B         5     0
kmem_alloc_122880         122880      4      4     491520B         4     0
kmem_alloc_131072         131072      9     11    1441792B        15     0
streams_mblk                  64  52382  52576    3473408B 124087963     0
streams_dblk_16              128    167    372      49152B    170530     0
streams_dblk_80              192    288    520     106496B   5051344     0
streams_dblk_144             256      0    150      40960B 183045108     0
streams_dblk_208             320    238    372     126976B     79036     0
streams_dblk_272             384  46408  46420   19013632B  12880159     0
streams_dblk_336             448   4902   4920    2519040B  22424719     0
streams_dblk_528             640      1    174     118784B   2021907     0
streams_dblk_1040           1152      0     21      24576B     49114     0
streams_dblk_1488           1600      0     40      65536B     11506     0
streams_dblk_1936           2048      0     18      36864B   8869135     0
streams_dblk_2576           2688      1     18      49152B       130     0
streams_dblk_3856           3968      0      5      20480B  28014049     0
streams_dblk_8192            112      0     31       4096B  17414618     0
streams_dblk_12048         12160      0      6      73728B   2859151     0
streams_dblk_16384           112      0     31       4096B      5004     0
streams_dblk_20240         20352      0      4      81920B      1631     0
streams_dblk_24576           112      0     31       4096B      1410     0
streams_dblk_28432         28544      0      4     114688B      1502     0
streams_dblk_32768           112      0     62       8192B   5543552     0
streams_dblk_36624         36736      0      0          0B         0     0
streams_dblk_40960           112      0      0          0B         0     0
streams_dblk_44816         44928      0      0          0B         0     0
streams_dblk_49152           112      0      0          0B         0     0
streams_dblk_53008         53120      0      0          0B         0     0
streams_dblk_57344           112      0      0          0B         0     0
streams_dblk_61200         61312      0      0          0B         0     0
streams_dblk_65536           112      0      0          0B         0     0
streams_dblk_69392         69504      0      0          0B         0     0
streams_dblk_73728           112      0      0          0B         0     0
streams_dblk_esb             112      0      0          0B         0     0
streams_fthdr                408      0      0          0B         0     0
streams_ftblk                376      0      0          0B         0     0
multidata                    248      0      0          0B         0     0
multidata_pdslab            7112      0      0          0B         0     0
multidata_pattbl              32      0      0          0B         0     0
log_cons_cache                48      9     83       4096B        97     0
taskq_ent_cache               56   2423   3408     196608B     92673     0
taskq_cache                  280    139    154      45056B       239     0
kmem_io_64G_128              128      0     31       4096B      1683     0
kmem_io_64G_256              256      0     30       8192B        24     0
kmem_io_64G_512              512      0     16       8192B        33     0
kmem_io_64G_1024            1024      0      0          0B         0     0
kmem_io_64G_2048            2048   5250   5250   10752000B      5250     0
kmem_io_64G_4096            4096      0      0          0B         0     0
kmem_io_4G_128               128      2     31       4096B         2     0
kmem_io_4G_256               256      1     15       4096B       105     0
kmem_io_4G_512               512     26     32      16384B        26     0
kmem_io_4G_1024             1024      3     16      16384B       945     0
kmem_io_4G_2048             2048      0      8      16384B       155     0
kmem_io_4G_4096             4096   1023   1023    4190208B      1023     0
kmem_io_2G_128               128      1     31       4096B         1     0
kmem_io_2G_256               256      0      0          0B         0     0
kmem_io_2G_512               512      0      0          0B         0     0
kmem_io_2G_1024             1024      0      0          0B         0     0
kmem_io_2G_2048             2048      0      0          0B         0     0
kmem_io_2G_4096             4096      2      4      16384B        54     0
kmem_io_16M_128              128      0      0          0B         0     0
kmem_io_16M_256              256      0      0          0B         0     0
kmem_io_16M_512              512      0      0          0B         0     0
kmem_io_16M_1024            1024      0      0          0B         0     0
kmem_io_16M_2048            2048      0      0          0B         0     0
kmem_io_16M_4096            4096      0      0          0B         0     0
id32_cache                    32      0      0          0B         0     0
bp_map_4096                 4096      0      0          0B         0     0
bp_map_8192                 8192      0     16     131072B      1307     0
bp_map_12288               12288      0     10     131072B      1411     0
bp_map_16384               16384      0      8     131072B      1478     0
bp_map_20480               20480      0      6     131072B      1511     0
bp_map_24576               24576      0     10     262144B      1425     0
bp_map_28672               28672      0      8     262144B      1459     0
bp_map_32768               32768      0     12     393216B   6293728     0
htable_t                      72   4133   4510     335872B   1249104     0
hment_t                       64  49000  72292    4775936B  60650543     0
hat_t                        176    116    176      32768B    166519     0
HatHash                     8192      2      8      65536B      1919     0
HatVlpHash                  4096    114    117     479232B    184807     0
zfs_file_data_4096          4096   1732   1920    7864320B   1364826     0
zfs_file_data_8192          8192 268345 347328 2845310976B  11958066     0
zfs_file_data_12288        12288    977    980   12845056B     13104     0
zfs_file_data_16384        16384    156    160    2621440B      4665     0
zfs_file_data_20480        20480    786    786   17170432B     15196     0
zfs_file_data_24576        24576    210    210    5505024B      4807     0
zfs_file_data_28672        28672    378    380   12451840B      8164     0
zfs_file_data_32768        32768    108    108    3538944B      3574     0
segkp_4096                  4096    111    160     655360B    117669     0
segkp_8192                  8192      0      0          0B         0     0
segkp_12288                12288      0      0          0B         0     0
segkp_16384                16384      0      0          0B         0     0
segkp_20480                20480      0      0          0B         0     0
umem_np_4096                4096      0     32     131072B       143     0
umem_np_8192                8192      0     16     131072B         9     0
umem_np_12288              12288      0      0          0B         0     0
umem_np_16384              16384      0      8     131072B        84     0
umem_np_20480              20480      0      6     131072B        15     0
umem_np_24576              24576      0      5     131072B         3     0
umem_np_28672              28672      0      0          0B         0     0
umem_np_32768              32768      0      4     131072B         9     0
mod_hash_entries              24    280    501      12288B      2706     0
ipp_mod                      304      0      0          0B         0     0
ipp_action                   368      0      0          0B         0     0
ipp_packet                    64      0      0          0B         0     0
seg_cache                     96   4796   6232     622592B   6690154     0
seg_pcache                   104     14     76       8192B        60     0
fnode_cache                  176      5     20       4096B        41     0
pipe_cache                   320     24     48      16384B     14178     0
snode_cache                  152    239    312      49152B    262637     0
dv_node_cache                176     86    220      40960B      1658     0
mac_impl_cache             13568      1      3      40960B         2     0
mac_ring_cache               192      4     20       4096B         8     0
flow_entry_cache           27112      3      8     217088B         6     0
flow_tab_cache               216      1     18       4096B         2     0
mac_soft_ring_cache          376      6     10       4096B        12     0
mac_srs_cache               3240      3      5      16384B         6     0
mac_bcast_grp_cache           80      2     50       4096B         4     0
mac_client_impl_cache       1328      1      3       4096B         2     0
mac_promisc_impl_cache       112      0      0          0B         0     0
dls_link_cache               344      1     11       4096B         2     0
dls_devnet_cache             360      1     11       4096B         2     0
sdev_node_cache              256   1551   1560     425984B      1658     0
dev_info_node_cache          680    181    240     163840B       876     0
ndi_fm_entry_cache            32    288    500      16384B       624     0
thread_cache                 912    368    504     516096B    621086     0
lwp_cache                   1760    792    837    1523712B    127844     0
turnstile_cache               64   1088   1488      98304B    538059     0
tslabel_cache                 48      2     83       4096B         2     0
cred_cache                   184     94    147      28672B     51549     0
rctl_cache                    48   1689   2241     110592B   1056083     0
rctl_val_cache                64   3364   4464     294912B   2270889     0
task_cache                   160     41     75      12288B       147     0
kmem_defrag_cache            216      2     18       4096B         2     0
kmem_move_cache               56      0    142       8192B     56646     0
rootnex_dmahdl              2592   6994   7086   19349504B  78414192     0
timeout_request              128      2     31       4096B         3     0
cyclic_id_cache               72     22     55       4096B        28     0
callout_cachebabecafe         80    501    527      69632B       501     0
callout_lcachebabecafe        48    874    930      61440B       874     0
dnlc_space_cache              24      0      0          0B         0     0
vfs_cache                    208     53     95      20480B      6192     0
vn_cache                     208  69611 125400   34242560B    933980     0
vsk_anchor_cache              40     21    100       4096B        36     0
file_cache                    56    462    639      36864B   2431053     0
stream_head_cache            376    139    170      69632B     41489     0
queue_cache                  656    281    300     204800B     63067     0
syncq_cache                  160     11     25       4096B        31     0
qband_cache                   64      0      0          0B         0     0
linkinfo_cache                48      7     83       4096B         9     0
ciputctrl_cache              256      0      0          0B         0     0
serializer_cache              64     27     62       4096B        56     0
as_cache                     232    115    187      45056B    166518     0
marker_cache                 120      0     33       4096B      4126     0
anon_cache                    48  48027  82834    4087808B  15986891     0
anonmap_cache                112   3438   4760     557056B   3375306     0
segvn_cache                  168   4796   6279    1118208B   5013234     0
segvn_szc_cache1            4096      0      0          0B         0     0
flk_edges                     48      0      0          0B         0     0
fdb_cache                    104      0      0          0B         0     0
timer_cache                  136      1     29       4096B         1     0
vmu_bound_cache               56      0      0          0B         0     0
vmu_object_cache              64      0      0          0B         0     0
physio_buf_cache             248      0      0          0B         0     0
process_cache               3880    121    125     512000B     97691     0
kcf_sreq_cache                56      0      0          0B         0     0
kcf_areq_cache               296      0      0          0B         0     0
kcf_context_cache            112      0     31       4096B     11248     0
clnt_clts_endpnt_cache        88      0      0          0B         0     0
zio_cache                    856     13   1107    1007616B 576590051     0
zio_link_cache                48      0   3320     163840B 622661688     0
zio_buf_512                  512  86480 118672   60760064B  33898508     0
zio_data_buf_512             512    995   1952     999424B    130223     0
zio_buf_1024                1024   2626   2876    2945024B  27507063     0
zio_data_buf_1024           1024    673   1056    1081344B     68046     0
zio_buf_1536                1536    550    624     958464B  47635687     0
zio_data_buf_1536           1536   2337   2600    3993600B     56112     0
zio_buf_2048                2048    297    322     659456B  38513745     0
zio_data_buf_2048           2048   1636   1642    3362816B     39457     0
zio_buf_2560                2560    152    176     450560B  25778624     0
zio_data_buf_2560           2560   1108   1112    2846720B     29162     0
zio_buf_3072                3072    111    128     393216B  19144907     0
zio_data_buf_3072           3072    641    648    1990656B     24087     0
zio_buf_3584                3584     74     96     344064B  12761791     0
zio_data_buf_3584           3584    471    480    1720320B     31594     0
zio_buf_4096                4096   1826   1831    7499776B   9181311     0
zio_data_buf_4096           4096    401    403    1650688B   1369951     0
zio_buf_5120                5120     79     96     491520B   7889481     0
zio_data_buf_5120           5120    642    644    3297280B     38339     0
zio_buf_6144                6144     55     70     430080B   5713191     0
zio_data_buf_6144           6144    419    420    2580480B     27961     0
zio_buf_7168                7168     49     56     401408B   1595006     0
zio_data_buf_7168           7168    299    304    2179072B     19621     0
zio_buf_8192                8192     18     48     393216B  24749683     0
zio_data_buf_8192           8192 268295 268345 2198282240B  35581653     0
zio_buf_10240              10240     26     34     348160B    444648     0
zio_data_buf_10240         10240    420    420    4300800B     31172     0
zio_buf_12288              12288     17     21     258048B    385183     0
zio_data_buf_12288         12288    279    280    3440640B     20533     0
zio_buf_14336              14336     16     24     344064B    260375     0
zio_data_buf_14336         14336    258    260    3727360B     15297     0
zio_buf_16384              16384  57156  57177  936787968B   9347304     0
zio_data_buf_16384         16384    152    156    2555904B     13260     0
zio_buf_20480              20480     77     81    1658880B    531087     0
zio_data_buf_20480         20480    276    276    5652480B     21820     0
zio_buf_24576              24576     12     18     442368B    460786     0
zio_data_buf_24576         24576    210    210    5160960B     15773     0
zio_buf_28672              28672     11     16     458752B    388570     0
zio_data_buf_28672         28672    112    112    3211264B     11596     0
zio_buf_32768              32768      6     11     360448B    446748     0
zio_data_buf_32768         32768    107    108    3538944B     13006     0
zio_buf_36864              36864     23     28    1032192B    235438     0
zio_data_buf_36864         36864     76     78    2875392B      8573     0
zio_buf_40960              40960      1      5     204800B    209746     0
zio_data_buf_40960         40960     61     63    2580480B      6769     0
zio_buf_45056              45056      2      8     360448B    170825     0
zio_data_buf_45056         45056     47     49    2207744B      5656     0
zio_buf_49152              49152      3      8     393216B    156767     0
zio_data_buf_49152         49152     47     50    2457600B      5080     0
zio_buf_53248              53248     10     15     798720B    134202     0
zio_data_buf_53248         53248     38     41    2183168B      4856     0
zio_buf_57344              57344      1      4     229376B    118866     0
zio_data_buf_57344         57344     56     58    3325952B      4287     0
zio_buf_61440              61440      0      4     245760B     98056     0
zio_data_buf_61440         61440     64     66    4055040B      3909     0
zio_buf_65536              65536      0      5     327680B   1521718     0
zio_data_buf_65536         65536     36     39    2555904B      2959     0
zio_buf_69632              69632      0      5     348160B     92961     0
zio_data_buf_69632         69632     28     29    2019328B      2907     0
zio_buf_73728              73728      0      5     368640B     85055     0
zio_data_buf_73728         73728     27     29    2138112B      2586     0
zio_buf_77824              77824      3      8     622592B     84550     0
zio_data_buf_77824         77824     25     27    2101248B      2583     0
zio_buf_81920              81920      0      5     409600B     86278     0
zio_data_buf_81920         81920      7     11     901120B      2231     0
zio_buf_86016              86016      1      7     602112B     84425     0
zio_data_buf_86016         86016     20     22    1892352B      2111     0
zio_buf_90112              90112      1      7     630784B     76066     0
zio_data_buf_90112         90112     16     18    1622016B      1696     0
zio_buf_94208              94208      0      5     471040B     74458     0
zio_data_buf_94208         94208     13     16    1507328B      1711     0
zio_buf_98304              98304      0      5     491520B     79836     0
zio_data_buf_98304         98304     12     14    1376256B      1335     0
zio_buf_102400            102400      0      5     512000B     96068     0
zio_data_buf_102400       102400     15     18    1843200B      1223     0
zio_buf_106496            106496      0      4     425984B     80083     0
zio_data_buf_106496       106496      6      9     958464B      1182     0
zio_buf_110592            110592      0      5     552960B     73802     0
zio_data_buf_110592       110592     12     15    1658880B      1339     0
zio_buf_114688            114688      0      6     688128B     69899     0
zio_data_buf_114688       114688      3      8     917504B      1060     0
zio_buf_118784            118784      0      3     356352B     64731     0
zio_data_buf_118784       118784      4      7     831488B      1221     0
zio_buf_122880            122880      0      5     614400B     68852     0
zio_data_buf_122880       122880      2      9    1105920B      1017     0
zio_buf_126976            126976      0      5     634880B     93213     0
zio_data_buf_126976       126976      3      7     888832B       903     0
zio_buf_131072            131072    285    290   38010880B   1378932     0
zio_data_buf_131072       131072   1560   1562  204734464B    291569     0
  sa_cache                      56  65154 119919    6918144B    370950     0
dnode_t                      744  99992 100001   74473472B    404068     0
dmu_buf_impl_t               192 110729 153740   31485952B  25952186     0
arc_buf_hdr_t                176 561027 828256  154206208B 121785562     0
arc_buf_t                     56 355682 757002   43671552B 116102653     0
zil_lwb_cache                192      3     80      16384B       443     0
zfs_znode_cache              248  65154  65232   16699392B    429429     0
audit_proc                    40    120    300      12288B     85933     0
drv_secobj_cache             296      0      0          0B         0     0
dld_str_cache                304      2     13       4096B         4     0
ip_minor_arena_sa_1            1     21    128        128B      7814     0
ip_minor_arena_la_1            1      3     64         64B      4323     0
ip_conn_cache                720      0      5       4096B         2     0
tcp_conn_cache              1808     46     60     122880B      2042     0
udp_conn_cache              1256     20     39      53248B      9052     0
rawip_conn_cache            1096      0      0          0B         0     0
rts_conn_cache               816      5      9       8192B        11     0
ire_cache                    352     29     33      12288B       146     0
ncec_cache                   200     16    100      20480B       249     0
nce_cache                     88     16    135      12288B       385     0
rt_entry                     152     21     52       8192B       127     0
radix_mask                    32      3    125       4096B         5     0
radix_node                   120      2     33       4096B         2     0
ipsec_actions                 72      0      0          0B         0     0
ipsec_selectors               80      0      0          0B         0     0
ipsec_policy                  80      0      0          0B         0     0
tcp_timercache                88     95    315      28672B      7943     0
tcp_notsack_blk_cache         24      0    167       4096B       791     0
squeue_cache                 168      6     20       4096B         6     0
sctp_conn_cache             2528      0      0          0B         0     0
sctp_faddr_cache             176      0      0          0B         0     0
sctp_set_cache                24      0      0          0B         0     0
sctp_ftsn_set_cache           16      0      0          0B         0     0
dce_cache                    152      8     26       4096B        17     0
ire_gw_secattr_cache          24      0      0          0B         0     0
socket_cache                 640     45     66      45056B      9500     0
socktpi_cache                944      0      4       4096B         1     0
socktpi_unix_cache           944     22     28      28672B        41     0
sock_sod_cache               648      0      0          0B         0     0
exacct_object_cache           40      0      0          0B         0     0
kssl_cache                  1624      0      0          0B         0     0
fctl_cache                   112      0      0          0B         0     0
tl_cache                     432     40     45      20480B       138     0
keysock_1                      1      0      0          0B         0     0
spdsock_1                      1      0     64         64B         1     0
namefs_inodes_1                1     27    128        128B      6142     0
port_cache                    80      5     50       4096B         5     0
softmac_cache                568      1      7       4096B         2     0
softmac_upper_cache          232      0      0          0B         0     0
Hex0xffffff024e6a4558_minor_1      1      0      0          0B         0     0
Hex0xffffff024e6a4560_minor_1      1      0      0          0B         0     0
lnode_cache                   32      2    125       4096B       960     0
idm_buf_cache                240      0      0          0B         0     0
idm_task_cache              2432      0      0          0B         0     0
idm_tx_pdu_cache             464      0      0          0B         0     0
idm_rx_pdu_cache             513      0      0          0B         0     0
idm_128k_buf_cache        131072      0      0          0B         0     0
authkern_cache                72      0     55       4096B   5453453     0
authnone_cache                72      0      0          0B         0     0
authloopback_cache            72      0      0          0B         0     0
authdes_cache_handle          80      0      0          0B         0     0
dtrace_state_cache         16384      0      1      16384B         3     0
ufs_inode_cache              368      0      0          0B         0     0
directio_buf_cache           272      0      0          0B         0     0
lufs_save                     24      0      0          0B         0     0
lufs_bufs                    256      0      0          0B         0     0
lufs_mapentry_cache          112      0      0          0B         0     0
pty_map                       64      2     62       4096B        12     0
sppptun_map                  440      0      0          0B         0     0
smb_share_cache              136      4     29       4096B         4     0
smb_unexport_cache           272      0      0          0B         0     0
smb_vfs_cache                 48      2     83       4096B         2     0
smb_mbc_cache                 56      0      0          0B         0     0
rnode_cache                  656      0      6       4096B         3     0
nfs_access_cache              56      0     71       4096B         1     0
client_handle_cache           32      0    125       4096B        17     0
rnode4_cache                 968      0      0          0B         0     0
svnode_cache                  40      0      0          0B         0     0
nfs4_access_cache             56      0      0          0B         0     0
client_handle4_cache          32      0      0          0B         0     0
nfs4_ace4vals_cache           48      0      0          0B         0     0
nfs4_ace4_list_cache         264      0      0          0B         0     0
NFS_idmap_cache               48      0      0          0B         0     0
smb_node_cache               752      1     20      16384B       137     0
smb_oplock_break_cache        32      0      0          0B         0     0
smb_txreq                  66592      0      7     466944B       772     0
smb_dtor_cache                40      0    100       4096B       210     0
smb_request_cache           1608      0     10      16384B       794     0
smb_session_cache           1744      1     14      24576B        99     0
smb_user_cache               288      0     13       4096B        91     0
smb_tree_cache              1752      0      7      12288B        52     0
smb_ofile_cache              480      0      8       4096B        67     0
smb_odir_cache              8600      0      0          0B         0     0
smb_opipe_cache              216      0     18       4096B        66     0
smb_event_cache               72      3     55       4096B       297     0
crypto_session_cache         104      0      0          0B         0     0
lm_vnode                     184      0      0          0B         0     0
lm_xprt                       32      0      0          0B         0     0
lm_sysid                     160      0      0          0B         0     0
lm_client                    128      0      0          0B         0     0
lm_async                      32      0      0          0B         0     0
lm_sleep                      96      0      0          0B         0     0
lm_config                     80      2     50       4096B         2     0
------------------------- ------ ------ ------ ---------- --------- -----
Total [hat_memload]                               5111808B  61899647     0
Total [kmem_msb]                                104321024B  29877295     0
Total [kmem_va]                                1801715712B    640009     0
Total [kmem_default]                           1583808512B 3214471011     0
Total [kmem_io_64G]                              10772480B      6990     0
Total [kmem_io_4G]                                4247552B      2256     0
Total [kmem_io_2G]                                  20480B        55     0
Total [bp_map]                                    1441792B   6302319     0
Total [umem_np]                                    786432B       263     0
Total [zfs_file_data]                          2907308032B  13372402     0
Total [zfs_file_data_buf]                      2504310784B  37917426     0
Total [segkp]                                      655360B    117669     0
Total [ip_minor_arena_sa]                             128B      7814     0
Total [ip_minor_arena_la]                              64B      4323     0
Total [spdsock]                                        64B         1     0
Total [namefs_inodes]                                 128B      6142     0
------------------------- ------ ------ ------ ---------- --------- -----

vmem                         memory     memory    memory     alloc alloc
name                         in use      total    import   succeed  fail
------------------------- ---------- ----------- ---------- --------- -----
heap                      2356097024B 1088595951616B         0B     53642     0
    vmem_metadata         125435904B  125435904B 125435904B     29483     0
        vmem_seg          120279040B  120279040B 120279040B     29350     0
        vmem_hash           4904448B    4911104B   4911104B        84     0
        vmem_vmem            254800B     282160B    245760B        80     0
    static                        0B          0B         0B         0     0
        static_alloc              0B          0B         0B         0     0
    hat_memload             5111808B    5111808B   5111808B      6500     0
    kstat                    453128B     471040B    405504B      1494     0
    kmem_metadata         111153152B  134873088B 134873088B     47895     0
        kmem_msb          104321024B  104321024B 104321024B     47551     0
        kmem_cache           350200B     438272B    438272B       481     0
        kmem_hash           6373376B    6393856B   6393856B       786     0
    kmem_log                 131808B     135168B    135168B         6     0
    kmem_firewall_va       99987456B   99987456B  99987456B      5848     0
        kmem_firewall             0B          0B         0B         0     0
        kmem_oversize      99895134B   99987456B  99987456B      5848     0
    mod_sysfile                 265B       4096B      4096B         7     0
    kmem_va               1860415488B 1860415488B 1860415488B     33954     0
        kmem_default      1583808512B 1583808512B 1583808512B    465612     0
    kmem_io_64G            10772480B   10772480B  10772480B      2631     0
    kmem_io_4G              4247552B    4247552B   4247552B      1045     0
    kmem_io_2G                20480B      20480B     20480B         6     0
    kmem_io_16M                   0B          0B         0B         0     0
    bp_map                  1441792B    1441792B   1441792B        17     0
    umem_np                  786432B     786432B    786432B        21     0
    ksyms                   2182106B    2445312B   2445312B       296     0
    ctf                     2145463B    2158592B   2158592B       338     0
heap_core                   1765376B  931115008B         0B        57     0
    heaptext               10715136B   67108864B         0B       255     0
        module_text        10464984B   10723328B  10715136B       331     0
    id32                          0B          0B         0B         0     0
    module_data              913855B    2072576B   1765376B       448     0
logminor_space                   25B     262137B         0B       113     0
taskq_id_arena                   83B 2147483647B         0B       162     0
zfs_file_data             3156045824B 3181379584B         0B    154761     0
zfs_file_data_buf     2504310784B 2504310784B 2504310784B   8577092     0
device                      1536000B 1073741824B         0B     32959     0
segkp                      27553792B 2147483648B         0B      4444     0
mac_minor_ids                     3B     130070B         0B         4     0
rctl_ids                         39B      32767B         0B        39     0
zoneid_space                      0B       9998B         0B         0     0
taskid_space                     41B     999999B         0B       147     0
pool_ids                          0B     999998B         0B         0     0
contracts                        44B 2147483646B         0B       153     0
ip_minor_arena_sa               128B     262140B         0B         2     0
ip_minor_arena_la                64B 4294705152B         0B         1     0
lib_va_32                   7483392B 2138820608B         0B        20     0
lport-instances                   2B      65536B         0B         2     0
rport-instances                   7B      65536B         0B         7     0
tl_minor_space                   40B     262138B         0B       137     0
keysock                           0B 4294967295B         0B         0     0
spdsock                          64B 4294967295B         0B         1     0
namefs_inodes                   128B      65536B         0B         2     0
lib_va_64                  15134720B 137988695961600B         0B        68     0
Hex0xffffff024e6a4558_minor         0B 4294967294B         0B         0     0
Hex0xffffff024e6a4560_minor         0B 4294967294B         0B         0     0
devfsadm_event_channel            1B        101B         0B         1     0
devfsadm_event_channel            1B          2B         0B         1     0
syseventconfd_door                0B        101B         0B         0     0
syseventconfd_door                1B          2B         0B         1     0
syseventd_channel                 3B        101B         0B         3     0
syseventd_channel                 1B          2B         0B         1     0
dtrace                        68449B 4294967295B         0B     77712     0
dtrace_minor                      0B 4294967293B         0B         3     0
logdmux_minor                     0B        256B         0B         0     0
ptms_minor                        2B         16B         0B        12     0
sppptun_minor                     0B         16B         0B         0     0
lmsysid_space                     1B      16383B         0B         1     0
crypto                            0B         16B         0B        26     0
------------------------- ---------- ----------- ---------- --------- -----

Updated by George Wilson almost 2 years ago

  • Category set to zfs - Zettabyte File System
  • Assignee set to George Wilson

As Bryan indicated this is a case of memory fragmentation. The proposed fix makes the arc evict memory when the free space in the arena becomes scarce and provides a new VM interface to reap quantum caches.

Updated by George Wilson almost 2 years ago

  • Subject changed from zfs causing system to hang in vmem_xalloc() to zfs causing system to hang in vmem_xalloc()

Updated by Piotr Jasiukajtis almost 2 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

Fixed in changeset 13682:e7836650181b

1618 zfs causing system to hang in vmem_xalloc()
Reviewed by: Eric Schrock <>
Reviewed by: Adam Leventhal <>
Reviewed by: Matt Ahrens <>
Reviewed by: Richard Elling <>
Reviewed by: Garrett D'Amore <>
Reviewed by: Steve Gonczi <>
Approved by: Gordon Ross <>

Also available in: Atom PDF