Project

General

Profile

Bug #4088

use after free in arc_release()

Added by Boris Protopopov about 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Category:
-
Start date:
2013-08-30
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

A race-induced use after free occurs in arc_release() where the ARC header is used outside the critical section protected by the hash_lock. At the very bottom of the function, the following section accesses the header (hdr):

if (l2hdr) {
ARCSTAT_INCR(arcstat_l2_asize, l2hdr>b_asize);
list_remove(l2hdr->b_dev->l2ad_buflist, hdr); /* <--- */
kmem_free(l2hdr, sizeof (l2arc_buf_hdr_t));
ARCSTAT_INCR(arcstat_l2_size, -buf_size);
mutex_exit(&l2arc_buflist_mtx);
}

The following is an exerpt from the mdb session that shows use after free:

$C
ffffff0172a8c920 list_remove+0x1b(ffffff345c40e770, ffffff39327d18e0)
ffffff0172a8c9a0 arc_release+0x247(ffffff419de4b460, ffffff3b9ae218a0)
ffffff0172a8c9f0 dbuf_assign_arcbuf+0x12a(ffffff3b9ae218a0, ffffff4b0985cf60, ffffff989cae5dc8)
ffffff0172a8ca60 dmu_assign_arcbuf+0xb5(ffffff35985cb430, cfbe30000, ffffff4b0985cf60, ffffff989cae5dc8)
ffffff0172a8caf0 sbd_zvol_rele_write_bufs+0x100(ffffff34a0449458, ffffff8788a36178)
ffffff0172a8cb70 sbd_handle_sgl_write_xfer_completion+0xed(ffffff363c66d800, ffffff495ee43a80, ffffff8788a36178)
ffffff0172a8cbb0 sbd_dbuf_xfer_done+0x131(ffffff363c66d800, ffffff8788a36178)
ffffff0172a8cc40 stmf_worker_task+0x39c(ffffff349ce2a0c0)
ffffff0172a8cc50 thread_start+8()

Looking at the ARC header ffffff39327d18e0, one can see that it was freed:

ffffff39327d18e0/26K
0xffffff39327d18e0: deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef
deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef
deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef deadbeefdeadbeef
deadbeefdeadbeef deadbeefdeadbeef feedfacefeedface ffffff38c79e4c18 ffffff383370b8f6 e00000003

by the following thread:

ffffff38c79e4c18::bufctl
ADDR BUFADDR TIMESTAMP THREAD CALLER
ffffff38c79e4c18 ffffff39327d18e0 4a73a1fafef ffffff0172b49c60 arc_hdr_destroy+0x1a7 ffffff38c79e4c18::bufctl -v
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff38c79e4c18 ffffff39327d18e0 4a73a1fafef ffffff0172b49c60
ffffff3438f53008 ffffff330dba61c0 ffffff339a6f6538
kmem_cache_free_debug+0x12f
kmem_cache_free+0xb4
arc_hdr_destroy+0x1a7
arc_buf_free+0x108
arc_buf_remove_ref+0xcb
dbuf_assign_arcbuf+0x136
dmu_assign_arcbuf+0xb5
sbd_zvol_rele_write_bufs+0x100
sbd_handle_sgl_write_xfer_completion+0xed
sbd_dbuf_xfer_done+0x131
stmf_worker_task+0x39c
thread_start+8

History

#1

Updated by Boris Protopopov about 6 years ago

The simplest solution is to move the offending

list_remove(l2hdr->b_dev->l2ad_buflist, hdr);

up to the place where the proper lock is held:

if (l2hdr) {
mutex_enter(&l2arc_buflist_mtx);
hdr->b_l2hdr = NULL;
list_remove(l2hdr->b_dev->l2ad_buflist, hdr);
}

#2

Updated by Marcel Telka almost 6 years ago

  • Status changed from New to Resolved
commit ccc22e130479b5bd7c0002267fee1e0602d3f772
Author:     Boris Protopopov <boris.protopopov@nexenta.com>
AuthorDate: Fri Aug 30 12:12:45 2013 -0700
Commit:     Dan McDonald <danmcd@nexenta.com>
CommitDate: Fri Oct 18 20:26:52 2013 -0400

    4088 use after free in arc_release()
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: Garrett D'Amore <garrett@damore.org>
    Reviewed by: Saso Kiselkov <skiselkov.ml@gmail.com>
    Approved by: Dan McDonald <danmcd@nexenta.com>

Also available in: Atom PDF