Bug #6653

dtrace modifies ELF string table causing problems for linker

Added by Richard PALO about 3 years ago. Updated about 1 year ago.

Status:ClosedStart date:2016-02-14
Priority:UrgentDue date:
Assignee:-% Done:

100%

Category:DTrace
Target version:-
Difficulty:Medium Tags:needs-triage

Description

this may be somewhat related to https://www.illumos.org/issues/6590

Since binutils 2.26, in particular with gas, I came across issues building pkgsrc
modular-xorg-server with the dtrace option on illumos i386
as documented https://sourceware.org/bugzilla/show_bug.cgi?id=19576

What is noticeable during build is that the symbol referencing free() is seeming changed to ree()

using greadelf -aW the following is the before and after state of the afflicted object file
(.libs/resource.o)

--- /export/zone/dev64/root/chroot/test/tmp/pkgsrc/x11/modular-xorg-server/work/xorg-server-1.18.1/dix/resource.elf0
+++ /export/zone/dev64/root/chroot/test/tmp/pkgsrc/x11/modular-xorg-server/work/xorg-server-1.18.1/dix/resource.elf1
@@ -10,7 +10,7 @@
   Version:                           0x1
   Entry point address:               0x0
   Start of program headers:          0 (bytes into file)
-  Start of section headers:          11712 (bytes into file)
+  Start of section headers:          11680 (bytes into file)
   Flags:                             0x0
   Size of this header:               52 (bytes)
   Size of program headers:           0 (bytes)
@@ -23,20 +23,20 @@
   [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
   [ 0]                   NULL            00000000 000000 000000 00      0   0  0
   [ 1] .text             PROGBITS        00000000 000040 001987 00  AX  0   0 16
-  [ 2] .rel.text         REL             00000000 00266c 0005b0 08   I 14   1  4
-  [ 3] .data             PROGBITS        00000000 0019c7 000000 00  WA  0   0  1
-  [ 4] .bss              NOBITS          00000000 001a00 003008 00  WA  0   0 64
-  [ 5] .text.unlikely    PROGBITS        00000000 001a00 000000 00  AX  0   0  1
-  [ 6] .rodata.str1.1    PROGBITS        00000000 001a00 00002b 01 AMS  0   0  1
-  [ 7] .rodata           PROGBITS        00000000 001a2c 000027 00   A  0   0  4
-  [ 8] .rel.rodata       REL             00000000 002c1c 000030 08   I 14   7  4
-  [ 9] .rodata.str1.4    PROGBITS        00000000 001a54 00005d 01 AMS  0   0  4
-  [10] .data.rel.ro      PROGBITS        00000000 001ac0 0000a0 00  WA  0   0 64
-  [11] .rel.data.rel.ro  REL             00000000 002c4c 0000f0 08   I 14  10  4
-  [12] .comment          PROGBITS        00000000 001b60 000012 01  MS  0   0  1
-  [13] .shstrtab         STRTAB          00000000 002d3c 000083 00      0   0  1
-  [14] .symtab           SYMTAB          00000000 001b74 0005f0 10     15  36  4
-  [15] .strtab           STRTAB          00000000 002164 000507 00      0   0  1
+  [ 2] .rel.text         REL             00000000 0019c8 0005b0 08   I 14   1  4
+  [ 3] .data             PROGBITS        00000000 001f78 000000 00  WA  0   0  1
+  [ 4] .bss              NOBITS          00000000 001f80 003008 00  WA  0   0 64
+  [ 5] .text.unlikely    PROGBITS        00000000 001f78 000000 00  AX  0   0  1
+  [ 6] .rodata.str1.1    PROGBITS        00000000 001f78 00002b 01 AMS  0   0  1
+  [ 7] .rodata           PROGBITS        00000000 001fa4 000027 00   A  0   0  4
+  [ 8] .rel.rodata       REL             00000000 001fcc 000030 08   I 14   7  4
+  [ 9] .rodata.str1.4    PROGBITS        00000000 001ffc 00005d 01 AMS  0   0  4
+  [10] .data.rel.ro      PROGBITS        00000000 002080 0000a0 00  WA  0   0 64
+  [11] .rel.data.rel.ro  REL             00000000 002120 0000f0 08   I 14  10  4
+  [12] .comment          PROGBITS        00000000 002210 000012 01  MS  0   0  1
+  [13] .shstrtab         STRTAB          00000000 002222 000083 00      0   0  1
+  [14] .symtab           SYMTAB          00000000 0022a8 0005f0 10     15  36  4
+  [15] .strtab           STRTAB          00000000 002898 000507 00      0   0  1
 Key to Flags:
   W (write), A (alloc), X (execute), M (merge), S (strings)
   I (info), L (link order), G (group), T (TLS), E (exclude), x (unknown)
@@ -46,14 +46,14 @@

 There are no program headers in this file.

-Relocation section '.rel.text' at offset 0x266c contains 182 entries:
+Relocation section '.rel.text' at offset 0x19c8 contains 182 entries:
  Offset     Info    Type                Sym. Value  Symbol's Name
 00000141  0000240a R_386_GOTPC            00000000   _GLOBAL_OFFSET_TABLE_
 0000014a  00002503 R_386_GOT32            00000004   ResourceStateCallback
 00000176  00002604 R_386_PLT32            00000000   _CallCallbacks
 00000183  00002703 R_386_GOT32            00000004   TypeMask
 0000019a  00000409 R_386_GOTOFF           00000000   .bss
-000001a8  00002804 R_386_PLT32            00000000   free
+000001a8  00002804 R_386_PLT32            00000000   ree
 000001cd  0000240a R_386_GOTPC            00000000   _GLOBAL_OFFSET_TABLE_
 000001d6  00002a03 R_386_GOT32            00000004   lastResourceType
 000001e1  00000409 R_386_GOTOFF           00000000   .bss
@@ -95,7 +95,7 @@
 000005cd  00000409 R_386_GOTOFF           00000000   .bss
 000005dd  00002703 R_386_GOT32            00000004   TypeMask
 000005e9  00000409 R_386_GOTOFF           00000000   .bss
-000005ee  00002804 R_386_PLT32            00000000   free
+000005ee  00002804 R_386_PLT32            00000000   ree
 000005fa  00003604 R_386_PLT32            00000000   malloc
 00000605  00000409 R_386_GOTOFF           00000000   .bss
 0000060d  00001709 R_386_GOTOFF           00000000   .data.rel.ro
@@ -124,7 +124,7 @@
 00000a4f  00003d04 R_386_PLT32            00000000   FatalError
 00000a72  0000240a R_386_GOTPC            00000000   _GLOBAL_OFFSET_TABLE_
 00000a7d  00003f04 R_386_PLT32            00000000   LookupResourceName
-00000a8c  00004004 R_386_PLT32            00000000   __dtrace_Xserver___resource__alloc
+00000a8c  00004004 R_386_PLT32            00000000   __dtrace_Xserver___resource-alloc
 00000a94  00003204 R_386_PLT32            000004c0   ResourceClientBits
 00000a9b  00003204 R_386_PLT32            000004c0   ResourceClientBits
 00000ab5  00003204 R_386_PLT32            000004c0   ResourceClientBits
@@ -136,11 +136,11 @@
 00000bab  00002b04 R_386_PLT32            00000000   xreallocarray
 00000bc6  00002b04 R_386_PLT32            00000000   xreallocarray
 00000c5d  00003704 R_386_PLT32            00000630   HashResourceID
-00000c89  00002804 R_386_PLT32            00000000   free
-00000ca0  00002804 R_386_PLT32            00000000   free
+00000c89  00002804 R_386_PLT32            00000000   ree
+00000ca0  00002804 R_386_PLT32            00000000   ree
 00000cb8  00002703 R_386_GOT32            00000004   TypeMask
 00000cd1  00000409 R_386_GOTOFF           00000000   .bss
-00000cf5  00002804 R_386_PLT32            00000000   free
+00000cf5  00002804 R_386_PLT32            00000000   ree
 00000d08  00001c09 R_386_GOTOFF           00000030   .LC23
 00000d1d  00004104 R_386_PLT32            00000000   ErrorF
 00000d23  00001d09 R_386_GOTOFF           00000018   .LC24
@@ -153,7 +153,7 @@
 00000d83  00000409 R_386_GOTOFF           00000000   .bss
 00000db7  00003704 R_386_PLT32            00000630   HashResourceID
 00000df8  00003f04 R_386_PLT32            00000000   LookupResourceName
-00000e07  00004304 R_386_PLT32            00000000   __dtrace_Xserver___resource__free
+00000e07  00004304 R_386_PLT32            00000000   __dtrace_Xserver___resource-free
 00000e6d  0000240a R_386_GOTPC            00000000   _GLOBAL_OFFSET_TABLE_
 00000e84  00003204 R_386_PLT32            000004c0   ResourceClientBits
 00000e8b  00003204 R_386_PLT32            000004c0   ResourceClientBits
@@ -162,7 +162,7 @@
 00000ec7  00000409 R_386_GOTOFF           00000000   .bss
 00000ef5  00003704 R_386_PLT32            00000630   HashResourceID
 00000f2f  00003f04 R_386_PLT32            00000000   LookupResourceName
-00000f3e  00004304 R_386_PLT32            00000000   __dtrace_Xserver___resource__free
+00000f3e  00004304 R_386_PLT32            00000000   __dtrace_Xserver___resource-free
 00000f82  0000240a R_386_GOTPC            00000000   _GLOBAL_OFFSET_TABLE_
 00000f8d  00003204 R_386_PLT32            000004c0   ResourceClientBits
 00000f95  00003204 R_386_PLT32            000004c0   ResourceClientBits
@@ -185,13 +185,13 @@
 0000134d  0000240a R_386_GOTPC            00000000   _GLOBAL_OFFSET_TABLE_
 00001360  00000409 R_386_GOTOFF           00000000   .bss
 000013dd  00003f04 R_386_PLT32            00000000   LookupResourceName
-000013ec  00004304 R_386_PLT32            00000000   __dtrace_Xserver___resource__free
+000013ec  00004304 R_386_PLT32            00000000   __dtrace_Xserver___resource-free
 0000144d  0000240a R_386_GOTPC            00000000   _GLOBAL_OFFSET_TABLE_
 00001467  00000409 R_386_GOTOFF           00000000   .bss
 0000146c  00004c04 R_386_PLT32            00000000   HandleSaveSet
 000014af  00003f04 R_386_PLT32            00000000   LookupResourceName
-000014be  00004304 R_386_PLT32            00000000   __dtrace_Xserver___resource__free
-00001511  00002804 R_386_PLT32            00000000   free
+000014be  00004304 R_386_PLT32            00000000   __dtrace_Xserver___resource-free
+00001511  00002804 R_386_PLT32            00000000   ree
 0000155d  0000240a R_386_GOTPC            00000000   _GLOBAL_OFFSET_TABLE_
 00001566  00004e03 R_386_GOT32            00000000   currentMaxClients
 00001572  00000409 R_386_GOTOFF           00000000   .bss
@@ -231,7 +231,7 @@
 00001952  00003503 R_386_GOT32            00000000   serverClient
 00001967  00005104 R_386_PLT32            00001720   dixLookupResourceByClass

-Relocation section '.rel.rodata' at offset 0x2c1c contains 6 entries:
+Relocation section '.rel.rodata' at offset 0x1fcc contains 6 entries:
  Offset     Info    Type                Sym. Value  Symbol's Name
 00000000  00001e09 R_386_GOTOFF           000006a0   .L89
 00000004  00001f09 R_386_GOTOFF           000006c0   .L91
@@ -240,7 +240,7 @@
 00000010  00002209 R_386_GOTOFF           00000720   .L94
 00000014  00001809 R_386_GOTOFF           00000680   .L100

-Relocation section '.rel.data.rel.ro' at offset 0x2c4c contains 30 entries:
+Relocation section '.rel.data.rel.ro' at offset 0x2120 contains 30 entries:
  Offset     Info    Type                Sym. Value  Symbol's Name
 00000000  00005501 R_386_32               00000000   NoopDDA
 00000004  00000201 R_386_32               00000000   .text
@@ -317,7 +317,7 @@
     37: 00000004     4 OBJECT  GLOBAL DEFAULT  COM ResourceStateCallback
     38: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND _CallCallbacks
     39: 00000004     4 OBJECT  GLOBAL DEFAULT  COM TypeMask
-    40: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND free
+    40: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND ree
     41: 000001c0   153 FUNC    GLOBAL DEFAULT    1 CreateNewResourceType
     42: 00000004     4 OBJECT  GLOBAL DEFAULT  COM lastResourceType
     43: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND xreallocarray
@@ -341,10 +341,10 @@
     61: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND FatalError
     62: 00000a60   720 FUNC    GLOBAL DEFAULT    1 AddResource
     63: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND LookupResourceName
-    64: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __dtrace_Xserver___resource__alloc
+    64: 00000000     0 NOTYPE  GLOBAL DEFAULT OS [0xff3f] __dtrace_Xserver___resource-alloc
     65: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND ErrorF
     66: 00000d30   298 FUNC    GLOBAL DEFAULT    1 FreeResource
-    67: 00000000     0 NOTYPE  GLOBAL DEFAULT  UND __dtrace_Xserver___resource__free
+    67: 00000000     0 NOTYPE  GLOBAL DEFAULT OS [0xff3f] __dtrace_Xserver___resource-free
     68: 00000e60   267 FUNC    GLOBAL DEFAULT    1 FreeResourceByType
     69: 00000f70   184 FUNC    GLOBAL DEFAULT    1 ChangeResourceValue
     70: 00001030   244 FUNC    GLOBAL DEFAULT    1 FindClientResourcesByType

Xserver.d (2.69 KB) Richard PALO, 2016-07-07 09:58 AM

resource.o.orig (12.1 KB) Richard PALO, 2016-07-07 09:58 AM

History

#1 Updated by Richard PALO about 3 years ago

BTW, there are three other dtrace strobed files in dix/ name getevents, events and dispatch.
These have modifications of the same sort, such as the following snippet:

@@ -424,12 +424,12 @@
 00003959  00006303 R_386_GOT32            00000000   screenInfo
 00003982  0000e103 R_386_GOT32            00000004   EventCallback
 000039a3  0000a404 R_386_PLT32            00000000   _CallCallbacks
-000039b1  0000e204 R_386_PLT32            00000000   __dtraceenabled_Xserver___send__event
+000039b1  0000e204 R_386_PLT32            00000000   __dtraceenabled_Xserver___send-event
 00003a16  00000409 R_386_GOTOFF           00000000   .bss
 00003a3d  00000409 R_386_GOTOFF           00000000   .bss
 00003a46  0000e303 R_386_GOT32            00000000   EventSwapVector
 00003a69  0000e404 R_386_PLT32            00000000   WriteToClient
-00003aaf  0000e504 R_386_PLT32            00000000   __dtrace_Xserver___send__event
+00003aaf  0000e504 R_386_PLT32            00000000   __dtrace_Xserver___send-event
 00003aca  00003b09 R_386_GOTOFF           00000114   .LC80
 00003ad3  00008204 R_386_PLT32            00000000   ErrorF
 00003af8  0000e404 R_386_PLT32            00000000   WriteToClient

but they seem to be able to link in fine as no non-dtrace symbol name is modified.

#2 Updated by Andrew Stormont over 2 years ago

Any idea if 2.26.1 fixes this problem?

#3 Updated by Richard PALO over 2 years ago

Andrew Stormont wrote:

Any idea if 2.26.1 fixes this problem?

well, running omnios-master-7397b49 and updating to 2.26.1
I see the same output.

I'm reasonably convinced that this is a dtrace issue, at least interworking
with recent binutils, but have not ventured into figuring out how to debug it.

What bothers me here are for example the name changes from
"__dtrace_Xserver___resource__free" to
"__dtrace_Xserver___resource-free".

perhaps some counter is off (there is one character difference from "__" to "-")
when dtrace writes updates to the object file.

Also, if I add '-save-temps' to CFLAGS, I see [naturally] that the correct name is
generated by gas, but perhaps the section and/or section attributes are a bit different.

#4 Updated by Richard PALO over 2 years ago

Just to simplify reproduction of this, the attached resource.o (.orig) is the output of the compile phase.
Copy it to resource.o and execute:

dtrace -G -C -S -s Xserver.d resource.o

now the corruption is noticeable.

#5 Updated by Richard PALO over 2 years ago

just for grins, I tried just dtrace on an old OI 151a9 system, with the same result.
It's probably related to some crazy elf structure and/or section change in recent binutils
that the gate elf tools doesn't digest correctly.

Also, for more chuckles, I notice that strings -a prints a bit of garbage, whereas gstrings
or elftoolchain strings seems okay... quite possibly related.

#6 Updated by Richard PALO over 2 years ago

one thing I just noticed via strings is

richard@omnis:/home/richard/src/tdtrace$ strings -a resource.o.orig |grep free
__dtrace_Xserver___resource__free

hmm

richard@omnis:/home/richard/src/tdtrace$ dump -c resource.o.orig 

resource.o.orig:

     **** STRING TABLE INFORMATION ****

.shstrtab:
   <offset>      Name
   <0>      
   <1>      .symtab
   <9>      .strtab
   <17>      .shstrtab
   <27>      .rel.text
   <37>      .data
   <43>      .bss
   <48>      .text.unlikely
   <63>      .rodata.str1.1
   <78>      .rel.rodata
   <90>      .rodata.str1.4
   <105>      .rel.data.rel.ro
   <122>      .comment

.strtab:
   <offset>      Name
   <0>      
   <1>      resource.c
   <12>      GetDefaultBytes
   <28>      DefaultFindSubRes
   <46>      FindWindowSubRes
   <63>      FindGCSubRes
   <76>      GetPixmapBytes
   <91>      doFreeResource
   <106>      resourceTypes
   <120>      lastResourceClass
   <138>      GetGcBytes
   <149>      GetWindowBytes
   <164>      clientTable
   <176>      predefTypes
   <188>      __func__.15628
   <203>      AvailableID.part.1
   <222>      .L100
   <228>      .LC16
   <234>      .LC17
   <240>      .LC21
   <246>      .LC23
   <252>      .LC24
   <258>      .L89
   <263>      .L91
   <268>      .L92
   <273>      .L93
   <278>      .L94
   <283>      _GLOBAL_OFFSET_TABLE_
   <305>      ResourceStateCallback
   <327>      _CallCallbacks
   <342>      TypeMask
   <351>      CreateNewResourceType
   <373>      lastResourceType
   <390>      xreallocarray
   <404>      RegisterResourceName
   <425>      GetResourceTypeSizeFunc
   <449>      SetResourceTypeSizeFunc
   <473>      SetResourceTypeFindSubResFunc
   <503>      SetResourceTypeErrorValue
   <529>      CreateNewResourceClass
   <552>      ResourceClientBits
   <571>      LimitClients
   <584>      InitClientResources
   <604>      serverClient
   <617>      malloc
   <624>      HashResourceID
   <639>      __assert_c99
   <652>      GetXIDRange
   <664>      FakeClientID
   <677>      clients
   <685>      MarkClientException
   <705>      FatalError
   <716>      AddResource
   <728>      LookupResourceName
   <747>      __dtrace_Xserver___resource__alloc
   <782>      ErrorF
   <789>      __dtrace_Xserver___resource__free
   <823>      FreeResourceByType
   <842>      ChangeResourceValue
   <862>      FindClientResourcesByType
   <888>      FindSubResources
   <905>      FindAllClientResources
   <928>      LookupClientResourceComplex
   <956>      FreeClientNeverRetainResources
   <987>      FreeClientResources
   <1007>      HandleSaveSet
   <1021>      FreeAllResources
   <1038>      currentMaxClients
   <1056>      dixLookupResourceByType
   <1080>      XaceHook
   <1089>      dixLookupResourceByClass
   <1114>      GetXIDList
   <1125>      LegalNewID
   <1136>      noPanoramiXExtension
   <1157>      NoopDDA
   <1165>      DeleteWindow
   <1178>      dixDestroyPixmap
   <1195>      FreeGC
   <1202>      CloseFont
   <1212>      FreeCursor
   <1223>      FreeColormap
   <1236>      FreeClientPixels
   <1253>      OtherClientGone
   <1269>      DeletePassiveGrab

looks perhaps as if the symbol 'free' is shared in the string table from '__dtrace_Xserver___resource__free'
which would explain that if dtrace likes to change the name from '__dtrace_Xserver___resource__free'
to '__dtrace_Xserver___resource-free' without heeding the possible overlap, then effectively 'free' is shifted left
one character.

apparently strhyphenate() is potentially dangerous to call without verifying first that there are no overlapping
referenced strings from the string table...

#7 Updated by Richard PALO over 2 years ago

according to http://wiki.osdev.org/ELF_Tutorial

  • The String Table*

The string table conceptually is quite simple: it's just a number of consecutive zero-terminated strings. String literals used in the program are stored in one of the tables. There are a number of different string tables that may be present in an ELF object such as .strtab (the default string table), .shstrtab (the section string table) and .dynstr (string table for dynamic linking). Anytime the loading process needs access to a string, it uses an offset into one of the string tables. The offset may point to the beginning of a zero-terminated string or somewhere in the middle or even to the zero terminator itself, depending on usage and scenario. The size of the string table itself is specified by sh_size in the corresponding section header entry.

The simplest program loader may copy all string tables into memory, but a more complete solution would omit any that are not necessary during runtime such, notably those not flagged with SHF_ALLOC in their respective section header (such as .shstrtab, since section names aren't used in program runtime).

#8 Updated by Richard PALO over 2 years ago

  • Subject changed from dtrace issues with [patched] binutils gas 2.26 to dtrace modifies ELF string table causing problems for linker
  • Priority changed from Normal to Urgent
  • Category set to DTrace

#9 Updated by Steve Wills about 2 years ago

Just an FYI, I ran into this issue on FreeBSD too. I was using clang 3.8. I found that avoiding clang's integrated assembler using -no-integrated-as avoids it since the external assembler is older.

#10 Updated by Dan McDonald almost 2 years ago

Adding myself to the watchlist, as OmniOS is frozen on binutils 2.25 until such time as this issue is addressed.

#11 Updated by Andy Fiddaman about 1 year ago

As already suspected in some of the previous comments, the underlying problem does turn out to be with dtrace as it modifies (corrupts?) the string table in the binary when resolving probes.

The problem was first exposed by a commit to binutils that made it into 2.26 - see https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=commit;h=ef10c3ace00674e8c3599c3bf95f06c87d68898b - which introduced string suffix merging to save space in the string table.

A fix for this has been in FreeBSD for a while ( https://github.com/freebsd/freebsd/commit/bdbe7eaee6d095d91694b16f10fd0c3b90c0de20 ) and in Joyent too (see https://smartos.org/bugview/OS-5946 ).

We've pulled it into OmniOS bloody and updated binutils to 2.29.1 and run full end-to-end builds and tests with dtrace (including testing with the files attached to this issue). All seems fine so far.

#12 Updated by Andy Fiddaman about 1 year ago

Oh, and here's the original FreeBSD fix commit message for reference:

dtrace converts pairs of consecutive underscores in a probe name to dashes.
When dtrace -G processes relocations corresponding to USDT probe sites, it
performs this conversion on the corresponding symbol names prior to looking
up the resulting probe names in the USDT provider definition. However, in
so doing it would actually modify the input object's string table, which
breaks the string suffix merging done by recent binutils. Because we don't
care about the symbol name once the probe site is recorded, just perform the
probe lookup using a temporary copy.

#13 Updated by Electric Monk about 1 year ago

  • % Done changed from 0 to 100
  • Status changed from New to Closed

git commit 6c19201566ac520f0d4eef6ed2f70bf9a4815eae

commit  6c19201566ac520f0d4eef6ed2f70bf9a4815eae
Author: Jerry Jelinek <jerry.jelinek@joyent.com>
Date:   2018-02-12T10:56:17.000Z

    6653 dtrace modifies ELF string table causing problems for linker
    Reviewed by: Andy Fiddaman <andy@omniosce.org>
    Reviewed by: Rich Lowe <richlowe@richlowe.net>
    Reviewed by: Yuri Pankov <yuripv@yuripv.net>
    Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
    Reviewed by: Dominik Hassler <hadfl@omniosce.org>
    Approved by: Hans Rosenfeld <hans.rosenfeld@joyent.com>

Also available in: Atom