Project

General

Profile

Bug #995

boot message: No randomness provider enabled for /dev/random

Added by Jon Strabala over 8 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
driver - device drivers
Start date:
2011-05-06
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

The boot message I see (on a fully installed server) is as follows:

WARNING: No randomness provider enabled for /dev/random. Use cryptoadm(1M) to enable a provider.

I have two systems that have the same illumos based OS install oi_148b (respin). The slow system is an Athlon X2. The faster system is a Supermico X9SCA-F with a E3-1280, where the OS is installed onto mirrored OCZ Vertex 3 SSD's this is a very fast box (a spec.org CPU 2006 result of 49.3 very close to topping the list). I only see this message on the "faster" system

Refer to:
https://defect.opensolaris.org/bz/show_bug.cgi?id=13547
and more interestingly
https://defect.opensolaris.org/bz/show_bug.cgi?id=38
The later states - "this is likely due to a race between KCF"

Note /dev/random is working just fine after the boot

root@lab10:~# dd if=/dev/random of=/tmp/random.out count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 1.5819e-05 s, 32.4 MB/s

===========================
The specs of the fast system:

root@lab10:~# cat /etc/release
OpenIndiana Development oi_148b X86 (powered by illumos)
Copyright 2010 Oracle and/or its affiliates. All rights reserved.
Use is subject to license terms.
Assembled 11 April 2011

root@lab10:~# uname -a
SunOS openindiana 5.11 oi_148b i86pc i386 i86pc Solaris

root@lab10:~# prtdiag
System Configuration: Supermicro X9SCI/X9SCA
BIOS Configuration: American Megatrends Inc. 4.6.4 02/24/2011
BMC Configuration: IPMI 2.0 (KCS: Keyboard Controller Style)

==== Processor Sockets ====================================

Version Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU E31280 @ 3.50GHz SOCKET 0

==== Memory Device Sockets ================================

Type Status Set Device Locator Bank Locator
----------- ------ --- ------------------- ----------------
Unknown in use 0 DIMM_1A BANK0
Unknown in use 0 DIMM_2A BANK0
Unknown in use 0 DIMM_1B BANK0
Unknown in use 0 DIMM_2B BANK0

==== On-Board Devices =====================================
To Be Filled By O.E.M.

==== Upgradeable Slots ====================================

ID Status Type Description
--- --------- ---------------- ----------------------------
0 in use PCI Express J6B2
1 in use PCI Express J6B1
2 in use PCI Express J6D1
3 in use PCI Express J7B1
4 in use PCI Express J8B4
root@lab10:~#


Related issues

Related to illumos gate - Feature #9640: /dev/random should not blockNew2018-07-05

Actions
Has duplicate illumos gate - Bug #1115: kcf should possibly depend upon swrandClosed2011-06-15

Actions

History

#1

Updated by Rich Lowe over 8 years ago

This has been seen, on and off, for a while. It's an easy symptom to induce.

One thing that might be useful is anonymous dtrace'ing for the consumer (or even just early callers of cmn_err) to try to nail down precisely when the attempt at use is being made, compare to when (I think) cryptoadm is run via cryptosvc.

#2

Updated by Garrett D'Amore over 8 years ago

I think this requires a fairly fast system to induce, because it requires that someone is trying to use /dev/random before the code to set up the underlying plumbing is done. Perhaps there should be an SMF dependency here?

#3

Updated by Rich Lowe about 8 years ago

This looks to be KCF trying to initialize randomness in kcf_rnd_init() prior to swrand being fully initialized. This looks likely to be because swrand depends upon KCF.

You can see the path by which KCF will consume randomness by following kcf_rnd_init() down into rnd_get_bytes via rnd_alloc_magazines().

Is this something Garrett stirred up with the kcfpoold changes?

#4

Updated by Richard PALO almost 8 years ago

for info https://www.illumos.org/issues/1425
I'm having this [still] after upgrading to oi_151a

Sep 20 16:08:44 X3200 genunix: [ID 365267 kern.notice] ^MOpenIndiana Build oi_151a 64-bit (illumos f342d051b376)
Sep 20 16:08:44 X3200 genunix: [ID 107366 kern.notice] SunOS Release 5.11 - Copyright 1983-2010 Oracle and/or its affiliates.
Sep 20 16:08:44 X3200 genunix: [ID 864463 kern.notice] All rights reserved. Use is subject to license terms.
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: lgpg
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: tsc
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: msr
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: mtrr
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: pge
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: de
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: cmov
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: mmx
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: mca
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: pae
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: cv8
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: pat
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: sse
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: sse2
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: asysc
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: nx
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: sse3
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: cx16
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: cmp
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: tscp
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: mwait
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: sse4a
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: cpuid
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: 1gpg
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: clfsh
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: 64
Sep 20 16:08:44 X3200 unix: [ID 223955 kern.info] x86_feature: svm
Sep 20 16:08:44 X3200 unix: [ID 168242 kern.info] mem = 8124984K (0x1efe8e000)
Sep 20 16:08:44 X3200 acpica: [ID 652648 kern.notice] ACPI: RSDP f8280 00014 (v0 ACRSYS)
Sep 20 16:08:44 X3200 acpica: [ID 197704 kern.notice] ACPI: RSDT afef3000 0003C (v1 ACRSYS ACRPRDCT 42302E31 NVDA 00000000)
Sep 20 16:08:44 X3200 acpica: [ID 226197 kern.notice] ACPI: FACP afef3080 00074 (v1 ACRSYS ACRPRDCT 42302E31 NVDA 00000000)
Sep 20 16:08:44 X3200 acpica: [ID 503918 kern.notice] ACPI Warning: Optional field Pm2ControlBlock has zero address or length:        0       0/1 (20091112/tbfadt-652)
Sep 20 16:08:44 X3200 acpica: [ID 536170 kern.notice] ACPI: DSDT afef3100 0AE39 (v1 ACRSYS ACRPRDCT 00001000 MSFT 03000000)
Sep 20 16:08:44 X3200 acpica: [ID 261815 kern.notice] ACPI: FACS afef1800 00040
Sep 20 16:08:44 X3200 acpica: [ID 910869 kern.notice] ACPI: SSDT afefe000 00544 (v1 PTLTD  POWERNOW 00000001  LTP 00000001)
Sep 20 16:08:44 X3200 acpica: [ID 338520 kern.notice] ACPI: HPET afefe580 00038 (v1 ACRSYS ACRPRDCT 42302E31 NVDA 00000098)
Sep 20 16:08:44 X3200 acpica: [ID 829985 kern.notice] ACPI: SLIC afefe5c0 00176 (v1 ACRSYS ACRPRDCT 42302E31 NVDA 00000000)
Sep 20 16:08:44 X3200 acpica: [ID 175781 kern.notice] ACPI: MCFG afefe740 0003C (v1 ACRSYS ACRPRDCT 42302E31 NVDA 00000000)
Sep 20 16:08:44 X3200 acpica: [ID 255862 kern.notice] ACPI: APIC afefdf40 0008E (v1 ACRSYS ACRPRDCT 42302E31 NVDA 00000000)
Sep 20 16:08:44 X3200 unix: [ID 190185 kern.info] SMBIOS v2.5 loaded (2063 bytes)
Sep 20 16:08:44 X3200 unix: [ID 972737 kern.info] Skipping psm: xpv_psm
Sep 20 16:08:44 X3200 rootnex: [ID 466748 kern.info] root nexus = i86pc
Sep 20 16:08:44 X3200 iommulib: [ID 321598 kern.info] NOTICE: iommulib_nexus_register: rootnex-1: Succesfully registered NEXUS i86pc nexops=fffffffffbd135e0
Sep 20 16:08:44 X3200 rootnex: [ID 349649 kern.info] pseudo0 at root
Sep 20 16:08:44 X3200 genunix: [ID 936769 kern.info] pseudo0 is /pseudo
Sep 20 16:08:44 X3200 rootnex: [ID 349649 kern.info] scsi_vhci0 at root
Sep 20 16:08:44 X3200 genunix: [ID 936769 kern.info] scsi_vhci0 is /scsi_vhci
Sep 20 16:08:44 X3200 genunix: [ID 596552 kern.info] Reading Intel IOMMU boot options
Sep 20 16:08:44 X3200 rootnex: [ID 349649 kern.info] npe0 at root: space 0 offset 0
Sep 20 16:08:44 X3200 genunix: [ID 936769 kern.info] npe0 is /pci@0,0
Sep 20 16:08:44 X3200 npe: [ID 236367 kern.info] PCI Express-device: isa@1, isa0
Sep 20 16:08:44 X3200 pcplusmp: [ID 615120 kern.info] NOTICE: apic: local nmi: 0 0x5 1
Sep 20 16:08:44 X3200 pcplusmp: [ID 615120 kern.info] NOTICE: apic: local nmi: 1 0x5 1
Sep 20 16:08:44 X3200 pcplusmp: [ID 615120 kern.info] NOTICE: apic: local nmi: 2 0x5 1
Sep 20 16:08:44 X3200 pcplusmp: [ID 615120 kern.info] NOTICE: apic: local nmi: 3 0x5 1
Sep 20 16:08:44 X3200 pcplusmp: [ID 801116 kern.info] NOTICE: ACPI HPET table query failed
Sep 20 16:08:44 X3200 pcplusmp: [ID 419660 kern.info] pcplusmp: irq 0x9 vector 0x80 ioapic 0x4 intin 0x9 is bound to cpu 1
Sep 20 16:08:44 X3200 amd_iommu: [ID 251261 kern.info] NOTICE: amd_iommu: No AMD IOMMU ACPI IVRS table
Sep 20 16:08:44 X3200 pseudo: [ID 129642 kern.info] pseudo-device: acpippm0
Sep 20 16:08:44 X3200 genunix: [ID 936769 kern.info] acpippm0 is /pseudo/acpippm@0
Sep 20 16:08:44 X3200 pseudo: [ID 129642 kern.info] pseudo-device: ppm0
Sep 20 16:08:44 X3200 genunix: [ID 936769 kern.info] ppm0 is /pseudo/ppm@0
Sep 20 16:08:44 X3200 ahci: [ID 405770 kern.info] NOTICE: ahci0: hba AHCI version = 1.20
Sep 20 16:08:44 X3200 pcplusmp: [ID 805372 kern.info] pcplusmp: pciclass,010601 (ahci) instance 0 irq 0x18 vector 0x40 ioapic 0xff intin 0xff is bound to cpu 2
Sep 20 16:08:44 X3200 sata: [ID 663010 kern.info] /pci@0,0/pci1025,157@9 :
Sep 20 16:08:44 X3200 sata: [ID 761595 kern.info]     SATA disk device at port 1
Sep 20 16:08:44 X3200 sata: [ID 846691 kern.info]     model WDC WD3200AAJS-22B4A0                   
Sep 20 16:08:44 X3200 sata: [ID 693010 kern.info]     firmware 01.03A01
Sep 20 16:08:44 X3200 sata: [ID 163988 kern.info]     serial number      WD-WCAT13982913
Sep 20 16:08:44 X3200 sata: [ID 594940 kern.info]     supported features:
Sep 20 16:08:44 X3200 sata: [ID 981177 kern.info]      48-bit LBA, DMA, Native Command Queueing, SMART, SMART self-test
Sep 20 16:08:44 X3200 sata: [ID 643337 kern.info]     SATA Gen2 signaling speed (3.0Gbps)
Sep 20 16:08:44 X3200 sata: [ID 349649 kern.info]     Supported queue depth 32
Sep 20 16:08:44 X3200 sata: [ID 349649 kern.info]     capacity = 625142448 sectors
Sep 20 16:08:44 X3200 scsi: [ID 583861 kern.info] sd1 at ahci0: target 1 lun 0
Sep 20 16:08:44 X3200 genunix: [ID 936769 kern.info] sd1 is /pci@0,0/pci1025,157@9/disk@1,0
Sep 20 16:08:44 X3200 genunix: [ID 408114 kern.info] /pci@0,0/pci1025,157@9/disk@1,0 (sd1) online
Sep 20 16:08:44 X3200 sata: [ID 663010 kern.info] /pci@0,0/pci1025,157@9 :
Sep 20 16:08:44 X3200 sata: [ID 761595 kern.info]     SATA CD/DVD (ATAPI) device at port 2
Sep 20 16:08:44 X3200 sata: [ID 846691 kern.info]     model HL-DT-ST DVDRAM GH15F                   
Sep 20 16:08:44 X3200 sata: [ID 693010 kern.info]     firmware EG00    
Sep 20 16:08:44 X3200 sata: [ID 163988 kern.info]     serial number K5588PJ4101         
Sep 20 16:08:44 X3200 sata: [ID 594940 kern.info]     supported features:
Sep 20 16:08:44 X3200 sata: [ID 981177 kern.info]      DMA
Sep 20 16:08:44 X3200 sata: [ID 514995 kern.info]     SATA Gen1 signaling speed (1.5Gbps)
Sep 20 16:08:45 X3200 scsi: [ID 583861 kern.info] sd2 at ahci0: target 2 lun 0
Sep 20 16:08:45 X3200 genunix: [ID 936769 kern.info] sd2 is /pci@0,0/pci1025,157@9/cdrom@2,0
*Sep 20 16:08:46 X3200 kcf: [ID 415456 kern.warning] WARNING: No randomness provider enabled for /dev/random. Use cryptoadm(1M) to enable a provider.*
...
#5

Updated by Albert Lee almost 8 years ago

This race seems to be preexisting? We could use kcf_rnd_get_pseudo_bytes in KCF as long as this doesn't violate any FIPS-140 rules.

I would also delay rnd_handler, or at least printing this message, until the first real reader appears.

#6

Updated by Richard PALO almost 8 years ago

I just noticed this on a 32-bit nvidia (shuttle FN41) system.
Seems prevalent on nvidia systems and has nothing to do with speed,
this system is butt slow.

Albert Lee wrote:

This race seems to be preexisting? We could use kcf_rnd_get_pseudo_bytes in KCF as long as this doesn't violate any FIPS-140 rules.

I would also delay rnd_handler, or at least printing this message, until the first real reader appears.

#7

Updated by Rene Nieuwenhuizen almost 8 years ago

I see this message also when oi_151a is running in a virtualbox on a oi_151a host. There was one instance this message was not shown.

#8

Updated by Rich Lowe almost 8 years ago

Albert Lee wrote:

This race seems to be preexisting? We could use kcf_rnd_get_pseudo_bytes in KCF as long as this doesn't violate any FIPS-140 rules.

I would also delay rnd_handler, or at least printing this message, until the first real reader appears.

The circular dependency didn't look pre-existing when I was checking it out, I don't think, but I could be wrong. We're not FIPS-140 certified, and won't ever be really, so I wouldn't worry too much about the the letter, so much as the spirit.

I would, however, certainly be nervous about screwing this up. I'd rather have the warning and availibility delay than actually compromised randomness.

#9

Updated by Alexander Eremin over 7 years ago

Could we at least put this warning into #ifdef DEBUG?
This story has been going on already 4 years...

#10

Updated by Rich Lowe over 7 years ago

Other instances of this message are different (often real) bugs.

Garrett, can we break the circular dependence here?

#11

Updated by Richard PALO over 7 years ago

just noticed this remark (from http://wiki.openindiana.org/display/oi/Workstations+and+Desktops?focusedCommentId=22446223#comment-22446223 )

Anonymous says:
On an HP xw8200 it boots and runs, but you must disable ACPI from BIOS (otherwis...

On an HP xw8200 it boots and runs, but you must disable ACPI from BIOS (otherwise you get the AE_NO_MEMORY ACPI exception and the system crashes randomly) and disable the internal SCSI RAID controller. For the rest, use the latest BIOS version (2.10) and SCSI controller firmware to get rid of the "/dev/random: no randomness provider" error.

#12

Updated by Garrett D'Amore over 6 years ago

  • Category set to driver - device drivers
  • Assignee set to Garrett D'Amore
  • % Done changed from 0 to 80
  • Tags deleted (needs-triage)

I have a fix.

http://cr.illumos.org/~gdamore/random-fix/

Please review.

#13

Updated by Garrett D'Amore over 6 years ago

Per Gordon's request:

The problem is that there are two entry points here that cause the check, and timing can determine which you get.

If you get a random consumer asking for data, you'll execute the check. The previous code handled that well. But if the timeout to initialize things finishes first, then you wind up with another path. The old path wasn't covered.

You can't just use module initializations, or dependencies, because circular dependencies cannot be resolved.

Other less elegant approaches involving the use of another level of asynchronicity can be used, but frankly they are more complex, and require more thorough analysis to ensure all code paths are race free.

The approach I've suggested is free from races because the code paths are idempotent. It uses a familiar singleton pattern that OOPS people will recognize.

More specifically, the races are between:

rnd_open() > kcf_rngprov_check() (called as part of /dev/random's open path)
kcf_rand_schedule_timeout()
>/*timeout*/->rnd_handler()->cmn_err()

Either of these can happen during early boot, before cryptoadm arranges to plumb in swrand. The challenge is to modload the swrand early enough to cover the checks in kcf_rngprov_check(), and to make sure that this is executed before the variable checks that are done in rnd_handler(). My solution is to add the "swrand" modload in kcf_rngprov_check(), if it hasn't already occurred. I also make sure that if the kcf_rngprov_check() hasn't already been done, we do it at least once before running the cmn_err().

#14

Updated by Garrett D'Amore over 6 years ago

After discussing with danmcd, following the pattern established in kcpc, the use of an atomic prevents any possibility of a race. So I'm changing the code as follows. Note that this specifically isn't required, since the code is idempotent, but it should avoid any confusion about thread safety.

I've also improved the comments around this slightly.

diff --git a/usr/src/uts/common/crypto/api/kcf_random.c b/usr/src/uts/common/crypto/api/kcf_random.c
index e566862..90fdad3 100644
--- a/usr/src/uts/common/crypto/api/kcf_random.c
+++ b/usr/src/uts/common/crypto/api/kcf_random.c
@ -166,15 +166,15 @ kcf_rngprov_check(void) {
int rv;
kcf_provider_desc_t *pd;
- static int inited = 0;
+ static uint32_t inited = 0;

- if (!inited) {
- /*
- * swrand should always be available unless another
- * provider is initialized.
- /
+ /

+ * The first time through, make sure that we have loaded the
+ * swrand module. This will normally be the default provider
+ * for random data.
+ */
+ if (atomic_cas_32(&inited, 0, 1) == 0) {
(void) modload("crypto", "swrand");
- inited = 1;
}

if ((pd = kcf_get_mech_provider(rngmech_type, NULL, NULL, &rv,
@ -910,15 +910,15 @ static void
rnd_handler(void *arg) {
int len = 0;
- static int inited = 0;
+ static uint32_t inited = 0;

- if (!inited) {
- /*
- * First time through, make sure that we have done the work
- * to load swrand by default.
- /
+ /

+ * Make sure we've checked for random providers (including
+ * loading up swrand by default first) before we start sending
+ * complaints (prematurely) to the console.
+ */
+ if (atomic_cas_32(&inited, 0, 1) == 0) {
(void) kcf_rngprov_check();
- inited = 1;
}

if (!rng_prov_found && rng_ok_to_log) {
#15

Updated by Gordon Ross over 6 years ago

  • Status changed from New to In Progress
  • Assignee changed from Garrett D'Amore to Gordon Ross

I had remaining questions about the root cause of this, so I
took a look for myself. It appears that the race is between
rnd_handler (called via timeout) and the rnd_mechid (called
via the system taskq). Here's the race in action. Note that
the rnd_handler call happens before the rnd_mechid call.

[0]> $C
ffffff0003efbae0 kcf`rnd_handler+8(38)
ffffff0003efbb20 callout_list_expire+0x77(ffffff01479dab80, ffffff01491daf40)
ffffff0003efbb50 callout_expire+0x31(ffffff01479dab80)
ffffff0003efbb70 callout_execute+0x1e(ffffff01479dab80)
ffffff0003efbc20 taskq_thread+0x285(ffffff0148f36350)
ffffff0003efbc30 thread_start+8()
[0]>

WARNING: No randomness provider enabled for /dev/random. Use cryptoadm(1M) to enable a provider.
kmdb: stop at kcf`rnd_mechid
kmdb: target stopped at:
kcf`rnd_mechid: pushq %rbp

[0]> $C
ffffff0003f19b90 kcf`rnd_mechid+8(38)
ffffff0003f19c20 taskq_d_thread+0xb1(ffffff0149144a48)
ffffff0003f19c30 thread_start+8()

On slower machines (i.e. in a VM) the timeout fires before
the system taskq gets around to calling rnd_mechid().

It's the latter function that calls crypto_mech2id() to
find and load a randomness provider (also setting the
no-auto-unload flag for the module) and hooks it into
the provider tables, etc. That needs to happen before
the rnd_handler timeouts begin.

There are also calls to rnd_get_bytes() before the
swrand provider is loaded. Here's the backtrace:
rnd_get_bytes
rnd_alloc_magazines
kcf_rnd_init
kcf`_init

Proposed fix:

Move a little more of the initialization into the existing
taskq call back. That taskq call back function was named
rnd_mechid, scheduled in kcf_rnd_schedule_timeout.
In the proposed fix, the callback is renamed rnd_init2,
and is scheduled by kcf_rnd_init (one level down in the
call stack from where the existing code schedules it.)
Also move the first call to kcf_rnd_schedule_timeout
into the taskq callback, so that rnd_handler calls will
not commence until the provider is loaded.

The early call to rnd_get_bytes is solved by moving that
(FIPS related) initialization from rnd_alloc_magazines to
a separate function: rnd_fips_discard_initial, which is
called from the taskq callback (after we have a provider).

#16

Updated by Gordon Ross over 6 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 80 to 100

https://github.com/illumos/illumos-gate/commit/717fae565868e87932422076eb52d99f1b7af64b

commit 717fae565868e87932422076eb52d99f1b7af64b
Author: Gordon Ross <gwr@nexenta.com>
Date:   Thu Dec 20 23:45:23 2012 -0500

    995 boot message: No randomness provider enabled for /dev/random
    Reviewed by: Garrett D'Amore <garrett@damore.org>
    Reviewed by: Dan McDonald <danmcd@nexenta.com>
    Reviewed by: Hans Rosenfeld <hans.rosenfeld@nexenta.com>
    Reviewed by: Boris Protopopov <boris.protopopov@nexenta.com>
    Approved by: Garrett D'Amore <garrett@damore.org>

#17

Updated by Dan McDonald about 1 year ago

Also available in: Atom PDF