Project

General

Profile

Actions

Bug #16544

closed

udp_output_lastdst() can end up with wrong ixa

Added by Robert Mustacchi 17 days ago. Updated 3 days ago.

Status:
Closed
Priority:
Normal
Category:
networking
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:

Description

We have an application that leverages IPv6 UDP multicast that leverages scope IDs to determine the correct interface to go out on. We were able to reproduce an issue where we had a packet going out the wrong interface for the scope ID. Between a combination of snoop and truss, we were able to see what the application was doing and then we started looking at the kernel.

To start with, we first confirmed easy ways to get the scope ID on entry to udp send and on dls sending the packet. Here's the first few things I did:

BRM42220014 # dtrace -n 'fbt:dld:str_mdata_fastpath_put:entry/execname == "faux-mgs"/{ this->mac = (mac_client_impl_t *)args[0]->ds_mch; @[this->mac->mci_name] = count(); }'
dtrace: description 'fbt:dld:str_mdata_fastpath_put:entry' matched 1 probe
^C

  sidecar1                                                         13
  psc0                                                             15

BRM42220014 # dtrace -n 'fbt::udp_send:entry/execname == "faux-mgs"/{ this->sin6 = (sin6_t *)args[2]->msg_name; @[this->sin6->sin6_scope_id] = count(); }'
dtrace: description 'fbt::udp_send:entry' matched 1 probe
^C

       79               14
       76               15

Along the way, I looked at the stacks that this application was getting to here, which were generally the same other than whether they used udp_output_newdest() or udp_output_lastdst(). The example stacks are:


              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
              ip`conn_ip_output+0x18f
              ip`udp_output_lastdst+0xff
              ip`udp_send+0x4ba
              sockfs`so_sendmsg+0x201
              sockfs`socket_sendmsg+0x65
              sockfs`sendit+0x186
              sockfs`sendto+0x9b
              unix`sys_syscall+0x17d
                6

              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
              ip`conn_ip_output+0x18f
              ip`udp_output_newdst+0x69d
              ip`udp_send+0x4e8
              sockfs`so_sendmsg+0x201
              sockfs`socket_sendmsg+0x65
              sockfs`sendit+0x186
              sockfs`sendto+0x9b
              unix`sys_syscall+0x17d
              430

With this, I then started to put together a script to prove that we were seeing this mismatch between application scope ID and the sending datalink at the far end. This D script was evolved a bunch, so I don't have it entirely, but it looked approximately like:

fbt::udp_send:entry
/execname == "faux-mgs"/
{
        this->sin6 = (sin6_t *)args[2]->msg_name;
        self->scope = this->sin6->sin6_scope_id;
}

fbt::udp_send:return
{
        self->scope = 0;
}

fbt:dld:str_mdata_fastpath_put:entry
/self->scope/
{
        this->mac = (mac_client_impl_t *)args[0]->ds_mch;
        @[self->scope, this->mac->mci_name] = count();
}

And it has output close to:

CPU     ID                    FUNCTION:NAME
 58  81704                         :tick-5s
       76  sidecar1                                                          1
       76  psc0                                                            217
       79  sidecar1                                                        218

 58  81704                         :tick-5s
       79  sidecar1                                                        196
       76  psc0                                                            197

So we did see a few mixups. Next, we changed things around so we could see what the stack was when this happened. Notably the stacks when we have mismatches were consistent. Here was the bit of D to do that:

fbt:dld:str_mdata_fastpath_put:entry
/(self->scope == 76 && ((mac_client_impl_t *)args[0]->ds_mch)->mci_name == "sidecar1" ||
      (self->scope == 79 && ((mac_client_impl_t *)args[0]->ds_mch)->mci_name == "psc0"))/
{
        stack();
}

...

CPU     ID                    FUNCTION:NAME
103  54334     str_mdata_fastpath_put:entry
              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
              ip`conn_ip_output+0x18f
              ip`udp_output_lastdst+0xff
              ip`udp_send+0x4ba
              sockfs`so_sendmsg+0x201
              sockfs`socket_sendmsg+0x65
              sockfs`sendit+0x186
              sockfs`sendto+0x9b
              unix`sys_syscall+0x17d

111  54334     str_mdata_fastpath_put:entry
              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
              ip`conn_ip_output+0x18f
              ip`udp_output_lastdst+0xff
              ip`udp_send+0x4ba
              sockfs`so_sendmsg+0x201

So everything was always coming from us having the lastdst path. We took a look at what the aggregation of which stacks were there and we didn't always hit lastdst. From there I started staring at conn_same_as_last_v6() but that didn't really prove fruitful. There was nothing consistent there with when we saw errors. This ends up looking like:

BRM42220014 # cat wat.d
fbt::udp_send:entry
/execname == "faux-mgs"/
{
        this->sin6 = (sin6_t *)args[2]->msg_name;
        self->scope = this->sin6->sin6_scope_id;
}

fbt::udp_send:return
{
        self->scope = 0;
}

fbt:dld:str_mdata_fastpath_put:entry
/self->scope/
{
        this->mac = (mac_client_impl_t *)args[0]->ds_mch;
        @[self->scope, this->mac->mci_name] = count();
        @s[stack()] = count();
}

fbt:dld:str_mdata_fastpath_put:entry
/(self->scope == 76 && ((mac_client_impl_t *)args[0]->ds_mch)->mci_name == "sidecar1" ||
      (self->scope == 79 && ((mac_client_impl_t *)args[0]->ds_mch)->mci_name == "psc0"))/
{
        stack();
}

fbt::conn_same_as_last_v6:return
/execname == "faux-mgs"/
{
        @conn[arg1] = count();
}

With this in mind, we ended up trying to work down could we prove an earlier place where this wasn't working. So the first place we did this was by asking if by the time we hit the ire_sendfn() indirect function, was this wrong. That is, could have this been changed or not. Unfortunately the D script for this isn't entirely there; however, here's a bit of what the output was:

BRM42220014 # dtrace -s mcast.d
dtrace: script 'mcast.d' matched 3 probes
^C

       76  sidecar1                                                          1
       76  psc0                                                            220
       79  sidecar1                                                        220
BRM42220014 # dtrace -s mcast.d -n 'tick-30s{ printa(@); exit(0); }'
dtrace: script 'mcast.d' matched 3 probes
dtrace: description 'tick-30s' matched 1 probe
CPU     ID                    FUNCTION:NAME
  1  81700                        :tick-30s
       76  sidecar1                                                          5
       76  psc0                                                           1030
       79  sidecar1                                                       1030

This showed we were hitting it, so we changed to move it sooner eventually and it looked a bit like this:

fbt::udp_send:entry
/execname == "faux-mgs"/
{
        this->sin6 = (sin6_t *)args[2]->msg_name;
        self->scope = this->sin6->sin6_scope_id;
}

fbt::udp_send:return

{
        self->scope = 0;
}

fbt::udp_output_lastdst:entry
/self->scope != 0 && args[4]->ixa_ire != NULL/
{
        @[self->scope, stringof(args[4]->ixa_ire->ire_ill->ill_name)] = count();
}
BRM42220014 # dtrace -s mcast.d
dtrace: script 'mcast.d' matched 3 probes
^C

       79  sidecar1                                                          1
       76  psc0                                                              5

So this wasn't quite as informative as we wanted. We started trying to work closer through it, but a lot of the intermediate scripts were off. So a lot of our focus was trying to figure out how to look at where the wrong ill came from. Based on the ixa, that told us a lot there. From there, the question that wasn't clear was how was it wrong. We went through a bunch of blank ends for example, by trying to see if the ixa->ixa_ire == NULL case, but the bit we found from that script above, that didn't quite tell us much usefully there because it turned out the ire was NULL a fair chunk of the time.

We evolved our main script a bunch more to get more information and confirm certain things were off:

# cat wat.d
fbt::udp_send:entry
/execname == "faux-mgs"/
{
        this->sin6 = (sin6_t *)args[2]->msg_name;
        self->scope = this->sin6->sin6_scope_id;
}

fbt::ip_attr_connect:entry
/self->scope/
{       
        self->ire = args[1];
}

fbt::ip_attr_connect:return
/self->scope && self->ire/
{
        self->attr_ret = self->ire->ixa_ire->ire_ill->ill_name;
        self->ire_scope = self->ire->ixa_scopeid;
}

fbt::udp_output_lastdst:entry
/self->scope != 0 && args[4]->ixa_ire != NULL/
{       
        self->ire_ill = args[4]->ixa_ire->ire_ill->ill_name;
}

fbt::udp_send:return
{
        self->scope = 0;
        self->attr_ret = 0;
        self->attr_ire = 0;
}

fbt:dld:str_mdata_fastpath_put:entry
/self->scope/
{
        this->mac = (mac_client_impl_t *)args[0]->ds_mch;
        @[self->scope, this->mac->mci_name] = count();
        @s[stack()] = count();
}

fbt:dld:str_mdata_fastpath_put:entry
/(self->scope == 76 && ((mac_client_impl_t *)args[0]->ds_mch)->mci_name == "sidecar1" ||
      (self->scope == 79 && ((mac_client_impl_t *)args[0]->ds_mch)->mci_name == "psc0"))/
{
        stack();
        trace(self->scope);
        trace(self->ire_scope);
        trace(self->ire_ill);
        trace(stringof(self->attr_ret));
}

fbt::conn_same_as_last_v6:return
/execname == "faux-mgs"/
{
        @conn[arg1] = count();
}

BRM42220014 # dtrace -s wat.d
dtrace: script 'wat.d' matched 8 probes
CPU     ID                    FUNCTION:NAME
  9  54334     str_mdata_fastpath_put:entry
              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
              ip`conn_ip_output+0x18f
              ip`udp_output_lastdst+0xff
              ip`udp_send+0x4ba
              sockfs`so_sendmsg+0x201
              sockfs`socket_sendmsg+0x65
              sockfs`sendit+0x186
              sockfs`sendto+0x9b
              unix`sys_syscall+0x17d
       76       79                0  sidecar1
 18  54334     str_mdata_fastpath_put:entry
              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
              ip`conn_ip_output+0x18f
              ip`udp_output_lastdst+0xff
              ip`udp_send+0x4ba
              sockfs`so_sendmsg+0x201
              sockfs`socket_sendmsg+0x65
              sockfs`sendit+0x186
              sockfs`sendto+0x9b
              unix`sys_syscall+0x17d
       76       79                0  sidecar1
 37  54334     str_mdata_fastpath_put:entry
              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
fbt::udp_send:entry
              ip`conn_ip_output+0x18f
              ip`udp_output_lastdst+0xff
              ip`udp_send+0x4ba
              sockfs`so_sendmsg+0x201
              sockfs`socket_sendmsg+0x65
              sockfs`sendit+0x186
              sockfs`sendto+0x9b
              unix`sys_syscall+0x17d
       76       79                0  sidecar1
  9  54334     str_mdata_fastpath_put:entry
              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
              ip`conn_ip_output+0x18f
              ip`udp_output_lastdst+0xff
              ip`udp_send+0x4ba
              sockfs`so_sendmsg+0x201
              sockfs`socket_sendmsg+0x65
              sockfs`sendit+0x186
              sockfs`sendto+0x9b
              unix`sys_syscall+0x17d
       76       79                0  sidecar1
^C

       76  sidecar1                                                          4
       79  sidecar1                                                        256
       76  psc0                                                            257

              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
              ip`conn_ip_output+0x18f
              ip`udp_output_lastdst+0xff
              ip`udp_send+0x4ba
              sockfs`so_sendmsg+0x201
              sockfs`socket_sendmsg+0x65
              sockfs`sendit+0x186
              sockfs`sendto+0x9b
              unix`sys_syscall+0x17d
               11

              dld`dld_wput+0x1ab
              unix`putnext+0x2d5
              ip`ip_xmit+0x422
              ip`ip_postfrag_loopcheck+0x99
              ip`ire_send_wire_v6+0x12e
              ip`ire_send_multicast_v6+0xb0
              ip`conn_ip_output+0x18f
              ip`udp_output_newdst+0x69d
              ip`udp_send+0x4e8
              sockfs`so_sendmsg+0x201
              sockfs`socket_sendmsg+0x65
              sockfs`sendit+0x186
              sockfs`sendto+0x9b
              unix`sys_syscall+0x17d
              506
                1               11
                0              506

So this gave us some cases where we could see that things were off, but the why wasn't clear. Earlier I had been staring at conn_get_ixa() and the fact that things were coming from there. Given that we were still routing to the wrong place, one of the things I was trying to figure out with that above D script was basically where that was coming from and we could see that the moment after we did an ip_attr_connect() in udp_output_lastdst() that we had things wrong. So why was more of a mystery.

One thing that had bothered me was the conn_lock related locking and a thing that seemed weird, but couldn't prove entirely: the ixa is obtained by copying information while the conn_lock is held; however, it's dropped between then and when we acquire it to do the comparison of conn_same_as_last_v6(). When looking at ip_attr_connect() I noticed it used the scope ID from the ixa. So the following D script was the latest hypothesis:


fbt::udp_send:entry
/execname == "faux-mgs"/
{
        this->sin6 = (sin6_t *)args[2]->msg_name;
        self->scope = this->sin6->sin6_scope_id;
}

fbt::udp_send:return
{
        self->scope = 0;
}

fbt::udp_output_lastdst:entry
/self->scope != 0/
{
        @[self->scope, args[4]->ixa_scopeid] = count();
}
BRM42220014 # dtrace -s mcast.d -n 'tick-10s{ printa(@); trunc(@); }'
dtrace: script 'mcast.d' matched 3 probes
dtrace: description 'tick-10s' matched 1 probe
CPU     ID                    FUNCTION:NAME
 53  81701                        :tick-10s
       76       79                2
       76       76                5

 53  81701                        :tick-10s
       76       79                1
       79       76                1
       79       79                1
       76       76                4

 53  81701                        :tick-10s
       79       79                1
       76       79                2
       76       76                8

And this was the proof of what happened and now made all the suspicions clear: there's an inherent assumption that when conn_same_as_last_v6() is called, if it returns true, the ixa should be accurate. However, that's not the case at all and there is a narrow window that can cause that to be wrong as we've seen here because we have a lot of stuff bouncing across this socket. Therefore the thing we need to figure out is how to make sure that the ixa is actually correct for this and otherwise we enter the udp_output_newdest() path.

Specifically:

                if (msg->msg_controllen == 0) {
                        ixa = conn_get_ixa(connp, B_FALSE);
                        if (ixa == NULL) {
                                UDPS_BUMP_MIB(us, udpOutErrors);
                                return (ENOMEM);
                        }
                } else {
                        ixa = NULL;
                }

        ---- Window where the conn get updated but ixa remains wrong

                mutex_enter(&connp->conn_lock);
Actions #1

Updated by Bryan Cantrill 9 days ago

Exploring the window that Robert Mustacchi identified to more explicitly understand the timeline:

#pragma D option quiet
#pragma D option bufpolicy=ring

BEGIN
{
    start = timestamp;
}

syscall::write:entry
/arg0 == 2 && execname == "faux-mgs" &&
    strstr(copyinstr(arg1, arg2), "WARN") != NULL/
{
    exit(0);
}

fbt::udp_send:entry
/execname == "faux-mgs"/
{
    self->udp = 1;
    this->sin6 = (sin6_t *)args[2]->msg_name;

    printf("%d tid=%-3d => scopeid=%d\n", timestamp - start, tid,
        this->sin6->sin6_scope_id);
}

fbt::*udp*:entry
/self->udp/
{
    printf("%d tid=%-3d -> %s\n", timestamp - start, tid, probefunc);
}

fbt::*udp*:return
/self->udp/
{
    printf("%d tid=%-3d <- %s\n", timestamp - start, tid, probefunc);
}

fbt::udp_output_lastdst:entry
/self->udp/
{
    printf("%d tid=%-3d -> udp_output_lastdst ixa=0x%p scopeid=%d\n",
        timestamp - start, tid, arg4, args[4]->ixa_scopeid);
}

fbt::conn_ip_output:entry
/self->udp/
{
    printf("%d tid=%-3d -> %s ixa=0x%p scopeid=%d\n", timestamp - start,
        tid, probefunc, arg1, args[1]->ixa_scopeid);
}

fbt::conn_get_ixa_impl:return
/self->udp/
{
    printf("%d tid=%-3d <- conn_get_ixa_impl ixa=0x%p scopeid=%d\n",
        timestamp - start, tid, arg1, args[1]->ixa_scopeid);
}

conn_same_as_last_v6:entry
/self->udp/
{
    printf("%d tid=%-3d -> conn_same_as_last_v6 scopeid=%d\n",
        timestamp - start, tid, args[0]->conn_lastscopeid);

}

conn_replace_ixa:entry
/self->udp/
{
    printf("%d tid=%-3d -> conn_replace_ixa oldscopeid=%d newscopeid=%d\n",
        timestamp - start, tid, args[0]->conn_ixa->ixa_scopeid,
        args[1]->ixa_scopeid);
}

fbt::udp_send:return
/self->udp/
{
    self->udp = 0;
}

Running this with the reproducer based on faux-mgs:

faux-mgs --log-level=warn --interface sidecar1 --interface psc0 state

Output:

...
7820150278 tid=129 => scopeid=76
7820153795 tid=129 -> udp_send
7820154847 tid=3   => scopeid=79
7820156200 tid=129 <- conn_get_ixa_impl ixa=0xfffffcfa3d3c0240 scopeid=79
7820157703 tid=129 -> conn_same_as_last_v6 scopeid=79
7820158063 tid=3   -> udp_send
7820158855 tid=129 -> udp_output_newdst
7820161069 tid=3   <- conn_get_ixa_impl ixa=0xfffffcfa9b603900 scopeid=76
7820162772 tid=3   -> conn_same_as_last_v6 scopeid=79
7820164906 tid=3   -> udp_output_lastdst
7820165938 tid=3   -> udp_output_lastdst ixa=0xfffffcfa9b603900 scopeid=76
7820170086 tid=3   -> udp_prepend_header_template
7820171559 tid=3   <- udp_prepend_header_template
7820173903 tid=129 -> conn_replace_ixa oldscopeid=76 newscopeid=76
7820175366 tid=129 -> udp_prepend_header_template
7820176488 tid=3   -> conn_ip_output ixa=0xfffffcfa9b603900 scopeid=76
7820177160 tid=129 <- udp_prepend_header_template
7820179264 tid=129 -> conn_ip_output ixa=0xfffffcfa3d3c0240 scopeid=76
7820188541 tid=3   <- udp_output_lastdst
7820189874 tid=3   <- udp_send
7820193571 tid=129 <- udp_output_newdst
7820194973 tid=129 <- udp_send

This very clearly demonstrates the race: thread 3 (sending to scope 79) seemingly matches the last connection at 7820162772 -- but in fact, thread 129 (sending to scope 76) has already called udp_output_newdst at 7820158855. Thread 129 has already taken an important action before it dropped conn_lock: it has set the scope in the ixa to its desired value of 76. Because thread 3 called udp_output_lastdst after thread 129 had called into udp_output_newdst, the decision that thread 3 made is no longer valid -- and the ixa that it has obtained has already copied the incorrect value in conn_get_ixa_impl, sending thread 3 on the path to ruin.

Fortunately, fixing this seems to be straightforward: once in udp_output_newdst, the cached connection state should be invalidated (i.e., by assigning ipv6_all_zeros to the last address), assuring that subsequent threads won't spuriousl match in conn_same_as_last_v6 (sending them to the correct udp_output_newdst rather than udp_output_lastdst).

Note that icmp_output_lastdst and icmp_output_newdst have a similar issue and icmp_output_newdst needs the same fix.

Diffs:

diff --git a/usr/src/uts/common/inet/ip/icmp.c b/usr/src/uts/common/inet/ip/icmp.c
index 57ee0c5585..e86dcbdac3 100644
--- a/usr/src/uts/common/inet/ip/icmp.c
+++ b/usr/src/uts/common/inet/ip/icmp.c
@@ -4424,6 +4424,16 @@ icmp_output_newdst(conn_t *connp, mblk_t *data_mp, sin_t *sin, sin6_t *sin6,
         goto ud_error;
     }

+    /*
+     * Before we modify the ixa at all, invalidate our most recent address
+     * to assure that any subsequent call to conn_same_as_last_v6() will
+     * not indicate a match: any thread that picks up conn_lock after we
+     * drop it (but before we pick it up again and properly set the most
+     * recent address) must not associate the ixa with the (now old) last
+     * address.
+     */
+    connp->conn_v6lastdst = ipv6_all_zeros;
+
     /* In case previous destination was multicast or multirt */
     ip_attr_newdst(ixa);

diff --git a/usr/src/uts/common/inet/udp/udp.c b/usr/src/uts/common/inet/udp/udp.c
index 5d42a69fa2..c3bcd5c802 100644
--- a/usr/src/uts/common/inet/udp/udp.c
+++ b/usr/src/uts/common/inet/udp/udp.c
@@ -3855,6 +3855,16 @@ udp_output_newdst(conn_t *connp, mblk_t *data_mp, sin_t *sin, sin6_t *sin6,
         goto ud_error;
     }

+    /*
+     * Before we modify the ixa at all, invalidate our most recent address
+     * to assure that any subsequent call to conn_same_as_last_v6() will
+     * not indicate a match: any thread that picks up conn_lock after we
+     * drop it (but before we pick it up again and properly set the most
+     * recent address) must not associate the ixa with the (now old) last
+     * address.
+     */
+    connp->conn_v6lastdst = ipv6_all_zeros;
+
     /* In case previous destination was multicast or multirt */
     ip_attr_newdst(ixa);


Actions #2

Updated by Robert Mustacchi 8 days ago

  • Assignee set to Bryan Cantrill
Actions #3

Updated by Electric Monk 4 days ago

  • Gerrit CR set to 3525
Actions #4

Updated by Bryan Cantrill 3 days ago

Verified that this change does indeed prevent code flow from entering the lastdst variants after newdst has already been called. (As it turns out, this bug isn't seen more broadly because it really needs a bunch of the specific conditions that we rely upon with respect to IPv6 scope.)

Actions #5

Updated by Electric Monk 3 days ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 4022e346ec1f26a6be01575e105bf0c465edd476

commit  4022e346ec1f26a6be01575e105bf0c465edd476
Author: Bryan Cantrill <bryan@oxide.computer>
Date:   2024-05-23T19:34:07.000Z

    16544 udp_output_lastdst() can end up with wrong ixa
    Reviewed by: Robert Mustacchi <rm@fingolfin.org>
    Reviewed by: Dan McDonald <danmcd@mnx.io>
    Approved by: Patrick Mooney <pmooney@pfmooney.com>

Actions

Also available in: Atom PDF