illumos gate

Based on extensive analysis by Brendan Gregg we found that we were often doing many more steals to other CPUs that would cause a performance hit than we expected. One of these cases is a bug in pg_cmt_can_migrate(). Here's a bit more of the background:

In usr/src/uts/common/disp/cmt.c:

pg_cmt_can_migrate(cpu_t *from, cpu_t *to)
{
        if (from->cpu_physid->cpu_cacheid ==
            to->cpu_physid->cpu_cacheid)
                return (1);
        return (0);
}

This is testing a cache that is shared between all CPUs:

[root@headnode (bh1-kvm4:0) /zones/brendan]# pginfo
PG  RELATIONSHIP                   CPUs
0   System                         0-15
3    Socket                        0 2 4 6 8 10 12 14
4     CPU_PM_Active_Power_Domain   0 2 4 6 8 10 12 14
2      Cache                       0 2 4 6 8 10 12 14
1       Integer_Pipeline           0 8
9       Integer_Pipeline           2 10
11      Integer_Pipeline           4 12
13      Integer_Pipeline           6 14
7    Socket                        1 3 5 7 9 11 13 15
8     CPU_PM_Active_Power_Domain   1 3 5 7 9 11 13 15
6      Cache                       1 3 5 7 9 11 13 15
5       Integer_Pipeline           1 9
10      Integer_Pipeline           3 11
12      Integer_Pipeline           5 13
14      Integer_Pipeline           7 15

And was a contributing reason for threads skipping between all CPUs on one socket. This function, and its usage in disp_getbest(), could be cleverer.

--

Prior to this change, a sample of return values from this function looked like:

                0              292
                1             1308

Implying that between arbitrary pairs of CPUs (logical processors in Intel speak), penalty-free migration was possible about 82% of the time. That's clearly false since migration is penalty-free only if the L1 cache is shared, which applies only to the two threads in a core.

After the change, we now see the much more reasonable:

                1               36
                0           179360