Bug #4247
openReduce thread migrations caused by pg_cmt_can_migrate()
100%
Description
Based on extensive analysis by Brendan Gregg we found that we were often doing many more steals to other CPUs that would cause a performance hit than we expected. One of these cases is a bug in pg_cmt_can_migrate(). Here's a bit more of the background:
In usr/src/uts/common/disp/cmt.c:
pg_cmt_can_migrate(cpu_t *from, cpu_t *to) { if (from->cpu_physid->cpu_cacheid == to->cpu_physid->cpu_cacheid) return (1); return (0); }
This is testing a cache that is shared between all CPUs:
[root@headnode (bh1-kvm4:0) /zones/brendan]# pginfo PG RELATIONSHIP CPUs 0 System 0-15 3 Socket 0 2 4 6 8 10 12 14 4 CPU_PM_Active_Power_Domain 0 2 4 6 8 10 12 14 2 Cache 0 2 4 6 8 10 12 14 1 Integer_Pipeline 0 8 9 Integer_Pipeline 2 10 11 Integer_Pipeline 4 12 13 Integer_Pipeline 6 14 7 Socket 1 3 5 7 9 11 13 15 8 CPU_PM_Active_Power_Domain 1 3 5 7 9 11 13 15 6 Cache 1 3 5 7 9 11 13 15 5 Integer_Pipeline 1 9 10 Integer_Pipeline 3 11 12 Integer_Pipeline 5 13 14 Integer_Pipeline 7 15
And was a contributing reason for threads skipping between all CPUs on one socket. This function, and its usage in disp_getbest(), could be cleverer.
--
Prior to this change, a sample of return values from this function looked like:
0 292 1 1308
Implying that between arbitrary pairs of CPUs (logical processors in Intel speak), penalty-free migration was possible about 82% of the time. That's clearly false since migration is penalty-free only if the L1 cache is shared, which applies only to the two threads in a core.
After the change, we now see the much more reasonable:
1 36 0 179360
Related issues
Updated by Josef Sipek about 7 years ago
- Related to Bug #5992: pg info lacks enough detail for dispatcher to decide what to do added