In my new server (NUMA) have one core reaching 40% and other cores 2%
Tried to solve using default intel default kernel affinity one txrx queue by core or Installed irqbalance and doesn't seen to work.
So I uninstall irqbalance and use a script to allocate interrupts for all cores.
Looks like maybe I can have more interrupts i don't know. But at least all cores are working balanced and now I see peaks at most of 15%.
I would like to share the script, hope be useful
for irq in `grep ens1f0 /proc/interrupts | cut -d: -f1`; do echo ffffff > /proc/irq/$irq/smp_affinity; done
for irq in `grep ens1f1 /proc/interrupts | cut -d: -f1`; do echo ffffff > /proc/irq/$irq/smp_affinity; done
for i in `seq 0 7`; do echo fffffff > /sys/class/net/ens1f0/queues/rx-$i/rps_cpus ; done
for i in `seq 0 7`; do echo fffffff > /sys/class/net/ens1f1/queues/rx-$i/rps_cpus ; done
Using Debian Stretch.
Balancing CPUs intel 10 GB
Re: Balancing CPUs intel 10 GB
show
Code: Select all
cat /proc/interrupts
Re: Balancing CPUs intel 10 GB
Code: Select all
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
0: 17 0 0 0 0 0 0 0 IO-APIC 2-edge timer
8: 0 1 0 0 0 0 0 0 IO-APIC 8-edge rtc0
9: 0 1 0 0 0 0 0 0 IO-APIC 9-fasteoi acpi
18: 0 0 0 59 0 0 0 0 IO-APIC 18-fasteoi i801_smbus, ehci_hcd:usb1, ehci_hcd:usb3
19: 0 0 0 0 0 0 0 407 IO-APIC 19-fasteoi uhci_hcd:usb2, hpilo
25: 0 0 0 0 0 0 0 0 PCI-MSI 16384-edge PCIe PME
26: 0 0 0 0 0 0 0 0 PCI-MSI 18432-edge PCIe PME
28: 0 0 0 0 0 0 0 0 PCI-MSI 32768-edge PCIe PME
29: 0 0 0 0 0 0 0 0 PCI-MSI 34816-edge PCIe PME
30: 0 0 0 0 0 0 0 0 PCI-MSI 36864-edge PCIe PME
31: 0 0 0 0 0 0 0 0 PCI-MSI 38912-edge PCIe PME
33: 0 0 0 0 0 0 0 0 PCI-MSI 49152-edge PCIe PME
34: 0 0 0 0 0 0 0 0 PCI-MSI 51200-edge PCIe PME
35: 0 0 0 0 0 0 0 0 PCI-MSI 53248-edge PCIe PME
36: 0 0 0 0 0 0 0 0 PCI-MSI 55296-edge PCIe PME
37: 0 0 0 0 0 0 0 0 PCI-MSI 462848-edge PCIe PME
38: 0 0 0 0 0 0 0 0 PCI-MSI 466944-edge PCIe PME
39: 0 0 0 0 0 0 0 0 PCI-MSI 471040-edge PCIe PME
40: 0 0 0 0 0 0 0 0 PCI-MSI 473088-edge PCIe PME
41: 31202 0 1 0 0 0 0 0 PCI-MSI 1572864-edge hpsa0-msix0
42: 0 19899 1 0 0 0 0 0 PCI-MSI 1572865-edge hpsa0-msix1
43: 0 0 21648 1 0 0 0 0 PCI-MSI 1572866-edge hpsa0-msix2
44: 0 0 0 12293 0 0 0 0 PCI-MSI 1572867-edge hpsa0-msix3
45: 0 0 0 0 24890 0 0 0 PCI-MSI 1572868-edge hpsa0-msix4
46: 0 0 0 0 1 32690 0 0 PCI-MSI 1572869-edge hpsa0-msix5
47: 0 0 0 0 0 1 35089 0 PCI-MSI 1572870-edge hpsa0-msix6
48: 0 0 0 0 0 1 0 53071 PCI-MSI 1572871-edge hpsa0-msix7
49: 0 0 0 0 0 0 25 0 PCI-MSI 327680-edge xhci_hcd
50: 0 0 0 0 0 0 687261 0 PCI-MSI 512000-edge ahci[0000:00:1f.2]
51: 19 2185464873 0 0 0 0 0 0 PCI-MSI 4194304-edge ens1f0-TxRx-0
52: 0 19 2193765831 0 0 0 0 0 PCI-MSI 4194305-edge ens1f0-TxRx-1
53: 0 0 2201278684 0 0 0 0 0 PCI-MSI 4194306-edge ens1f0-TxRx-2
54: 0 0 0 2192157378 0 0 0 0 PCI-MSI 4194307-edge ens1f0-TxRx-3
55: 0 0 0 2193464481 19 0 0 0 PCI-MSI 4194308-edge ens1f0-TxRx-4
56: 0 0 0 0 2233232382 19 0 0 PCI-MSI 4194309-edge ens1f0-TxRx-5
57: 0 0 0 0 2262997278 0 19 0 PCI-MSI 4194310-edge ens1f0-TxRx-6
58: 0 0 0 0 0 2215993925 0 19 PCI-MSI 4194311-edge ens1f0-TxRx-7
59: 0 0 0 0 0 51176 0 0 PCI-MSI 4194312-edge ens1f0
60: 16 0 0 0 0 0 1627941912 0 PCI-MSI 4196352-edge ens1f1-TxRx-0
61: 0 16 0 0 0 0 890242770 0 PCI-MSI 4196353-edge ens1f1-TxRx-1
62: 0 0 16 0 0 0 0 856293999 PCI-MSI 4196354-edge ens1f1-TxRx-2
63: 0 884549281 0 16 0 0 0 0 PCI-MSI 4196355-edge ens1f1-TxRx-3
64: 0 0 853715405 0 16 0 0 0 PCI-MSI 4196356-edge ens1f1-TxRx-4
65: 0 0 808481297 0 0 16 0 0 PCI-MSI 4196357-edge ens1f1-TxRx-5
66: 0 0 0 862327096 0 0 16 0 PCI-MSI 4196358-edge ens1f1-TxRx-6
67: 0 0 0 970827965 0 0 0 16 PCI-MSI 4196359-edge ens1f1-TxRx-7
68: 0 0 0 0 12 0 0 0 PCI-MSI 4196360-edge ens1f1
70: 0 0 0 0 0 2 0 0 PCI-MSI 65536-edge ioat-msix
72: 0 0 0 0 0 0 2 0 PCI-MSI 67584-edge ioat-msix
73: 0 0 0 0 0 0 2 0 PCI-MSI 69632-edge ioat-msix
74: 0 0 0 0 0 0 0 2 PCI-MSI 71680-edge ioat-msix
75: 0 2 0 0 0 0 0 0 PCI-MSI 73728-edge ioat-msix
76: 0 0 2 0 0 0 0 0 PCI-MSI 75776-edge ioat-msix
77: 0 0 2 0 0 0 0 0 PCI-MSI 77824-edge ioat-msix
78: 0 0 0 2 0 0 0 0 PCI-MSI 79872-edge ioat-msix
79: 0 0 0 1 0 0 0 0 PCI-MSI 1054720-edge eno4-tx-0
80: 0 0 0 0 1 0 0 0 PCI-MSI 1054721-edge eno4-rx-1
81: 0 0 0 0 1 0 0 0 PCI-MSI 1054722-edge eno4-rx-2
82: 0 0 0 0 0 1 0 0 PCI-MSI 1054723-edge eno4-rx-3
83: 0 0 0 0 0 1 0 0 PCI-MSI 1054724-edge eno4-rx-4
84: 0 0 0 0 0 0 10982251 0 PCI-MSI 1048576-edge eno1-tx-0
85: 0 0 0 0 0 0 3415978 0 PCI-MSI 1048577-edge eno1-rx-1
86: 0 0 0 0 0 0 0 2494484 PCI-MSI 1048578-edge eno1-rx-2
87: 0 3135707 0 0 0 0 0 0 PCI-MSI 1048579-edge eno1-rx-3
88: 0 0 2266534 0 0 0 0 0 PCI-MSI 1048580-edge eno1-rx-4
89: 0 0 1 0 0 0 0 0 PCI-MSI 1050624-edge eno2-tx-0
90: 0 0 0 1 0 0 0 0 PCI-MSI 1050625-edge eno2-rx-1
91: 0 0 0 1 0 0 0 0 PCI-MSI 1050626-edge eno2-rx-2
92: 0 0 0 0 1 0 0 0 PCI-MSI 1050627-edge eno2-rx-3
93: 0 0 0 0 1 0 0 0 PCI-MSI 1050628-edge eno2-rx-4
NMI: 265001 326956 285943 343646 270669 256472 252049 205589 Non-maskable interrupts
LOC: 4203682264 4289977452 4225735740 4288788873 4255323177 4183586837 4129026023 4206340721 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 265001 326956 285943 343646 270669 256472 252049 205589 Performance monitoring interrupts
IWI: 20071623 16883400 38553433 17531562 36602854 32277089 43646559 47224834 IRQ work interrupts
RTR: 7 0 0 0 0 0 0 0 APIC ICR read retries
RES: 10765529 4082308 7799076 1983476 8795262 9584047 8329673 9204928 Rescheduling interrupts
CAL: 2752579806 2268719961 1725290636 1688816829 1708263433 2220634928 4247569113 2885008141 Function call interrupts
TLB: 3909 4882 4880 5129 4712 4290 4662 4170 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 2229 2229 2229 2229 2229 2229 2229 2229 Machine check polls
ERR: 0
MIS: 0
PIN: 0 0 0 0 0 0 0 0 Posted-interrupt notification event
PIW: 0 0 0 0 0 0 0 0 Posted-interrupt wakeup event
Re: Balancing CPUs intel 10 GB
don't work for me
using debian stretch
This is exmaple of a process after running script
would you help me ?
in cpu usage, showing this (HIGH CPU USAGE):
using debian stretch
Code: Select all
/sys/class/net/eth0.2001/queues# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 9 0 0 0 IO-APIC 2-edge timer
1: 10 0 0 0 IO-APIC 1-edge i8042
8: 1 0 0 0 IO-APIC 8-edge rtc0
9: 0 0 0 0 IO-APIC 9-fasteoi acpi
12: 16 0 0 0 IO-APIC 12-edge i8042
14: 0 0 0 0 IO-APIC 14-edge ata_piix
15: 0 0 0 0 IO-APIC 15-edge ata_piix
16: 1 0 0 0 IO-APIC 16-fasteoi vmwgfx
18: 64 0 0 0 IO-APIC 18-fasteoi uhci_hcd:usb1
19: 0 0 0 0 IO-APIC 19-fasteoi ehci_hcd:usb2
24: 0 0 0 0 PCI-MSI 344064-edge PCIe PME, pciehp
25: 0 0 0 0 PCI-MSI 346112-edge PCIe PME, pciehp
26: 0 0 0 0 PCI-MSI 348160-edge PCIe PME, pciehp
27: 0 0 0 0 PCI-MSI 350208-edge PCIe PME, pciehp
28: 0 0 0 0 PCI-MSI 352256-edge PCIe PME, pciehp
29: 0 0 0 0 PCI-MSI 354304-edge PCIe PME, pciehp
30: 0 0 0 0 PCI-MSI 356352-edge PCIe PME, pciehp
31: 0 0 0 0 PCI-MSI 358400-edge PCIe PME, pciehp
32: 0 0 0 0 PCI-MSI 360448-edge PCIe PME, pciehp
33: 0 0 0 0 PCI-MSI 362496-edge PCIe PME, pciehp
34: 0 0 0 0 PCI-MSI 364544-edge PCIe PME, pciehp
35: 0 0 0 0 PCI-MSI 366592-edge PCIe PME, pciehp
36: 0 0 0 0 PCI-MSI 368640-edge PCIe PME, pciehp
37: 0 0 0 0 PCI-MSI 370688-edge PCIe PME, pciehp
38: 0 0 0 0 PCI-MSI 372736-edge PCIe PME, pciehp
39: 0 0 0 0 PCI-MSI 374784-edge PCIe PME, pciehp
40: 0 0 0 0 PCI-MSI 376832-edge PCIe PME, pciehp
41: 0 0 0 0 PCI-MSI 378880-edge PCIe PME, pciehp
42: 0 0 0 0 PCI-MSI 380928-edge PCIe PME, pciehp
43: 0 0 0 0 PCI-MSI 382976-edge PCIe PME, pciehp
44: 0 0 0 0 PCI-MSI 385024-edge PCIe PME, pciehp
45: 0 0 0 0 PCI-MSI 387072-edge PCIe PME, pciehp
46: 0 0 0 0 PCI-MSI 389120-edge PCIe PME, pciehp
47: 0 0 0 0 PCI-MSI 391168-edge PCIe PME, pciehp
48: 0 0 0 0 PCI-MSI 393216-edge PCIe PME, pciehp
49: 0 0 0 0 PCI-MSI 395264-edge PCIe PME, pciehp
50: 0 0 0 0 PCI-MSI 397312-edge PCIe PME, pciehp
51: 0 0 0 0 PCI-MSI 399360-edge PCIe PME, pciehp
52: 0 0 0 0 PCI-MSI 401408-edge PCIe PME, pciehp
53: 0 0 0 0 PCI-MSI 403456-edge PCIe PME, pciehp
54: 0 0 0 0 PCI-MSI 405504-edge PCIe PME, pciehp
55: 0 0 0 0 PCI-MSI 407552-edge PCIe PME, pciehp
56: 19567526 0 0 0 PCI-MSI 5767168-edge eth0-rxtx-0
57: 8335140 0 0 0 PCI-MSI 5767169-edge eth0-rxtx-1
58: 8442909 0 0 0 PCI-MSI 5767170-edge eth0-rxtx-2
59: 8356847 0 0 0 PCI-MSI 5767171-edge eth0-rxtx-3
60: 0 0 0 0 PCI-MSI 5767172-edge eth0-event-4
66: 12747717 0 0 0 PCI-MSI 14155776-edge eth4-rxtx-0
67: 13007137 0 0 0 PCI-MSI 14155777-edge eth4-rxtx-1
68: 12870746 0 0 0 PCI-MSI 14155778-edge eth4-rxtx-2
69: 13032285 0 0 0 PCI-MSI 14155779-edge eth4-rxtx-3
70: 0 0 0 0 PCI-MSI 14155780-edge eth4-event-4
71: 22268 0 0 0 PCI-MSI 1572864-edge vmw_pvscsi
72: 2386 0 0 0 PCI-MSI 1097728-edge ahci[0000:02:03.0]
73: 0 0 0 0 PCI-MSI 129024-edge vmw_vmci
74: 0 0 0 0 PCI-MSI 129025-edge vmw_vmci
NMI: 0 0 0 0 Non-maskable interrupts
LOC: 23950209 2027132 1761533 1808457 Local timer interrupts
SPU: 0 0 0 0 Spurious interrupts
PMI: 0 0 0 0 Performance monitoring interrupts
IWI: 2 1 0 0 IRQ work interrupts
RTR: 0 0 0 0 APIC ICR read retries
RES: 478799 746756 725165 730054 Rescheduling interrupts
CAL: 46474 1965965 1949109 1935292 Function call interrupts
TLB: 37895 54853 52594 52308 TLB shootdowns
TRM: 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 Threshold APIC interrupts
DFR: 0 0 0 0 Deferred Error APIC interrupts
MCE: 0 0 0 0 Machine check exceptions
MCP: 17 17 17 17 Machine check polls
ERR: 0
MIS: 0
PIN: 0 0 0 0 Posted-interrupt notification event
PIW: 0 0 0 0 Posted-interrupt wakeup event
Code: Select all
root@bras-1:# cat /proc/irq/56/smp_affinity
f
root@bras-1:#
in cpu usage, showing this (HIGH CPU USAGE):
Code: Select all
top - 06:40:25 up 1:22, 1 user, load average: 0.62, 0.54, 0.55
Tasks: 166 total, 2 running, 164 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1.3 us, 2.5 sy, 0.0 ni, 40.5 id, 0.0 wa, 0.0 hi, 55.7 si, 0.0 st
%Cpu1 : 2.0 us, 6.9 sy, 0.0 ni, 51.5 id, 0.0 wa, 0.0 hi, 39.6 si, 0.0 st
%Cpu2 : 2.2 us, 4.5 sy, 0.0 ni, 68.5 id, 0.0 wa, 0.0 hi, 24.7 si, 0.0 st
%Cpu3 : 2.2 us, 7.6 sy, 0.0 ni, 55.4 id, 0.0 wa, 0.0 hi, 34.8 si, 0.0 st
KiB Mem : 2052176 total, 1530168 free, 278116 used, 243892 buff/cache
KiB Swap: 2095100 total, 2095100 free, 0 used. 1604160 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3 root 20 0 0 0 0 S 7.7 0.0 5:20.51 ksoftirqd/0
16 root 20 0 0 0 0 S 5.8 0.0 0:23.88 ksoftirqd/1
28 root 20 0 0 0 0 R 4.8 0.0 0:24.30 ksoftirqd/3
863 root 20 0 346004 21656 4580 S 4.8 1.1 4:45.55 accel-pppd
3607 www-data 20 0 262972 12440 6364 S 2.9 0.6 0:05.11 apache2
4709 root 20 0 44924 3740 3068 R 2.9 0.2 0:00.16 top
22 root 20 0 0 0 0 S 1.9 0.0 0:23.35 ksoftirqd/2
23053 www-data 20 0 262748 12012 5992 S 1.9 0.6 0:02.22 apache2
25273 www-data 20 0 262748 12012 5992 S 1.9 0.6 0:01.46 apache2
7 root 20 0 0 0 0 S 1.0 0.0 0:15.64 rcu_sched
2009 root 20 0 0 0 0 S 1.0 0.0 0:00.01 kworker/1:2
3572 www-data 20 0 262972 12320 6244 S 1.0 0.6 0:05.54 apache2
4478 www-data 20 0 262976 12440 6360 S 1.0 0.6 0:32.96 apache2
28704 www-data 20 0 262748 12012 5992 S 1.0 0.6 0:01.08 apache2
1 root 20 0 206984 9164 5232 S 0.0 0.4 0:10.63 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root rt 0 0 0 0 S 0.0 0.0 0:01.76 migration/0
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:00.01 watchdog/0
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0
Re: Balancing CPUs intel 10 GB
try bind static irq to cpu.
Re: Balancing CPUs intel 10 GB
or irqblance service is running