Balancing CPUs intel 10 GB

Any topics
Post Reply
hugleo
Posts: 59
Joined: 13 Apr 2016, 14:28

Balancing CPUs intel 10 GB

Post by hugleo »

In my new server (NUMA) have one core reaching 40% and other cores 2%
Tried to solve using default intel default kernel affinity one txrx queue by core or Installed irqbalance and doesn't seen to work.

So I uninstall irqbalance and use a script to allocate interrupts for all cores.

Looks like maybe I can have more interrupts i don't know. But at least all cores are working balanced and now I see peaks at most of 15%.

I would like to share the script, hope be useful ;-)

for irq in `grep ens1f0 /proc/interrupts | cut -d: -f1`; do echo ffffff > /proc/irq/$irq/smp_affinity; done

for irq in `grep ens1f1 /proc/interrupts | cut -d: -f1`; do echo ffffff > /proc/irq/$irq/smp_affinity; done

for i in `seq 0 7`; do echo fffffff > /sys/class/net/ens1f0/queues/rx-$i/rps_cpus ; done

for i in `seq 0 7`; do echo fffffff > /sys/class/net/ens1f1/queues/rx-$i/rps_cpus ; done


Using Debian Stretch.
dimka88
Posts: 866
Joined: 13 Oct 2014, 05:51
Contact:

Re: Balancing CPUs intel 10 GB

Post by dimka88 »

show

Code: Select all

cat /proc/interrupts
hugleo
Posts: 59
Joined: 13 Apr 2016, 14:28

Re: Balancing CPUs intel 10 GB

Post by hugleo »

Code: Select all


           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  0:         17          0          0          0          0          0          0          0   IO-APIC   2-edge      timer
  8:          0          1          0          0          0          0          0          0   IO-APIC   8-edge      rtc0
  9:          0          1          0          0          0          0          0          0   IO-APIC   9-fasteoi   acpi
 18:          0          0          0         59          0          0          0          0   IO-APIC  18-fasteoi   i801_smbus, ehci_hcd:usb1, ehci_hcd:usb3
 19:          0          0          0          0          0          0          0        407   IO-APIC  19-fasteoi   uhci_hcd:usb2, hpilo
 25:          0          0          0          0          0          0          0          0   PCI-MSI 16384-edge      PCIe PME
 26:          0          0          0          0          0          0          0          0   PCI-MSI 18432-edge      PCIe PME
 28:          0          0          0          0          0          0          0          0   PCI-MSI 32768-edge      PCIe PME
 29:          0          0          0          0          0          0          0          0   PCI-MSI 34816-edge      PCIe PME
 30:          0          0          0          0          0          0          0          0   PCI-MSI 36864-edge      PCIe PME
 31:          0          0          0          0          0          0          0          0   PCI-MSI 38912-edge      PCIe PME
 33:          0          0          0          0          0          0          0          0   PCI-MSI 49152-edge      PCIe PME
 34:          0          0          0          0          0          0          0          0   PCI-MSI 51200-edge      PCIe PME
 35:          0          0          0          0          0          0          0          0   PCI-MSI 53248-edge      PCIe PME
 36:          0          0          0          0          0          0          0          0   PCI-MSI 55296-edge      PCIe PME
 37:          0          0          0          0          0          0          0          0   PCI-MSI 462848-edge      PCIe PME
 38:          0          0          0          0          0          0          0          0   PCI-MSI 466944-edge      PCIe PME
 39:          0          0          0          0          0          0          0          0   PCI-MSI 471040-edge      PCIe PME
 40:          0          0          0          0          0          0          0          0   PCI-MSI 473088-edge      PCIe PME
 41:      31202          0          1          0          0          0          0          0   PCI-MSI 1572864-edge      hpsa0-msix0
 42:          0      19899          1          0          0          0          0          0   PCI-MSI 1572865-edge      hpsa0-msix1
 43:          0          0      21648          1          0          0          0          0   PCI-MSI 1572866-edge      hpsa0-msix2
 44:          0          0          0      12293          0          0          0          0   PCI-MSI 1572867-edge      hpsa0-msix3
 45:          0          0          0          0      24890          0          0          0   PCI-MSI 1572868-edge      hpsa0-msix4
 46:          0          0          0          0          1      32690          0          0   PCI-MSI 1572869-edge      hpsa0-msix5
 47:          0          0          0          0          0          1      35089          0   PCI-MSI 1572870-edge      hpsa0-msix6
 48:          0          0          0          0          0          1          0      53071   PCI-MSI 1572871-edge      hpsa0-msix7
 49:          0          0          0          0          0          0         25          0   PCI-MSI 327680-edge      xhci_hcd
 50:          0          0          0          0          0          0     687261          0   PCI-MSI 512000-edge      ahci[0000:00:1f.2]
 51:         19 2185464873          0          0          0          0          0          0   PCI-MSI 4194304-edge      ens1f0-TxRx-0
 52:          0         19 2193765831          0          0          0          0          0   PCI-MSI 4194305-edge      ens1f0-TxRx-1
 53:          0          0 2201278684          0          0          0          0          0   PCI-MSI 4194306-edge      ens1f0-TxRx-2
 54:          0          0          0 2192157378          0          0          0          0   PCI-MSI 4194307-edge      ens1f0-TxRx-3
 55:          0          0          0 2193464481         19          0          0          0   PCI-MSI 4194308-edge      ens1f0-TxRx-4
 56:          0          0          0          0 2233232382         19          0          0   PCI-MSI 4194309-edge      ens1f0-TxRx-5
 57:          0          0          0          0 2262997278          0         19          0   PCI-MSI 4194310-edge      ens1f0-TxRx-6
 58:          0          0          0          0          0 2215993925          0         19   PCI-MSI 4194311-edge      ens1f0-TxRx-7
 59:          0          0          0          0          0      51176          0          0   PCI-MSI 4194312-edge      ens1f0
 60:         16          0          0          0          0          0 1627941912          0   PCI-MSI 4196352-edge      ens1f1-TxRx-0
 61:          0         16          0          0          0          0  890242770          0   PCI-MSI 4196353-edge      ens1f1-TxRx-1
 62:          0          0         16          0          0          0          0  856293999   PCI-MSI 4196354-edge      ens1f1-TxRx-2
 63:          0  884549281          0         16          0          0          0          0   PCI-MSI 4196355-edge      ens1f1-TxRx-3
 64:          0          0  853715405          0         16          0          0          0   PCI-MSI 4196356-edge      ens1f1-TxRx-4
 65:          0          0  808481297          0          0         16          0          0   PCI-MSI 4196357-edge      ens1f1-TxRx-5
 66:          0          0          0  862327096          0          0         16          0   PCI-MSI 4196358-edge      ens1f1-TxRx-6
 67:          0          0          0  970827965          0          0          0         16   PCI-MSI 4196359-edge      ens1f1-TxRx-7
 68:          0          0          0          0         12          0          0          0   PCI-MSI 4196360-edge      ens1f1
 70:          0          0          0          0          0          2          0          0   PCI-MSI 65536-edge      ioat-msix
 72:          0          0          0          0          0          0          2          0   PCI-MSI 67584-edge      ioat-msix
 73:          0          0          0          0          0          0          2          0   PCI-MSI 69632-edge      ioat-msix
 74:          0          0          0          0          0          0          0          2   PCI-MSI 71680-edge      ioat-msix
 75:          0          2          0          0          0          0          0          0   PCI-MSI 73728-edge      ioat-msix
 76:          0          0          2          0          0          0          0          0   PCI-MSI 75776-edge      ioat-msix
 77:          0          0          2          0          0          0          0          0   PCI-MSI 77824-edge      ioat-msix
 78:          0          0          0          2          0          0          0          0   PCI-MSI 79872-edge      ioat-msix
 79:          0          0          0          1          0          0          0          0   PCI-MSI 1054720-edge      eno4-tx-0
 80:          0          0          0          0          1          0          0          0   PCI-MSI 1054721-edge      eno4-rx-1
 81:          0          0          0          0          1          0          0          0   PCI-MSI 1054722-edge      eno4-rx-2
 82:          0          0          0          0          0          1          0          0   PCI-MSI 1054723-edge      eno4-rx-3
 83:          0          0          0          0          0          1          0          0   PCI-MSI 1054724-edge      eno4-rx-4
 84:          0          0          0          0          0          0   10982251          0   PCI-MSI 1048576-edge      eno1-tx-0
 85:          0          0          0          0          0          0    3415978          0   PCI-MSI 1048577-edge      eno1-rx-1
 86:          0          0          0          0          0          0          0    2494484   PCI-MSI 1048578-edge      eno1-rx-2
 87:          0    3135707          0          0          0          0          0          0   PCI-MSI 1048579-edge      eno1-rx-3
 88:          0          0    2266534          0          0          0          0          0   PCI-MSI 1048580-edge      eno1-rx-4
 89:          0          0          1          0          0          0          0          0   PCI-MSI 1050624-edge      eno2-tx-0
 90:          0          0          0          1          0          0          0          0   PCI-MSI 1050625-edge      eno2-rx-1
 91:          0          0          0          1          0          0          0          0   PCI-MSI 1050626-edge      eno2-rx-2
 92:          0          0          0          0          1          0          0          0   PCI-MSI 1050627-edge      eno2-rx-3
 93:          0          0          0          0          1          0          0          0   PCI-MSI 1050628-edge      eno2-rx-4
NMI:     265001     326956     285943     343646     270669     256472     252049     205589   Non-maskable interrupts
LOC: 4203682264 4289977452 4225735740 4288788873 4255323177 4183586837 4129026023 4206340721   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:     265001     326956     285943     343646     270669     256472     252049     205589   Performance monitoring interrupts
IWI:   20071623   16883400   38553433   17531562   36602854   32277089   43646559   47224834   IRQ work interrupts
RTR:          7          0          0          0          0          0          0          0   APIC ICR read retries
RES:   10765529    4082308    7799076    1983476    8795262    9584047    8329673    9204928   Rescheduling interrupts
CAL: 2752579806 2268719961 1725290636 1688816829 1708263433 2220634928 4247569113 2885008141   Function call interrupts
TLB:       3909       4882       4880       5129       4712       4290       4662       4170   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:       2229       2229       2229       2229       2229       2229       2229       2229   Machine check polls
ERR:          0
MIS:          0
PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event



catweb
Posts: 9
Joined: 07 Mar 2018, 14:35

Re: Balancing CPUs intel 10 GB

Post by catweb »

don't work for me :(

using debian stretch

Code: Select all

/sys/class/net/eth0.2001/queues# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  0:          9          0          0          0   IO-APIC   2-edge      timer
  1:         10          0          0          0   IO-APIC   1-edge      i8042
  8:          1          0          0          0   IO-APIC   8-edge      rtc0
  9:          0          0          0          0   IO-APIC   9-fasteoi   acpi
 12:         16          0          0          0   IO-APIC  12-edge      i8042
 14:          0          0          0          0   IO-APIC  14-edge      ata_piix
 15:          0          0          0          0   IO-APIC  15-edge      ata_piix
 16:          1          0          0          0   IO-APIC  16-fasteoi   vmwgfx
 18:         64          0          0          0   IO-APIC  18-fasteoi   uhci_hcd:usb1
 19:          0          0          0          0   IO-APIC  19-fasteoi   ehci_hcd:usb2
 24:          0          0          0          0   PCI-MSI 344064-edge      PCIe PME, pciehp
 25:          0          0          0          0   PCI-MSI 346112-edge      PCIe PME, pciehp
 26:          0          0          0          0   PCI-MSI 348160-edge      PCIe PME, pciehp
 27:          0          0          0          0   PCI-MSI 350208-edge      PCIe PME, pciehp
 28:          0          0          0          0   PCI-MSI 352256-edge      PCIe PME, pciehp
 29:          0          0          0          0   PCI-MSI 354304-edge      PCIe PME, pciehp
 30:          0          0          0          0   PCI-MSI 356352-edge      PCIe PME, pciehp
 31:          0          0          0          0   PCI-MSI 358400-edge      PCIe PME, pciehp
 32:          0          0          0          0   PCI-MSI 360448-edge      PCIe PME, pciehp
 33:          0          0          0          0   PCI-MSI 362496-edge      PCIe PME, pciehp
 34:          0          0          0          0   PCI-MSI 364544-edge      PCIe PME, pciehp
 35:          0          0          0          0   PCI-MSI 366592-edge      PCIe PME, pciehp
 36:          0          0          0          0   PCI-MSI 368640-edge      PCIe PME, pciehp
 37:          0          0          0          0   PCI-MSI 370688-edge      PCIe PME, pciehp
 38:          0          0          0          0   PCI-MSI 372736-edge      PCIe PME, pciehp
 39:          0          0          0          0   PCI-MSI 374784-edge      PCIe PME, pciehp
 40:          0          0          0          0   PCI-MSI 376832-edge      PCIe PME, pciehp
 41:          0          0          0          0   PCI-MSI 378880-edge      PCIe PME, pciehp
 42:          0          0          0          0   PCI-MSI 380928-edge      PCIe PME, pciehp
 43:          0          0          0          0   PCI-MSI 382976-edge      PCIe PME, pciehp
 44:          0          0          0          0   PCI-MSI 385024-edge      PCIe PME, pciehp
 45:          0          0          0          0   PCI-MSI 387072-edge      PCIe PME, pciehp
 46:          0          0          0          0   PCI-MSI 389120-edge      PCIe PME, pciehp
 47:          0          0          0          0   PCI-MSI 391168-edge      PCIe PME, pciehp
 48:          0          0          0          0   PCI-MSI 393216-edge      PCIe PME, pciehp
 49:          0          0          0          0   PCI-MSI 395264-edge      PCIe PME, pciehp
 50:          0          0          0          0   PCI-MSI 397312-edge      PCIe PME, pciehp
 51:          0          0          0          0   PCI-MSI 399360-edge      PCIe PME, pciehp
 52:          0          0          0          0   PCI-MSI 401408-edge      PCIe PME, pciehp
 53:          0          0          0          0   PCI-MSI 403456-edge      PCIe PME, pciehp
 54:          0          0          0          0   PCI-MSI 405504-edge      PCIe PME, pciehp
 55:          0          0          0          0   PCI-MSI 407552-edge      PCIe PME, pciehp
 56:   19567526          0          0          0   PCI-MSI 5767168-edge      eth0-rxtx-0
 57:    8335140          0          0          0   PCI-MSI 5767169-edge      eth0-rxtx-1
 58:    8442909          0          0          0   PCI-MSI 5767170-edge      eth0-rxtx-2
 59:    8356847          0          0          0   PCI-MSI 5767171-edge      eth0-rxtx-3
 60:          0          0          0          0   PCI-MSI 5767172-edge      eth0-event-4
 66:   12747717          0          0          0   PCI-MSI 14155776-edge      eth4-rxtx-0
 67:   13007137          0          0          0   PCI-MSI 14155777-edge      eth4-rxtx-1
 68:   12870746          0          0          0   PCI-MSI 14155778-edge      eth4-rxtx-2
 69:   13032285          0          0          0   PCI-MSI 14155779-edge      eth4-rxtx-3
 70:          0          0          0          0   PCI-MSI 14155780-edge      eth4-event-4
 71:      22268          0          0          0   PCI-MSI 1572864-edge      vmw_pvscsi
 72:       2386          0          0          0   PCI-MSI 1097728-edge      ahci[0000:02:03.0]
 73:          0          0          0          0   PCI-MSI 129024-edge      vmw_vmci
 74:          0          0          0          0   PCI-MSI 129025-edge      vmw_vmci
NMI:          0          0          0          0   Non-maskable interrupts
LOC:   23950209    2027132    1761533    1808457   Local timer interrupts
SPU:          0          0          0          0   Spurious interrupts
PMI:          0          0          0          0   Performance monitoring interrupts
IWI:          2          1          0          0   IRQ work interrupts
RTR:          0          0          0          0   APIC ICR read retries
RES:     478799     746756     725165     730054   Rescheduling interrupts
CAL:      46474    1965965    1949109    1935292   Function call interrupts
TLB:      37895      54853      52594      52308   TLB shootdowns
TRM:          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0   Machine check exceptions
MCP:         17         17         17         17   Machine check polls
ERR:          0
MIS:          0
PIN:          0          0          0          0   Posted-interrupt notification event
PIW:          0          0          0          0   Posted-interrupt wakeup event
This is exmaple of a process after running script

Code: Select all

root@bras-1:# cat /proc/irq/56/smp_affinity
f
root@bras-1:#
would you help me ?

in cpu usage, showing this (HIGH CPU USAGE):

Code: Select all

top - 06:40:25 up  1:22,  1 user,  load average: 0.62, 0.54, 0.55
Tasks: 166 total,   2 running, 164 sleeping,   0 stopped,   0 zombie
%Cpu0  :  1.3 us,  2.5 sy,  0.0 ni, 40.5 id,  0.0 wa,  0.0 hi, 55.7 si,  0.0 st
%Cpu1  :  2.0 us,  6.9 sy,  0.0 ni, 51.5 id,  0.0 wa,  0.0 hi, 39.6 si,  0.0 st
%Cpu2  :  2.2 us,  4.5 sy,  0.0 ni, 68.5 id,  0.0 wa,  0.0 hi, 24.7 si,  0.0 st
%Cpu3  :  2.2 us,  7.6 sy,  0.0 ni, 55.4 id,  0.0 wa,  0.0 hi, 34.8 si,  0.0 st
KiB Mem :  2052176 total,  1530168 free,   278116 used,   243892 buff/cache
KiB Swap:  2095100 total,  2095100 free,        0 used.  1604160 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    3 root      20   0       0      0      0 S   7.7  0.0   5:20.51 ksoftirqd/0
   16 root      20   0       0      0      0 S   5.8  0.0   0:23.88 ksoftirqd/1
   28 root      20   0       0      0      0 R   4.8  0.0   0:24.30 ksoftirqd/3
  863 root      20   0  346004  21656   4580 S   4.8  1.1   4:45.55 accel-pppd
 3607 www-data  20   0  262972  12440   6364 S   2.9  0.6   0:05.11 apache2
 4709 root      20   0   44924   3740   3068 R   2.9  0.2   0:00.16 top
   22 root      20   0       0      0      0 S   1.9  0.0   0:23.35 ksoftirqd/2
23053 www-data  20   0  262748  12012   5992 S   1.9  0.6   0:02.22 apache2
25273 www-data  20   0  262748  12012   5992 S   1.9  0.6   0:01.46 apache2
    7 root      20   0       0      0      0 S   1.0  0.0   0:15.64 rcu_sched
 2009 root      20   0       0      0      0 S   1.0  0.0   0:00.01 kworker/1:2
 3572 www-data  20   0  262972  12320   6244 S   1.0  0.6   0:05.54 apache2
 4478 www-data  20   0  262976  12440   6360 S   1.0  0.6   0:32.96 apache2
28704 www-data  20   0  262748  12012   5992 S   1.0  0.6   0:01.08 apache2
    1 root      20   0  206984   9164   5232 S   0.0  0.4   0:10.63 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
    9 root      rt   0       0      0      0 S   0.0  0.0   0:01.76 migration/0
   10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.01 watchdog/0
   12 root      20   0       0      0      0 S   0.0  0.0   0:00.00 cpuhp/0
dimka88
Posts: 866
Joined: 13 Oct 2014, 05:51
Contact:

Re: Balancing CPUs intel 10 GB

Post by dimka88 »

try bind static irq to cpu.
hugleo
Posts: 59
Joined: 13 Apr 2016, 14:28

Re: Balancing CPUs intel 10 GB

Post by hugleo »

or irqblance service is running
Post Reply