Balancing CPUs intel 10 GB

Any topics
Post Reply
hugleo
Posts: 55
Joined: 13 Apr 2016, 14:28

Balancing CPUs intel 10 GB

Post by hugleo » 05 Apr 2017, 13:24

In my new server (NUMA) have one core reaching 40% and other cores 2%
Tried to solve using default intel default kernel affinity one txrx queue by core or Installed irqbalance and doesn't seen to work.

So I uninstall irqbalance and use a script to allocate interrupts for all cores.

Looks like maybe I can have more interrupts i don't know. But at least all cores are working balanced and now I see peaks at most of 15%.

I would like to share the script, hope be useful ;-)

for irq in `grep ens1f0 /proc/interrupts | cut -d: -f1`; do echo ffffff > /proc/irq/$irq/smp_affinity; done

for irq in `grep ens1f1 /proc/interrupts | cut -d: -f1`; do echo ffffff > /proc/irq/$irq/smp_affinity; done

for i in `seq 0 7`; do echo fffffff > /sys/class/net/ens1f0/queues/rx-$i/rps_cpus ; done

for i in `seq 0 7`; do echo fffffff > /sys/class/net/ens1f1/queues/rx-$i/rps_cpus ; done


Using Debian Stretch.

dimka88
Posts: 261
Joined: 13 Oct 2014, 05:51
Contact:

Re: Balancing CPUs intel 10 GB

Post by dimka88 » 05 Apr 2017, 13:56

show

Code: Select all

cat /proc/interrupts

hugleo
Posts: 55
Joined: 13 Apr 2016, 14:28

Re: Balancing CPUs intel 10 GB

Post by hugleo » 05 Apr 2017, 14:05

Code: Select all


           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
  0:         17          0          0          0          0          0          0          0   IO-APIC   2-edge      timer
  8:          0          1          0          0          0          0          0          0   IO-APIC   8-edge      rtc0
  9:          0          1          0          0          0          0          0          0   IO-APIC   9-fasteoi   acpi
 18:          0          0          0         59          0          0          0          0   IO-APIC  18-fasteoi   i801_smbus, ehci_hcd:usb1, ehci_hcd:usb3
 19:          0          0          0          0          0          0          0        407   IO-APIC  19-fasteoi   uhci_hcd:usb2, hpilo
 25:          0          0          0          0          0          0          0          0   PCI-MSI 16384-edge      PCIe PME
 26:          0          0          0          0          0          0          0          0   PCI-MSI 18432-edge      PCIe PME
 28:          0          0          0          0          0          0          0          0   PCI-MSI 32768-edge      PCIe PME
 29:          0          0          0          0          0          0          0          0   PCI-MSI 34816-edge      PCIe PME
 30:          0          0          0          0          0          0          0          0   PCI-MSI 36864-edge      PCIe PME
 31:          0          0          0          0          0          0          0          0   PCI-MSI 38912-edge      PCIe PME
 33:          0          0          0          0          0          0          0          0   PCI-MSI 49152-edge      PCIe PME
 34:          0          0          0          0          0          0          0          0   PCI-MSI 51200-edge      PCIe PME
 35:          0          0          0          0          0          0          0          0   PCI-MSI 53248-edge      PCIe PME
 36:          0          0          0          0          0          0          0          0   PCI-MSI 55296-edge      PCIe PME
 37:          0          0          0          0          0          0          0          0   PCI-MSI 462848-edge      PCIe PME
 38:          0          0          0          0          0          0          0          0   PCI-MSI 466944-edge      PCIe PME
 39:          0          0          0          0          0          0          0          0   PCI-MSI 471040-edge      PCIe PME
 40:          0          0          0          0          0          0          0          0   PCI-MSI 473088-edge      PCIe PME
 41:      31202          0          1          0          0          0          0          0   PCI-MSI 1572864-edge      hpsa0-msix0
 42:          0      19899          1          0          0          0          0          0   PCI-MSI 1572865-edge      hpsa0-msix1
 43:          0          0      21648          1          0          0          0          0   PCI-MSI 1572866-edge      hpsa0-msix2
 44:          0          0          0      12293          0          0          0          0   PCI-MSI 1572867-edge      hpsa0-msix3
 45:          0          0          0          0      24890          0          0          0   PCI-MSI 1572868-edge      hpsa0-msix4
 46:          0          0          0          0          1      32690          0          0   PCI-MSI 1572869-edge      hpsa0-msix5
 47:          0          0          0          0          0          1      35089          0   PCI-MSI 1572870-edge      hpsa0-msix6
 48:          0          0          0          0          0          1          0      53071   PCI-MSI 1572871-edge      hpsa0-msix7
 49:          0          0          0          0          0          0         25          0   PCI-MSI 327680-edge      xhci_hcd
 50:          0          0          0          0          0          0     687261          0   PCI-MSI 512000-edge      ahci[0000:00:1f.2]
 51:         19 2185464873          0          0          0          0          0          0   PCI-MSI 4194304-edge      ens1f0-TxRx-0
 52:          0         19 2193765831          0          0          0          0          0   PCI-MSI 4194305-edge      ens1f0-TxRx-1
 53:          0          0 2201278684          0          0          0          0          0   PCI-MSI 4194306-edge      ens1f0-TxRx-2
 54:          0          0          0 2192157378          0          0          0          0   PCI-MSI 4194307-edge      ens1f0-TxRx-3
 55:          0          0          0 2193464481         19          0          0          0   PCI-MSI 4194308-edge      ens1f0-TxRx-4
 56:          0          0          0          0 2233232382         19          0          0   PCI-MSI 4194309-edge      ens1f0-TxRx-5
 57:          0          0          0          0 2262997278          0         19          0   PCI-MSI 4194310-edge      ens1f0-TxRx-6
 58:          0          0          0          0          0 2215993925          0         19   PCI-MSI 4194311-edge      ens1f0-TxRx-7
 59:          0          0          0          0          0      51176          0          0   PCI-MSI 4194312-edge      ens1f0
 60:         16          0          0          0          0          0 1627941912          0   PCI-MSI 4196352-edge      ens1f1-TxRx-0
 61:          0         16          0          0          0          0  890242770          0   PCI-MSI 4196353-edge      ens1f1-TxRx-1
 62:          0          0         16          0          0          0          0  856293999   PCI-MSI 4196354-edge      ens1f1-TxRx-2
 63:          0  884549281          0         16          0          0          0          0   PCI-MSI 4196355-edge      ens1f1-TxRx-3
 64:          0          0  853715405          0         16          0          0          0   PCI-MSI 4196356-edge      ens1f1-TxRx-4
 65:          0          0  808481297          0          0         16          0          0   PCI-MSI 4196357-edge      ens1f1-TxRx-5
 66:          0          0          0  862327096          0          0         16          0   PCI-MSI 4196358-edge      ens1f1-TxRx-6
 67:          0          0          0  970827965          0          0          0         16   PCI-MSI 4196359-edge      ens1f1-TxRx-7
 68:          0          0          0          0         12          0          0          0   PCI-MSI 4196360-edge      ens1f1
 70:          0          0          0          0          0          2          0          0   PCI-MSI 65536-edge      ioat-msix
 72:          0          0          0          0          0          0          2          0   PCI-MSI 67584-edge      ioat-msix
 73:          0          0          0          0          0          0          2          0   PCI-MSI 69632-edge      ioat-msix
 74:          0          0          0          0          0          0          0          2   PCI-MSI 71680-edge      ioat-msix
 75:          0          2          0          0          0          0          0          0   PCI-MSI 73728-edge      ioat-msix
 76:          0          0          2          0          0          0          0          0   PCI-MSI 75776-edge      ioat-msix
 77:          0          0          2          0          0          0          0          0   PCI-MSI 77824-edge      ioat-msix
 78:          0          0          0          2          0          0          0          0   PCI-MSI 79872-edge      ioat-msix
 79:          0          0          0          1          0          0          0          0   PCI-MSI 1054720-edge      eno4-tx-0
 80:          0          0          0          0          1          0          0          0   PCI-MSI 1054721-edge      eno4-rx-1
 81:          0          0          0          0          1          0          0          0   PCI-MSI 1054722-edge      eno4-rx-2
 82:          0          0          0          0          0          1          0          0   PCI-MSI 1054723-edge      eno4-rx-3
 83:          0          0          0          0          0          1          0          0   PCI-MSI 1054724-edge      eno4-rx-4
 84:          0          0          0          0          0          0   10982251          0   PCI-MSI 1048576-edge      eno1-tx-0
 85:          0          0          0          0          0          0    3415978          0   PCI-MSI 1048577-edge      eno1-rx-1
 86:          0          0          0          0          0          0          0    2494484   PCI-MSI 1048578-edge      eno1-rx-2
 87:          0    3135707          0          0          0          0          0          0   PCI-MSI 1048579-edge      eno1-rx-3
 88:          0          0    2266534          0          0          0          0          0   PCI-MSI 1048580-edge      eno1-rx-4
 89:          0          0          1          0          0          0          0          0   PCI-MSI 1050624-edge      eno2-tx-0
 90:          0          0          0          1          0          0          0          0   PCI-MSI 1050625-edge      eno2-rx-1
 91:          0          0          0          1          0          0          0          0   PCI-MSI 1050626-edge      eno2-rx-2
 92:          0          0          0          0          1          0          0          0   PCI-MSI 1050627-edge      eno2-rx-3
 93:          0          0          0          0          1          0          0          0   PCI-MSI 1050628-edge      eno2-rx-4
NMI:     265001     326956     285943     343646     270669     256472     252049     205589   Non-maskable interrupts
LOC: 4203682264 4289977452 4225735740 4288788873 4255323177 4183586837 4129026023 4206340721   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:     265001     326956     285943     343646     270669     256472     252049     205589   Performance monitoring interrupts
IWI:   20071623   16883400   38553433   17531562   36602854   32277089   43646559   47224834   IRQ work interrupts
RTR:          7          0          0          0          0          0          0          0   APIC ICR read retries
RES:   10765529    4082308    7799076    1983476    8795262    9584047    8329673    9204928   Rescheduling interrupts
CAL: 2752579806 2268719961 1725290636 1688816829 1708263433 2220634928 4247569113 2885008141   Function call interrupts
TLB:       3909       4882       4880       5129       4712       4290       4662       4170   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:       2229       2229       2229       2229       2229       2229       2229       2229   Machine check polls
ERR:          0
MIS:          0
PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event




Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest