high wait

Questions related to general functionality
Post Reply
proca
Posts: 14
Joined: 07 Dec 2018, 17:24

high wait

Post by proca »

Hello all,

I'm running a debian 9 with 4.9.0-8-amd64 and accel-ppp 1.11.99, 10Gbe X710 with RSS/RPS set on 11 out of 12 cores.
I notice a constant cpu usage on waiting for io for one core when i increase the number of sessions on the box:

top - 19:45:57 up 40 min, 2 users, load average: 1.04, 1.40, 1.66
Tasks: 173 total, 3 running, 170 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1.3 us, 2.7 sy, 0.0 ni, 0.0 id, 95.3 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni, 96.0 id, 0.0 wa, 0.0 hi, 4.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni, 95.0 id, 0.0 wa, 0.0 hi, 5.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni, 94.7 id, 0.0 wa, 0.0 hi, 5.3 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni, 92.4 id, 0.0 wa, 0.0 hi, 7.6 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni, 95.3 id, 0.0 wa, 0.0 hi, 4.7 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni, 92.1 id, 0.0 wa, 0.0 hi, 7.9 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni, 92.0 id, 0.0 wa, 0.0 hi, 8.0 si, 0.0 st
%Cpu8 : 0.0 us, 0.0 sy, 0.0 ni, 88.5 id, 0.0 wa, 0.0 hi, 11.5 si, 0.0 st
%Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 90.7 id, 0.0 wa, 0.0 hi, 9.3 si, 0.0 st
%Cpu10 : 0.0 us, 0.0 sy, 0.0 ni, 94.3 id, 0.0 wa, 0.0 hi, 5.7 si, 0.0 st
%Cpu11 : 0.0 us, 0.0 sy, 0.0 ni, 92.3 id, 0.0 wa, 0.0 hi, 7.7 si, 0.0 st
KiB Mem : 32938048 total, 20664000 free, 11008612 used, 1265436 buff/cache
KiB Swap: 33540092 total, 33540092 free, 0 used. 20401100 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

10819 root 20 0 44940 3708 2976 S 1.7 0.0 0:08.10 top
1389 root 20 0 735676 35608 7772 S 1.0 0.1 0:27.24 accel-pppd

The wait fluctuates from 0 to ver 90% every few seconds, for this reason i excluded this core from the packet forwarding cpus.
This is regardless of the throughput.

When I go over 2000 sessions i start seeing:
ioctl(PPPIOCCONNECT): Transport endpoint is not connected
or
radius: server(2) not responding, terminating session...

It looks like the single cpu0 is handling control packets, is this the case ?
Any idea how to improve ?
dimka88
Posts: 866
Joined: 13 Oct 2014, 05:51
Contact:

Re: high wait

Post by dimka88 »

Hello @proca, try to decrease log level.
Why do you use too old accel-ppp version?
proca
Posts: 14
Joined: 07 Dec 2018, 17:24

Re: high wait

Post by proca »

I set verbose to 0 everywhere and level to 1 and still have a high wa.
Do you think an update to the latest git version would help ?
dimka88
Posts: 866
Joined: 13 Oct 2014, 05:51
Contact:

Re: high wait

Post by dimka88 »

You need to figure out what happens, try to use `iotop` program
proca
Posts: 14
Joined: 07 Dec 2018, 17:24

Re: high wait

Post by proca »

Seems some disk write action but I don't know who is triggering it:

Total DISK READ : 0.00 B/s | Total DISK WRITE : 80.64 K/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 8.66 M/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND

28136 be/4 root 0.00 B/s 0.00 B/s 0.00 % 99.82 % [kworker/u24:1]
742 be/3 root 0.00 B/s 80.64 K/s 0.00 % 10.71 % [jbd2/sda1-8]
1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init rcu_novb_poll

still looking, i don't have anything else installed on the system than snmpd, rest is standard debian.
Post Reply