Accel-ppp service rester

Questions related to general functionality
Post Reply
haniaro
Posts: 23
Joined: 29 Dec 2019, 14:18

Accel-ppp service rester

Post by haniaro »

Dears,
I have deployed Accel-PPP as ISP LNS on ESXi 6.5 VM.
I have two VMs (A, B) on two different ESXi (Same version, same resources)
B deployed as import from A.
VM OS is Debian 9.

A working well.
B suddenly restart.
I can not get any hint from log (system and accel)

syslog:
Jul 12 18:21:31 Accel-LNS5 systemd[1]: apt-daily.timer: Adding 35min 37.290135s random time.
Jul 12 19:17:01 Accel-LNS5 CRON[16800]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 12 20:17:01 Accel-LNS5 CRON[16990]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 12 21:17:01 Accel-LNS5 CRON[17360]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 12 22:17:01 Accel-LNS5 CRON[17711]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jul 12 22:37:44 Accel-LNS5 systemd[1]: accel-ppp.service: Main process exited, code=killed, status=11/SEGV
Jul 12 22:37:44 Accel-LNS5 systemd[1]: accel-ppp.service: Unit entered failed state.
Jul 12 22:37:44 Accel-LNS5 systemd[1]: accel-ppp.service: Failed with result 'signal'.
Jul 12 22:37:44 Accel-LNS5 systemd[1]: accel-ppp.service: Service hold-off time over, scheduling restart.
Jul 12 22:37:44 Accel-LNS5 systemd[1]: Stopped Accel-PPP.
Jul 12 22:37:44 Accel-LNS5 systemd[1]: Starting Accel-PPP...
Jul 12 22:37:44 Accel-LNS5 systemd[1]: accel-ppp.service: Failed to read PID from file /var/run/accel-pppd.pid: Invalid argument
Jul 12 22:37:44 Accel-LNS5 systemd[1]: Started Accel-PPP.
Jul 12 22:44:10 Accel-LNS5 systemd[1]: accel-ppp.service: Service hold-off time over, scheduling restart.
Jul 12 22:44:10 Accel-LNS5 systemd[1]: Stopped Accel-PPP.
Jul 12 22:44:10 Accel-LNS5 systemd[1]: Starting Accel-PPP...
Jul 12 22:44:10 Accel-LNS5 systemd[1]: accel-ppp.service: PID file /var/run/accel-pppd.pid not readable (yet?) after start: No such file or directory
Jul 12 22:44:10 Accel-LNS5 systemd[1]: Started Accel-PPP.



Accel-ppp.log
ESC[1;32m[2020-07-12 22:36:36]: info: ESC[0;39ml2tp tunnel 44072-8413 (178.253.122.28:1701): send [L2TP tid=8413 sid=29281 Ns=10443 Nr=11181
<Message-Type Call-Disconnect-Notify> <Assigned-Session-ID 4117> <Result-Code>]
ESC[1;34m[2020-07-12 22:36:36]: debug: : ESC[0;39mauth_layer_free
ESC[1;34m[2020-07-12 22:36:36]: debug: : ESC[0;39mccp_layer_free
ESC[1;34m[2020-07-12 22:36:36]: debug: : ESC[0;39mipcp_layer_free
ESC[1;34m[2020-07-12 22:36:36]: debug: : ESC[0;39mipv6cp_layer_free
ESC[1;34m[2020-07-12 22:36:36]: debug: ESC[0;39ml2tp tunnel 44072-8413 (178.253.122.28:1701): 1 message sent from send queue
ESC[1;32m[2020-07-12 22:36:36]: info: ESC[0;39ml2tp session 44072-8413, 4117-29281: deleting session
ESC[1;32m[2020-07-12 22:36:36]: info: ESC[0;39ml2tp session 44072-8413, 4117-29281: deleting data channel
ESC[1;33m[2020-07-12 22:37:44]: warn: ESC[0;39ml2tp: iprange module disabled, improper IP configuration of PPP interfaces may cause kernel so
ft lockup
ESC[1;33m[2020-07-12 22:37:44]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.4: invalid tid 12285
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.28: invalid tid 44072
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.32: invalid tid 36857
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
ESC[1;33m[2020-07-12 22:37:45]: warn: ESC[0;39ml2tp: discarding unexpected message from 178.253.122.7: invalid tid 12124
.....

.....
ESC[1;32m[2020-07-12 22:39:38]: info: ESC[0;39ml2tp: recv [L2TP tid=0 sid=0 Ns=0 Nr=0 <Message-Type Start-Ctrl-Conn-Request> <Protocol-Versio
n 256> <Host-Name SWA:AB_BRAS_ME60-X8_01> <Vendor-Name HuaWei> <Framing-Capabilities 3> <Assigned-Tunnel-ID 15192> <Recv-Window-Size 1024> <Ch
allenge>]
ESC[1;32m[2020-07-12 22:39:38]: info: ESC[0;39ml2tp: handling SCCRQ from 178.253.122.26
ESC[1;32m[2020-07-12 22:39:38]: info: ESC[0;39ml2tp: new tunnel 42423-15192 created following reception of SCCRQ from 178.253.122.26:1701
ESC[1;32m[2020-07-12 22:39:38]: info: ESC[0;39ml2tp tunnel 42423-15192 (178.253.122.26:1701): sending SCCRP
ESC[1;32m[2020-07-12 22:39:38]: info: ESC[0;39ml2tp tunnel 42423-15192 (178.253.122.26:1701): send [L2TP tid=15192 sid=0 Ns=0 Nr=1 <Message-Type Start-Ctrl-Conn-Reply> <Protocol-Version 256> <Host-Name accel-ppp> <Framing-Capabilities 3> <Assigned-Tunnel-ID -23113> <Vendor-Name accel-ppp> <Recv-Window-Size 16> <Challenge-Response> <Challenge>]
ESC[1;34m[2020-07-12 22:39:38]: debug: ESC[0;39ml2tp tunnel 42423-15192 (178.253.122.26:1701): 1 message sent from send queue
ESC[1;34m[2020-07-12 22:39:38]: debug: ESC[0;39ml2tp tunnel 42423-15192 (178.253.122.26:1701): 1 message added to reception queue
ESC[1;34m[2020-07-12 22:39:38]: debug: ESC[0;39ml2tp tunnel 42423-15192 (178.253.122.26:1701): 1 message acked by peer
ESC[1;32m[2020-07-12 22:39:38]: info: ESC[0;39ml2tp tunnel 42423-15192 (178.253.122.26:1701): recv [L2TP tid=42423 sid=0 Ns=1 Nr=1 <Message-Type Start-Ctrl-Conn-Connected> <Challenge-Response>]

any help please.
Thank you.
dimka88
Posts: 866
Joined: 13 Oct 2014, 05:51
Contact:

Re: Accel-ppp service rester

Post by dimka88 »

Hi @haniaro, I propose to run accel-ppp in GDB instead of systemd unit for better understanding of what happened. Read how to do this https://accel-ppp.readthedocs.io/en/lat ... index.html
haniaro
Posts: 23
Joined: 29 Dec 2019, 14:18

Re: Accel-ppp service rester

Post by haniaro »

hi @dimka88 the above log is from Debian9
here is a systemd core dump from Oracle Linux 8 if can help, a dump fro Debian9 in debug mode coming soon.

Jul 25 15:13:36 lns7 systemd-coredump[26035]: Process 8617 (accel-pppd) of user 0 dumped core.#012#012Stack trace of thread 8619:#012#0 0x00007f0a9f8a42a0 mempool_free (libtriton.so)#012#1 0x0000000000427f1b log_free_msg (accel-pppd)#012#2 0x00007f0a9e0ab6a1 purge (liblog_file.so)#012#3 0x00007f0a9e0ab832 log_thread (liblog_file.so)#012#4 0x00007f0a9f47c2de start_thread (libpthread.so.0)#012#5 0x00007f0a9e5c1e93 __clone (libc.so.6)#012#012Stack trace of thread 8625:#012#0 0x00007f0a9e4fe4dc __sigtimedwait (libc.so.6)#012#1 0x00007f0a9f48695c sigwait (libpthread.so.0)#012#2 0x00007f0a9f8a1271 triton_thread (libtriton.so)#012#3 0x00007f0a9f47c2de start_thread (libpthread.so.0)#012#4 0x00007f0a9e5c1e93 __clone (libc.so.6)#012#012Stack trace of thread 8620:#012#0 0x00007f0a9f48248c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)#012#1 0x000000000040a2aa uc_thread (accel-pppd)#012#2 0x00007f0a9f47c2de start_thread (libpthread.so.0)#012#3 0x00007f0a9e5c1e93 __clone (libc.so.6)#012#012Stack trace of thread 8618:#012#0 0x00007f0a9e4fe4dc __sigtimedwait (libc.so.6)#012#1 0x00007f0a9f48695c sigwait (libpthread.so.0)#012#2 0x00007f0a9c277da9 sigchld_thread (libsigchld.so)#012#3 0x00007f0a9f47c2de start_thread (libpthread.so.0)#012#4 0x00007f0a9e5c1e93 __clone (libc.so.6)#012#012Stack trace of thread 8627:#012#0 0x00007f0a9e5c21c7 epoll_wait (libc.so.6)#012#1 0x00007f0a9f8a0562 timer_thread (libtriton.so)#012#2 0x00007f0a9f47c2de start_thread (libpthread.so.0)#012#3 0x00007f0a9e5c1e93 __clone (libc.so.6)#012#012Stack trace of thread 8624:#012#0 0x00007f0a9e4fe4dc __sigtimedwait (libc.so.6)#012#1 0x00007f0a9f48695c sigwait (libpthread.so.0)#012#2 0x00007f0a9f8a1271 triton_thread (libtriton.so)#012#3 0x00007f0a9f47c2de start_thread (libpthread.so.0)#012#4 0x00007f0a9e5c1e93 __clone (libc.so.6)#012#012Stack trace of thread 8617:#012#0 0x00007f0a9e4fe4dc __sigtimedwait (libc.so.6)#012#1 0x00007f0a9f48695c sigwait (libpthread.so.0)#012#2 0x00000000004293df main (accel-pppd)#012#3 0x00007f0a9e4e96a3 __libc_start_main (libc.so.6)#012#4 0x000000000040552e _start (accel-pppd)#012#012Stack trace of thread 8626:#012#0 0x00007f0a9e5c21c7 epoll_wait (libc.so.6)#012#1 0x00007f0a9f89fabd md_thread (libtriton.so)#012#2 0x00007f0a9f47c2de start_thread (libpthread.so.0)#012#3 0x00007f0a9e5c1e93 __clone (libc.so.6)
Jul 25 15:13:56 lns7 systemd[1]: accel-ppp.service: Main process exited, code=killed, status=11/SEGV
Jul 25 15:13:56 lns7 systemd[1]: accel-ppp.service: Failed with result 'signal'.
Jul 25 15:13:56 lns7 systemd[1]: accel-ppp.service: Service RestartSec=100ms expired, scheduling restart.
Jul 25 15:13:56 lns7 systemd[1]: accel-ppp.service: Scheduled restart job, restart counter is at 1.

Thank you.
haniaro
Posts: 23
Joined: 29 Dec 2019, 14:18

Re: Accel-ppp service rester

Post by haniaro »

Here are the core dump output:

coredumpctl dump 30544 --output=30544.dump
PID: 30544 (accel-pppd)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Sun 2020-07-26 13:50:36 EEST (1h 13min ago)
Command Line: /usr/sbin/accel-pppd -d -p /var/run/accel-pppd.pid -c /etc/accel-ppp.conf
Executable: /usr/sbin/accel-pppd
Control Group: /
Slice: -.slice
Boot ID: dcf2a076d5a143eb8df9c1051086a7e2
Machine ID: cd4e3fbd8ff245fabddcd9bb37ecd6dd
Hostname: lns7.ibaaco.net
Storage: /var/lib/systemd/coredump/core.accel-pppd.0.dcf2a076d5a143eb8df9c1051086a7e2.30544.1595760636000000.lz4
Message: Process 30544 (accel-pppd) of user 0 dumped core.

Stack trace of thread 30546:
#0 0x0000000000427f55 _log_free_msg (accel-pppd)
#1 0x0000000000427f27 log_free_msg (accel-pppd)
#2 0x00007fc3c42156a1 purge (liblog_file.so)
#3 0x00007fc3c4215967 log_thread (liblog_file.so)
#4 0x00007fc3c55e62de start_thread (libpthread.so.0)
#5 0x00007fc3c472be93 __clone (libc.so.6)

Stack trace of thread 30552:
#0 0x00007fc3c472c1c7 epoll_wait (libc.so.6)
#1 0x00007fc3c5a09abd md_thread (libtriton.so)
#2 0x00007fc3c55e62de start_thread (libpthread.so.0)
#3 0x00007fc3c472be93 __clone (libc.so.6)

Stack trace of thread 30553:
#0 0x00007fc3c472c1c7 epoll_wait (libc.so.6)
#1 0x00007fc3c5a0a562 timer_thread (libtriton.so)
#2 0x00007fc3c55e62de start_thread (libpthread.so.0)
#3 0x00007fc3c472be93 __clone (libc.so.6)

Stack trace of thread 30551:
#0 0x00007fc3c46684dc __sigtimedwait (libc.so.6)
#1 0x00007fc3c55f095c sigwait (libpthread.so.0)
#2 0x00007fc3c5a0b271 triton_thread (libtriton.so)
#3 0x00007fc3c55e62de start_thread (libpthread.so.0)
#4 0x00007fc3c472be93 __clone (libc.so.6)

Stack trace of thread 30547:
#0 0x00007fc3c55ec48c pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x000000000040a2aa uc_thread (accel-pppd)
#2 0x00007fc3c55e62de start_thread (libpthread.so.0)
#3 0x00007fc3c472be93 __clone (libc.so.6)

Stack trace of thread 30545:
#0 0x00007fc3c46684dc __sigtimedwait (libc.so.6)
#1 0x00007fc3c55f095c sigwait (libpthread.so.0)
#2 0x00007fc3c23e1da9 sigchld_thread (libsigchld.so)
#3 0x00007fc3c55e62de start_thread (libpthread.so.0)
#4 0x00007fc3c472be93 __clone (libc.so.6)

Stack trace of thread 30550:
#0 0x00007fc3c46684dc __sigtimedwait (libc.so.6)
#1 0x00007fc3c55f095c sigwait (libpthread.so.0)
#2 0x00007fc3c5a0b271 triton_thread (libtriton.so)
#3 0x00007fc3c55e62de start_thread (libpthread.so.0)
#4 0x00007fc3c472be93 __clone (libc.so.6)

Stack trace of thread 30544:
#0 0x00007fc3c46684dc __sigtimedwait (libc.so.6)
#1 0x00007fc3c55f095c sigwait (libpthread.so.0)
#2 0x00000000004293df main (accel-pppd)
#3 0x00007fc3c46536a3 __libc_start_main (libc.so.6)
#4 0x000000000040552e _start (accel-pppd)
dimka88
Posts: 866
Joined: 13 Oct 2014, 05:51
Contact:

Re: Accel-ppp service rester

Post by dimka88 »

Did accel-ppp built with Debug option like in the docs https://accel-ppp.readthedocs.io/en/lat ... index.html?
Provide please `bt full` output
Post Reply