Session Dissconnection

Shaper related questions
Post Reply
faiza94
Posts: 5
Joined: 06 Dec 2017, 05:37

Session Dissconnection

Post by faiza94 »

​I have an Accel-ppp[1] server that deals with a lot concurrent pppoe connections of about 4000 clients. These pppoe connections are generated in a way
that all come in a burst and Accel-ppp server does the pppoe connection hand shake followed by AAA. Accel-ppp server also assigns
HQOS to individual ppp connections. The utility tc (part of iproute2), combined with iptables setting marks on packets and
connections, is used to implement the Hierarchical Token Bucket. This implementation is placed in the /etc/ppp/ip-up script present
in the Accel-ppp server. This script is triggered in the pppd_compat module present in the accel-ppp.conf file at the start of every new connection.

The problem I am facing that when a burst of pppoe request comes in, the Accel-ppp server with the HQOS script is only able to
create a certain number of connections. Than it stops making connections. We have collected data in various configurations e.g.
with HQOS rules and without HQOS rules. Without HQOS rules, all the connections connect to the Accel-ppp server in 1 minute whereas
without HQOS, Accel-ppp is only able to make 15-20% of the total connections and taking a lot of time.

It looks like and TC and iptables are slowing down the connections. Hardware resources where also being monitored while collecting data.
Hardware resources where under utilized.

Hardware specs of server running Accel-ppp:
----------------------------------------------------------------------------------------
Processor |Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz | 32
Memory |64GB total memory | 8 x 8GB
Hard Drive |450GB 10K RPM SAS 2.5” | 8
Network Card |Intel Corporation X520-DA2 (2-ports) | 1

Does anyone have an idea what actually might be the reason for this behavior. What can be the alternative to implement HQOS without TC if it is the problem?
dimka88
Posts: 866
Joined: 13 Oct 2014, 05:51
Contact:

Re: Session Dissconnection

Post by dimka88 »

Hi.
Show: HQOS rules, perf top when set HQOS, and accel-ppp version.
faiza94
Posts: 5
Joined: 06 Dec 2017, 05:37

Re: Session Dissconnection

Post by faiza94 »

Hi dimka88
Following are the rules we are implementing in the ip-up script.

Code: Select all

#!/bin/bash
source /etc/ppp/setAttrs.sh
iptables -t mangle -F

# Interface facing the Internet
EXTDEV=ens3

PPP_IFACE="$1"
SPEED=$(getBandwidth $PPP_IFACE)
DOWNSPEED="$(cut -d'/' -f1 <<<"$SPEED")"
UPSPEED="$(cut -d'/' -f2 <<<"$SPEED")"
DOWNSPEED=$((DOWNSPEED/1000))
UPSPEED=$((UPSPEED/1000))
RSERVICE=$(getPackage $PPP_IFACE) 
CEIL1=3
CEIL2=4 

echo $DOWNSPEED
echo $UPSPEED

#custom services
LIVE_IP=172.30.228.1
VOD_IP=172.30.2.4
LIVE_CEILRATE=3
LIVE_RATE=2
VOD_RATE=5
VOD_CEILRATE=6

#variables to check if package includes these services
LIVE_FOUND=0
VOD_FOUND=0

#seperate Reply-Message from package details
PACKAGE="${RSERVICE:14}"

#seperate iface no from ppp interface
IFACE_NO="${PPP_IFACE:3}"

IFB_IFACE="ifb"$IFACE_NO

modprobe ifb numifbs=10
ip link set $IFB_IFACE up

tc qdisc del dev ${PPP_IFACE} root 
tc qdisc add dev ${PPP_IFACE} root handle 1:0 htb default 15 
tc class add dev ${PPP_IFACE} parent 1: classid 1:1 htb rate ${CEIL1}mbit ceil ${CEIL2}mbit burst 60mbit cburst 60mbit

tc qdisc del dev $IFB_IFACE root
tc qdisc add dev $IFB_IFACE root handle 1:0 htb default 15
tc class add dev $IFB_IFACE parent 1: classid 1:1 htb rate ${CEIL1}mbit ceil ${CEIL2}mbit

IFS=',' read -ra PK <<< "$PACKAGE"
for PKATTR in "${PK[@]}"; do
    echo "$PKATTR"
    if [ "$PKATTR" = "L" ]; then
	echo "$PKATTR"
	tc class add dev ${PPP_IFACE} parent 1:1 classid 1:10 htb rate ${LIVE_RATE}mbit ceil  ${LIVE_CEILRATE}mbit prio 0
	tc class add dev $IFB_IFACE parent 1:1 classid 1:10 htb rate ${LIVE_RATE}mbit ceil ${LIVE_CEILRATE}mbit prio 0 
	LIVE_FOUND=1
    elif [ "$PKATTR" = "V" ]; then
	echo "$PKATTR"
	tc class add dev ${PPP_IFACE} parent 1:1 classid 1:11 htb rate ${VOD_RATE}mbit ceil ${VOD_CEILRATE}mbit prio 0
	tc class add dev $IFB_IFACE parent 1:1 classid 1:11 htb rate ${VOD_RATE}mbit ceil ${VOD_CEILRATE}mbit prio 0
	VOD_FOUND=1
    fi     
done 

if [ "$LIVE_FOUND" = 0 ]; then
	echo "live not found"
	tc class add dev ${PPP_IFACE} parent 1:1 classid 1:10 htb rate 0.001kbps ceil 0.001kbps prio 0
	tc class add dev $IFB_IFACE parent 1:1 classid 1:10 htb rate 0.001kbps ceil 0.001kbps prio 0 
fi

if [ "$VOD_FOUND" = 0 ]; then
	echo "VOD not found"
	tc class add dev ${PPP_IFACE} parent 1:1 classid 1:11 htb rate 0.001kbps ceil 0.001kbps prio 0
	tc class add dev $IFB_IFACE parent 1:1 classid 1:11 htb rate 0.001kbps ceil 0.001kbps prio 0
fi 

tc class add dev ${PPP_IFACE} parent 1:1 classid 1:15 htb rate ${DOWNSPEED}mbit ceil ${DOWNSPEED}mbit prio 0 
tc class add dev $IFB_IFACE parent 1:1 classid 1:15 htb rate ${UPSPEED}mbit ceil ${UPSPEED}mbit prio 0   

#upload
tc filter add dev $IFB_IFACE protocol ip parent 1:0 prio 1 u32 match ip dst $LIVE_IP flowid 1:10
tc filter add dev $IFB_IFACE protocol ip parent 1:0 prio 1 u32 match ip dst $VOD_IP flowid 1:11

#upload ingress
tc qdisc del dev ${PPP_IFACE} handle ffff: ingress
tc qdisc add dev ${PPP_IFACE} handle ffff: ingress
tc filter add dev ${PPP_IFACE} parent ffff: u32 match u32 0 0 action mirred egress redirect dev $IFB_IFACE

tc qdisc add dev ${PPP_IFACE} parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev ${PPP_IFACE} parent 1:11 handle 120: sfq perturb 10
tc qdisc add dev ${PPP_IFACE} parent 1:15 handle 130: sfq perturb 10

tc filter add dev ${PPP_IFACE} parent 1:0 protocol ip prio 1 handle 2 fw classid 1:10
tc filter add dev ${PPP_IFACE} parent 1:0 protocol ip prio 1 handle 3 fw classid 1:11
tc filter add dev ${PPP_IFACE} parent 1:0 protocol ip prio 1 handle 15 fw classid 1:15

echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -s 10.10.0.0/255.255.0.0 -o $EXTDEV -j SNAT --to-source 172.30.222.60

iptables -t mangle -A PREROUTING  --source $LIVE_IP -j MARK --set-mark 0x2
iptables -t mangle -A PREROUTING  --source $VOD_IP -j MARK --set-mark 0x3 #vod
iptables -t mangle -A PREROUTING  --source 0.0.0.0 -j MARK --set-mark 0x15

tc -s class show dev ${PPP_IFACE}
tc -s class show dev $IFB_IFACE
Below is the image depicting our performance
Attachments
perf.PNG
perf.PNG (84.89 KiB) Viewed 6008 times
faiza94
Posts: 5
Joined: 06 Dec 2017, 05:37

Re: Session Dissconnection

Post by faiza94 »

The accel version is 1.10.3.
faiza94
Posts: 5
Joined: 06 Dec 2017, 05:37

Re: Session Dissconnection

Post by faiza94 »

@Dimitry. Can you give your views regarding this problem?
dimka88
Posts: 866
Joined: 13 Oct 2014, 05:51
Contact:

Re: Session Dissconnection

Post by dimka88 »

this script really run if each session up? If is not really, show ip-up script.
may be use tc polecer without ifb?
faiza94
Posts: 5
Joined: 06 Dec 2017, 05:37

Re: Session Dissconnection

Post by faiza94 »

Hello back
The ip-up script posted above runs whenever a new user connects or a session is made. Moreover, this script also runs whenever there is a time-shift. Ip-up script has our hqos rules for assigning bandwidth to a particular service (using Iptables and tc rules).
dimka88
Posts: 866
Joined: 13 Oct 2014, 05:51
Contact:

Re: Session Dissconnection

Post by dimka88 »

I think, this script does not have optimization, need change global logics in script. I dont have time for help you now, may be other forum users will help you.
Post Reply