I have an Accel-ppp[1] server that deals with a lot concurrent pppoe connections of about 4000 clients. These pppoe connections are generated in a way
that all come in a burst and Accel-ppp server does the pppoe connection hand shake followed by AAA. Accel-ppp server also assigns
HQOS to individual ppp connections. The utility tc (part of iproute2), combined with iptables setting marks on packets and
connections, is used to implement the Hierarchical Token Bucket. This implementation is placed in the /etc/ppp/ip-up script present
in the Accel-ppp server. This script is triggered in the pppd_compat module present in the accel-ppp.conf file at the start of every new connection.
The problem I am facing that when a burst of pppoe request comes in, the Accel-ppp server with the HQOS script is only able to
create a certain number of connections. Than it stops making connections. We have collected data in various configurations e.g.
with HQOS rules and without HQOS rules. Without HQOS rules, all the connections connect to the Accel-ppp server in 1 minute whereas
without HQOS, Accel-ppp is only able to make 15-20% of the total connections and taking a lot of time.
It looks like and TC and iptables are slowing down the connections. Hardware resources where also being monitored while collecting data.
Hardware resources where under utilized.
Hardware specs of server running Accel-ppp:
----------------------------------------------------------------------------------------
Processor |Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz | 32
Memory |64GB total memory | 8 x 8GB
Hard Drive |450GB 10K RPM SAS 2.5” | 8
Network Card |Intel Corporation X520-DA2 (2-ports) | 1
Does anyone have an idea what actually might be the reason for this behavior. What can be the alternative to implement HQOS without TC if it is the problem?
Session Dissconnection
Re: Session Dissconnection
Hi.
Show: HQOS rules, perf top when set HQOS, and accel-ppp version.
Show: HQOS rules, perf top when set HQOS, and accel-ppp version.
Re: Session Dissconnection
Hi dimka88
Following are the rules we are implementing in the ip-up script.
Below is the image depicting our performance
Following are the rules we are implementing in the ip-up script.
Code: Select all
#!/bin/bash
source /etc/ppp/setAttrs.sh
iptables -t mangle -F
# Interface facing the Internet
EXTDEV=ens3
PPP_IFACE="$1"
SPEED=$(getBandwidth $PPP_IFACE)
DOWNSPEED="$(cut -d'/' -f1 <<<"$SPEED")"
UPSPEED="$(cut -d'/' -f2 <<<"$SPEED")"
DOWNSPEED=$((DOWNSPEED/1000))
UPSPEED=$((UPSPEED/1000))
RSERVICE=$(getPackage $PPP_IFACE)
CEIL1=3
CEIL2=4
echo $DOWNSPEED
echo $UPSPEED
#custom services
LIVE_IP=172.30.228.1
VOD_IP=172.30.2.4
LIVE_CEILRATE=3
LIVE_RATE=2
VOD_RATE=5
VOD_CEILRATE=6
#variables to check if package includes these services
LIVE_FOUND=0
VOD_FOUND=0
#seperate Reply-Message from package details
PACKAGE="${RSERVICE:14}"
#seperate iface no from ppp interface
IFACE_NO="${PPP_IFACE:3}"
IFB_IFACE="ifb"$IFACE_NO
modprobe ifb numifbs=10
ip link set $IFB_IFACE up
tc qdisc del dev ${PPP_IFACE} root
tc qdisc add dev ${PPP_IFACE} root handle 1:0 htb default 15
tc class add dev ${PPP_IFACE} parent 1: classid 1:1 htb rate ${CEIL1}mbit ceil ${CEIL2}mbit burst 60mbit cburst 60mbit
tc qdisc del dev $IFB_IFACE root
tc qdisc add dev $IFB_IFACE root handle 1:0 htb default 15
tc class add dev $IFB_IFACE parent 1: classid 1:1 htb rate ${CEIL1}mbit ceil ${CEIL2}mbit
IFS=',' read -ra PK <<< "$PACKAGE"
for PKATTR in "${PK[@]}"; do
echo "$PKATTR"
if [ "$PKATTR" = "L" ]; then
echo "$PKATTR"
tc class add dev ${PPP_IFACE} parent 1:1 classid 1:10 htb rate ${LIVE_RATE}mbit ceil ${LIVE_CEILRATE}mbit prio 0
tc class add dev $IFB_IFACE parent 1:1 classid 1:10 htb rate ${LIVE_RATE}mbit ceil ${LIVE_CEILRATE}mbit prio 0
LIVE_FOUND=1
elif [ "$PKATTR" = "V" ]; then
echo "$PKATTR"
tc class add dev ${PPP_IFACE} parent 1:1 classid 1:11 htb rate ${VOD_RATE}mbit ceil ${VOD_CEILRATE}mbit prio 0
tc class add dev $IFB_IFACE parent 1:1 classid 1:11 htb rate ${VOD_RATE}mbit ceil ${VOD_CEILRATE}mbit prio 0
VOD_FOUND=1
fi
done
if [ "$LIVE_FOUND" = 0 ]; then
echo "live not found"
tc class add dev ${PPP_IFACE} parent 1:1 classid 1:10 htb rate 0.001kbps ceil 0.001kbps prio 0
tc class add dev $IFB_IFACE parent 1:1 classid 1:10 htb rate 0.001kbps ceil 0.001kbps prio 0
fi
if [ "$VOD_FOUND" = 0 ]; then
echo "VOD not found"
tc class add dev ${PPP_IFACE} parent 1:1 classid 1:11 htb rate 0.001kbps ceil 0.001kbps prio 0
tc class add dev $IFB_IFACE parent 1:1 classid 1:11 htb rate 0.001kbps ceil 0.001kbps prio 0
fi
tc class add dev ${PPP_IFACE} parent 1:1 classid 1:15 htb rate ${DOWNSPEED}mbit ceil ${DOWNSPEED}mbit prio 0
tc class add dev $IFB_IFACE parent 1:1 classid 1:15 htb rate ${UPSPEED}mbit ceil ${UPSPEED}mbit prio 0
#upload
tc filter add dev $IFB_IFACE protocol ip parent 1:0 prio 1 u32 match ip dst $LIVE_IP flowid 1:10
tc filter add dev $IFB_IFACE protocol ip parent 1:0 prio 1 u32 match ip dst $VOD_IP flowid 1:11
#upload ingress
tc qdisc del dev ${PPP_IFACE} handle ffff: ingress
tc qdisc add dev ${PPP_IFACE} handle ffff: ingress
tc filter add dev ${PPP_IFACE} parent ffff: u32 match u32 0 0 action mirred egress redirect dev $IFB_IFACE
tc qdisc add dev ${PPP_IFACE} parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev ${PPP_IFACE} parent 1:11 handle 120: sfq perturb 10
tc qdisc add dev ${PPP_IFACE} parent 1:15 handle 130: sfq perturb 10
tc filter add dev ${PPP_IFACE} parent 1:0 protocol ip prio 1 handle 2 fw classid 1:10
tc filter add dev ${PPP_IFACE} parent 1:0 protocol ip prio 1 handle 3 fw classid 1:11
tc filter add dev ${PPP_IFACE} parent 1:0 protocol ip prio 1 handle 15 fw classid 1:15
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -s 10.10.0.0/255.255.0.0 -o $EXTDEV -j SNAT --to-source 172.30.222.60
iptables -t mangle -A PREROUTING --source $LIVE_IP -j MARK --set-mark 0x2
iptables -t mangle -A PREROUTING --source $VOD_IP -j MARK --set-mark 0x3 #vod
iptables -t mangle -A PREROUTING --source 0.0.0.0 -j MARK --set-mark 0x15
tc -s class show dev ${PPP_IFACE}
tc -s class show dev $IFB_IFACE
- Attachments
-
- perf.PNG (84.89 KiB) Viewed 7132 times
Re: Session Dissconnection
The accel version is 1.10.3.
Re: Session Dissconnection
@Dimitry. Can you give your views regarding this problem?
Re: Session Dissconnection
this script really run if each session up? If is not really, show ip-up script.
may be use tc polecer without ifb?
may be use tc polecer without ifb?
Re: Session Dissconnection
Hello back
The ip-up script posted above runs whenever a new user connects or a session is made. Moreover, this script also runs whenever there is a time-shift. Ip-up script has our hqos rules for assigning bandwidth to a particular service (using Iptables and tc rules).
The ip-up script posted above runs whenever a new user connects or a session is made. Moreover, this script also runs whenever there is a time-shift. Ip-up script has our hqos rules for assigning bandwidth to a particular service (using Iptables and tc rules).
Re: Session Dissconnection
I think, this script does not have optimization, need change global logics in script. I dont have time for help you now, may be other forum users will help you.