Table of Contents
Why we do it
Trying to get a platform with firewall, loadbalancing, and many connections, we ended up not taking any appliance, but getting nice hardware with heavy processors and nice network cards.
It's actually cheaper to get 8 servers with dual 10Gbps interfaces, 4 10Gbps switches than it would be to get like 2 appliance load-balancers that could possibly handle 4Gbps total traffic.
More redundancy, probably higher limits, and more flexibility… and all that cheaper ? Yeah, nice. But what are the limits you can reach with those ?
Lets bench it !
What we want
At the end, we want :
- firewall/gateway with multiple vlans, and alot of failover IPs
- loadbalancers with hundreds of IPs
- client that connects to servers through fw/gw and lb
Lets check what are the limits we can reach, and how it reacts.
Basic setup
Some informations about the hardware used for our benches :
Servers
| cpu | 2 Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (two 6 cores, total of 24 threads) |
|---|---|
| memory | 64GB (memory is cheap) |
| network | Intel X520-DA2 – Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) |
Connectivity
| Switches | TurboIron x24 |
|---|---|
| Cables | Twinax cables (no reason to get fiber when your servers are so close from the switches) |
Operating System
Linux - Debian/wheezy (some hardware are not supported ot squeeze install)
Monitoring
We are using Munin, with 1s statistic plugins, including :
- CPU
- per cpu usage
- Network bandwidth
- bps
- pkt/s
- tcp (netstat -s, TCP part)
- established connections (current)
- new inboud tcp connections
- new outbound tcp connections
- sockets (/proc/net/sockstat)
- orphan
- timewait
- alloc
- memory
Softwares
Client
inject (found here)
As we try to handle alot of connections from a single server, we soon hit the source port limit. inject allows to bypass that limit, as it binds to a specific source ip/port for each outgoing connections.
Server
Nginx, as in production we will have reverse proxies using nginx.
Some issues with nginx, forced me to search for an other server to bench. httpterm was pointed out. Like inject, it is aimed to just do that: stress some http connections.
firewall
As we are running linux, it's obviously iptables.
Loadbalancing
As mentionned earlier, it's IPVS with direct routing (already used in our production).
