This is an old revision of the document!
Table of Contents
Why we do it
Trying to get a platform with firewall, loadbalancing, and many connections, we ended up not taking any appliance, but getting nice hardware with heavy processors and nice network cards.
It's actually cheaper to get 8 servers with dual 10Gbps interfaces, 4 10Gbps switches than it would be to get like 2 appliance load-balancers that could possibly handle 4Gbps total traffic.
More redundancy, probably higher limits, and more flexibility… and all that cheaper ? Yeah, nice. But what are the limits you can reach with those ?
Lets bench it !
What we want
At the end, we want :
- firewall/gateway with multiple vlans, and alot of failover IPs
- loadbalancers with hundreds of IPs
- client that connects to servers through fw/gw and lb
Lets check what are the limits we can reach, and how it reacts.
Basic setup
Some informations about the hardware used for our benches :
Servers
| cpu | 2 Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (two 6 cores, total of 24 threads) |
|---|---|
| memory | 64GB (memory is cheap) |
| network | Intel X520-DA2 – Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) |
Connectivity
| Switches | TurboIron x24 |
|---|---|
| Cables | Twinax cables (no reason to get fiber when your servers are so close from the switches) |
Operating System
Linux - Debian/wheezy (some hardware are not supported ot squeeze install)
Monitoring
We are using Munin, with 1s statistic plugins, including :
- CPU
- per cpu usage
- Network bandwidth
- bps
- pkt/s
- tcp (netstat -s, TCP part)
- established connections (current)
- new inboud tcp connections
- new outbound tcp connections
- sockets (/proc/net/sockstat)
- orphan
- timewait
- alloc
- memory
Softwares
Server
Nginx, as in production we will have reverse proxies using nginx
Client
inject (found here)
As we try to handle alot of connections from a single server, we soon hit the source port limit. inject allows to bypass that limit, as it binds to a specific source ip/port for each outgoing connections.
firewall
As we are running linux, it's obviously iptables.
Loadbalancing
As mentionned earlier, it's IPVS with direct routing (already used in our production).
