This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision | ||
system:benches10gbps:firewall [2012/10/12 12:26] ze |
system:benches10gbps:firewall [2012/11/02 15:41] (current) ze ipset |
||
---|---|---|---|
Line 2: | Line 2: | ||
Those bench start with the inject / httpterm configuration from direct | Those bench start with the inject / httpterm configuration from direct | ||
benches, with 270-290k connections/s between a client and a server. | benches, with 270-290k connections/s between a client and a server. | ||
- | |||
- | FIXME: graph not available yet. Will wait until the bench are over. | ||
Monitoring graphs for the different benches can be found | Monitoring graphs for the different benches can be found | ||
Line 26: | Line 24: | ||
we have (approximate reading on graphs) : | we have (approximate reading on graphs) : | ||
- | ^ what ^ per client ^ per server ^ total ^ | + | ^ what ^ per client ^ per server ^ total ^ graph ^ |
- | ^ conn/s | 400k | 266k | 800k | | + | ^ conn/s | 400k | 266k | 800k | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/tcp_stats_conn_out.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/tcp_stats_conn_out.png|cli2]] | |
- | ^ Gbps from cli/srv | 1.1/1.7 | 0.75/1.12 | 2.4/3.4 | | + | ^ Gbps from cli/srv | 1.1/1.7 | 0.75/1.12 | 2.4/3.4 | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/interfaces_eth1_bps.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/interfaces_eth1_bps.png|cli2]] | |
- | ^ Mpkt/s from cli/srv | 1.2/1.62 | 0.8/1.08 | 2.4/3.24 | | + | ^ Mpkt/s from cli/srv | 1.2/1.62 | 0.8/1.08 | 2.4/3.24 | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/interfaces_eth1_pkt.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/interfaces_eth1_pkt.png|cli2]] | |
Well, we can have about a little over 3Gbps, with 800k connections/s. | Well, we can have about a little over 3Gbps, with 800k connections/s. | ||
Line 79: | Line 77: | ||
earlier, which is explained by the fact both go in and out of our | earlier, which is explained by the fact both go in and out of our | ||
gateway via the same interface. | gateway via the same interface. | ||
+ | |||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/01-baseline-gateway/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/01-baseline-gateway/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
===== no rules ===== | ===== no rules ===== | ||
Line 93: | Line 94: | ||
down from 5.7 to 4.9 (both, Gbps and M pkt/s). That's down from 800k to | down from 5.7 to 4.9 (both, Gbps and M pkt/s). That's down from 800k to | ||
just under 700k conn/s. | just under 700k conn/s. | ||
+ | |||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/02-baseline-norule/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/02-baseline-norule/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
+ | |||
====== Firewall ====== | ====== Firewall ====== | ||
Line 121: | Line 126: | ||
seconds, then there is a drastic drop to an average of 135k for the rest | seconds, then there is a drastic drop to an average of 135k for the rest | ||
of the time. | of the time. | ||
+ | |||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/03-firewall-conntrack/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/03-firewall-conntrack/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
As the conntrack count increase in a linear form, up to about 21.2M, and | As the conntrack count increase in a linear form, up to about 21.2M, and | ||
Line 143: | Line 151: | ||
Testing with those values allowed us to get the break at 60 seconds. | Testing with those values allowed us to get the break at 60 seconds. | ||
Connections gets in time_wait, and expires after 60s instead of 120s. | Connections gets in time_wait, and expires after 60s instead of 120s. | ||
+ | |||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/04-firewall-conntrack2/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/04-firewall-conntrack2/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
+ | |||
Testing with nf_conntrack_tcp_timeout_time_wait set to 1s gives directly | Testing with nf_conntrack_tcp_timeout_time_wait set to 1s gives directly | ||
the low performances, even if the conntrack stay under 200k, instead of | the low performances, even if the conntrack stay under 200k, instead of | ||
a few millions. | a few millions. | ||
+ | |||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/05-firewall-conntrack3/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/05-firewall-conntrack3/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
For our heavy connections, we clearly need to be able to *not* track | For our heavy connections, we clearly need to be able to *not* track | ||
them. | them. | ||
+ | |||
===== notrack ===== | ===== notrack ===== | ||
Line 172: | Line 188: | ||
4.2M pkt/s, total of about 590k conn/s instead of our 800k without | 4.2M pkt/s, total of about 590k conn/s instead of our 800k without | ||
firewall. | firewall. | ||
+ | |||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/06-firewall-notrack/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/06-firewall-notrack/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
+ | |||
Trying to get only one rule for notrack get un slightly better | Trying to get only one rule for notrack get un slightly better | ||
Line 180: | Line 200: | ||
That give us about 620k conn/s. | That give us about 620k conn/s. | ||
+ | |||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/07-firewall-notrack2/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
+ | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/07-firewall-notrack2/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
===== simple rules ===== | ===== simple rules ===== | ||
Line 196: | Line 219: | ||
iptables -F ; for ((i=0;i<n;++i)) ; { iptables -A FORWARD -s 10.0.0.$i ; } | iptables -F ; for ((i=0;i<n;++i)) ; { iptables -A FORWARD -s 10.0.0.$i ; } | ||
- | ^ match rules ^ conn/s ^ pkt/s ^ | + | ^ match rules ^ conn/s ^ pkt/s ^ graph ^ |
- | | 0 | 800k | 5.7M | | + | | 0 | 800k | 5.7M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0000/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0000/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | 16 | 780k | 5.6M | | + | | 16 | 780k | 5.6M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0010/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0010/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | 64 | 730k | 5.1M | | + | | 64 | 730k | 5.1M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0040/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0040/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | 256 | 480k | 3.38M | | + | | 256 | 480k | 3.38M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0100/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0100/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | 1024 | 148k | 1.05M | | + | | 1024 | 148k | 1.05M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0400/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0400/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
Line 210: | Line 233: | ||
Tests done with 256 match rules. | Tests done with 256 match rules. | ||
- | ^ match rule ^ conn/s ^ pkt/s ^ | + | ^ match rule ^ conn/s ^ pkt/s ^ graph ^ |
- | | -m u32 --u32 ""0xc&0xffffffff=0xa0000`printf %02x $i`" | 67k | 480k | | + | | -m u32 --u32 ""0xc&0xffffffff=0xa0000`printf %02x $i`" | 67k | 480k | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-u32-src/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-u32-src/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | -p udp -m udp --dport 53 | 315k | 2.4M | | + | | -p udp -m udp --dport 53 | 315k | 2.4M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-udp/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-udp/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | -p tcp -m tcp --dport 443 | 155k | 1.1M | | + | | -p tcp -m tcp --dport 443 | 155k | 1.1M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-https/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-https/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | -p tcp -m tcp --dport 80 (does match) | 140k | 990k | | + | | -p tcp -m tcp --dport 80 (does match) | 140k | 990k | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-http/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-http/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | -d 10.0.0.$i | 460k | 3.2M | | + | | -d 10.0.0.$i | 460k | 3.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-dst/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-dst/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
Different kind of matches have different kind of impact. -d or -s have | Different kind of matches have different kind of impact. -d or -s have | ||
about the same impact. | about the same impact. | ||
+ | |||
+ | |||
+ | |||
===== other configs ===== | ===== other configs ===== | ||
Line 225: | Line 251: | ||
performances so far. | performances so far. | ||
- | ^ match rules ^ conn/s ^ pkt/s ^ | + | ^ match rules ^ conn/s ^ pkt/s ^ graph ^ |
- | | default | 480k | 3.38M | | + | | default | 480k | 3.38M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen1k/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen1k/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | ethtool -G eth1 {tx/rx} 512 | 505k | 3.6M | | + | | ethtool -G eth1 {tx/rx} 512 | 505k | 3.6M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool512/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool512/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | ethtool -G eth1 {tx/rx} 64 | 450k | 3.2M | | + | | ethtool -G eth1 {tx/rx} 64 | 450k | 3.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool64/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool64/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | ip link set eth1 txqueuelen 10000 | 470k | 3.3M | | + | | ip link set eth1 txqueuelen 10000 | 470k | 3.3M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen10k/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen10k/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
txqueuelen - no effect | txqueuelen - no effect | ||
Line 310: | Line 336: | ||
to the next entries. | to the next entries. | ||
- | ^ bits matched per level ^ check ^ match ^ conn/s ^ pkt/s ^ | + | ^ bits matched per level ^ check ^ match ^ conn/s ^ pkt/s ^ graph ^ |
- | | 2 | 39 | 11 | 560k | 3.9M | | + | | 2 | 39 | 11 | 560k | 3.9M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-2/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-2/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | 3 | 51 | 8 | 595k | 4.2M | | + | | 3 | 51 | 8 | 595k | 4.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-3/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-3/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | 4 | 73 | 6 | 580k | 4.0M | | + | | 4 | 73 | 6 | 580k | 4.0M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-4/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-4/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
- | | 5 | 113 | 5 | 575k | 4.0M | | + | | 5 | 113 | 5 | 575k | 4.0M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-5/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-5/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
Note: such high number of rules uses memory. Like 20GB+ of ram used. | Note: such high number of rules uses memory. Like 20GB+ of ram used. | ||
Line 327: | Line 353: | ||
Will have to seek for other solutions. | Will have to seek for other solutions. | ||
+ | |||
+ | ===== ipset ===== | ||
+ | |||
+ | Some people mentionned ipset. Lets bench that. | ||
+ | |||
+ | <code> | ||
+ | # lets create some sets we might use | ||
+ | ipset create ip hash:ip | ||
+ | ipset create net hash:net | ||
+ | ipset create ip,port hash:ip,port | ||
+ | ipset create net,port hash:net,port | ||
+ | </code> | ||
+ | |||
+ | Rules used for different tests : | ||
+ | <code> | ||
+ | -A FORWARD -m set --match-set ip src | ||
+ | -A FORWARD -m set --match-set net src | ||
+ | -A FORWARD -m set --match-set net,port src,src | ||
+ | -A FORWARD -m set --match-set ip,port src,dst | ||
+ | </code> | ||
+ | |||
+ | Lets see how a few match for hash:ip affects our traffic : | ||
+ | |||
+ | ^ # rules ^ conn/s ^ pkt/s ^ | ||
+ | | 1 | 570k | 3.6M | | ||
+ | | 2 | 340k | 2.05M | | ||
+ | | 3 | 240k | 1.45M | | ||
+ | | 4 | 184k | 1.1M | | ||
+ | |||
+ | Ok, so just a few ipset match affects us ALOT. What about other hashes ? | ||
+ | |||
+ | (tests done with 2 matches) | ||
+ | |||
+ | ^ ipset ^ conn/s ^ pkt/s ^ | ||
+ | | hash:ip | 340k | 2.05M | | ||
+ | | hash:net | 350k | 2.1M | | ||
+ | | hash:ip,port | 330k | 2M | | ||
+ | | hash:net,port | 330k | 2M | | ||
+ | |||
+ | Net or ip doesn't change much, and including the port is only a light overhead, | ||
+ | considering the overhead we already have. | ||
+ | |||
+ | What about ipset bitmasks ? | ||
+ | |||
+ | <code> | ||
+ | ipset create bip0 bitmap:ip range 10.136.0.0-10.136.255.255 | ||
+ | ipset create bip1 bitmap:ip range 10.140.0.0-10.140.255.255 | ||
+ | </code> | ||
+ | |||
+ | ^ # rules ^ conn/s ^ pkt/s ^ | ||
+ | | 2 | 550k | 3.5M | | ||
+ | | 4 | 320k | 1.9M | | ||
+ | |||
+ | |||
+ | Considering ipset is limited to 65k entries, and the results, I would advise | ||
+ | against using it, unless you really need the easy to manage set. | ||
+ | |||
+ | |||
+ | ===== interface irq affinity ===== | ||
+ | |||
+ | FIXME: add irq affinity matches with results | ||
+ | |||
+ | ====== Conclusion ====== | ||
+ | |||
+ | * Alot of matching reduce performances. | ||
+ | * u32 are costly | ||
+ | * if you can, try to match and segregate to different subchains, with like 8 to 16 match per chain (for src/dst match, maybe less with heavier match) | ||
+ | * irq affinity can change performances on high loads | ||