system:benches10gbps:firewall
Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
|
system:benches10gbps:firewall [2012/10/12 12:26] ze |
system:benches10gbps:firewall [2012/11/02 15:41] (current) ze ipset |
||
|---|---|---|---|
| Line 2: | Line 2: | ||
| Those bench start with the inject / httpterm configuration from direct | Those bench start with the inject / httpterm configuration from direct | ||
| benches, with 270-290k connections/s between a client and a server. | benches, with 270-290k connections/s between a client and a server. | ||
| - | |||
| - | FIXME: graph not available yet. Will wait until the bench are over. | ||
| Monitoring graphs for the different benches can be found | Monitoring graphs for the different benches can be found | ||
| Line 26: | Line 24: | ||
| we have (approximate reading on graphs) : | we have (approximate reading on graphs) : | ||
| - | ^ what ^ per client ^ per server ^ total ^ | + | ^ what ^ per client ^ per server ^ total ^ graph ^ |
| - | ^ conn/s | 400k | 266k | 800k | | + | ^ conn/s | 400k | 266k | 800k | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/tcp_stats_conn_out.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/tcp_stats_conn_out.png|cli2]] | |
| - | ^ Gbps from cli/srv | 1.1/1.7 | 0.75/1.12 | 2.4/3.4 | | + | ^ Gbps from cli/srv | 1.1/1.7 | 0.75/1.12 | 2.4/3.4 | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/interfaces_eth1_bps.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/interfaces_eth1_bps.png|cli2]] | |
| - | ^ Mpkt/s from cli/srv | 1.2/1.62 | 0.8/1.08 | 2.4/3.24 | | + | ^ Mpkt/s from cli/srv | 1.2/1.62 | 0.8/1.08 | 2.4/3.24 | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/interfaces_eth1_pkt.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/interfaces_eth1_pkt.png|cli2]] | |
| Well, we can have about a little over 3Gbps, with 800k connections/s. | Well, we can have about a little over 3Gbps, with 800k connections/s. | ||
| Line 79: | Line 77: | ||
| earlier, which is explained by the fact both go in and out of our | earlier, which is explained by the fact both go in and out of our | ||
| gateway via the same interface. | gateway via the same interface. | ||
| + | |||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/01-baseline-gateway/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/01-baseline-gateway/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
| ===== no rules ===== | ===== no rules ===== | ||
| Line 93: | Line 94: | ||
| down from 5.7 to 4.9 (both, Gbps and M pkt/s). That's down from 800k to | down from 5.7 to 4.9 (both, Gbps and M pkt/s). That's down from 800k to | ||
| just under 700k conn/s. | just under 700k conn/s. | ||
| + | |||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/02-baseline-norule/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/02-baseline-norule/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
| + | |||
| ====== Firewall ====== | ====== Firewall ====== | ||
| Line 121: | Line 126: | ||
| seconds, then there is a drastic drop to an average of 135k for the rest | seconds, then there is a drastic drop to an average of 135k for the rest | ||
| of the time. | of the time. | ||
| + | |||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/03-firewall-conntrack/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/03-firewall-conntrack/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
| As the conntrack count increase in a linear form, up to about 21.2M, and | As the conntrack count increase in a linear form, up to about 21.2M, and | ||
| Line 143: | Line 151: | ||
| Testing with those values allowed us to get the break at 60 seconds. | Testing with those values allowed us to get the break at 60 seconds. | ||
| Connections gets in time_wait, and expires after 60s instead of 120s. | Connections gets in time_wait, and expires after 60s instead of 120s. | ||
| + | |||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/04-firewall-conntrack2/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/04-firewall-conntrack2/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
| + | |||
| Testing with nf_conntrack_tcp_timeout_time_wait set to 1s gives directly | Testing with nf_conntrack_tcp_timeout_time_wait set to 1s gives directly | ||
| the low performances, even if the conntrack stay under 200k, instead of | the low performances, even if the conntrack stay under 200k, instead of | ||
| a few millions. | a few millions. | ||
| + | |||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/05-firewall-conntrack3/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/05-firewall-conntrack3/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
| For our heavy connections, we clearly need to be able to *not* track | For our heavy connections, we clearly need to be able to *not* track | ||
| them. | them. | ||
| + | |||
| ===== notrack ===== | ===== notrack ===== | ||
| Line 172: | Line 188: | ||
| 4.2M pkt/s, total of about 590k conn/s instead of our 800k without | 4.2M pkt/s, total of about 590k conn/s instead of our 800k without | ||
| firewall. | firewall. | ||
| + | |||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/06-firewall-notrack/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/06-firewall-notrack/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
| + | |||
| Trying to get only one rule for notrack get un slightly better | Trying to get only one rule for notrack get un slightly better | ||
| Line 180: | Line 200: | ||
| That give us about 620k conn/s. | That give us about 620k conn/s. | ||
| + | |||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/07-firewall-notrack2/rempart-firewall/interfaces_eth1_bps.png|bps]] | ||
| + | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/07-firewall-notrack2/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | ||
| ===== simple rules ===== | ===== simple rules ===== | ||
| Line 196: | Line 219: | ||
| iptables -F ; for ((i=0;i<n;++i)) ; { iptables -A FORWARD -s 10.0.0.$i ; } | iptables -F ; for ((i=0;i<n;++i)) ; { iptables -A FORWARD -s 10.0.0.$i ; } | ||
| - | ^ match rules ^ conn/s ^ pkt/s ^ | + | ^ match rules ^ conn/s ^ pkt/s ^ graph ^ |
| - | | 0 | 800k | 5.7M | | + | | 0 | 800k | 5.7M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0000/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0000/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | 16 | 780k | 5.6M | | + | | 16 | 780k | 5.6M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0010/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0010/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | 64 | 730k | 5.1M | | + | | 64 | 730k | 5.1M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0040/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0040/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | 256 | 480k | 3.38M | | + | | 256 | 480k | 3.38M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0100/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0100/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | 1024 | 148k | 1.05M | | + | | 1024 | 148k | 1.05M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0400/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0400/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| Line 210: | Line 233: | ||
| Tests done with 256 match rules. | Tests done with 256 match rules. | ||
| - | ^ match rule ^ conn/s ^ pkt/s ^ | + | ^ match rule ^ conn/s ^ pkt/s ^ graph ^ |
| - | | -m u32 --u32 ""0xc&0xffffffff=0xa0000`printf %02x $i`" | 67k | 480k | | + | | -m u32 --u32 ""0xc&0xffffffff=0xa0000`printf %02x $i`" | 67k | 480k | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-u32-src/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-u32-src/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | -p udp -m udp --dport 53 | 315k | 2.4M | | + | | -p udp -m udp --dport 53 | 315k | 2.4M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-udp/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-udp/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | -p tcp -m tcp --dport 443 | 155k | 1.1M | | + | | -p tcp -m tcp --dport 443 | 155k | 1.1M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-https/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-https/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | -p tcp -m tcp --dport 80 (does match) | 140k | 990k | | + | | -p tcp -m tcp --dport 80 (does match) | 140k | 990k | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-http/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-http/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | -d 10.0.0.$i | 460k | 3.2M | | + | | -d 10.0.0.$i | 460k | 3.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-dst/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-dst/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| Different kind of matches have different kind of impact. -d or -s have | Different kind of matches have different kind of impact. -d or -s have | ||
| about the same impact. | about the same impact. | ||
| + | |||
| + | |||
| + | |||
| ===== other configs ===== | ===== other configs ===== | ||
| Line 225: | Line 251: | ||
| performances so far. | performances so far. | ||
| - | ^ match rules ^ conn/s ^ pkt/s ^ | + | ^ match rules ^ conn/s ^ pkt/s ^ graph ^ |
| - | | default | 480k | 3.38M | | + | | default | 480k | 3.38M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen1k/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen1k/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | ethtool -G eth1 {tx/rx} 512 | 505k | 3.6M | | + | | ethtool -G eth1 {tx/rx} 512 | 505k | 3.6M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool512/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool512/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | ethtool -G eth1 {tx/rx} 64 | 450k | 3.2M | | + | | ethtool -G eth1 {tx/rx} 64 | 450k | 3.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool64/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool64/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | ip link set eth1 txqueuelen 10000 | 470k | 3.3M | | + | | ip link set eth1 txqueuelen 10000 | 470k | 3.3M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen10k/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen10k/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| txqueuelen - no effect | txqueuelen - no effect | ||
| Line 310: | Line 336: | ||
| to the next entries. | to the next entries. | ||
| - | ^ bits matched per level ^ check ^ match ^ conn/s ^ pkt/s ^ | + | ^ bits matched per level ^ check ^ match ^ conn/s ^ pkt/s ^ graph ^ |
| - | | 2 | 39 | 11 | 560k | 3.9M | | + | | 2 | 39 | 11 | 560k | 3.9M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-2/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-2/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | 3 | 51 | 8 | 595k | 4.2M | | + | | 3 | 51 | 8 | 595k | 4.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-3/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-3/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | 4 | 73 | 6 | 580k | 4.0M | | + | | 4 | 73 | 6 | 580k | 4.0M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-4/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-4/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| - | | 5 | 113 | 5 | 575k | 4.0M | | + | | 5 | 113 | 5 | 575k | 4.0M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-5/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-5/rempart-firewall/interfaces_eth1_pkt.png|pkt]] | |
| Note: such high number of rules uses memory. Like 20GB+ of ram used. | Note: such high number of rules uses memory. Like 20GB+ of ram used. | ||
| Line 327: | Line 353: | ||
| Will have to seek for other solutions. | Will have to seek for other solutions. | ||
| + | |||
| + | ===== ipset ===== | ||
| + | |||
| + | Some people mentionned ipset. Lets bench that. | ||
| + | |||
| + | <code> | ||
| + | # lets create some sets we might use | ||
| + | ipset create ip hash:ip | ||
| + | ipset create net hash:net | ||
| + | ipset create ip,port hash:ip,port | ||
| + | ipset create net,port hash:net,port | ||
| + | </code> | ||
| + | |||
| + | Rules used for different tests : | ||
| + | <code> | ||
| + | -A FORWARD -m set --match-set ip src | ||
| + | -A FORWARD -m set --match-set net src | ||
| + | -A FORWARD -m set --match-set net,port src,src | ||
| + | -A FORWARD -m set --match-set ip,port src,dst | ||
| + | </code> | ||
| + | |||
| + | Lets see how a few match for hash:ip affects our traffic : | ||
| + | |||
| + | ^ # rules ^ conn/s ^ pkt/s ^ | ||
| + | | 1 | 570k | 3.6M | | ||
| + | | 2 | 340k | 2.05M | | ||
| + | | 3 | 240k | 1.45M | | ||
| + | | 4 | 184k | 1.1M | | ||
| + | |||
| + | Ok, so just a few ipset match affects us ALOT. What about other hashes ? | ||
| + | |||
| + | (tests done with 2 matches) | ||
| + | |||
| + | ^ ipset ^ conn/s ^ pkt/s ^ | ||
| + | | hash:ip | 340k | 2.05M | | ||
| + | | hash:net | 350k | 2.1M | | ||
| + | | hash:ip,port | 330k | 2M | | ||
| + | | hash:net,port | 330k | 2M | | ||
| + | |||
| + | Net or ip doesn't change much, and including the port is only a light overhead, | ||
| + | considering the overhead we already have. | ||
| + | |||
| + | What about ipset bitmasks ? | ||
| + | |||
| + | <code> | ||
| + | ipset create bip0 bitmap:ip range 10.136.0.0-10.136.255.255 | ||
| + | ipset create bip1 bitmap:ip range 10.140.0.0-10.140.255.255 | ||
| + | </code> | ||
| + | |||
| + | ^ # rules ^ conn/s ^ pkt/s ^ | ||
| + | | 2 | 550k | 3.5M | | ||
| + | | 4 | 320k | 1.9M | | ||
| + | |||
| + | |||
| + | Considering ipset is limited to 65k entries, and the results, I would advise | ||
| + | against using it, unless you really need the easy to manage set. | ||
| + | |||
| + | |||
| + | ===== interface irq affinity ===== | ||
| + | |||
| + | FIXME: add irq affinity matches with results | ||
| + | |||
| + | ====== Conclusion ====== | ||
| + | |||
| + | * Alot of matching reduce performances. | ||
| + | * u32 are costly | ||
| + | * if you can, try to match and segregate to different subchains, with like 8 to 16 match per chain (for src/dst match, maybe less with heavier match) | ||
| + | * irq affinity can change performances on high loads | ||
system/benches10gbps/firewall.1350044817.txt.gz · Last modified: 2012/10/12 12:26 by ze
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International
