Differences

This shows you the differences between two versions of the page.

--- system:benches10gbps:firewall [2012/10/11 18:10]
ze
+++ system:benches10gbps:firewall [2012/11/02 15:41] (current)
ze ipset
@@ Line 2: / Line 2: @@
 Those bench start with the inject / httpterm configuration from direct
 benches, with 270-290k connections/s between a client and a server.
-FIXME: graph not available yet. Will wait until the bench are over.
 Monitoring graphs for the different benches can be found
@@ Line 26: / Line 24: @@
 we have (approximate reading on graphs) :
-^  what  ^  per client  ^  per server  ^  total  ^
+^  what  ^  per client  ^  per server  ^  total  ^  graph  ^
-^ conn/s |  400k  |  266k  |  800k  |
+^ conn/s |  400k  |  266k  |  800k  |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/tcp_stats_conn_out.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/tcp_stats_conn_out.png|cli2]]  |
-^ Gbps from cli/srv |  1.1/1.7  | 0.75/1.12 |  2.4/3.4  |
+^ Gbps from cli/srv |  1.1/1.7  | 0.75/1.12 |  2.4/3.4  |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/interfaces_eth1_bps.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/interfaces_eth1_bps.png|cli2]]  |
-^ Mpkt/s from cli/srv |  1.2/1.62  | 0.8/1.08 |  2.4/3.24  |
+^ Mpkt/s from cli/srv |  1.2/1.62  | 0.8/1.08 |  2.4/3.24  |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/interfaces_eth1_pkt.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/interfaces_eth1_pkt.png|cli2]]  |
 Well, we can have about a little over 3Gbps, with 800k connections/s.
@@ Line 79: / Line 77: @@
 earlier, which is explained by the fact both go in and out of our
 gateway via the same interface.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/01-baseline-gateway/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/01-baseline-gateway/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 ===== no rules =====
@@ Line 93: / Line 94: @@
 down from 5.7 to 4.9 (both, Gbps and M pkt/s). That's down from 800k to
 just under 700k conn/s.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/02-baseline-norule/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/02-baseline-norule/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 ====== Firewall ======
@@ Line 121: / Line 126: @@
 seconds, then there is a drastic drop to an average of 135k for the rest
 of the time.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/03-firewall-conntrack/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/03-firewall-conntrack/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 As the conntrack count increase in a linear form, up to about 21.2M, and
@@ Line 143: / Line 151: @@
 Testing with those values allowed us to get the break at 60 seconds.
 Connections gets in time_wait, and expires after 60s instead of 120s.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/04-firewall-conntrack2/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/04-firewall-conntrack2/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 Testing with nf_conntrack_tcp_timeout_time_wait set to 1s gives directly
 the low performances, even if the conntrack stay under 200k, instead of
 a few millions.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/05-firewall-conntrack3/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/05-firewall-conntrack3/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 For our heavy connections, we clearly need to be able to *not* track
 them.
 ===== notrack =====
@@ Line 172: / Line 188: @@
 .2M pkt/s, total of about 590k conn/s instead of our 800k without
 firewall.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/06-firewall-notrack/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/06-firewall-notrack/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 Trying to get only one rule for notrack get un slightly better
@@ Line 180: / Line 200: @@
 That give us about 620k conn/s.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/07-firewall-notrack2/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/07-firewall-notrack2/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 ===== simple rules =====
@@ Line 196: / Line 219: @@
   iptables -F ; for ((i=0;i<n;++i)) ; { iptables -A FORWARD -s 10.0.0.$i ; }
-^  match rules  ^  conn/s  ^  pkt/s  ^
+^  match rules  ^  conn/s  ^  pkt/s  ^  graph  ^
-|  0 |  800k |  5.7M |
+|  0 |  800k |  5.7M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0000/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0000/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
-|  16 |  780k |  5.6M |
+|  16 |  780k |  5.6M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0010/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0010/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
-|  64 |  730k |  5.1M |
+|  64 |  730k |  5.1M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0040/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0040/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
-|  256 |  480k |  3.38M |
+|  256 |  480k |  3.38M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0100/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0100/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
-|  1024 |  148k | 1.05M |
+|  1024 |  148k | 1.05M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0400/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0400/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
@@ Line 210: / Line 233: @@
 Tests done with 256 match rules.
-^  match rule  ^  conn/s  ^  pkt/s  ^
+^  match rule  ^  conn/s  ^  pkt/s  ^  graph  ^
-| -m u32 --u32 ""0xc&0xffffffff=0xa0000`printf %02x $i`" |  67k |  480k |
+| -m u32 --u32 ""0xc&0xffffffff=0xa0000`printf %02x $i`" |  67k |  480k |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-u32-src/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-u32-src/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
-| -p udp -m udp --dport 53 |  315k |  2.4M |
+| -p udp -m udp --dport 53 |  315k |  2.4M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-udp/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-udp/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
-| -p tcp -m tcp --dport 443 |  155k |  1.1M |
+| -p tcp -m tcp --dport 443 |  155k |  1.1M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-https/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-https/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
-| -p tcp -m tcp --dport 80 (does match) |  140k |  990k |
+| -p tcp -m tcp --dport 80 (does match) |  140k |  990k |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-http/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-http/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
-| -d 10.0.0.$i |  460k |  3.2M |
+| -d 10.0.0.$i |  460k |  3.2M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-dst/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-dst/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
 Different kind of matches have different kind of impact. -d or -s have
 about the same impact.
 ===== other configs =====
@@ Line 225: / Line 251: @@
 performances so far.
-^  match rules  ^  conn/s  ^  pkt/s  ^
+^  match rules  ^  conn/s  ^  pkt/s  ^  graph  ^
-| default |  480k |  3.38M |
+| default |  480k |  3.38M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen1k/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen1k/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
-| ethtool -G eth1 {tx/rx} 512 |  505k |  3.6M |
+| ethtool -G eth1 {tx/rx} 512 |  505k |  3.6M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool512/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool512/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
-| ethtool -G eth1 {tx/rx} 64 |  450k |  3.2M |
+| ethtool -G eth1 {tx/rx} 64 |  450k |  3.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool64/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool64/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
-| ip link set eth1 txqueuelen 10000 |  470k |  3.3M |
+| ip link set eth1 txqueuelen 10000 |  470k |  3.3M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen10k/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen10k/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
 txqueuelen - no effect
@@ Line 310: / Line 336: @@
 to the next entries.
-^  bits matched per level  ^  check  ^  match  ^  conn/s  ^  pkt/s  ^
+^  bits matched per level  ^  check  ^  match  ^  conn/s  ^  pkt/s  ^  graph  ^
-|  2 |  39 |  11 |  560k |  3.9M |
+|  2 |  39 |  11 |  560k |  3.9M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-2/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-2/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
-|  3 |  51 |  8 |  595k |  4.2M |
+|  3 |  51 |  8 |  595k |  4.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-3/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-3/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
-|  4 |  73 |  6 |  580k |  4.0M |
+|  4 |  73 |  6 |  580k |  4.0M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-4/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-4/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
-|  5 |  113 |  5 |  575k |  4.0M |
+|  5 |  113 |  5 |  575k |  4.0M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-5/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-5/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
+Note: such high number of rules uses memory. Like 20GB+ of ram used.
+===== nat =====
+Earlier, we already noticed that conntracking all our connections would
+be too much. What if we can have a main 1:1 mapping that would not
+require any tracking ?
+Well, iptables NOTRACK prevent any form of nat, so that can't be done...
+Will have to seek for other solutions.
+===== ipset =====
+Some people mentionned ipset. Lets bench that.
+<code>
+# lets create some sets we might use
+ipset create ip hash:ip
+ipset create net hash:net
+ipset create ip,port hash:ip,port
+ipset create net,port hash:net,port
+</code>
+Rules used for different tests :
+<code>
+-A FORWARD -m set --match-set ip src
+-A FORWARD -m set --match-set net src
+-A FORWARD -m set --match-set net,port src,src
+-A FORWARD -m set --match-set ip,port src,dst
+</code>
+Lets see how a few match for hash:ip affects our traffic :
+^  # rules  ^  conn/s  ^  pkt/s  ^
+|  1 |  570k |  3.6M |
+|  2 |  340k |  2.05M |
+|  3 |  240k |  1.45M |
+|  4 |  184k |  1.1M |
+Ok, so just a few ipset match affects us ALOT. What about other hashes ?
+(tests done with 2 matches)
+^  ipset  ^  conn/s  ^  pkt/s  ^
+| hash:ip |  340k |  2.05M |
+| hash:net |  350k |  2.1M |
+| hash:ip,port |  330k |  2M |
+| hash:net,port |  330k |  2M |
+Net or ip doesn't change much, and including the port is only a light overhead,
+considering the overhead we already have.
+What about ipset bitmasks ?
+<code>
+ipset create bip0 bitmap:ip range 10.136.0.0-10.136.255.255
+ipset create bip1 bitmap:ip range 10.140.0.0-10.140.255.255
+</code>
+^  # rules  ^  conn/s  ^  pkt/s  ^
+|  2 |  550k |  3.5M |
+|  4 |  320k |  1.9M |
+Considering ipset is limited to 65k entries, and the results, I would advise
+against using it, unless you really need the easy to manage set.
+===== interface irq affinity =====
+FIXME: add irq affinity matches with results
+====== Conclusion ======
+  * Alot of matching reduce performances.
+  * u32 are costly
+  * if you can, try to match and segregate to different subchains, with like 8 to 16 match per chain (for src/dst match, maybe less with heavier match)
+  * irq affinity can change performances on high loads

ze's sandcastle

User Tools

Site Tools

Differences

Page Tools