Differences

This shows you the differences between two versions of the page.

--- system:benches10gbps:firewall [2012/10/09 09:46]
ze
+++ system:benches10gbps:firewall [2012/11/02 15:41] (current)
ze ipset
@@ Line 2: / Line 2: @@
 Those bench start with the inject / httpterm configuration from direct
 benches, with 270-290k connections/s between a client and a server.
-FIXME: graph not available yet. Will wait until the bench are over.
 Monitoring graphs for the different benches can be found
@@ Line 26: / Line 24: @@
 we have (approximate reading on graphs) :
-^  what  ^  per client  ^  per server  ^  total  ^
+^  what  ^  per client  ^  per server  ^  total  ^  graph  ^
-^ conn/s |  400k  |  266k  |  800k  |
+^ conn/s |  400k  |  266k  |  800k  |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/tcp_stats_conn_out.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/tcp_stats_conn_out.png|cli2]]  |
-^ Gbps from cli/srv |  1.1/1.7  | 0.75/1.12 |  2.4/3.4  |
+^ Gbps from cli/srv |  1.1/1.7  | 0.75/1.12 |  2.4/3.4  |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/interfaces_eth1_bps.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/interfaces_eth1_bps.png|cli2]]  |
-^ Mpkt/s from cli/srv |  1.2/1.62  | 0.8/1.08 |  2.4/3.24  |
+^ Mpkt/s from cli/srv |  1.2/1.62  | 0.8/1.08 |  2.4/3.24  |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/douves-client/interfaces_eth1_pkt.png|cli1]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/00-baseline-direct/muraille-client/interfaces_eth1_pkt.png|cli2]]  |
 Well, we can have about a little over 3Gbps, with 800k connections/s.
@@ Line 57: / Line 55: @@
   net.ipv4.conf.eth1.accept_redirects = 0
   net.ipv4.conf.eth1.send_redirects = 0
+And something "heard" as being a good idea (checked in later bench) :
+  gateway#/etc/rc.local
+  ethtool -G eth1 rx 4096
+  ethtool -G eth1 tx 4096
 As we don't have any process that will be running here, and only the
@@ Line 72: / Line 77: @@
 earlier, which is explained by the fact both go in and out of our
 gateway via the same interface.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/01-baseline-gateway/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/01-baseline-gateway/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 ===== no rules =====
@@ Line 86: / Line 94: @@
 down from 5.7 to 4.9 (both, Gbps and M pkt/s). That's down from 800k to
 just under 700k conn/s.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/02-baseline-norule/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/02-baseline-norule/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 ====== Firewall ======
@@ Line 114: / Line 126: @@
 seconds, then there is a drastic drop to an average of 135k for the rest
 of the time.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/03-firewall-conntrack/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/03-firewall-conntrack/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 As the conntrack count increase in a linear form, up to about 21.2M, and
@@ Line 136: / Line 151: @@
 Testing with those values allowed us to get the break at 60 seconds.
 Connections gets in time_wait, and expires after 60s instead of 120s.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/04-firewall-conntrack2/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/04-firewall-conntrack2/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 Testing with nf_conntrack_tcp_timeout_time_wait set to 1s gives directly
 the low performances, even if the conntrack stay under 200k, instead of
 a few millions.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/05-firewall-conntrack3/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/05-firewall-conntrack3/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 For our heavy connections, we clearly need to be able to *not* track
 them.
 ===== notrack =====
@@ Line 165: / Line 188: @@
 .2M pkt/s, total of about 590k conn/s instead of our 800k without
 firewall.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/06-firewall-notrack/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/06-firewall-notrack/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 Trying to get only one rule for notrack get un slightly better
@@ Line 173: / Line 200: @@
 That give us about 620k conn/s.
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/07-firewall-notrack2/rempart-firewall/interfaces_eth1_bps.png|bps]]
+[[http://www.hagtheil.net/files/system/benches10gbps/firewall/07-firewall-notrack2/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
 ===== simple rules =====
@@ Line 185: / Line 215: @@
 considering.
-^  match rules  ^  conn/s  ^  pkt/s  ^
+  # with n being the matching rules...
-|  0 |  800k |  5.7M |
+  n=64
-|  16 |  780k |  5.6M |
+  iptables -F ; for ((i=0;i<n;++i)) ; { iptables -A FORWARD -s 10.0.0.$i ; }
-|  64 |  730k |  5.1M |
-|  256 |  480k |  3.38M |
+^  match rules  ^  conn/s  ^  pkt/s  ^  graph  ^
+|  0 |  800k |  5.7M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0000/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0000/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+|  16 |  780k |  5.6M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0010/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0010/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+|  64 |  730k |  5.1M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0040/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0040/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+|  256 |  480k |  3.38M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0100/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0100/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+|  1024 |  148k | 1.05M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0400/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/08-firewall-simple-rules_0400/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+===== other matches =====
+Source match has an impact on the requests. Lets check other kind of match.
+Tests done with 256 match rules.
+^  match rule  ^  conn/s  ^  pkt/s  ^  graph  ^
+| -m u32 --u32 ""0xc&0xffffffff=0xa0000`printf %02x $i`" |  67k |  480k |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-u32-src/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-u32-src/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+| -p udp -m udp --dport 53 |  315k |  2.4M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-udp/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-udp/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+| -p tcp -m tcp --dport 443 |  155k |  1.1M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-https/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-https/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+| -p tcp -m tcp --dport 80 (does match) |  140k |  990k |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-http/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-tcp-http/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+| -d 10.0.0.$i |  460k |  3.2M |  [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-dst/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/09-firewall-rule-dst/rempart-firewall/interfaces_eth1_pkt.png|pkt]]  |
+Different kind of matches have different kind of impact. -d or -s have
+about the same impact.
+===== other configs =====
+Tests done with 256 -s xxx matches, as it's the one that gave the best
+performances so far.
+^  match rules  ^  conn/s  ^  pkt/s  ^  graph  ^
+| default |  480k |  3.38M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen1k/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen1k/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
+| ethtool -G eth1 {tx/rx} 512 |  505k |  3.6M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool512/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool512/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
+| ethtool -G eth1 {tx/rx} 64 |  450k |  3.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool64/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-ethtool64/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
+| ip link set eth1 txqueuelen 10000 |  470k |  3.3M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen10k/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/10-firewall-txqueuelen10k/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
+txqueuelen - no effect
+ring parameters rx/tx, do have an effect, and neither too big nor too
+small is the best.
+===== tree search =====
+netfilter allow to use chains, with a few match, you can send to a
+chain, and avoid doing what is in the chain if it does not match.
+As seen previously, having alot of match in a single chain means that
+the packet is tested for all possible match.
+Lets see how we could have per ip match for a whole /13. (yeah, that
+mean 512k different IPs)
+Using iptables to generate that many entry is just too slow, it would
+take days. generating a result, and using iptables-restore can provide
+way better performances (5-10 minutes, instead of days).
+Having 512k rules to match for each packet would slow down the trafic
+alot. A way to do it would is to get a few matches, and send to a
+specific rule. Then match a few more bits, and send to other rules.
+Ideas for that Jesper Dangaard Brouer slides about
+[[http://www.slideshare.net/brouer/netfilter-making-large-iptables-rulesets-scale|Making large iptables rulesets scale]].
+Unfortunatly, the perl library has not been updated to build with wheezy
+iptables version yet.
+Exemple of rules that would be checked/matched for 10.139.5.43 :
+<code>
+-A FORWARD -s 10.128.0.0/12 -j cidr_12_176160768 (match)
+ -A cidr_12_176160768 -s 10.136.0.0/14 -j cidr_14_176685056 (match)
+  -A cidr_14_176685056 -s 10.136.0.0/16 -j cidr_16_176685056
+  -A cidr_14_176685056 -s 10.137.0.0/16 -j cidr_16_176750592
+  -A cidr_14_176685056 -s 10.138.0.0/16 -j cidr_16_176816128
+  -A cidr_14_176685056 -s 10.139.0.0/16 -j cidr_16_176881664 (match)
+   -A cidr_16_176881664 -s 10.139.0.0/18 -j cidr_18_176881664 (match)
+    -A cidr_18_176881664 -s 10.139.0.0/20 -j cidr_20_176881664 (match)
+     -A cidr_20_176881664 -s 10.139.0.0/22 -j cidr_22_176881664
+     -A cidr_20_176881664 -s 10.139.4.0/22 -j cidr_22_176882688 (match)
+      -A cidr_22_176882688 -s 10.139.4.0/24 -j cidr_24_176882688
+      -A cidr_22_176882688 -s 10.139.5.0/24 -j cidr_24_176882944 (match)
+       -A cidr_24_176882944 -s 10.139.5.0/26 -j cidr_26_176882944 (match)
+        -A cidr_26_176882944 -s 10.139.5.0/28 -j cidr_28_176882944
+        -A cidr_26_176882944 -s 10.139.5.16/28 -j cidr_28_176882960
+        -A cidr_26_176882944 -s 10.139.5.32/28 -j cidr_28_176882976 (match)
+         -A cidr_28_176882976 -s 10.139.5.32/30 -j cidr_30_176882976
+         -A cidr_28_176882976 -s 10.139.5.36/30 -j cidr_30_176882980
+         -A cidr_28_176882976 -s 10.139.5.40/30 -j cidr_30_176882984 (match)
+          -A cidr_30_176882984 -s 10.139.5.40/32 -j cidr_32_176882984
+          -A cidr_30_176882984 -s 10.139.5.41/32 -j cidr_32_176882985
+          -A cidr_30_176882984 -s 10.139.5.42/32 -j cidr_32_176882986
+          -A cidr_30_176882984 -s 10.139.5.43/32 -j cidr_32_176882987 (match)
+           -A cidr_32_176882987 ...
+         -A cidr_28_176882976 -s 10.139.5.44/30 -j cidr_30_176882988
+        -A cidr_26_176882944 -s 10.139.5.48/28 -j cidr_28_176882992
+       -A cidr_24_176882944 -s 10.139.5.64/26 -j cidr_26_176883008
+       -A cidr_24_176882944 -s 10.139.5.128/26 -j cidr_26_176883072
+       -A cidr_24_176882944 -s 10.139.5.192/26 -j cidr_26_176883136
+      -A cidr_22_176882688 -s 10.139.6.0/24 -j cidr_24_176883200
+      -A cidr_22_176882688 -s 10.139.7.0/24 -j cidr_24_176883456
+     -A cidr_20_176881664 -s 10.139.8.0/22 -j cidr_22_176883712
+     -A cidr_20_176881664 -s 10.139.12.0/22 -j cidr_22_176884736
+    -A cidr_18_176881664 -s 10.139.16.0/20 -j cidr_20_176885760
+    -A cidr_18_176881664 -s 10.139.32.0/20 -j cidr_20_176889856
+    -A cidr_18_176881664 -s 10.139.48.0/20 -j cidr_20_176893952
+   -A cidr_16_176881664 -s 10.139.64.0/18 -j cidr_18_176898048
+   -A cidr_16_176881664 -s 10.139.128.0/18 -j cidr_18_176914432
+   -A cidr_16_176881664 -s 10.139.192.0/18 -j cidr_18_176930816
+ -A cidr_12_176160768 -s 10.140.0.0/14 -j cidr_14_176947200
+</code>
+With at most 39 check, and 11 jump, any IP within the /13 arrives it its
+own chain (or merged chained, if you have serveral IP that need the same
+rules). Anything not even in the /12 would get just one check, and get
+to the next entries.
+^  bits matched per level  ^  check  ^  match  ^  conn/s  ^  pkt/s  ^  graph  ^
+|  2 |  39 |  11 |  560k |  3.9M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-2/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-2/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
+|  3 |  51 |  8 |  595k |  4.2M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-3/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-3/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
+|  4 |  73 |  6 |  580k |  4.0M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-4/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-4/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
+|  5 |  113 |  5 |  575k |  4.0M | [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-5/rempart-firewall/interfaces_eth1_bps.png|bps]] [[http://www.hagtheil.net/files/system/benches10gbps/firewall/11-fw-sourcetree-5/rempart-firewall/interfaces_eth1_pkt.png|pkt]] |
+Note: such high number of rules uses memory. Like 20GB+ of ram used.
+===== nat =====
+Earlier, we already noticed that conntracking all our connections would
+be too much. What if we can have a main 1:1 mapping that would not
+require any tracking ?
+Well, iptables NOTRACK prevent any form of nat, so that can't be done...
+Will have to seek for other solutions.
+===== ipset =====
+Some people mentionned ipset. Lets bench that.
+<code>
+# lets create some sets we might use
+ipset create ip hash:ip
+ipset create net hash:net
+ipset create ip,port hash:ip,port
+ipset create net,port hash:net,port
+</code>
+Rules used for different tests :
+<code>
+-A FORWARD -m set --match-set ip src
+-A FORWARD -m set --match-set net src
+-A FORWARD -m set --match-set net,port src,src
+-A FORWARD -m set --match-set ip,port src,dst
+</code>
+Lets see how a few match for hash:ip affects our traffic :
+^  # rules  ^  conn/s  ^  pkt/s  ^
+|  1 |  570k |  3.6M |
+|  2 |  340k |  2.05M |
+|  3 |  240k |  1.45M |
+|  4 |  184k |  1.1M |
+Ok, so just a few ipset match affects us ALOT. What about other hashes ?
+(tests done with 2 matches)
+^  ipset  ^  conn/s  ^  pkt/s  ^
+| hash:ip |  340k |  2.05M |
+| hash:net |  350k |  2.1M |
+| hash:ip,port |  330k |  2M |
+| hash:net,port |  330k |  2M |
+Net or ip doesn't change much, and including the port is only a light overhead,
+considering the overhead we already have.
+What about ipset bitmasks ?
+<code>
+ipset create bip0 bitmap:ip range 10.136.0.0-10.136.255.255
+ipset create bip1 bitmap:ip range 10.140.0.0-10.140.255.255
+</code>
+^  # rules  ^  conn/s  ^  pkt/s  ^
+|  2 |  550k |  3.5M |
+|  4 |  320k |  1.9M |
+Considering ipset is limited to 65k entries, and the results, I would advise
+against using it, unless you really need the easy to manage set.
+===== interface irq affinity =====
+FIXME: add irq affinity matches with results
+====== Conclusion ======
+  * Alot of matching reduce performances.
+  * u32 are costly
+  * if you can, try to match and segregate to different subchains, with like 8 to 16 match per chain (for src/dst match, maybe less with heavier match)
+  * irq affinity can change performances on high loads

ze's sandcastle

User Tools

Site Tools

Differences

Page Tools