ze's sandcastle

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
Like for firewall, those benches starts with inject/httpterm configuration
allowing us to get 800k connections/s (2 clients, 3 servers) without
loadbalancing them.

Monitoring graphs for the different benches can be found
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing|here]]. 

====== Baseline ======

===== configuration =====

Keepalive configuration :

<code>
global_defs {
   router_id rempart.nbs-system.com
}

virtual_server_group pool_test_0 {
        10.128.0.0 80
        10.128.0.1 80
        10.128.0.2 80
        10.128.0.3 80
        10.128.0.4 80
        10.128.0.5 80
        10.128.0.6 80
        10.128.0.7 80
        10.128.0.8 80
        10.128.0.9 80
        10.128.0.10 80
        10.128.0.11 80
        10.128.0.12 80
        10.128.0.13 80
        10.128.0.14 80
        10.128.0.15 80
}

virtual_server group pool_test_0 {
  delay_loop 5
  lb_algo rr
  lb_kind DR
  protocol TCP
  real_server 10.127.0.1 80 {
    weight 1
#    HTTP_GET {
#      url {
#        path /
#        status_code 200
#      }
#      connect_port 80
#      connect_timeout 20
#      nb_get_retry 1
#      delay_before_retry 0
#    }
  }
  real_server 10.127.0.2 80 {
    weight 1
#    HTTP_GET {
#      url {
#        path /
#        status_code 200
#      }
#      connect_port 80
#      connect_timeout 20
#      nb_get_retry 1
#      delay_before_retry 0
#    }
  }
  real_server 10.127.0.6 80 {
    weight 1
#    HTTP_GET {
#      url {
#        path /
#        status_code 200
#      }
#      connect_port 80
#      connect_timeout 20
#      nb_get_retry 1
#      delay_before_retry 0
#    }
  }
}

[...]

vrrp_instance INTERNAL {
  interface eth1
  virtual_router_id 1

  state MASTER
  priority 150

  advert_int 1

  garp_master_delay 5

  virtual_ipaddress_excluded {
        10.128.0.0/32 dev eth1
        10.128.0.1/32 dev eth1
        10.128.0.2/32 dev eth1
        10.128.0.3/32 dev eth1
        10.128.0.4/32 dev eth1
        10.128.0.5/32 dev eth1
        10.128.0.6/32 dev eth1
        10.128.0.7/32 dev eth1
        10.128.0.8/32 dev eth1
        10.128.0.9/32 dev eth1
        10.128.0.10/32 dev eth1
        10.128.0.11/32 dev eth1
        10.128.0.12/32 dev eth1
        10.128.0.13/32 dev eth1
        10.128.0.14/32 dev eth1
        10.128.0.15/32 dev eth1
  }
}
</code>

Checks are disabled for now, as we mainly want to check the ipvs impact
first.

16 VIP pointing to our 3 servers.


===== baseline =====

First attempt gives very poor results, a high start, but drops to about
40-45k conn/s.

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit12/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit12/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit12/rempart-firewall/ipvs_conn.png|conn per sec]]

===== ipvs_tab_bits =====

Like for conntrack, ipvs need to keep track of the connections. There is
also a hash table, and size can also be changed.

Default value (used in previous test) is 12. Documentation states the
maximum value is 20 (bits, with means 1M entries).

  # cat /etc/modprobe.d/ipvs.conf
  options ip_vs conn_tab_bits=20

That give us some better overall performances, with like 350k conn/s,
but there is some weird downfall some time in the beginning.

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20/rempart-firewall/ipvs_conn.png|conn per sec]]

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20b/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20b/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20b/rempart-firewall/ipvs_conn.png|conn per sec]]

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20c/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20c/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20c/rempart-firewall/ipvs_conn.png|conn per sec]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20c/rempart-firewall/ipvs_cur.png|ipvs connection tracking]]


===== smp affinity =====

The statistics shows that only the 16 first cpu are used.  Basic
affinity that was in place was a simple interrupt n to thread n.


With interrups mapped to cpu#0, core#0-3, and then getting on the other
thread, and back on the same (0-3, 12-15, 0-3, 12-15), all connections
get on the same 8 threads, but the performances is more stable. and gets
stable around 280k conn/s.

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-af
finity-a/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-af
finity-a/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-af
finity-a/rempart-firewall/ipvs_conn.png|conn per sec]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-af
finity-a/rempart-firewall/ipvs_cur.png|ipvs connection tracking]]


With interrupts mapped the same, but then to the other cpu (0-3, 12-15,
6-9, 18-21), we use more threads, and get better performances, stable
around 370k/s.

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-af
finity-b/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-af
finity-b/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-af
finity-b/rempart-firewall/ipvs_conn.png|conn per sec]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-af
finity-b/rempart-firewall/ipvs_cur.png|ipvs connection tracking]]

If we change cpu before changing thread (0-3, 6-9, 12-15, 18-21), the
performances are lightly better, around 380k/s.

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-af
finity-c/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-affinity-c/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-affinity-c/rempart-firewall/ipvs_conn.png|conn per sec]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit20-affinity-c/rempart-firewall/ipvs_cur.png|ipvs connection tracking]]

===== higher hash size =====

Increasing the hashsize (24 or 28) tends to increase the overall conn/s,
but with worse downfall at times, especially with 28.


with 24 bits:

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit24-affinity-c/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit24-affinity-c/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit24-affinity-c/rempart-firewall/ipvs_conn.png|conn per sec]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit24-affinity-c/rempart-firewall/ipvs_cur.png|ipvs connection tracking]]

with 28 bits (same test, twice) :

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit28-affinity-c-bis/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit28-affinity-c-bis/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit28-affinity-c-bis/rempart-firewall/ipvs_conn.png|conn per sec]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit28-affinity-c-bis/rempart-firewall/ipvs_cur.png|ipvs connection tracking]]

[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit28-affinity-c/rempart-firewall/interfaces_eth1_bps.png|bps]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit28-affinity-c/rempart-firewall/interfaces_eth1_pkt.png|pkt]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit28-affinity-c/rempart-firewall/ipvs_conn.png|conn per sec]]
[[http://www.hagtheil.net/files/system/benches10gbps/loadbalancing/ipvs-bit28-affinity-c/rempart-firewall/ipvs_cur.png|ipvs connection tracking]]

====== Conclusion ======

  * As always, affinity helps. In this case, not only with performance, but with stability.
  * if you have alot of connection to balance, increase the bits to at least 20. It doesn't take much memory, and it helps alot.
ze's sandcastle

User Tools

Site Tools

Page Tools
