Load balancing

From securityrouter.org, an OpenBSD-based firewall
Jump to: navigation, search

The load balancer can dynamically redirect and route traffic. Halon Security's implementation is based on, and slightly extends, OpenBSD's relayd which makes relayd.conf's manual page[1] a great source of information.

Introduction

The load balancer can operate as

  • Load balancer
  • Application layer gateway
  • SSL accelerator
  • Transparent proxy
  • Internet failover

It's most commonly used to forward traffic to multiple servers with load distribution and health checking. This functionality can, with some generalization, be divided into layer 3 (called redirects) and layer 4+ (called relays). They are configured in much the same way, but have some striking technical differences. Both the redirects and relays are sometimes referred to as "virtual servers" in other products.

Check and distribution methods

The following methods can be used to check the health of a host.

Syntax Comment
http ... code HTTP(S) response code
http ... digest HTTP(S) response content digest
send Raw data response to configurable request (possibly using SSL)
script Shell or Perl script placed in the /cfg/relayd/ folder
ssl Complete SSL handshake on the forwarded port
tcp Complete TCP handshake on the forwarded port
icmp Ping the host

Traffic is distributed over the active hosts using one of the following methods. Persistence is sometimes provided by the distribution method, and otherwise (in case of for example least states) by the source tracking mechanism (if sticky address is enabled).

Syntax Supported by Comment
loadbalance Relays Source IP address of the client, and the IP address and port of the relay
source-hash Relays Source IP address of the client
hash Relays Input fed from the protocol specification, for example HTTP headers and GET variables
least-states Redirects The host with the fewest active connections (firewall states)
roundrobin Relays, Redirects Distributes connections using round robin to all active hosts
random Relays Distributes connections randomly to all active hosts

Layer 3 (redirects)

Layer 3 load balancing is implemented as a firewall port forward, which is very efficient, and therefore gives layer 3 load balancing a performance advantage over layer 4+. Because of its implementation, it requires the firewall to be started (simply check if there are any firewall rules in the configuration). Also because if this, it's normally not necessary to add any additional firewall rules. Further, it's necessary that the load balancer is the default route (gateway) of the servers being load balanced, since the source packets of the incoming traffic is maintained. In order words, the topology would look something like:

Internet -> Halon -> Switch -> Server1
                            -> Server2
                            -> Server3                

To add a layer 3 load balancer, follow these steps

  1. Check that the firewall is enabled by looking for a firewall statement in the configuration, or rules on the Network > Firewall page (if not, add a rule)
  2. Go to the Network > Load balancer page, click the "Pools (tables)" button and then click "Add pool"
  3. Give the pool a name, type an IP address in the table's node column (Add multiple hosts by pressing the + icon to the right in the table header) and finally click "Save"
  4. If you want, you can click the "Node (hosts)" button and give names to the IP addresses you just added and then click "Save"
  5. Click the "Relays (servers)" button, click "Add..." and select "Redirect (layer 3)"
  6. Give the redirect a name and add one or more listen addresses (the external virtual IP) and listen ports (By pressing the + icon to the right in the table header)
  7. Add the forward pool you created in step 3. It's possible to specifiy more than one forward pool (by adding an additional row). The second row's nodes will be used as backup if all nodes in the main forward's pool are down.
  8. When you're done, apply the changes by clicking "Save" and then "Deploy working copy".

Filtering traffic

If you want to create your own firewall rules for a particular redirect (for example, if you wish to block a certain IP from accessing the redirect) you'll need to enter a "Redirect firewall tag" and enable the "Redirect match-only" checkbox in the settings for the redirect. Please keep in mind that if you enable this setting you most likely have to add rules on your own allowing traffic to the redirect.

Redirect sticky address

Some protocols need sticky-address in order to load-balance properly, like FTP where the control- (21) and data-connection must end up on the same backend server, or web-sessions which doesn't have a shared session store. The load balancer uses the firewalls source tracking cache to keep track of previous redirects from a particular source address (IP). This cache has two parameters, one of which you may need to change primarily for web applications.

In the configuration editor, you may set these values in the firewall { section.

set timeout src.track X
set limit src-nodes Y

src.track defines the number of seconds a src-track entry is stored in memory after the last active state has expired. By default this feature is not used. For protocol like FTP this is exactly what you want. However HTTP may suffer from bad behavior if this value doesn't match you web applications session timeout (eg. PHP session timeout). Do not set this value higher than what is necessary.

src-nodes defines how many src-track (nodes) entries are stored in memory. By default 10000. This value should not be changed if you do not hit this limitation.

Caveats

Source tracking is stored per-rule. Thus in the load-balancer each redirect has it's own source tracking. So in order to load-balance FTP both listen on has to be added to the same redirect.

load-balancer {
    table <ftp_servers> { 10.2.0.30 10.2.0.31 }
    redirect "FTP" {
         listen on 1.2.3.4 port 21
         listen on 1.2.3.4 port 5000:5100
         forward to <ftp_servers> check tcp
         sticky-address
    }
}

Layer 4+ (relays)

Unlike redirects, relays doesn't require the firewall to be enabled, nor that the load balancer is the servers' default route (gateway). In fact, the most common scenario is to perform layer 4 load balancing with the firewall disabled (removing the firewall configuration) and to not use the load balancer as a router. Please keep in mind that if the firewall is enabled (simply check if there are any firewall rules in the configuration), you most likely have to add rules on your own allowing traffic to the relay. The topology would look something like:

Internet -> Router -> Switch <-> Halon
                              -> Server1
                              -> Server2         

Note that unlike redirects, you can only specify one listen port for each relay you create.

To add a layer 4+ load balancer, follow these steps

  1. Check if the firewall is enabled by looking for a firewall statement in the configuration, or rules on the Network > Firewall page (if it is, add rules for the relay)
  2. Go to the Network > Load balancer page, click the "Pools (tables)" button and then click "Add pool"
  3. Give the pool a name, type an IP address in the table's node column (Add multiple hosts by pressing the + icon to the right in the table header) and finally click "Save"
  4. If you want, you can click the "Node (hosts)" button and give names to the IP addresses you just added and then click "Save"
  5. Click the "Relays (servers)" button, click "Add..." and select "Relay (layer 4+)"
  6. Give the relay a name and add one listen address (the external virtual IP) and set one listen port
  7. Add the forward pool you created in step 3. It's possible to specifiy more than one forward pool (by adding an additional row). The second row's nodes will be used as backup if all nodes in the main forward's pool are down.
  8. When you're done, apply the changes by clicking "Save" and then "Deploy working copy".

Configuration examples

Below are a few clear-text configuration examples. The load balancer is implemented using OpenBSD's relayd and therefore its manual page[2] is useful for clear-text configuration.

HTTPS (SSL) acceleration

This very simple example provides an HTTPS accelerator. If you are using the 64-bit version (amd64) on a router with AES-NI instructions, you can expect gigabit performance. Below is a more or less complete example, using the router exclusively as a layer 7 load balancer, utilizing only one Ethernet interface.

interface em0 {
	address 192.168.0.100/24
	route default 192.168.0.1
}
load-balancer {
	table <servers> { 192.168.0.101 192.168.0.102 }
	relay "webservers" {
		listen on 192.168.0.100 port 443 ssl
		forward to <servers> port 80 check tcp
	}
}
system {
	http-server {
		port 4433
	}
	authentication {
		root-password "extremelyhardpassword"
		user "admin" {
			password "veryhardpassword"
		}
	}
	dns {
		name-server 8.8.8.8
	}
}

Then, upload the certificate and private key. Currently, these are not in the configuration file. Instead, enable root access (already enabled by the above example) and upload the file using for example scp according to the skeleton files guidelines. You can also try out the load balancer by using the web administration's self-signed certificate, by issuing the following commands when logged in as root:

# cp /etc/ssl/server.crt /cfg/skel/ssl/192.168.0.100.crt
# cp /etc/ssl/private/server.key /cfg/skel/ssl/private/192.168.0.100.key

Internet failover

The load balancer can be used to select one of several default routers (gateways) which is useful for outbound internet failover when more sophisticated protocols such as BGP is unavailable. Below is an incomplete example of such a configuration using one main gateway and one fallback gateway. Note the different priorities used for the gateways in this example as these are the lowest and highest priorities that you can assign to individual hosts (gateways in this case) in the load balancer.

load-balancer {
	main_gateway = 212.37.18.193
	fallback_gateway = 213.12.48.1
	table <gateways> { $main_gateway retry 2 ip ttl 1 priority 8 $fallback_gateway ip ttl 1 priority 52 }
	router "internetfailover" {
		forward to <gateways> check icmp
		route 0.0.0.0/0
	}
}

Note that this function does not require a load-balancer license.

SSL stripping

If you would like to strip the SSL from a service that is only available over SSL (eg. the web administration, even though it's not recommended nor good practice), this example shows how to make the web administration available for unsecure HTTP connections.

load-balancer {
	relay "webui" {
		listen on 0.0.0.0 port 80
		forward with tls to 127.0.0.1 port 443
	}
}

Be aware that some web services send "secure" cookies (Set-Cookie), that needs to be disabled on the backend web server. This is also true for our built-in web server. Add this to your skeleton files rc.local for the example above to work.

#!/bin/sh
perl -pi -e 's/session.cookie_secure=True/session.cookie_secure=False/g' /etc/php-5.3.ini
pkill -9 httpd

Microsoft Exchange

We have a separate article on Load balancing Microsoft Exchange.

SNMP traps

The preferred way of creating notifications is using SNMP traps; notifying all trap receivers about hostUp and hostDown event. Configure a trap receiver on the System > SNMP page, and then enable SNMP traps on the Network > Load balancer page. In raw configuration, this equals to

load-balanacer {
   send traps
   ...
}
system {
   snmp-server {
      trap receiver X.X.X.X
   }
   ...
}
...

The relayd daemon sends a enterprise.30155.3.1 (relaydStateChange), with descriptive object values. Please note that this MIB file's layout is not final and may be revised in the future. The MIB file is available from the Network > Load balancer page.

+--openBSD(30155)
   |
   +--relaydMIBObjects(3)
      |
      +--relaydStateChange(1)
         +-- -R-- String    hostName(1)
         +-- -R-- EnumVal   hostStatus(2)
         |        Values: hostDown(-1), hostUnknown(0), hostUp(1)
         +-- -R-- EnumVal   hostLastStatus(3)
         |        Values: hostDown(-1), hostUnknown(0), hostUp(1)
         +-- -R-- INTEGER   checksSuccessful(4)
         +-- -R-- INTEGER   checks(5)
         +-- -R-- String    tableName(6)
         +-- -R-- INTEGER   tableHosts(7)
         +-- -R-- INTEGER   hostRetriesLeft(8)
         +-- -R-- INTEGER   hostRetries(9)

Net-SNMP guide

This short guide describes how to set up a trap receiver with Net-SNMP[3] and run a custom script when a load balancer host is up/down. Net-SNMP works on many platforms including Win32, OSX and Linux. SNMP authentication and security is out of scope for this guide.

  1. Install the HALON-RELAYD-TRAP-MIB.txt file to your mibs/ folder.
  2. Create a configuration file for snmptrapd (snmptrapd.conf). Run whatever script/interpreter you prefer (eg. PHP).
    authCommunity log,execute,net public
    traphandle HALON-RELAYD-TRAP-MIB::relaydStateChange /path/to/php /path/trap.php
  3. Start snmptrapd.

Below is an example trap.php script file.

<?php
$result = array();
$result['host'] = trim(fgets(STDIN));
$result['ip'] = trim(fgets(STDIN));
while(!feof(STDIN)) {
   list($type, $value) = explode(' ', trim(fgets(STDIN)), 2);
   if ($type == "") break;
   $result[$type] = $value;
}
mail("snmp@example.org", "SNMP trap", var_export($result, true));
?>