References: http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-DR.html http://kb.linuxvirtualserver.org/wiki/LVS/DR
Test Setup: http://lb-www.club.cc.cmu.edu
IPVS kernel support is already in the default Debian kernel. All that needs to be installed is the administration tool (ipvsadm) and a management tool. The only management tool that I have tested, and thus the one I like most, is keepalived.
Install the tools on the load balancer only.
apt-get install ipvsadm keepalived
All (most) configuration is done on the load balancer side. In particular, /etc/keepalived/keepalived.conf. Below is the current configuration.
global_def {
notification_email {
awesie@club.cc.cmu.edu
}
notification_email_from root@club.cc.cmu.edu
smtp_server localhost
smtp_connect_timeout 30
router_id lvs-b6
}
vrrp_instance VI_B6 {
! state BACKUP
state MASTER
interface eth0
lvs_sync_daemon_inteface eth0
virtual_router_id 200
! priority 100
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass badpass
}
virtual_ipaddress {
128.237.157.89
}
}
! ftp active/passive
virtual_server fwmark 1 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 128.237.157.9 21 {
weight 100
TCP_CHECK {
connect_port 21
connect_timeout 3
}
}
real_server 128.237.157.10 21 {
weight 100
TCP_CHECK {
connect_port 21
connect_timeout 3
}
}
}
! rsync
virtual_server 128.237.157.89 873 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
persistence_timeout 300
protocol TCP
persistence_timeout 300
real_server 128.237.157.9 873 {
weight 100
TCP_CHECK {
connect_port 873
connect_timeout 3
}
}
real_server 128.237.157.10 873 {
weight 100
TCP_CHECK {
connect_port 873
connect_timeout 3
}
}
}
! nntp
virtual_server 128.237.157.89 119 {
delay_loop 6
lb_algo rr
lb_kind DR
protocol TCP
real_server 128.237.157.36 119 {
weight 100
TCP_CHECK {
connect_timeout 10
}
}
real_server 128.237.157.69 119 {
weight 100
TCP_CHECK {
connect_timeout 10
}
}
}
! web servers
virtual_server 128.237.157.89 80 {
delay_loop 6
lb_algo wlc
lb_kind DR
protocol TCP
virtualhost www.club.cc.cmu.edu
! Can't have a server be the local computer, must be remote.
! sorry_server 127.0.0.1 8080
real_server 128.237.157.9 80 {
weight 100
HTTP_GET {
url {
path /index.cgi
url {
path /index.cgi
status_code 200
! digest 523d93c9f140610c309061167f92a4b2
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 2
}
}
real_server 128.237.157.10 80 {
weight 100
HTTP_GET {
url {
path /index.cgi
status_code 200
#digest 523d93c9f140610c309061167f92a4b2
}
connect_timeout 3
nb_get_retry 3
delay_before_retry 2
}
}
}
! web (ssl) servers
virtual_server 128.237.157.89 443 {
delay_loop 6
lb_algo wlc
lb_kind DR
protocol TCP
persistence_timeout 360
virtualhost www.club.cc.cmu.edu
! Can't have a server be the local computer, must be remote.
! sorry_server 127.0.0.1 8080
real_server 128.237.157.49 443 {
weight 100
SSL_GET {
url {
path /index.cgi # replace with a file that does exist
status_code 404
! digest 523d93c9f140610c309061167f92a4b2
}
connect_timeout 3
connect_port 443
nb_get_retry 3
delay_before_retry 2
}
}
real_server 128.237.157.50 443 {
weight 100
SSL_GET {
url {
path /index.cgi # replace with a file that does exist
status_code 404
#digest 523d93c9f140610c309061167f92a4b2
status_code 404
#digest 523d93c9f140610c309061167f92a4b2
}
connect_timeout 3
connect_port 443
nb_get_retry 3
delay_before_retry 2
}
}
}
! dns server (not working)
virtual_server 128.237.157.89 53 {
delay_loop 10
lb_algo wrr
lb_kind DR
protocol UDP
real_server 128.237.157.12 53 {
weight 100
MISC_CHECK {
misc_path "/usr/bin/dig -b 128.237.157.89 a www.club.cc.cmu.edu @128.237.157.12 +time=1 +tries=5 +fail > /dev/null"
misc_timeout 6
}
}
real_server 128.237.157.14 53 {
weight 100
MISC_CHECK {
misc_path "/usr/bin/dig -b 128.237.157.89 a www.club.cc.cmu.edu @128.237.157.14 +time=1 +tries=5 +fail > /dev/null"
misc_timeout 6
}
}
}Most of the configuration is relatively intuitive. More documentation can be found at the keepalived website (http://www.keepalived.org).
That finishes all of the load balancer server configuration. So go ahead and tell keepalived to reload it's configuration.
On the backends, you need to tell Linux to accept packets for your virtual server IP. In the configuration above, this IP is 128.237.157.89. There are several ways to do this, as described in the references. Also, as described in the references, since we have our load balancer on the same network as the backends, we need to prevent our backends from broadcasting ARP packets for the virtual server IP, and this is all documented in the references. Below is the way I prefer to do this.
iptables -t nat -A PREROUTING -p tcp -d <virtual_server_ip> --dport <virtual_server_port> -j REDIRECT --to-port <local_port>
It's simple. Doesn't require adding another address to an interface. Note that djbdns doesn't seem to like this at all.
Congratulations! The load balancing is now setup with failover. Both IPVS and keepalived are flexible systems, with support for failover load balancers as well, via VRRP2.
Current firewall configuration files:
load-balancer
# Flush chains iptables -F # Set chain policies iptables -P INPUT ACCEPT iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT # Delete old chains iptables -X PROPERREJECT 2> /dev/null # Create new chains iptables -N PROPERREJECT # Drop invalid packets iptables -A INPUT -m state --state INVALID -j DROP # Accept connections on the local interface. iptables -A INPUT -i lo -j ACCEPT # Drop connections to the local interface from outside iptables -A INPUT -d 127.0.0.0/8 -j DROP # Reject connections from evil sources # iptables -A INPUT -s 72.20.0.0/18 -j REJECT # 72.20.0.14 # Reject all TCP & UDP connections, drop others iptables -A PROPERREJECT -p tcp -j REJECT --reject-with tcp-reset iptables -A PROPERREJECT -p udp -j REJECT --reject-with icmp-port-unreachable iptables -A PROPERREJECT -j DROP # mark ftp passive/active packets iptables -t mangle -A PREROUTING -i eth0 -p tcp -s 0.0.0.0/0 -d 128.237.157.89 --dport ftp -j MARK --set-mark 1 iptables -t mangle -A PREROUTING -i eth0 -p tcp -s 0.0.0.0/0 -d 128.237.157.89 --dport ftp-data -j MARK --set-mark 1 iptables -t mangle -A PREROUTING -i eth0 -p tcp -s 0.0.0.0/0 -d 128.237.157.89 --dport 60000: -j MARK --set-mark 1
www
# Flush chains iptables -F # Set chain policies iptables -P INPUT ACCEPT iptables -P FORWARD DROP iptables -P OUTPUT ACCEPT # Delete old chains iptables -X PROPERREJECT 2> /dev/null # Create new chains iptables -N PROPERREJECT # Drop invalid packets iptables -A INPUT -m state --state INVALID -j DROP # Accept connections on the local interface. iptables -A INPUT -i lo -j ACCEPT # Drop connections to the local interface from outside iptables -A INPUT -d 127.0.0.0/8 -j DROP # Reject connections from evil sources iptables -A INPUT -s 72.20.0.0/18 -j REJECT # 72.20.0.14 # Reject all TCP & UDP connections, drop others iptables -A PROPERREJECT -p tcp -j REJECT --reject-with tcp-reset iptables -A PROPERREJECT -p udp -j REJECT --reject-with icmp-port-unreachable iptables -A PROPERREJECT -j DROP # Forward load balancing ip iptables -t nat -A PREROUTING -p tcp -d 128.237.157.89 -j REDIRECT
ftp configuration additions (vsftpd.conf)
# load balancing pasv_min_port=60000 pasv_max_port=65535 pasv_address=128.237.157.89