Packages to Install
apt-get install ipvsadm keepalived
Keepalived
This daemon is pretty confusingly named and has confusing terminology.
The following articles are useful in discussing what it actually does:
And VRRP. Virtual Router Redundancy Protocol. What? We're not setting up a router. Well, perhaps the protocol was named with too much emphasis on a particular application. The main thing the protocol does is ensure exactly one machine has a "virtual" IP address configured. Which is what we need.
BFD
I think this is completely irrelevant for what we're trying to do. Ignore it!
Our Configuration
Keepalived
I tried the SMTP notifications on a test instance. They didn't seem to have enough content to be terribly useful, so I've disabled them.
The VRRP configuration determines which "virtual" IP addresses are assigned to exactly one of the participating hosts.
I set the new loadbalancers up a little differently the old ones. The lb-03 host is the primary for .9, .49, and .139 IPs, while the lb-04 host is the primary for .10, .50, and .140 IPs. While this won't necessarily decrease the traffic that a loadbalancer will route to a www-node on a different physical host, it makes the routing more uniform, and means lb-03 doesn't necessarily handle all traffic if it is online.
It's possible to define virtual_server_group constructs, where all the packets identified by the group members can be handled by the same backend real_server without causing additional health checks.
Otherwise the only thing that's sort of interesting is that certain services need persistence. Anything using SSL, to support session resumption. And FTP, to deal with the separate data transfer connection (both replies to port 20 for active and new connections to high ports for passive).
TODO: look into synchronization for the LVS persistence state. I don't think we had it synchronized on lb-1/lb-2, but perhaps it would nice to set it up.
Firewall Marking
Is just via a /etc/rc.firewall script. It's started up via /etc/systemd/system/firewall.service, with the appropriate directives so that it's sequenced correctly in the boot process.
The firewall marks are based on units where persistence is required. HTTP, which does not require persistence, gloms all the destination IPs together. HTTPS, on the other hand, has a different fwmark for each destination IP. FTP has a different fwmark for each destination IP, but uses the same mark for all possible destination ports that might be used with the IP.