floof.org
I just tried to tell my dad what I'd achieved today, and had one of those "first create the universe" moments. I pretty much ended up describing (simplified) the whole technical history of my hosting network, what BGP, OSPF and VRRP do and how they fit together, how my router setup has evolved over time, and quite a few sidetracks too, to explain that what I'd done today was to complete a script that sets up Linux "virtual MAC interfaces" for the VRRP implementation I'm hoping to soon migrate to, because the vrrp keyword built into the ifupdown2 package I've been using doesn't do a few things quite exactly the way I need and it's easier to reimplement than to fix.
Explaining all that took probably an hour and left my throat really dry and with the realisation of just how *much* technical stuff I have to look after to run a whole hosting ISP single-handedly. I really *really* ought to actually try to get some more customers in - pretty much all the customers I have, I got by accident.
FYI I do server colocation in London, UK, plus most of the other stuff a hoster does: domains and DNS, email (well, not really at the moment, email's one of the more horrible things to host and the current setup is very old and crusty and needs modernising before I'll be happy taking on email customers), web hosting, virtual servers, etc. My main selling points are service and reliability - if cost is your main factor in choosing a host, you might want to try elsewhere. ;) If I haven't scared you off and you're interested, feel free to ask.
Looks like FRR's VRRP implementation has a lot of weirdnesses. It works by having you configure a macvlan interface attached to the actual interface VRRP is running on. The macvlan interface has the VRRP virtual router MAC address on it, plus the virtual IP addresses. VRRPd doesn't touch the addresses or interfaces, other than to set or remove "protodown on" on the macvlan interface.
Weirdnesses discovered so far:
1. Packets that should transit the router running VRRP which are addressed to a virtual IP address on a VRRP macvlan interface in backup mode get received by that router rather than being routed on and ending up at the active VRRP router. Not a huge problem, but can be annoyingly misleading when, e.g., tracerouting.
2. Packets addressed to the VRRP subnet get routed out the macvlan interface even in backup mode unless you turn on "ignore_routes_with_linkdown" for the interface in sysctl. Again, not a huge problem (it's nice to have ingress and egress from a given subnet on the same router for statistics purposes but it's not a deal-breaker if not). Solved, but just the first indication that there may be any number of other related weirdnesses.
3. The one that's currently bugging me. Since the macvlan interface is still "up" (albeit also "protodown on") while VRRP is in the backup state, so that VRRPd can listen for VRRP announcements from other routers, the kernel still receives packets addressed to the VRRP virtual MAC address. Unfortunately, it also still *forwards* those packets. That's not actually a problem as long as your switches correctly learn where the VRRP MAC address lives and send packets only to that interface, but if (as mine have) they decide they are confused as they've seen that MAC address in two different places, throw up their little hands and say "I have no idea, let's just flood frames for this MAC address" you'll get all VRRP routers in a subnet (active and backup) all receiving all packets going out the default router, and therefore all routing those packets out. So your outbound bandwidth suddenly doubles if you have two VRRP routers, or triples or more if you have three or more. Ouch.
4. Can't remember if there was anything else. But… it feels like there are more that won't come to mind as I write this up.
I've been pondering how to fix number 3, and so far I have a tiny shell script which has to run continuously all the time on every VRRP macvlan interface (and if it breaks, the world ends), which uses "ip monitor" to see when "protodown on" is turned on or off. It would then do something to fix things while in backup state — potentially remove/re-add IP addresses, or possibly turn off/on forwarding for the interface using the kernel sysctl interface, or maybe move the interface between groups so that firewall rules can block forwarding when it's in backup state, or… something else. It feels like I shouldn't need to do this kind of thing just to get the protocol to work properly, though. I wonder if there's a better VRRP implementation I should be using instead, or if I should be reporting bugs to FRR (but their VRRPd has had very few changes in years, which suggests they either consider it works properly or it's disused. And I lack the energy to work out how to report bugs, and to write them up properly.)
*sigh* Why is nothing ever simple?
