Anycast: DGRAM vs. STREAM

As most read­ers are aware, there are really only two layer 4 pro­to­cols that peo­ple have to deal with on a daily basis: UDP and TCP. The pri­mary dif­fer­ence is that UDP is a data­gram pro­to­col, which means that your pack­ets are just sent off to the far end, whereas TCP is a connection-oriented stream protocol.

There’s actu­ally another way to look at this: session-oriented vs. session-less com­mu­ni­ca­tions. I’m sure there is research lit­er­a­ture on this, and I’ll rewrite this if some­one can point me at the appro­pri­ate terms. Until then, how­ever, I’m going to some­what arbi­trar­ily define a session-less com­mu­ni­ca­tions as one which:

1. Contains only two pack­ets: a request, and a response. That is, you send a query packet, the server sends a response, and that’s it.
2. Uses trans­ac­tions capa­ble of being processed independently

Now, with a TCP socket, when you actu­ally call connect(), your TCP stack sends a SYN packet (“hey, I want a con­nec­tion”), the des­ti­na­tion sends a SYNACK back (“hey, I see you want a con­nec­tion, I want one too”), and then your stack sends an ACK back to the server (“OK, let’s do this”). All that hap­pens trans­par­ently, before your appli­ca­tion request/response is sent. This means a TCP con­nec­tion is never session-less as I’ve defined it: you’ve got an absolute min­i­mum of four pack­ets (and a prac­ti­cal min­i­mum of 5) in order to do any­thing useful.

Similarly, no VoIP, IPTV, and very few finan­cial mul­ti­cast appli­ca­tions are session-less: even though they’re built using UDP, and so don’t have session-oriented fea­tures built into the layer 4 pro­to­col, the appli­ca­tions them­selves build some sort of ses­sion inter­nally — there are mul­ti­ple pack­ets required to do actual work with these appli­ca­tions, so they’re session-oriented, as defined.

Intuitively, appli­ca­tions which use session-less com­mu­ni­ca­tions are the ideal users for any­cast: you send your request, you get your response, and it’s a one-off. If your server dies and there’s a failover, it will be trans­par­ent to your appli­ca­tion, since each trans­ac­tion is inde­pen­dent (yes, that’s not strictly true, see the last post).

Unfortunately, only DNS and Kerberos-via-UDP are truly session-less (as opposed to “state­less,” which Kerberos plainly is not, and DNS isn’t in prac­tice due to caching), which means they’re the only things that are con­sid­ered truly ideal for an any­cast application.

DNS, in par­tic­u­lar, is nice because of the fail­ure path in most DNS clients’ state dia­grams, which look like this:

1. Server is already dead
2. Client sends a query
3. Client waits for n sec­onds (20 – 60 sec­onds, typ­i­cally) in order to fig­ure out the server is dead
4. Client sends a query to the alter­nate server

Without any­cast, any user try­ing to hit a dead name­server will incur that mas­sive multi-second penalty, every time, for as long as the name­server is dead. However, when you have any­cast, the user has to be per­form­ing a query at exactly the moment of death (that is, between the time the DNS server fails, and before IP SLA detects it and your IGP has re-converged to send the traf­fic to your alter­nate — around 5 – 10 sec­onds, depend­ing on how you’ve tuned IP SLA and your IGP re-convergence timers) for them to be impacted.

So doing DNS via any­cast is almost a no-brainer: you allow users to con­tinue doing what­ever it is they’re doing unaf­fected by out­ages, main­te­nance, etc. Almost makes you wish cer­tain oner­ous, US-based, cable-Internet providers could fig­ure it out.

This isn’t to say that you can only use any­cast for DNS and Kerberos/UDP. While the def­i­n­i­tion I gave of a session-oriented vs. session-less com­mu­ni­ca­tions is cor­rect as far as it goes, and demon­strates that any­cast is best suited to doing dis­trib­uted UDP-based ser­vices, that doesn’t mean any­cast is only suited to dis­trib­uted session-less services.

In fact, it’s just as well suited to any ser­vice that you’d want geo­graphic mir­rors of, and can rea­son­ably suf­fer a multi-second failover time. The rea­son for this is sim­ple: if a session-oriented ser­vice fails, you’re still going to have to dis­cover the fail­ure and re-establish your ses­sion — and it’s actu­ally a lot eas­ier to rely on any given appli­ca­tion, frame­work, or library to do a sim­plis­tic recon­nect than it is to expect them to allow you to con­fig­ure a failover policy.

Otherwise, some things that are “session-oriented” are sim­ply well suited to any­cast: an n-way mul­ti­mas­ter LDAP direc­tory, for exam­ple, works well with any­cast. As does a site-local yum repos­i­tory (you don’t have to do stu­pid DNS tricks, or con­fig­ure each machine with knowl­edge of the “clos­est” mir­ror — you just need to let the net­work know the mir­rors, it and let it han­dle the rout­ing for you). I’m likely going to look at this for a dis­trib­uted Cassandra-based project I’m work­ing on as well.

In the next instal­ment I’ll post the com­bined con­fig­u­ra­tion from the last few posts in a sin­gle place, in case it was hard to fol­low through all the back­ground and explanation:

1. Anycast: Networking Introduction
2. Anycast: The Loophole
3. Anycast: The Interface
4. Anycast: Handling Routes
5. Anycast: DGRAM vs. STREAM
6. Anycast: IP-SLA HOWTO