The uselessness of "Round Robin" resource record set shuffling

You've come to this page because you've said something similar to the following:
One can use "Round Robin", a feature that shuffles the resource records in a resource record set so that they occur in different orders, to achieve load balancing, fault tolerance, and traffic locality.

This is the Frequently Given Answer to such statements.

(You can read a different, and on one point flawed, approach to this same answer on the ISC's web page about load balancing and its own DNS server.)

No, one cannot.

Round-robin isn't a feature. It's nothing more than a bodge, and doesn't deserve to be promoted to the rank of feature. It attempts, without success, to produce the same sort of behaviour as would occur if the true features were properly implemented. However, it does not achieve any of the goals that one actually wants to achieve, because it is based upon premises that are actually false.

Round-robin has no real use. It's not even a poor man's load balancer.

The flawed foundations that underpin round-robin

The erroneous notion that shuffling resource records will affect service clients' connection behaviour

The primary idea that underpins the notion of round-robin achieving anything is that the order of the resource records that content and proxy DNS servers use, when sending an A or an AAAA resource record set in their responses to DNS queries, will determine the order of the IP addresses that service clients attempt to connect to when using the service servers whose IP addresses have been looked up.

This tenet is false. The DNS protocol and the mechanics of query resolution simply do not work this way.

The DNS protocol imposes no ordering on the resource records within a resource record set. When handling a response datagram, a DNS client (including the DNS client that forms the back-end of a proxy DNS server) is under no obligation to preserve the ordering of the resource record set as it is received. (DNS clients may be configured to re-order resource record sets. The sortlist directive in BIND's DNS Client library, used by many applications, can be used to do this, for example. Indeed, in certain cases DNS clients are specifically obliged to re-order resource record sets. The DNS clients in SMTP Relay clients are obliged to re-order resource record sets, for example.) Moreover, a caching proxy DNS server is under no obligation to preserve the orderings of resource record sets held in its cache, or to preserve the orderings of resource record sets as received from content DNS servers (or upstream proxy DNS servers).

This completely defeats the idea that the sender of a DNS response datagram can specify a particular ordering of the resource records in a resource record set. Whatever ordering a content DNS server may specify in its responses can be overridden by a proxy DNS server (by its back-end re-ordering the resource record set as it receives the response, by its cache, and — most ironically — by whatever round-robin or other ordering "features" the proxy DNS server itself may have), and whatever ordering a proxy DNS server may specify in its responses can be overriden by the DNS client within the service client application itself (by its re-ordering the resource record as it receives the response, and by whatever round-robin or other ordering "features" the DNS client library itself may have).

There is no ordering on the resource records within a resource record set, and thus specifying an order cannot be used to convey information from DNS servers to service client applications.

The erroneous notion that shuffling resource records provides an even, or a predictable, distribution of connection attempts to service servers

A secondary idea that underpins the notion of round-robin achieving anything is that (temporarily assuming the primary tenet to hold true for the sake of argument) the shuffling of resource records with every DNS response ensures that service clients will connect to different service servers across multiple transactions and that different service clients will see and use different orderings for service servers.

This tenet is false. It is defeated by the existence of caching.

Caching occurs in caching proxy DNS servers. In the situation where a content DNS server is employing round-robin shuffling of resource records, the caching proxy DNS server will only receive a response (with the resource records in a different order) whenever its back-end issues a fresh query. But this, in turn will only occur whenever the previous resource record set has exceeded its TTL. Until that time, the cached resource record set, in whatever order it happens to be in, will continue to be used. Now, of course, there's no guarantee that the proxy DNS server will preserve the resource record set ordering that its back-end received from the content DNS server, but if it just so happens that it does all of the DNS clients querying that proxy DNS server will only see the ordering of the resource record set change whenever the cached copy expires and a fresh copy is fetched by the proxy DNS server's back-end from the content DNS server. Moreover, all of the DNS clients querying that proxy DNS server will see the same ordering. Again, of course, there's no guarantee that the DNS clients within the service client applications will preserve the resource record set ordering that they receive from proxy DNS servers, but if it just so happens that they do then all of the service clients whose DNS clients share that same proxy DNS server will see the same ordering.

Now consider the case of service client applications run by a large number of customers of a single large ISP, where the DNS clients on the customer's machines are configured to use that ISP's proxy DNS service (and to do no re-ordering or caching of their own — of which more later). If the ISP has a million customers actively and repeatedly using the service, then a million service clients will all be connecting to the service server IP addresses in the exact same order. And every TTL seconds, all of those million service clients will switch to a new order, and all hit on a different service server IP address first. Caching by proxy DNS servers (that preserve the received ordering) causes a disproportionate weight to be given to the resource record set orderings that happen to be seen by proxy DNS servers that serve a lot of clients.

To avoid this problem, people employing round-robin resource record set shuffling will set the TTL of the resource record set very low, to attempt to ensure that every connection to the service server is preceded by a fresh fetch of the newly shuffled resource record set from the content DNS server. However, this creates further problems of its own, and doesn't work anyway.

In theory, a TTL of zero prevents proxy DNS servers from caching a resource record set, and thus every use of a service server would be preceded by the DNS lookup for that server's IP address(es), passing all of the way through from the DNS client to the actual content DNS servers since nothing cached. Ironically, this often doesn't actually work and causes problems. TTLs of zero can cause DNS lookups to fail entirely. Hence it is common for a TTL no lower than 60 seconds to be used. However, low TTLs cause extra DNS lookup traffic, of course, using up more network bandwidth and putting a greater load on content and proxy DNS servers. Eliminating this overhead is why resource record sets are cached in the first place, after all.

Furthermore, TTLs are not always respected. This is because DNS clients can also perform caching. Even the applications themselves often perform caching, of the data that they receive from the DNS clients. Such caching often does not operate at the level of of DNS lookups and does not retain the DNS TTL information. For examples:

You can read more on this subject on Pete Tenereillo's Browser DNS Caching page.

The goals that people actually want to achieve

There are three goals that people generally have in mind when they consider round-robin. None of the three goals is achieved by making content or proxy DNS servers rotate or randomise resource record sets.

Goal #1: load balancing among the service servers themselves

This goal is achieved in one of two ways:

Some half-baked load balancing solutions

The ISC's web page about load balancing and its own DNS server recommends an additional way of achieving this goal, namely of having a content DNS server that is aware of the load on each of the service server machines, by some private channel, and that orders the resource record sets that it publishes in ascending order of load. However, this is one point on which the ISC's web page is flawed. As described earlier, there is simply no guarantee that the ordering used by the content DNS server when it publishes a resource record set will even reach the service client applications, let alone affect the order in which they attempt to connect to the service server IP addresses.

One way that people attempt to work around this difficulty is to have the specialized content DNS server publish a resource record set comprising just the first address, selected, according to load, from the list of service server IP addresses. This variation on the scheme is often called "Global Server Load Balancing", and one can buy expensive systems to implement it. However, as Pete Tenereillo points out, it too causes problems, since by eliminating in this way the possibility of proxy DNS servers and DNS clients re-ordering the resource record set, one creates single-point failure modes for one's service and loses any hope of fault tolerance.

Goal #2: fault tolerance in the event of network failure or packet loss

This goal is reached by having service clients that attempt to connect to each address in turn. Such clients are not hard to write. It's not hard to iterate over all of the addresses returned from gethostbyname().

Goal #3: ensuring better traffic locality by ensuring that service clients try the nearest service servers first

This goal is reached …


© Copyright 2004–2004 Jonathan de Boyne Pollard. "Moral" rights asserted.
Permission is hereby granted to copy and to distribute this web page in its original, unmodified form as long as its last modification datestamp is preserved.