Matt’s geolocation, anycast, and server location notes

So this is here mainly for my own future reference, but I’ve added some details and made it public in case others find it useful.

I’ll provide a slight bit of background.

I’ve frequently tested various server locations for latency in an effort to improve site speed around the world. I’ve played with Anycast, Geo DNS, Amazon’s Latency Based Routing, IPv4, IPv6, and so on. At times I’ve had nearly 20 servers located around the world. Of course sometimes I don’t have time to maintain it all and I end up back at 1.

I’ve used numerous providers, but have trended towards those with hourly billing and low cost VPSs, particularly since some locations only end up with a handful of visitors a day (but at least I can get the site to load fast for them!).

These notes were taken while I was setting up a hybrid Anycast/GeoDNS configuration.

Alright, so let’s start with the US, since it’s a large land mass that tends to get heavier traffic:

Latency Server Map USA (United States)

The best server locations I’ve found are generally in these areas, listed in a (somewhat) prioritized order if not filling out all locations:

  • New York (New York)
  • LA (California) – ties with NY for priority
  • Chicago (Illinois)
  • Atlanta (Georgia) – ties with Chicago for priority
  • Dallas (Texas)
  • Seattle (Washington)

A lot of traffic travels through these major hubs on the way to the destination regardless, so having a server housed in these hubs rather than in a location just-outside will shave off a bit of latency.

Be very cautious targeting other smaller locales. For example if you have a lot of traffic from Phoenix, AZ you might be tempted to use a server there. The problem is that unless you’re well-peered with all the local ISPs, your Phoenix traffic might take a detour through LAX on its way to your Phoenix server. This is extremely common and thus you want to thoroughly test any location that isn’t inside what would be considered a major hub before committing to it.

You’ll see color-coded bits in the map above which are applicable if attempting to Geolocate. After heavy testing, they encompass the surrounding states that should roughly be Geolocated to the corresponding server to obtain lowest latency with the fewest visitors being crazy-routed.

Roughly is the key word here. This also depends on who your Server/VPS provider peers or has transit with. It’s also easy to end up in a situation where… say… a Nebraska ISP routes all their traffic to Chicago. Then you’re left with the choice of selecting Dallas so that most Nebraskans get lower latency, or choosing Chicago so that those few people on that ISP don’t have really bad latency.

This setup is not without its flaws. Texas is a massive pain, and I had to go through 4 providers before I found a server that wasn’t inundated with bad routing and massive TCP retransmissions. Georgia isn’t far behind when it comes to routing… half the time neighbouring states have better latency to the server than the people actually in Georgia. These could simply be congested networks, as testing the ping to specific users seems to show better than the actual results. But either way I’ve found it hard to get the page-delivered-time in these states to compete with the others.

To clear up confusion regarding oddball colors in the above map, you’ll notice that Virginia and Maryland are yellow: this was doing some Vultr vs AWS testing. You can assume it’s blue unless you’re also using AWS. Wyoming and West Virginia are uncolored because the were… very inconsistent in terms of which route was the fastest.

As far as Anycast goes, in the US it generally works pretty well. In the Anycast vs GeoDNS writeup I did last year, GeoDNS came out ahead by about 5 ms in total-page-delivery time. In my most recent testing GeoDNS was now behind by about 2 ms in the US, and I found numerous cases where visitors were Geolocated to the wrong place compared to the odd case here and there previously.

If only going with only 2 servers, NY and California give the most “bang for the buck” and also provide routes to Europe and Oceania/Asia.

If using one of the major cloud providers (AWS, Google, etc), they don’t build datacenters in the middle of major cities and thus tend to have higher latency to most of the population. However they’re fairly well-connected which lessens the impact somewhat.

Canada

Canada is a tough one to Geolocate because most DNS services that offer Geolocation and don’t cost an arm/leg only go down to the “state/province” level in the US. Ideally, west coast traffic would go down to Seattle, central traffic to Chicago, and east coast traffic would go down to New York (or perhaps Toronto/Montreal if locating a server there). If using Amazon Route 53 for DNS, you can somewhat “fake” this behavior via Latency Based Routing by using a Seattle server as “us-west-1” or “us-west-2”, Chicago as “us-east-2” and New York as “us-east-1”.

Since the major Canadian telco monopolies avoid peering at the public exchanges in Canada as much as possible, placing servers in Canada and then geolocating Canadian traffic to them can be a risky move.

In any case, Anycast will currently provide the lowest latency to Canadian visitors most of the time. If you happen to have a server in Canada and can utilize Anycast, you’ll probably get direct traffic from the smaller ISPs, while the monopolies in a number of provinces will continue to send their traffic down to the US. But at least you’ll avoid the extra round trip.

 

Europe

Europe’s an interesting one. The main locations for most servers (England, France, Germany, Netherlands) are all pretty well interconnected and I’d go so far as to say you probably don’t need a lot of servers for a more typical distribution of worldwide traffic. You can usually pick 1 of those 4 locations and get away with it because latency is so low between them.

In fact, they’re so well connected that if you place a server in each, I’ve found you’re best off Geolocating them to keep them “in bounds”. With Anycast it’s easy for traffic to spill into 1 of the other 4. The United Kingdom (GB) region also has a tendency to spill into the US via Anycast.

If choosing 1 server, I’d probably go with Amsterdam.

If going with 2 servers, I’d probably go either Germany/France or Germany/England depending on your audience.

If placing servers in some of the other surrounding countries, routing can get interesting depending on the provider… you might have everything backhauled to Germany for example. So if you have servers in countries that neighbour the 4 main hubs, Anycast tends to work a little bit better.

Brazil and South America

I haven’t tested South American countries beyond Brazil very heavily.

Brazil is the “easiest” location to find a server in. However, peering between South American countries is not always great. If you try to force all the traffic to Brazil, some traffic might go to the US (generally Miami) first. This is a situation where AWS Route 53 and “Latency Based Routing” seems to work well.

Anycast is not a great option here if you have a server in Brazil: some South American countries/visitors have very short routes to the US yet have longer but faster/direct routes to Brazil. However, using anycast is usually preferable to “forcing” all South America traffic to Brazil.

If not placing any servers in Brazil, then using a server in Miami (Florida) isn’t uncommon and works reasonably well. Or you could Anycast your US locations.

 

Africa

I haven’t done any real testing here yet, partly because my traffic from that region is so minimal that it’s harder to test, and partly because there isn’t much in the way of options right now anyway. It looks like South Africa (ZA) might be on its way to becoming a hub which is fantastic, as it is currently very challenging to bring low latencies to visitors from that location. In the interim, London seems to be the best major hub for the majority of ZA traffic though I use the term “best” very loosely here (almost 200ms RTT).

 

India

If you’ve ever checked a traceroute from another country to an India IP address…. well… you probably saw a lot of hops. Unsurprisingly, Anycast does not work well with India at all.

India in general has decent connections to Singapore, Europe, and the Western US. Beyond that it’s really hit-and-miss.

The best solution here is to get a well-connected server in India and Geolocate to make sure the traffic stays “in bounds”. If you try to Anycast the server even from *within* India, some traffic will spill out into other countries far far away.

If you can’t get a server inside India, the next best location to keep things speedy for visitors tends to be Singapore. Again, you have to Geolocate to it.

If Singapore isn’t an option either, then find a place in Europe (much tends to go through London). Keep in mind though that at this point your visitors from India are seeing well over 100 ms RTT. Note that some locations like Tokyo tend to have better latency to India than Europe, but these locations aren’t as well connected so you risk traffic going through the US before it gets to Tokyo.

If even Europe isn’t an option, try to at least send them to LAX since chances are some traffic will be going through there anyway if you’ve eliminated both Singapore and Europe as options.

You may want to test the location you’ve chosen, as some cloud providers (Google) historically did not had a direct link west of India, and thus if you Geolocated from India to Europe there is a good chance it would travel to and through the US.

 

Pakistan

I haven’t used servers in Pakistan. In any case, do not send to India: it may pass through Europe on its way even though India is right next door.

Easiest to either send this traffic to Europe, Singapore (slower) or to Anycast from other locations and let the chips fall where they may.

 

Australia and New Zealand

If placing a server inside Australia, you need a provider that peers/transits (or otherwise connects with) with the big monopolies (Telstra and Opus). Otherwise, much of your traffic is likely taking a trip from Australia-> country_across_ocean -> Australia.

Anycast allows spillage, so Geolocate this one.

If not inside Australia, you’re probably fine to Anycast although there’s no guarantee where it’ll end up if you have a few locations in the relative vicinity. The US West Coast is a real possibility.

Order of preference if Australia is not possible: Singapore, Tokyo, Hong Kong/USWestCoast. Note that ~150 ms RTT is a real possibility with these, depending on the visitors ISP and which location is hit.

Next up… a few quirky locations…

Latency Server Map - Some other locations

Focusing on the stuff in the map above, many of these locations will not Anycast well. I won’t go through alternate options for all of these because it becomes quite messy. If you don’t have a server in the appropriate place, either do some testing or some best-effort guessing.

Note that I’m focusing on server locations provided by some of the more common providers (Vultr, AWS, etc) which means Singapore, Tokyo, and possibly HK.

 

Philippines

Don’t Anycast this. Geolocate it to Tokyo. Singapore can be less latency for some visitors, but I found that some others (perhaps a certain ISP) will go through Tokyo regardless. If you Geolocate to Tokyo, all traffic seems to be happy to go directly there.

 

Indonesia

Geolocate to Singapore.

 

Vietnam

Geolocate to Hong Kong.

 

Thailand and Malaysia

Geolocate to Singapore (fastest) or Hong Kong (next fastest). Anycast does reasonably well here but not quite as well as Geolocation.

 

Taiwan

Hong Kong, Tokyo, Singapore. In that order.

 

Others in the APAC region and/or Single Location

Singapore is usually a fairly safe bet if you can’t micro-optimize for each region or only get to pick 1 location to serve the region.

I should note that in this region I found it nearly impossible to get absolutely everyone routed perfectly.

 

 

Final Notes

Since routing/peering/etc all change over time, relying too heavily on anything I’ve mentioned above is a high risk. You should probably test first before implementing. There’s nothing worse than plopping in a new Anycast server only to find that it’s become a high-latency magnet for places far away.

Additionally, keep in mind that the testing was largely for my own usage, intended to lower latency worldwide for my web visitors, and generally relied on testing via low cost providers (AWS, Vultr, etc) who tend to have datacenters in similar common regions. Obviously it may not be as helpful if you’re picking up servers in some of the less-common hubs.

Since someone who is trying to decide between Anycast and GeoDNS will likely come across this page, I previously did a GeoDNS vs Anycast write-up that you might find interesting if you’re in that position.

3 Comments | Leave a Comment

  1. Anders on April 20, 2020 - click here to reply
    Very useful findings. I'm about to test Rage 4 anycast service that doesn't look too expensive for us without our own IPv4 range. They also offer GeoDNS. Do you have any experience using them?
    • I played with the GeoDNS offered by Rage4 years ago (back when there was a free variant), but that was about it. Their anycast service certainly looks interesting and it appears the locations are pretty well chosen.
  2. Poppy on April 25, 2020 - click here to reply
    Great! It's really helpful!

Leave a Comment

You can use an alias and fake email. However, if you choose to use a real email, "gravatars" are supported. You can check the privacy policy for more details.