I’m hoping that some of our European contributers can help us out with this one. As you may know, Tim is over in Europe right now. He has had a terrible time connecting to Sanityville while there. It seems like he’s able to connect pretty consistently through wifi, but when he tries through his phone’s data connection, there are times when the site is entirely inaccesible. He is currently on Telekom. It sounds like the problem is pretty specific to Sanityville. (I just checked, and we aren’t doing any ip or country-level blocking, so far as I can tell.)
What’s more, it seems like it may be country dependent. He just got to Spain from Italy (via France), and he couldn’t connect to the site using his phone data in either France or Italy.
Is Sanityville hosted in a way that is very US centric?
If it is distributed to help global users, is it done the right/common/modern/commercial way? Even if the hosting is using content distribution services etc., I wonder if it’s done the “cheap” way, then maybe international mobile data plans aren’t so good at using sites that don’t use CDN properly.
Just a hunch. I’ve not experienced this situation.
If Tim only only has trouble with Sanityville, but not other US sites, what about other US sites that aren’t “big?”
Interesting. At first it worked in Spain. For a few hours. Then dropped again. So I tried the fix Joseph suggested of turning on beta of iCloud Private Relay. Now seems to be working again.
I’m in the US but I’m not surprised iCloud Private Relay was involved in the problem. I’ve had more than one random site struggle to load when I had it enabled.
Yup. And I just removed a VPN I had on for a couple weeks b/c an online order I made was cancelled as suspected fraud, and their email informing me said one reason your order might be cancelled is if they can’t identify your IP address. So choose your poison, apparently.
Because the VPN solved it. VPN always encapsulates traffic and lowers the MTU of the traffic passing through it - directly on the device. The upper boundary is lower than whatever caused the problem in the first place, thereby fixing it.
The same applies to IPv4 traffic being carried in IPv6-only networks, the latter making up the majority of European access traffic (big mobile and residential providers): The site only does IPv4, the packets get encapsulated into IPv6 and at the transit exits “uncapsuled” (CGNAT). Normally this doesn’t happen, but it can mess up things. Many people had problems during Covid time to access their datacenters at work from home.
From my setup (big cable provider IPv6-only) the MTU to sanityville seems to be 1460:
$ ping -s 1430 35.208.128.7
PING 35.208.128.7 (35.208.128.7) 1430(1458) bytes of data.
1438 bytes from 35.208.128.7: icmp_seq=1 ttl=114 time=129 ms
1438 bytes from 35.208.128.7: icmp_seq=2 ttl=114 time=126 ms
^C
--- 35.208.128.7 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 126.324/127.465/128.607/1.141 ms
$ ping -s 1440 35.208.128.7
PING 35.208.128.7 (35.208.128.7) 1440(1468) bytes of data.
From 35.208.9.207 icmp_seq=1 Frag needed and DF set (mtu = 1460)
1448 bytes from 35.208.128.7: icmp_seq=2 ttl=114 time=129 ms
^C
--- 35.208.128.7 ping statistics ---
2 packets transmitted, 1 received, +1 errors, 50% packet loss, time 1001ms
rtt min/avg/max/mdev = 128.684/128.684/128.684/0.000 ms
This is why I don’t actually run any servers anymore. Lol.
We’ve paid a hosting provider, and in situations like this, I just can’t do anything. Which is nice, because I don’t know the first thing about MTU and servers, and I really shouldn’t spend time learning that. But I would dive in and learn it if I was running my own server.
Interesting. So, could one summarize loosely and say that the additional byte overhead of things like CGNAT and 6in4-type encapsulation involved in international traffic creates additional MTU constraints, and that the VPN’s inherent ability to fragment traffic is able to atone for it?
I know we have plenty of CGNAT going on over here in the US now, and ISPs have done a slow-but-steady transition toward IPv6, but I wonder if there are additional factors involves in Europe’s routing landscape that make this a uniquely international issue.
Yes, that’s what I think. The setup won’t allow to validate my hypothesis. But so often in the last few years I had to debug network problems that sounded exactly like this one. Especially with China. They even don’t allow VPNs from big VPN providers anymore.
Yes, definitely. I guess Tim got a German Telekom SIM card for his phone, making Germany his home network. Sanityville probably works fine in Germany. In Italy the German SIM card logs in to an Italian provider, and since EU roaming is mandatory and “same cost as home” he can make calls and use the internet there too. Now the providers do their network stuff, and the settings are just a little bit different in Italy than in Germany. Or they route his traffic back to Germany first and then to the real destination - resulting in yet another encapsulation of course…
Thanks. It’s interesting to hear that your cable provider does IPv6 only. In the US, it’s still commonplace to be able to acquire static IPv4 space from ISPs if you request it and are willing to pay a few extra bucks. I know of no providers who are v6 only in the US.
Earlier this year, I switched from cable to a relatively new and regional GPON-based provider who is aggressively building out their plant in metro areas. They do CGNAT by default, but even as a residential subscriber I was able to request a static v4 IP (that doesn’t pass through their stateful CGNAT appliance) and they obliged for another $5 a month.
Money well spent from my perspective as a network engineer who works from home.
Sorry I was unclear, it’s called “dual-stack-lite”, you get IPv6 plus your router/modem gives you some RFC 1918 IPv4 address. Any traffic with those IPs get encapsuled / CGNATted. The devices on the home network therefore can do IPv4. This led directly to VPNs problems for people working from home for the first time in Covid. The solution was always to reduce the MTU on the company side’s VPN endpoint.
My cable provider is - after a series of acquisitions - Vodafone. Residential users can get “real” dual-stack, like you have. Business customers (my company office) either get ds-lite or ipv4 only. We chose the latter. Telekom offers all options, but they don’t have cable here, only DSL… BUT we deployed Starlink recently and it works very well as a backup for the cable connection. It’s CGNAT of course.
So you’re on T-Mobile US? That changes the picture a bit, but it’s still probably some mobile roaming glitch. I think T-Mobile US and Europe are very different.
You know you can get a cheap prepaid SIM from your favorite Albrecht Discount?
