Real Infrastructure Is The Enemies We Made Along The Way

Welcome to Infrastructure Week July 2018! New articles and tools every day this week.

• day 1: lematt
• day 2: netmatt
• day 3: email 2018
• day 4: web 2018
• day 5: reliable local nets

The Internet is always working against us both literally and metaphorically.

When the Internet is working against us literally, we have some tools to wield against it. When the Internet works against us metaphorically, we have to either fight back or walk away.

Stability Problems

As a traveling Internet user, so much can go wrong:

• poor building cabling
  • excessive noise (insecure connectors, busted cables) leads to to inadequate signal
• poor wireless quality
  • can you see 40 access points advertising names? There’s going to be so much noise in the air you won’t get reliable performance.
• unexpected network conditions
  • at some hotel / airport / coffee shop, but their captive portal fights with their ISP gateway?
    • enjoy getting ICMP Redirect packets all the time with connections randomly getting refused
      • enjoy explaining to a minimum wage front desk clerk how you diagnosed misconfigured gateways due to excessive ICMP Redirect packets and you’d like it fixed please
        • you’ll be about as successful as asking a TSA worker why you can’t take your water bottle through security.
    • or, even better, maybe their captive portal rate limits are misconfigured!
      • they have a gigabit business-class ISP link, but nobody updated their captive portal config in 8 years, so each client is still restricted to 3 Mbps with no burstable features.
        • enjoy explaining to a minimum wage front desk clerk how you think the Internet is too slow.
  • or, you’re behind a misconfigured NAT and all connections get terminated after 30 seconds
    • better remember to use resumable downloads!
  • or, your location has a cheap off-brand wireless access point so your devices can only stay connected for 20 seconds at a time before you have to power cycle something
  • and other fun situations like:
    • your location captures/proxies/filters all DNS traffic
    • your country captures/proxies/filters all DNS traffic and blocks encrypted DNS too
    • you’re about 10m too far away from the wireless antenna, so you have to compute against your hotel door for good signal
    • location doesn’t understand wireless design and named each of their 40 access points different names (Floor1A-1, Floor1A-2, Floor2A-1, Floor2A-2, …)

And, sadly, each of those problems is completely “invisible” to normies.

If you go to a hotel and the water doesn’t work, sure, you can call the front desk to fix it. If you go to a hotel advertising “high speed Internet” and end up with 3 Mbps in 2018, you have no way to complain because nobody understands or cares. Internet access is a completely “invisible” service, so the front desk will just reload google.com on their desktop and say “it works for me! here’s our overseas tech support number for guests who don’t know how to use the Internet!”

Easy To Use Network Reliability Defenses

Since network quality is impossible to predict before we reach a new location, how can we protect ourselves from most broken network conditions?

Problem: DNS Is Slow or Unreliable

I’ve stayed at a hotel where any initial outbound request was unreliable, but connected TCP sessions were reliable, which doesn’t help us for UDP 53 at all.

Using wireshark, I saw about 80% of my DNS queries just got no responses–sacrebleu!

Fortunately, our friend dnsmasq can fix the problem for us.

dnsmasq has a command line param enabling it to request in parallel from multiple DNS servers at the same time then return the fastest result.

So, I added these to my /usr/local/etc/dnsmasq.conf:

server=8.8.8.8
server=4.2.2.1
server=4.2.2.2
server=4.2.2.3
server=4.2.2.4
server=4.2.2.5
server=4.2.2.6
server=8.8.4.4

Then add command line parameter --all-servers to the dnsmasq command. (sadly, you can’t seem to configure --all-servers from the config file, it must be a command line option, which means editing startup/daemon files if you want it on restart)

Last step: configure your local DNS server to just be 127.0.0.1 (assuming dnsmasq is listening on localhost obviously).

Now, each DNS request gets sent to 8 servers and the fastest reply wins.

Using wireshark, I verified it works — send 8 requests and, because the network was garbage, get about 2-4 responses back for every request.

As an added bonus: DNS resolvers don’t always have the same latency, so you’ll often get the fastest response back from different servers. It’s the best of all worlds, and you’ve even got a local on-device resolver cache now¹.

Also see the sshuttle section below for an alternative DNS improvement method which tunnels all your DNS requests over ssh with firewall magic.

Problem: Connections Can’t Establish (sometimes)

Does your Internet access work, but just feels unreliable? Maybe you can stream YouTube at 1080p, but it takes 90 seconds to connect to your mail server? Your network probably has poor connection management.

Your challenge will be to establish a reliable connection, keep it up, then send traffic to workaround your network’s broken connection establishment problems.

Connection Establishment Fix: sshuttle

sshuttle is a really fancy pseudo-VPN/proxy capable of intelligently disassembling traffic on your client and re-assembling it on your remote endpoint to avoid typical “TCP-in-TCP” buffer issues.

You can install sshuttle with pip3 install sshuttle then it’s quite easy to configure.

sshuttle is most useful if you have remote hosts/subnets you want to tunnel locally (e.g. a remote mail server or a remote hosting service you want to tunnel over one connection).

sshuttle also has a very useful --dns option to automatically proxy all local DNS requests over the established tunnel too (useful to avoid DNS DPI or censorship/blocking).

The one major flaw of sshuttle is it refuses to implement² an in-process reconnect mechanism, which seems like a big oversight for a reliable network tunneling transport. If your network is unstable, you have to run sshuttle in a loop under sudo:

sudo bash -c 'while [[ true ]]; do shuttle @sshuttle.conf; done'

Connection Establishment Fix: ssh Tunnels / Proxies / Port Forwarding

ssh has many tunneling features including port forwarding (-L) and creating an entire SOCKS5 proxy (-D). There are thousands of tutorials for each feature online, so ask around.

Connection Establishment Fix: autossh

A downside to ssh tunnels and proxies and port forwarding is: if your network gets interrupted the connection will most likely break, which is annoying for tunnels, proxies, and forwards.

The problem was so common we have a wrapper tool for ssh called autossh for monitoring remote connections and re-establishing upon breakage.

The simplest usage of autossh is autossh -M0 [remotehost] but autossh supports every complex ssh usage and more complex reachability measurements (the required -M option) to reliably re-establish all features of ssh.

Connection Establishment Fix: 3rd Party Proxy/VPN

Another tool for your traveling network reliability arsenal is of course any of the thousands of cheap “buy a lifetime subscription for $30” VPN companies out there. They typically offer a browser proxy service and maybe even system-level VPNs too.

When operating on unstable networks, find ways to move your connection establishment off your local network and onto (hopefully more reliable) long lived connections (if they aren’t constantly breaking too).

Connection Retry Fix: Loop It

Sometimes your connections work at first, then break after a minute or two (broken NAT timers in a captive portal? who knows).

But, we can fight against the breakage by just retrying resumable downloads until they complete. Your CPUs don’t care care if they have to retry a million times over night. They are very persistent.

Here’s a shell function I use to loop youtube-dl until downloads complete:

 y() {
     PATH="/usr/local/opt/python/libexec/bin:$PATH"
     while ! youtube-dl --no-mtime --socket-timeout 5 $1; do
         echo DISCONNECTED;                
         sleep 5;                          
     done                                  
 }

Usage: y https://www.youtube.com/watch?v=ei3cj0re0Kk

…and as for regular file downloads, here’s a script I use for wrapping curl to auto-resume partial downloads (which curl does not do by default! You must enable the resume options manually as below):

#!/usr/bin/env perl

use File::Basename;

my $url = $ARGV[0];

# Transform $url into a local filename without any query parameters.
my $filename = $url;
$filename =~ s/\?.*//;

# Cheap and dirty way of getting only a filename out of the URL
$filename = basename $filename;

print "Saving $url as $filename\n";

# Download URL $url to local name $filename
do {
    # Continue attempting download if it fails
    # or is interrupted/cutoff (-C -).
    # Download will resume from its last saved
    # position each time (if server supports it).
    `curl -L -o $filename -C - $url`;
} while (($? >> 8) != 0);

Usage: [scriptname] [url]

Problem: ICMP Garbage From Local Gateways

Another weirdness in the never-ending list of network problems: one place I stayed at had every connection attempt accompanied by ICMP Redirect packets sent from the router because it was misconfigured with respect to the hotel’s captive portal.

To stop the extra network garbage hitting my applications, I just blocked all ICMP from the local management network (as discovered with a combination of wireshark and the payload in the ICMP Redirects themselves) by adding this to /etc/pf.conf:

block in quick on en0 proto icmp from 4.4.4.0/24 to any

After the line is added, reload pf with sudo pfctl -f /etc/pf.conf. You may need to manually enable Firewall through Security & Privacy preferences if it isn’t already running.

Conclusion

Will we ever live in a world where you can visit a hotel and get Internet speeds faster than your phone? Right now LTE easily delivers 50 Mbps to individual handsets, but in most public wifi places you’re lucky to get 10 Mbps.

Is this like the TV situation where places advertise “HD TVs” and it’s some 720p screen with sextuple analog converted signals that look like 144p YouTube videos playing under an After Effects static generator effect?

How do we get non-technical people to understand “Internet” things when they can’t see it or touch it? There’s no easy answer other than “wait 25 more years and maybe we’ll get 2018 speed public Internet access by 2043.”

Happy Being Frustrated With Public Networks Forever,

-Matt — @mattsta — ☁mattsta

Infrastructure Week July 2018!

• day 1: lematt
• day 2: netmatt
• day 3: email 2018
• day 4: web 2018
• day 5: reliable local nets