When making my home NAS reachable via Tailscale I experienced very slow TCP speeds, but only when uploading from the NAS and only when going through the Tailscale tunnel.
The journey
I was trying to get some data from my NAS, and noticed that the connection seemed quite slow, around 8-10 MBit. I started out by making sure that this is not an issue with my internet connection. I ran the following from various machines on that network, and got consistent results:
$ docker run --rm moutten/speedtest-cli --accept-gdpr
Speedtest by Ookla
Server: <redacted>
ISP: <redacted>
Latency: 1.65 ms (0.04 ms jitter)
Download: 389.15 Mbps (data used: 473.9 MB )
Upload: 383.03 Mbps (data used: 425.9 MB )
Packet Loss: 0.0%
At first I suspected the issue was with my mounted SMB share, so I switched to SFTP and was seeing the same issues. I then tried to use iperf3 to get more insights into whats going on. On the NAS, I ran the following to start an iperf3 server:
docker run -it --rm --network=host networkstatic/iperf3 -s
Then, from the remote machine, I ran the following tests. First, in the direction from the remote machine to the NAS, which showed good speeds:
$ iperf3 -c 100.101.102.104
Connecting to host 100.101.102.104, port 5201
[ 5] local 100.85.100.101 port 56494 connected to 100.101.102.104 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 54.7 MBytes 459 Mbits/sec 102 1.04 MBytes
[ 5] 1.00-2.00 sec 26.2 MBytes 220 Mbits/sec 4 297 KBytes
[ 5] 2.00-3.00 sec 22.5 MBytes 189 Mbits/sec 0 343 KBytes
[ 5] 3.00-4.00 sec 26.2 MBytes 220 Mbits/sec 0 391 KBytes
[ 5] 4.00-5.00 sec 28.8 MBytes 241 Mbits/sec 0 437 KBytes
[ 5] 5.00-6.00 sec 32.5 MBytes 273 Mbits/sec 0 482 KBytes
[ 5] 6.00-7.00 sec 36.2 MBytes 304 Mbits/sec 0 529 KBytes
[ 5] 7.00-8.00 sec 38.8 MBytes 325 Mbits/sec 0 576 KBytes
[ 5] 8.00-9.00 sec 43.8 MBytes 367 Mbits/sec 0 622 KBytes
[ 5] 9.00-10.00 sec 46.2 MBytes 388 Mbits/sec 0 669 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 356 MBytes 299 Mbits/sec 106 sender
[ 5] 0.00-10.01 sec 353 MBytes 296 Mbits/sec receiver
However, examining traffic in the other direction (note the -R in the iperf command) showed a very different picture, with slow and variable connection speeds as well as intermittent stalling (sometimes):
Connecting to host 100.101.102.104, port 5201
Reverse mode, remote host 100.101.102.104 is sending
[ 5] local 100.85.100.101 port 37272 connected to 100.101.102.104 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.37 MBytes 11.5 Mbits/sec
[ 5] 1.00-2.00 sec 803 KBytes 6.58 Mbits/sec
[ 5] 2.00-3.00 sec 838 KBytes 6.87 Mbits/sec
[ 5] 3.00-4.00 sec 620 KBytes 5.08 Mbits/sec
[ 5] 4.00-5.00 sec 788 KBytes 6.45 Mbits/sec
[ 5] 5.00-6.00 sec 680 KBytes 5.57 Mbits/sec
[ 5] 6.00-7.00 sec 1.25 MBytes 10.5 Mbits/sec
[ 5] 7.00-8.00 sec 548 KBytes 4.49 Mbits/sec
[ 5] 8.00-9.00 sec 937 KBytes 7.67 Mbits/sec
[ 5] 9.00-10.00 sec 1.08 MBytes 9.06 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.02 sec 9.00 MBytes 7.54 Mbits/sec 68 sender
[ 5] 0.00-10.00 sec 8.79 MBytes 7.38 Mbits/sec receiver
Notably, this was not an issue over UDP:
$ iperf3 --udp --client 100.101.102.104 --bitrate 350M -R
Connecting to host 100.101.102.104, port 5201
Reverse mode, remote host 100.101.102.104 is sending
[ 5] local 100.85.100.101 port 38418 connected to 100.101.102.104 port 5201
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-1.00 sec 41.7 MBytes 350 Mbits/sec 0.017 ms 0/35630 (0%)
[ 5] 1.00-2.00 sec 41.7 MBytes 350 Mbits/sec 0.018 ms 21/35626 (0.059%)
[ 5] 2.00-3.00 sec 41.7 MBytes 350 Mbits/sec 0.032 ms 0/35625 (0%)
[ 5] 3.00-4.00 sec 41.7 MBytes 350 Mbits/sec 0.015 ms 0/35633 (0%)
[ 5] 4.00-5.00 sec 41.7 MBytes 350 Mbits/sec 0.021 ms 0/35619 (0%)
[ 5] 5.00-6.00 sec 41.7 MBytes 350 Mbits/sec 0.022 ms 0/35628 (0%)
[ 5] 6.00-7.00 sec 41.7 MBytes 350 Mbits/sec 0.036 ms 0/35617 (0%)
[ 5] 7.00-8.00 sec 41.7 MBytes 350 Mbits/sec 0.020 ms 59/35639 (0.17%)
[ 5] 8.00-9.00 sec 41.7 MBytes 350 Mbits/sec 0.017 ms 0/35628 (0%)
[ 5] 9.00-10.00 sec 41.7 MBytes 350 Mbits/sec 0.028 ms 0/35620 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.01 sec 418 MBytes 350 Mbits/sec 0.000 ms 0/356725 (0%) sender
[ 5] 0.00-10.00 sec 417 MBytes 350 Mbits/sec 0.028 ms 80/356265 (0.022%) receiver
The solution
So it turns out that Tailscale tunnels all of its traffic over UDP, which was easy to see running tcpdump on either of the machines running iperf. Not being very versed in networking, I’m not really sure what’s at play here specifically, but I think it’s something along the lines of:
- UDP upstream from my connection is slightly lossy
- TCP tunneled over UDP interprets dropped / missed packages as congestion, and throttles
What helped in the end was to increase TCP socket buffers on my NAS:
sudo sysctl -w net.ipv4.tcp_rmem="4096 131072 50000000"
sudo sysctl -w net.ipv4.tcp_wmem="4096 131072 50000000"
The specific values I decided based on this page.
This worked okay for a comparatively low-latency scenario (~20ms). However, for higher latencies (>100ms) the I saw very slow speeds and stalls once more. Using BBR congestion control helped in that situation, which is apparently designed to work well for situations where TCP traffic is known to be tunneled. Unfortunately my NAS does not support it.
sudo sysctl -w net.ipv4.tcp_congestion_control=bbr
sudo sysctl -w net.core.default_qdisc=fq