Skip to content

TLS handshake failing until I set the tcp-mss to 1322 or lower #455

Open
Open
TLS handshake failing until I set the tcp-mss to 1322 or lower#455

Description

@ruffy91
ruffy91
opened on Jul 15, 2026

I am trying to wrap my head around a problem I have when visiting my banks webservers.
I use Jool 4.1.15 as part ov VyOS 1.5 rolling release for NAT64 in a IPv6-only network.

Most of the websites just work. But accessing https://www.ubs.com fails because the TLS handshake can not complete.

I clamped tcp-mss to a lower value until the connection succeeded.

Why is the clamping needed in this case?

With mss 1330:
I send them a packet cut to 1330 (DF) + 425, while their answer is cut to 1330+1330+1436+768 (why 1436? why DF on the 1330 packet?)

With mss 1322:
I send them 1755, they send me 3700. In this case the connection is successful (why no segmentation? Is this TSO/GRO/GSO at work?)

I am trying to find out if this is a error on my side (VyOS/NIC/Config?) of the connection or if the bank is doing something wrong/strange.

When clamping tcp-mss to 1330 the TLS handshake fails:

20:11:48.814802 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 50031, offset 0, flags [none], proto TCP (6), length 52)
    192.0.2.6.62910 > 193.134.111.71.443: Flags [S], cksum 0x5622 (correct), seq 1189319263, win 65535, options [mss 1330,nop,wscale 8,nop,nop,sackOK], length 0
20:11:48.828719 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 238, id 60063, offset 0, flags [none], proto TCP (6), length 52)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [S.], cksum 0x47a4 (correct), seq 4062747715, ack 1189319264, win 32768, options [mss 1330,nop,wscale 8,nop,nop,sackOK], length 0
20:11:48.829095 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 50036, offset 0, flags [none], proto TCP (6), length 40)
    192.0.2.6.62910 > 193.134.111.71.443: Flags [.], cksum 0x06f7 (correct), seq 1, ack 1, win 255, length 0
20:11:48.829254 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 1384: (tos 0x0, ttl 63, id 50037, offset 0, flags [DF], proto TCP (6), length 1370)
    192.0.2.6.62910 > 193.134.111.71.443: Flags [.], cksum 0x2c1d (correct), seq 1:1331, ack 1, win 255, length 1330
20:11:48.829258 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 479: (tos 0x0, ttl 63, id 50038, offset 0, flags [none], proto TCP (6), length 465)
    192.0.2.6.62910 > 193.134.111.71.443: Flags [P.], cksum 0x7b7d (correct), seq 1331:1756, ack 1, win 255, length 425
20:11:48.843131 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 238, id 60064, offset 0, flags [none], proto TCP (6), length 40)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xff20 (correct), seq 1, ack 1756, win 506, length 0
20:11:48.844234 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 238, id 27550, offset 0, flags [none], proto TCP (6), length 40)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xff20 (correct), seq 1, ack 1756, win 506, length 0
20:11:48.846632 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 1384: (tos 0x0, ttl 238, id 27551, offset 0, flags [none], proto TCP (6), length 1370)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xdcc3 (correct), seq 1:1331, ack 1756, win 506, length 1330
20:11:48.846767 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 1384: (tos 0x0, ttl 238, id 27552, offset 0, flags [none], proto TCP (6), length 1370)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xa43e (correct), seq 1331:2661, ack 1756, win 506, length 1330
20:11:48.846842 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 1490: (tos 0x0, ttl 238, id 27553, offset 0, flags [none], proto TCP (6), length 1476)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [P.], cksum 0x4e6a (incorrect -> 0xaf84), seq 2661:4097, ack 1756, win 506, length 1436
20:11:48.848554 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 822: (tos 0x0, ttl 238, id 27555, offset 0, flags [none], proto TCP (6), length 808)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [P.], cksum 0x7a06 (correct), seq 4097:4865, ack 1756, win 506, length 768
20:11:48.848683 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 50046, offset 0, flags [none], proto TCP (6), length 52)
    192.0.2.6.62910 > 193.134.111.71.443: Flags [.], cksum 0x8a29 (correct), seq 1756, ack 1, win 255, options [nop,nop,sack 1 {4097:4865}], length 0
20:11:48.853431 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 822: (tos 0x0, ttl 238, id 27556, offset 0, flags [none], proto TCP (6), length 808)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [P.], cksum 0x7a06 (correct), seq 4097:4865, ack 1756, win 506, length 768
20:11:48.853532 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 50047, offset 0, flags [none], proto TCP (6), length 60)
    192.0.2.6.62910 > 193.134.111.71.443: Flags [.], cksum 0x2a3e (correct), seq 1756, ack 1, win 255, options [nop,nop,sack 2 {4097:4865}{4097:4865}], length 0
20:11:48.863247 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 1384: (tos 0x0, ttl 238, id 27557, offset 0, flags [none], proto TCP (6), length 1370)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xdcc3 (correct), seq 1:1331, ack 1756, win 506, length 1330
20:11:48.867794 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 1384: (tos 0x0, ttl 238, id 27558, offset 0, flags [none], proto TCP (6), length 1370)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xa43e (correct), seq 1331:2661, ack 1756, win 506, length 1330
20:11:49.070587 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 1384: (tos 0x0, ttl 238, id 27559, offset 0, flags [none], proto TCP (6), length 1370)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xdcc3 (correct), seq 1:1331, ack 1756, win 506, length 1330
20:11:49.486599 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 1384: (tos 0x0, ttl 238, id 27560, offset 0, flags [none], proto TCP (6), length 1370)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xdcc3 (correct), seq 1:1331, ack 1756, win 506, length 1330
20:11:50.350589 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 1384: (tos 0x0, ttl 238, id 27561, offset 0, flags [none], proto TCP (6), length 1370)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xdcc3 (correct), seq 1:1331, ack 1756, win 506, length 1330
20:11:52.014438 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 1384: (tos 0x0, ttl 238, id 27562, offset 0, flags [none], proto TCP (6), length 1370)
    193.134.111.71.443 > 192.0.2.6.62910: Flags [.], cksum 0xdcc3 (correct), seq 1:1331, ack 1756, win 506, length 1330
20:11:54.831392 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 53498, offset 0, flags [none], proto TCP (6), length 52)

Clamping it to 1322 makes the connection succeed:

20:15:59.255177 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 47122, offset 0, flags [none], proto TCP (6), length 52)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [S], cksum 0x3cdf (correct), seq 245490650, win 65535, options [mss 1322,nop,wscale 8,nop,nop,sackOK], length 0
20:15:59.266836 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 238, id 33952, offset 0, flags [none], proto TCP (6), length 52)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [S.], cksum 0x9203 (correct), seq 1990702216, ack 245490651, win 32768, options [mss 1220,nop,wscale 8,nop,nop,sackOK], length 0
20:15:59.267254 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 47134, offset 0, flags [none], proto TCP (6), length 40)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [.], cksum 0x50e8 (correct), seq 1, ack 1, win 255, length 0
20:15:59.267370 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 1809: (tos 0x0, ttl 63, id 47135, offset 0, flags [none], proto TCP (6), length 1795)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [P.], cksum 0x4fa9 (incorrect -> 0xf487), seq 1:1756, ack 1, win 255, length 1755
20:15:59.279033 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 238, id 33953, offset 0, flags [none], proto TCP (6), length 40)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [.], cksum 0x4912 (correct), seq 1, ack 1756, win 506, length 0
20:15:59.280111 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 238, id 60651, offset 0, flags [none], proto TCP (6), length 40)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [.], cksum 0x4912 (correct), seq 1, ack 1756, win 506, length 0
20:15:59.282579 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 3714: (tos 0x0, ttl 238, id 60652, offset 0, flags [none], proto TCP (6), length 3700)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [.], cksum 0x571a (incorrect -> 0x04ee), seq 1:3661, ack 1756, win 506, length 3660
20:15:59.282689 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 490: (tos 0x0, ttl 238, id 60655, offset 0, flags [none], proto TCP (6), length 476)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [P.], cksum 0xebe0 (correct), seq 3661:4097, ack 1756, win 506, length 436
20:15:59.282720 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 47143, offset 0, flags [none], proto TCP (6), length 40)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [.], cksum 0x3bc1 (correct), seq 1756, ack 3661, win 255, length 0
20:15:59.282824 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 47144, offset 0, flags [none], proto TCP (6), length 40)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [.], cksum 0x3a0e (correct), seq 1756, ack 4097, win 254, length 0
20:15:59.284399 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 822: (tos 0x0, ttl 238, id 60656, offset 0, flags [none], proto TCP (6), length 808)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [P.], cksum 0x9a61 (correct), seq 4097:4865, ack 1756, win 506, length 768
20:15:59.285328 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 134: (tos 0x0, ttl 63, id 47145, offset 0, flags [none], proto TCP (6), length 120)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [P.], cksum 0x651b (correct), seq 1756:1836, ack 4865, win 251, length 80
20:15:59.285382 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 781: (tos 0x0, ttl 63, id 47146, offset 0, flags [none], proto TCP (6), length 767)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [P.], cksum 0x0fd5 (correct), seq 1836:2563, ack 4865, win 251, length 727
20:15:59.288400 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 822: (tos 0x0, ttl 238, id 60657, offset 0, flags [none], proto TCP (6), length 808)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [P.], cksum 0x9a61 (correct), seq 4097:4865, ack 1756, win 506, length 768
20:15:59.288637 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 63, id 47149, offset 0, flags [none], proto TCP (6), length 52)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [.], cksum 0x8470 (correct), seq 2563, ack 4865, win 251, options [nop,nop,sack 1 {4097:4865}], length 0
20:15:59.298875 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 133: (tos 0x0, ttl 238, id 60658, offset 0, flags [none], proto TCP (6), length 119)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [P.], cksum 0x0abc (correct), seq 4865:4944, ack 2563, win 503, length 79
20:15:59.300187 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 133: (tos 0x0, ttl 238, id 60659, offset 0, flags [none], proto TCP (6), length 119)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [P.], cksum 0xf8da (correct), seq 4944:5023, ack 2563, win 503, length 79
20:15:59.300329 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 47159, offset 0, flags [none], proto TCP (6), length 40)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [.], cksum 0x3348 (correct), seq 2563, ack 5023, win 255, length 0
20:15:59.302282 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 645: (tos 0x0, ttl 238, id 60660, offset 0, flags [none], proto TCP (6), length 631)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [P.], cksum 0xfbe7 (correct), seq 5023:5614, ack 2563, win 503, length 591
20:15:59.302609 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 238, id 60661, offset 0, flags [none], proto TCP (6), length 64)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [P.], cksum 0x6e98 (correct), seq 5614:5638, ack 2563, win 503, length 24
20:15:59.302724 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 47160, offset 0, flags [none], proto TCP (6), length 40)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [F.], cksum 0x30e2 (correct), seq 2563, ack 5638, win 253, length 0
20:15:59.302726 de:ad:be:ef:ca:fe > 00:00:f7:f7:ca:fe, ethertype IPv4 (0x0800), length 54: (tos 0x0, ttl 63, id 47161, offset 0, flags [none], proto TCP (6), length 40)
    192.0.2.6.62928 > 193.134.111.71.443: Flags [R.], cksum 0x31db (correct), seq 2564, ack 5638, win 0, length 0
20:15:59.302721 00:00:f7:f7:ca:fe > de:ad:be:ef:ca:fe, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 238, id 60662, offset 0, flags [none], proto TCP (6), length 40)
    193.134.111.71.443 > 192.0.2.6.62928: Flags [F.], cksum 0x2fe8 (correct), seq 5638, ack 2563, win 503, length 0

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions