F5 SSL handshake failure: troubleshooting TLS systematically

A service behind F5 BIG-IP is unavailable. The browser reports a certificate error, a connection reset or a failed TLS handshake, while the pool remains green. These observations can all be correct: pool health, TLS negotiation and application behaviour are separate parts of the same connection.

A useful investigation starts by locating the failure. Changing certificates or cipher settings before identifying the failing connection can introduce a second problem without resolving the first.

Separate client-side and server-side TLS

When BIG-IP terminates TLS and encrypts traffic again towards the backend, there are two independent TLS connections. On the client-facing connection, BIG-IP acts as a TLS server and uses a Client SSL profile. On the backend connection, it acts as a TLS client and uses a Server SSL profile.

These connections can use different certificates, protocol versions and cryptographic settings. The certificate received by the browser does not have to be the certificate installed on the backend. F5 describes these roles in its SSL Traffic Management documentation.

TLS passthrough is different: BIG-IP does not terminate that TLS session. Establish the actual traffic model before choosing the next test.

Test the VIP with the correct hostname

Testing only the IP address can produce misleading results. A service may depend on Server Name Indication, or SNI. A connection without the intended server name may receive a different certificate or reach a different TLS configuration.

The following OpenSSL test explicitly separates the connection address from the intended hostname:

openssl s_client \
  -connect 192.0.2.10:443 \
  -servername portal.example.nl \
  -verify_hostname portal.example.nl \
  -verify_return_error \
  -showcerts </dev/null

The hostname verification option checks the certificate identity. The verification-error option makes certificate verification failures fail the handshake rather than merely report them. For an internal PKI, add the appropriate trusted CA using the relevant OpenSSL trust options. See the official s_client reference.

Inspect the leaf certificate, its Subject Alternative Names, the supplied intermediate certificates and the negotiated TLS version. Record the actual error rather than grouping every failure under “SSL is broken”. A missing intermediate certificate and a hostname mismatch require different corrections.

Test HTTP without changing DNS

A successful handshake does not prove that the application responds correctly. Test a particular VIP while retaining the application hostname:

curl \
  --resolve portal.example.nl:443:192.0.2.10 \
  --connect-timeout 5 \
  --max-time 15 \
  --verbose \
  https://portal.example.nl/healthz

For this request, --resolve connects the specified hostname and port to the chosen IP. The URL hostname still supplies the intended HTTP and TLS context. The curl project explains this in Name resolve tricks.

Avoid adding -k as a routine troubleshooting step: it bypasses the certificate verification that this test is meant to assess.

Investigate the backend connection separately

If client-side TLS succeeds but the application remains unavailable, test the backend from an authorised investigation point. Match its expected server name and trust requirements as closely as possible.

A successful laptop test does not prove that BIG-IP can establish the same connection. The laptop may trust different certificate authorities or offer different protocol settings. Compare the test with the actual Server SSL configuration, including SNI and any client-certificate requirement.

Use these results to narrow the investigation. If the backend responds only when a particular server name is supplied, investigate that difference. If the connection cannot be established at all, examine the network path before changing application settings.

Produce a specific diagnosis

The useful outcome is a statement identifying which connection fails, at which stage and under which conditions. For example: client-side TLS presents the expected certificate, but backend negotiation fails when the expected server name is absent.

That finding identifies a configuration boundary and a repeatable verification test. After making the correction, repeat the TLS and application tests together. The service is recovered when the intended client can establish a trusted connection and complete a representative application request.