F5 pool member down: troubleshooting health monitors

A BIG-IP pool member is marked down even though the application appears reachable directly. Relaxing or disabling its monitor may restore traffic, but it also removes the control that prevents requests from reaching an unusable instance.

The investigation should explain why the monitor and the manual test produce different results. That difference is usually more informative than the red status indicator itself.

Examine status and configuration

Start with the current pool status and the relevant definitions:

tmsh show ltm pool /Common/POOL_PORTAL detail

tmsh list ltm pool /Common/POOL_PORTAL all-properties

tmsh list ltm monitor http /Common/MON_PORTAL all-properties

Pool status and pool configuration answer different questions. Check whether a member has its own monitor configuration, and whether administrative state or session settings restrict its use. The TMSH pool reference describes these settings separately.

Before changing anything, record the affected object, applied monitor and start time. Establish whether one instance, one service or the entire pool is affected. This provides a baseline for the next test and avoids confusing a new effect with the original fault.

A monitor sends a specific request

An HTTP monitor does not test an abstract “website”. It sends a request to an address and port, then evaluates the response according to its settings.

For a host-dependent application, the intended request might be:

GET /healthz HTTP/1.1
Host: portal.example.nl
Connection: close

In a monitor send string, the line endings and final blank line need to be represented correctly:

GET /healthz HTTP/1.1\r\nHost: portal.example.nl\r\nConnection: close\r\n\r\n

Without the expected Host header, a web server may return a default site. A browser using the application hostname can therefore succeed while the monitor evaluates a different response. Consult the HTTP monitor TMSH reference when checking the actual definition.

Decide what “healthy” must mean

A receive string should prove something meaningful about the instance. A word found on the homepage may also appear on a maintenance or error page.

A readiness endpoint can return a controlled response:

{
  "service": "portal",
  "status": "ready"
}

The chosen monitor must interpret that response deliberately. F5 supports regular-expression matching for certain monitor types; response status and other settings also affect the result. The Monitor Settings Reference explains those distinctions.

The endpoint itself also needs a defined purpose. An endpoint that always reports readiness adds little. Conversely, treating every optional dependency as critical may remove all instances during a shared external outage. Agree with the application team which conditions make this particular instance unsuitable for traffic.

Reproduce the request

Use the same path and Host header when testing manually. Run the example from a host with curl and permitted access to the backend; tool availability on BIG-IP depends on the installed release:

curl \
  --connect-timeout 5 \
  --max-time 10 \
  --header 'Host: portal.example.nl' \
  --dump-header - \
  http://10.20.30.11:8080/healthz

Inspect response content, redirects and response time as well as the status code. A redirect to authentication is different from a direct readiness response.

This reproduces the HTTP request, not automatically the monitor's source address, route or complete TLS behaviour. Keep that limitation explicit when interpreting a successful result.

Follow the network path

When the request succeeds from an investigation host but fails from BIG-IP, examine the path between the monitor and backend. Check the actual source address, firewall rules, service port and return route.

A capture during a monitor probe can distinguish unanswered SYN packets, resets and complete responses containing unexpected content. Those observations identify different fault categories; they should not all lead to weakening the receive condition.

Verify failure and recovery

After correcting the cause, test both directions. Make an instance unsuitable in a controlled way, verify its removal from service, then restore it and observe its return.

Record whether those transitions meet the application's operational needs. The final result should explain the original discrepancy and demonstrate that a green member represents a usable instance, not simply a reachable socket.