Runbooks for Verifying an Unexpected Listening Service

Runbooks for Verifying an Unexpected Listening Service

An external scan shows a port that wasn’t open yesterday, and the on-call engineer has to work out whether it’s a forgotten test container or an attacker’s foothold. A runbook for verifying an unexpected listening service turns that moment into a set of repeatable checks: confirm the exposure, find the process behind it, establish who owns it and decide what happens next. This article covers how to build that runbook, which commands it should contain and where teams usually go wrong.

The payoff is speed and consistency. With a tested runbook, a junior admin can get from alert to verified answer in 15-30 minutes. Without one, two senior engineers can spend an afternoon arguing in Slack about whether the port is real.

Why a Runbook Beats Improvisation

A new listener raises three questions: is it really reachable, what is it, and should it be there? Improvised investigations tend to answer the second question first and skip the first one completely. An engineer SSHs in, sees nginx, decides it’s fine, and nobody notices that the external scan actually flagged port 8443 rather than 443.

A good runbook fixes the order of checks. It also sets a clear point where verification stops and containment starts. That matters at 2 a.m. when nobody wants to make judgment calls.

Step-by-Step Runbook for an Unexpected Listening Service

An experienced responder works from the outside in and keeps a timestamped log of every command and its output. These are the core steps.

Step 1 – Re-confirm from outside. Run a targeted check from a separate vantage point, for example nmap -Pn -sV -p 8443 203.0.113.10 from a cloud VM outside your own network. One scan result can be a transient artefact. Two independent results showing the same banner make the exposure confirmed.

Step 2 – Identify the process on the host. On Linux, ss -tulpn | grep 8443 gives you the PID and binary. On Windows, Get-NetTCPConnection -LocalPort 8443 followed by Get-Process -Id does the same. If nothing on the host is listening on that port, look at NAT rules, load balancers or Docker before you suspect anything exotic.

Step 3 – Trace the binary. Check the executable path with readlink /proc/PID/exe, the parent process, the systemd unit or scheduled task, and the package owner (rpm -qf or dpkg -S). A binary in /tmp or /dev/shm, or one that was deleted from disk while still running, should send you straight to containment.

Step 4 – Fingerprint the service. Compare the version string against what the process claims to be. Proper service fingerprinting catches cases where a “web server” answers with an SSH banner, or where Redis 6.0 is running on a host that was supposed to be standardised on 7.2.

Step 5 – Match against change records. Look up deployments, Terraform applies, Ansible runs and tickets from the last 24-72 hours. Correlating port changes with deployment timelines explains most surprises within minutes.

A Scenario Most Ops Teams Will Recognise

On a Monday morning, a weekly external scan reports port 6379 open on a public VPS that runs a small SaaS backend. The firewall is UFW, and ufw status shows only 22, 80 and 443 allowed. The admin’s first instinct is that the scanner must be wrong.

It isn’t. A developer added Redis to a docker-compose file on Friday with ports: “6379:6379”. Docker writes its own iptables rules in the DOCKER chain, and those rules take effect before UFW’s rules can filter the traffic. Redis was running with no password, reachable from the entire internet for roughly 60 hours. The fix took one line: bind to 127.0.0.1:6379:6379. The verification work still mattered, because someone had to check Redis for unfamiliar keys, run CONFIG GET dir to see whether an attacker had redirected persistence, and review authorized_keys files.

This is why Step 2 includes docker ps and iptables -t nat -L -n. Host-level tools and the firewall frontend can disagree about what is actually exposed.

Myth: “If the Firewall Says Closed, the Port Is Closed”

This belief is responsible for more surprises than any other. A firewall configuration describes what you intended to allow. An external scan shows what is actually reachable. Docker, Kubernetes NodePorts (30000-32767 by default), cloud security groups changed outside Terraform and IPv6 rules that nobody mirrored all create gaps between the two.

IPv6 is especially easy to miss. A server with a strict iptables policy and an empty ip6tables policy is fully exposed on its IPv6 address. Your runbook should check both address families every time.

Common Mistakes When Verifying Listeners

Killing the process before capturing evidence. If the listener turns out to be malicious, you have just lost its memory, its open file handles and its network connections. Before you stop anything, capture ls -l /proc/PID/, lsof -p PID and the active connections. Once you suspect compromise, move to dedicated containment steps after finding a rogue listener.

Trusting the banner. Banners are just strings, and attackers routinely make backdoors look like Apache or OpenSSH. A process claiming to be sshd but running from /usr/local/share deserves suspicion, whatever it tells the scanner.

Closing the ticket at “it’s ours.” A known service that is exposed by mistake is still a finding. Verification should end with an explicit decision: keep the port open and document an owner, restrict it to specific source IPs, or close it.

Edge Cases Your Runbook Should Cover

Shared hosting and managed platforms limit what you can see. On AWS Fargate or Azure App Service you have no host shell, so Step 2 means checking task definitions and platform logs. In regulated environments under PCI DSS 4.0 or NIS2, policy may require you to notify a security lead before you touch a production system at all. The runbook should state that requirement up front, not bury it in step four.

Small teams of 2-5 people often skip formal runbooks because “everyone knows the servers.” That knowledge lasts until someone goes on holiday. A one-page runbook kept in the same repository as the infrastructure code is enough.

Air-gapped or internal-only systems need the same process run from an internal scanning host. The external confirmation step becomes a check from a different network segment.

FAQ

How quickly should an unexpected listening service be verified?
For internet-facing hosts, aim to have the exposure confirmed and the process identified within one hour. Database, remote management and admin-panel ports should be treated as urgent, because automated scanners often find new exposures within minutes.

Should the runbook be different for Windows and Linux servers?
The logic is the same, but the commands differ. Keep one decision flow and add OS-specific command blocks: ss, lsof and systemctl for Linux, and Get-NetTCPConnection, Get-Process and Get-ScheduledTask for Windows.

When does verification turn into incident response?
Escalate immediately if the binary runs from a temporary directory, has no package owner, has been deleted from disk while still running, or doesn’t match any change record. Any one of these is enough.

Final Practical Tip

Test the runbook before you need it. Once a quarter, start an unexpected listener yourself, for example python3 -m http.server 8999 on a staging host, and time how long the team takes to detect, trace and resolve it. Each gap you find in a drill is one you won’t hit during a real incident. A runbook that has never been tested is only a list of good intentions.