Honeypot service setup¶

Dr. Crucible designed the honeypot services with the same philosophy he applies to teaching: make it look legitimate enough that someone who does not know what they are looking at will not question it, and make sure everything that happens is observed. The three internal honeypot services on the Golem Trust Headscale network have realistic hostnames, plausible response times, and data that looks like it matters. None of it does. But the attacker who found the SSH honeypot and spent three hours working through it did not know that until it was too late.

Infrastructure overview¶

Three honeypot VMs run on Hetzner CX11 instances (the smallest tier, 2 vCPU, 2 GB RAM, €3.79/month each). Total cost: approximately €8/month including storage and bandwidth, which is the most cost-effective security control in the Golem Trust stack. All three VMs are enrolled in Headscale and appear in internal DNS as legitimate service hostnames.

The VMs have no access to real internal systems. They are enrolled in Headscale purely to appear in internal DNS and be reachable from the internal network. They cannot initiate connections to production systems.

OpenCanary SSH honeypot¶

The SSH honeypot runs OpenCanary on hpot-ssh-01. It intentionally accepts password authentication with weak credentials. This is the point: a real SSH server would not do this, but an attacker scanning the internal network who finds an SSH server accepting passwords will try common credentials, succeed, and believe they have found a legitimate server.

OpenCanary configuration for the SSH service:

{
    "device.node_id": "honeypot-ssh-01",
    "git.enabled": false,
    "ftp.enabled": false,
    "http.enabled": false,
    "http.port": 80,
    "httpproxy.enabled": false,
    "mysql.enabled": false,
    "mssql.enabled": false,
    "nmap.enabled": false,
    "portscan.enabled": true,
    "redis.enabled": false,
    "sip.enabled": false,
    "smb.enabled": false,
    "snmp.enabled": false,
    "ssh.enabled": true,
    "ssh.port": 22,
    "ssh.version": "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6",
    "telnet.enabled": false,
    "tftp.enabled": false,
    "vnc.enabled": false,
    "logger": {
        "class": "PyLogger",
        "kwargs": {
            "formatters": {
                "plain": {
                    "format": "%(message)s"
                }
            },
            "handlers": {
                "SyslogHandler": {
                    "class": "logging.handlers.SysLogHandler",
                    "level": "DEBUG",
                    "formatter": "plain",
                    "socktype": "socket.SOCK_DGRAM",
                    "address": ["graylog.golemtrust.am", 514]
                }
            }
        }
    }
}

The SSH service returns a realistic fake shell. When an attacker authenticates (with any of the weak credentials), they see a prompt resembling a legitimate Ubuntu server. Commands produce plausible output. The /etc/passwd file contains fake user accounts. The ps aux output shows realistic processes. All of this is generated by OpenCanary’s fake shell functionality.

Weak credentials accepted by the SSH honeypot (these are intentional vulnerabilities):

root:password
root:admin
root:root
admin:admin
admin:password
ubuntu:ubuntu

OpenCanary PostgreSQL honeypot¶

The database honeypot runs OpenCanary’s PostgreSQL service on hpot-db-01. It accepts connections from any host on the internal network, using the same weak credential pattern. Once connected, it returns fake data that looks like a customer database.

The fake data is generated using the Python Faker library and loaded into OpenCanary’s fake response templates. The data includes:

• A customers table with fake names, emails, and account numbers

• A transactions table with realistic-looking amounts and references

• A staff table with fake employee records

The data is regenerated monthly (first Monday of each month, via a cron job) to maintain naming conventions consistent with what a real database might look like.

# /opt/honeypot/refresh-fake-data.py
from faker import Faker
import json

fake = Faker('en_GB')

customers = []
for i in range(500):
    customers.append({
        "id": i + 1,
        "name": fake.name(),
        "email": fake.email(),
        "account_number": fake.bban(),
        "sort_code": fake.numerify("##-##-##"),
        "balance": round(fake.pyfloat(min_value=0, max_value=50000, right_digits=2), 2)
    })

with open('/etc/opencanary/fake-customers.json', 'w') as f:
    json.dump(customers, f, indent=2)

Custom Flask API honeypot¶

The API honeypot is a custom Flask application on hpot-api-01. It simulates an internal API with an “accidentally left in” debug endpoint: /api/v1/debug/users. This endpoint requires no authentication, which is the intentional vulnerability. A real API endpoint at this path would be a serious vulnerability; in a honeypot, it is a trap.

# /opt/honeypot/app.py (excerpts showing the deceptive structure)
from flask import Flask, request, jsonify
import logging
import syslog

app = Flask(__name__)

@app.route('/api/v1/users', methods=['GET'])
def get_users():
    # Normal authenticated endpoint
    auth = request.headers.get('Authorization')
    if not auth:
        return jsonify({"error": "Unauthorized"}), 401
    return jsonify({"users": [], "message": "No access"}), 403

@app.route('/api/v1/debug/users', methods=['GET'])
def debug_users():
    # "Forgotten" debug endpoint - no auth required
    # Every access triggers alert
    source_ip = request.headers.get('X-Forwarded-For', request.remote_addr)
    user_agent = request.headers.get('User-Agent', 'unknown')

    alert_data = {
        "event_type": "honeypot_api_access",
        "endpoint": "/api/v1/debug/users",
        "source_ip": source_ip,
        "user_agent": user_agent,
        "timestamp": datetime.utcnow().isoformat()
    }
    syslog.syslog(syslog.LOG_ALERT, json.dumps(alert_data))

    # Return convincing fake data
    return jsonify({
        "debug": True,
        "users": FAKE_USERS,
        "warning": "Debug endpoint - disable before production"
    })

The warning message in the response is intentional. It makes the endpoint look like a real developer oversight rather than a trap.

Alert transport¶

All three honeypots send alerts to Graylog via syslog (UDP 514) to graylog.golemtrust.am. The Graylog input is configured to accept syslog from the honeypot IP range (100.64.20.0/24). The Graylog pipeline parses the OpenCanary JSON format and routes events to the Deception stream.

Maintaining honeypot realism¶

The honeypots must remain believable. Monthly maintenance tasks:

• Update the fake data in the PostgreSQL honeypot to reflect current naming conventions (current quarter, current service names)

• Update the SSH honeypot’s fake /etc/hostname and /etc/hosts to match current internal naming patterns

• Review the Flask API honeypot’s fake user list and update names and roles to look like current staff structure

• Check that the SSH service banner version matches a plausible Ubuntu LTS release

These updates are documented in security/deception/maintenance-log.md. An attacker who notices that the “server” is running an operating system version inconsistent with the current Golem Trust standard might become suspicious. The goal is to delay suspicion for as long as possible.