Open in app

Sign in

Write

Sign in

Monitoring Adversaries at Your Trapdoor with Security Onion

Wes Lambert
11 min readApr 7, 2021

--

Introduction

There’s been quite a buzz 🐝 around honeypots and honeytokens recently. Not long ago, Chris Sanders released his latest book, Intrusion Detection Honeypots: Detection Through Deception. In the book, Chris does an excellent job of walking through the various ways that honeypots can be leveraged for detection purposes. In this article, we’ll briefly describe one way in which we can leverage a serverless honeypot/honeytokens to facilitate intrusion detection. 🍯

3CORESec has introduced a new serverless HTTP honeytoken framework for AWS, called Trapdoor. This application is inspired from the great work of Adel in honeyLambda. With Trapdoor, we can quickly and easily set up a detection and send notifications (ex. Slack, HTTP POST) when someone accesses a particular URL we’ve designated as our honeytoken. We can also set up friendly reminders to associate a particular URI or path with a label. The team at 3CORESec put out their own blog here describing Trapdoor in great detail, so we won’t go over all of the super cool things it can do here.

Depending on the level of operational security practiced by the adversary, the information obtained from the client accessing the URL can include things like:

  • Client IP
  • Client Browser/User Agent
  • Client Language(s)
  • Client/host machine specs
  • Client/host display characteristics
  • How many times (and at what times) a specific client has triggered our trapdoor
  • and much more! 👏

We can leverage the detection and notification capabilities of Trapdoor to provide greater context around honeytoken interaction by associating it with other types of data natively collected by Security Onion. Network connection data or transactional data provided by Zeek or Suricata, as well as host-based telemetry provided by Osquery or Wazuh, and file metadata provided by Strelka can assist in painting a more complete picture of adversary activity.

In order to facilitate the aforementioned, we’ll first need to set up a VPC, as well as an instance of Security Onion in AWS. Next we’ll need to configure Logstash, and an Elasticsearch Ingest Node pipeline to receive notifications from Trapdoor, then configure Trapdoor itself to send notifications to Security Onion (and ensure the appropriate firewall rules are in place to allow it to do so). Last, we will test our work and ensure everything is working as it should. We’ll discuss this process in more detail below.

VPC

Within AWS, set up a VPC with both a private and public subnet, and a NAT gateway, or alternative for outbound internet access from the private subnet.

VPN/connectivity to hosts in the private subnet won’t be covered here, but if not using VPN, you can set up an instance of Apache Guacamole or another type of bastion host in the public subnet to broker access to the Security Onion instance in the private subnet.

Security Onion

Launch the Security Onion AMI from the AWS Marketplace, or perform a network install of Security Onion on Ubuntu 18.04 or Centos 7, using an instance size of least t3a.xlarge, with 200GB storage, and place it in the private subnet.

If VPC mirroring is desired (not necessary for this demo), add another NIC to the Security Onion instance (along with appropriate security group associations for receiving mirrored traffic), and follow the guide here for setting it up.

Logstash

In this example, we will create a custom HTTP input for Logstash, and utilize the already available port that Logstash publishes for Beats communication on port 5044 in Security Onion.

To make these changes, we will do the following:

In /opt/so/saltstack/local/salt/logstash/pipelines/config/custom/ , we will create 0001_http_input.conf .

The file will contain the following:

input {
 http {
   id => "trapdoor_alerts"
   port => 5044
   tags => [“trapdoor”]
 }
}

We will also copy/opt/so/saltstack/default/pillar/logstash/manager.sls to /opt/so/saltstack/local/pillar/logstash/manager.sls , and modify the manager pillar to remove the existing so/0009_input_beats.conf , and add custom/0001_http_input.conf to the list.

We’ll need to create a special output config file, so that Logstash knows which pipeline to send alerts from Trapdoor. This file will be created in

/opt/so/saltstack/local/salt/logstash/config/custom/ , and and be called 9888_output_trapdoor.conf.jinja . Within this file, we’ll paste the following content:

{%- if grains['role'] == 'so-eval' -%}
{%- set ES = salt['pillar.get']('manager:mainip', '') -%}
{%- else %}
{%- set ES = salt['pillar.get']('elasticsearch:mainip', '') -%}
{%- endif %}
{% set FEATURES = salt['pillar.get']('elastic:features', False) %}
output {
  if "trapdoor" in [tags] {
    elasticsearch {
      pipeline => "trapdoor"
      hosts => "{{ ES }}"
      index => "so-honeypot"
      template_name => "so-common"
      template => "/templates/so-common-template.json"
      template_overwrite => true
      {%- if grains['role'] in ['so-node','so-heavynode'] %}
      ssl => true
      ssl_certificate_verification => false
      {%- endif %}
    }
  }
}

Next, we’ll copy/opt/so/saltstack/default/pillar/logstash/search.sls to /opt/so/saltstack/local/pillar/logstash/search.sls , and modify the file in the local directory (so that we are referencing our newly created output file).

We’ll simply add custom/9888_output_trapdoor.conf.jinja to the bottom of the list in the file, so that it looks like the following:

{%- set PIPELINE = salt['pillar.get']('global:pipeline', 'minio') %}
logstash:
  pipelines:
    search:
      config:
        - so/0900_input_redis.conf.jinja
        - so/9000_output_zeek.conf.jinja
        - so/9002_output_import.conf.jinja
        - so/9034_output_syslog.conf.jinja
        - so/9100_output_osquery.conf.jinja
        - so/9400_output_suricata.conf.jinja
        - so/9500_output_beats.conf.jinja
        - so/9600_output_ossec.conf.jinja
        - so/9700_output_strelka.conf.jinja
        - custom/9888_output_trapdoor.conf.jinja

Finally, we will restart Logstash with so-logstash-restart to bring in all of our changes.

Elastic Ingest

At this point, we have Logstash listening for alerts from Trapdoor, then forwarding them to the appropriate Elastic Ingest pipeline, however, we don’t yet have an ingest pipeline created.

To create a pipeline for alerts from Trapdoor:

  • Create a file called trapdoor in: /opt/so/saltstack/local/salt/elasticsearch/files/ingest
  • Within the file, include the following content:
{
  "description" : "Trapdoor honeypot alerts",
  "processors" : [
    { "set":         { "field": "event.module",        "value": "trapdoor"  } },
    { "set":         { "field": "event.dataset",        "value": "alert"  } },
    { "rename":      { "field": "host",                 "target_field": "destination.nat.ip", "ignore_missing": true } },
    { "rename":      { "field": "Source IP",                 "target_field": "source.ip", "ignore_missing": true } },
    { "set":         { "field": "client.ip",                 "value": "{{source.ip}}", "ignore_failure": true } },
    { "rename":      { "field": "Browser Language",                 "target_field": "client.browser_language", "ignore_missing": true } },
    { "rename":      { "field": "Full Path",                 "target_field": "http.uri", "ignore_missing": true } },
    { "rename":      { "field": "Host",                 "target_field": "observer.url", "ignore_missing": true } },
    { "rename":      { "field": "HTTP Method",                 "target_field": "http.method", "ignore_missing": true } },
    { "rename":      { "field": "IP Hits",                 "target_field": "client.times_seen", "ignore_missing": true } },
    { "rename":      { "field": "Hardware Concurrency",                 "target_field": "host.cpu.cores", "ignore_missing": true } },
    { "rename":      { "field": "Viewer Device",                 "target_field": "host.type", "ignore_missing": true } },
    { "rename":      { "field": "Viewer Country",                 "target_field": "host.geo.country", "ignore_missing": true } },
    { "rename":      { "field": "Hardware Concurrency",                 "target_field": "host.cpu.cores", "ignore_missing": true } },
    { "rename":      { "field": "User Agent",                 "target_field": "http.user_agent", "ignore_missing": true } },
    { "rename":      { "field": "Session ID",                 "target_field": "http.session_id", "ignore_missing": true } },
    { "rename":      { "field": "Session ID Hits",                 "target_field": "http.session_id_hits", "ignore_missing": true } },
    { "rename":      { "field": "Screen Width",                 "target_field": "host.screen_width", "ignore_missing": true } },
    { "rename":      { "field": "Screen Height",                 "target_field": "host.screen_height", "ignore_missing": true } },
    { "rename":      { "field": "Screen Orientation",                 "target_field": "host.screen_orientation", "ignore_missing": true } },
    { "rename":      { "field": "Screen Color Depth",                 "target_field": "host.screen_color_depth", "ignore_missing": true } },
    { "rename":      { "field": "Round Trip Delay",                 "target_field": "network.rtd", "ignore_missing": true } },
    { "rename":      { "field": "Other Languages",                 "target_field": "client.languages", "ignore_missing": true } },
    { "rename":      { "field": "Media Devices",                 "target_field": "host.media_devices", "ignore_missing": true } },
    { "rename":      { "field": "Java Enabled",                 "target_field": "client.java_enabled", "ignore_missing": true } },
    { "rename":      { "field": "Client Timezone",                 "target_field": "client.timezone", "ignore_missing": true } },
    { "rename":      { "field": "Clipboard",                 "target_field": "client.clipboard", "ignore_missing": true } },
    { "rename":      { "field": "Cookies Enabled",                 "target_field": "http.cookies_enabled", "ignore_missing": true } },
    { "rename":      { "field": "Device Memory",                 "target_field": "host.memory", "ignore_missing": true } },
    { "rename":      { "field": "Effective Type (Up to 4g)",     "target_field": "network.effective_type", "ignore_missing": true } },
    { "rename":      { "field": "Path",     "target_field": "client.access_path", "ignore_missing": true } },
    { "rename":      { "field": "Tor Network",     "target_field": "network.tor", "ignore_missing": true } },
    { "rename":      { "field": "AdBlock",     "target_field": "client.adblock_enabled", "ignore_missing": true } },
    { "rename":      { "field": "Bandwidth (Mbps)",     "target_field": "network.bandwidth", "ignore_missing": true } },
    { "rename":      { "field": "Battery Level",     "target_field": "host.battery_level", "ignore_missing": true } },
    { "rename":      { "field": "Battery Charging",     "target_field": "host.battery_charging", "ignore_missing": true } },
    { "rename":      { "field": "Browser",     "target_field": "client.browser", "ignore_missing": true } },
    { "rename":      { "field": "Client Time",     "target_field": "client.time", "ignore_missing": true } },
    { "rename":      { "field": "Friendly Reminder",     "target_field": "rule.name", "ignore_missing": true } },
    { "set":         { "field": "event.category", "value": "honeytoken", "override": true }  },
    { "set":         { "field": "event.severity", "value": 3, "override": true }  },
    { "remove":      { "field":  "headers" } },
    { "pipeline":    { "name": "common"                 } }
  ]
}
  • Restart Elasticsearch with so-elasticsearch-restart .

Trapdoor

To set up Trapdoor, head to the Serverless Application Repository, and search for Trapdoor.

Make sure to check the box next to Show apps that create custom IAM roles or resource policies (1 additional).

We’ll need to fill in some details once we’ve selected the Trapdoor application.

In this example, we’ll choose to use the HTTP POST option and supply our POSTURL value, which will be our Security Onion instance we’ve set up in our private subnet. The value for POSTURL will be supplied as:

http://$securityonionip:$port

The $port portion of this value should correspond to the port used for the Logstash HTTP input plugin.

In our example, our POSTURL value will end up being http://10.0.1.196:5044.

From here, we can click to accept the terms of the application and deploy it. It will take a few minutes for everything to be set up. You can monitor progress of the application deployment through the Cloud Formation console.

Once deployed, we’ll want to make a few more modifications. The modifications include:

  • Connecting the Trapdoor Lambda to our VPC
  • Specifying additional alert paths/friendly reminders for Trapdoor

To connect Trapdoor to our VPC, we’ll need to do the following:

  • From the AWS Lambda page, click the Functions tab on the left side of the page
  • Click the Trapdoor function
  • Under the Function overview, click Configuration
  • Click the VPC tab
  • Click Edit to add details for your VPC, private subnet, and security group to be associated with the Lambda interface to communicate with EC2 instances (the interface will need to be able to send to the Security Onion instance over port 5044)
  • Click Save (it will take several minutes to update the function after doing so)

NOTE: The AWSLambdaVPCAccessExecutionRole policy may also need to be attached to the trapdoor role via the AWS IAM console to ensure the Trapdoor lambda can create and manage the necessary EC2 interface for joining to a VPC.

While the function is updated, the Function Overview page will display a banner like the following:

After the status of this banner changes, we can continue with our customization.

If we click the Triggers tab, then expand the details under the API Gateway trigger, we can find the base URL that will be used for our honeytoken.

Anything within the proxy{+} path can be used to uniquely identify visits to the URL (for example, /123, /testing)

If we want to add our own label to a path, we can do so by modifying the function code. To do this within the AWS console, we’ll need to employ a slight workaround, however. Because the console does not provide an interactive editor for the C#/.Net/Powershell runtime, we’ll need to change the runtime to something different while we are editing the code.

From the Function Overview page, if we click on the Code tab, we can edit the runtime settings for the function.

After clicking Edit, we’ll choose to use Node.js 14.x, and click Save.

After saving, a directory structure and code editor will be displayed. We can click config.json, and modify it to add our custom path/friendly reminder.

Next, we’ll click Deploy to deploy our changes, then modify the runtime settings again, to .Net Core 3.1.

📢 IF YOU DO NOT CHANGE THE RUNTIME BACK TO THAT OF .NET CORE 3.1, THE APPLICATION WILL NOT WORK! 😉

Firewall

For Security Onion, we will add an allow entry for the IP address of the internally facing interface for the Trapdoor Lambda function (obtained from the EC2 interfaces page).

We will run so-allow, choosing the b option for Elastic Beats (port 5044), and supply the necessary IP address.

Testing Our Work

Now that we have our VPC, Security Onion, and Trapdoor set up, we can test the triggering of Trapdoor, and it’s notification to Security Onion.

To trigger a notification, simply visit the trigger URL/$Path.

In our case, this URL/Path will be:

NOTE: We would likely want to customize the domain used here when using in a production environment— we can do this by following the instructions here.

The default page content will be presented as follows:

However, if we take a look in Security Onion’s SOC Alerts pane, we see alerts from Trapdoor!

Digging deeper into to the results in Security Onion Hunt, we can see more detail around the alert(s):

From the above images, we can see that Trapdoor has done quite a job of ascertaining various details around the client accessing our honeytoken! 😍

NOTE: You may notice that you sometimes receive two alerts from Trapdoor for the same access/session. This is due to the fact that Trapdoor attempts to fingerprint a client using Javascript, and also uses a pure HTTP-based method. This is an effort to ensure we can at least retrieve some information other than simply the IP address or HTTP session information tied to the individual that accessed our honeytoken.

Summary

Depending on how/where our trapdoor is placed, being alerted by this activity can help us to be proactive in tracking a potential adversary’s actions. Assuming activity from the same IP or client is present within our environment, we can correlate activity from the adversary across our infrastructure, maybe through other Zeek connection or service-oriented records, or even NIDS alerts from Suricata, or other audit/cloud-oriented logs brought in to Security Onion from AWS. Having the additional information available from the HTTP and Javascript-based fingerprinting could also help us to paint a better picture of the adversary, their background/intent, and potential capabilities. 🔍

If you’ve read this far, thanks! 😉 If you have any feedback, questions, or suggestions, please feel free to respond below or give me a shout @therealwlambert on Twitter.

Also be sure to tell @0xtf, @3CORESec, and @securityonion how awesome they are! 🎊

--

--

Wes Lambert

Written by Wes Lambert

93 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams