Syslog

Syslog is both a protocol and common logging format that consolidate events to a central location for storage. On *nix systems, Syslog often outputs to predefined locations, such as /var/log. The LimaCharlie Adapter can be configured as a Syslog endpoint to collect events either via TCP or UDP.

Syslog data can also be ingested via other data platforms, such as an S3 bucket.

Syslog events are observed in LimaCharlie as the text platform.

Adapter Deployment

Given its ubiquity, Syslog can be ingested via a myriad of methods in both text/log and streaming formats. For non-streaming methods, please refer to the corresponding Adapter type (such as S3, GCP, etc.)

Syslog-specific Configurations

All Adapters have the same common client configuration options, found here. A syslog Adapter has a few unique configuration options not found with other Adapter types. These include:

• port: port to listen for syslog from.

• iface: the interface name to listen for new connections/packets from, defaults to all.

• is_udp: if true, listen over UDP instead of TCP.

• ssl_cert: path to a file with the SSL cert to use to receive logs over TCP.

• ssl_key: path to a file with the SSL key to use to receive logs over TCP.

Collecting Syslog via Docker

The following example walks through configuring a Docker container as a syslog Adapter.

docker run --rm -it -p 1514:1514 refractionpoint/lc-adapter:latest syslog port=1514 \
client_options.identity.installation_key=e9a3bcdf-efa2-47ae-b6df-579a02f3a54d \
client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd \
client_options.platform=text "client_options.mapping.parsing_re=(?P<date>... \d\d \d\d:\d\d:\d\d) (?P<host>.+) (?P<exe>.+?)\[(?P<pid>\d+)\]: (?P<msg>.*)" \
client_options.sensor_seed_key=testclient1 \
client_options.mapping.rename_only=true \
"client_options.mapping.mapping[0].src_field=host" \
"client_options.mapping.mapping[0].dst_field=syslog_hostname"

Here's a breakdown of the above example:

• docker run --rm: run a container and don't keep the contents around when it's stopped.

• -it: make the container interactive so you can ctrl-c to stop it.

• -p 1514:1514: allow the container to listen on port 1514 on the local host and use the same port within the container.

• refractionpoint/lc-adapter:latest: this is the name of the public container provided by LimaCharlie.

• syslog: the method the Adapter should use to collect data locally. The syslog value will operate as a syslog endpoint on the TCP port specified.

• port=1514: the TCP port the Adapter should listen on. By default this is a normal TCP connection (not SSL), although SSL options exist.

• client_options.identity.installation_key=....: the Installation Key from LimaCharlie.

• client_options.identity.oid=....: the Organization ID from LimaCharlie the installation key above belongs to.

• client_options.platform=text: this indicates the type of data that will be received from this adapter. In this case it's syslog, so text lines.

• client_options.mapping.parsing_re=....: this is the parsing expression describing how to interpret the text lines and how to convert them to JSON.

• client_options.sensor_seed_key=....: this is the value that identifies this instance of the Adapter. Record it to re-use the Sensor generated for this Adapter later if you have to re-install the Adapter.

• client_options.mapping.rename_only=true: only rename the field in mapping below, so keep the other original fields.

• client_options.mapping.mapping[0].src_field=....: the source field of the first mapping record.

• client_options.mapping.mapping[0].dst_field=....: the destination field of the first mapping record.

To test it, assuming we're on the same Debian box as the container, pipe the syslog to the container:

journalctl -f -q | netcat 127.0.0.1 1514

Collecting Syslog via Binary Adapter

The LimaCharlie binary Adapter can be deployed as a syslog listener. This option allows you to configure multiple syslog outputs to a single listener, and ingest multiple types of events with a single Adapter.

Step 1: Create an installation key

We recommend utilizing a unique installation key for this deployment, specifically with a syslog Tag. This allows for a level of delineation within rules and outputs via Tags.

Step 2: Create an Adapter config file

Syslog events are typically ingested as text, however often have specific structures to them. Utilizing a config file allows for easy management of a regex string to extract relevant fields from syslog output.

The following example config file can be a starting point. However, you might need to modify the regex to match your specific message.

# Syslog Specific Docs: https://docs.limacharlie.io/docs/adapter-types-syslog

sensor_type: "syslog"
syslog:
  # The method the Adapter should use to collect data locally. The syslog value will operate as a syslog endpoint on the TCP port specified.
  port: 1514 # port to listen for syslog from. (e.g., 514 is standard but often requires root, 1514 is a common alternative)
  iface: "0.0.0.0" # the interface name to listen for new connections/packets from, defaults to all. (e.g., "eth0", "127.0.0.1", or "0.0.0.0" for all)
  is_udp: false # if true, listen over UDP instead of TCP. (true or false)
  ssl_cert: "/opt/limacharlie_adapter/certs/syslog_server.pem" # path to a file with the SSL cert to use to receive logs over TCP. (e.g., /path/to/your/cert.pem)
  ssl_key: "/opt/limacharlie_adapter/certs/syslog_server.key" # path to a file with the SSL key to use to receive logs over TCP. (e.g., /path/to/your/key.pem)
  mutual_tls_cert: "/opt/limacharlie_adapter/certs/client_ca_bundle.pem" # (optional) Path to a CA certificate bundle for validating client certificates if mutual TLS is required. e.g., /path/to/ca_clients.pem or null
  write_timeout_sec: 30 # (optional) Timeout in seconds for writing data. e.g., 30 or null
  client_options:
    identity:
      oid: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # (required) Organization ID from LimaCharlie.
      installation_key: "YOUR_LC_INSTALLATION_KEY_SYSLOG" # (required) Installation key associated with the OID.
    hostname: "syslog-adapter-prod-01.example.com" # (required if not using sensor_hostname_path) A unique, static hostname for this sensor instance.
    platform: "linux" # (required) The platform where the adapter is running. e.g., "linux", "windows", "darwin"
    architecture: "x86_64" # (optional) The architecture of the adapter's host. e.g., "x86_64", "arm64", null
    mapping:
      # Example regex for RFC3164 syslog. Adjust based on your log format.
      parsing_re: "^<(?P<pri>\\d+)>(?P<timestamp>\\w{3}\\s+\\d{1,2}\\s+\\d{2}:\\d{2}:\\d{2})\\s+(?P<original_hostname>\\S+)\\s+(?P<process_name>[a-zA-Z0-9\\._-]+)(?:\\[(?P<pid>\\d+)\\])?:\\s+(?P<message>.*)$"
      sensor_key_path: "original_hostname" # (optional) JSON path from parsed data to uniquely ID the original sensor/device. e.g., use 'original_hostname' if distinct.
      # (optional) If client_options.hostname is NOT set, use this to dynamically extract hostname from event.
      sensor_hostname_path: null # e.g., "original_hostname" (if captured by parsing_re and client_options.hostname is not set).
      event_type_path: "SYSLOG_{{ .process_name | token | upper }}" # (optional) Example: "SYSLOG_NGINX", "SYSLOG_SSHD" based on parsed 'process_name'.
      event_time_path: "timestamp" # (optional) JSON path to the event's occurrence time. Assumes 'timestamp' is captured by parsing_re.
      investigation_id_path: "security_alert_id" # (optional) JSON path for a field to populate LimaCharlie's investigation_id. e.g., "alert.uuid" or null
      transform:
        "+adapter_source": "syslog_collector" # Add a static field for source identification
        "+event_priority": "{{ .pri }}" # Add priority field from parsed regex group
        "-pri": null # Remove the original 'pri' field after processing
        # "+message_length": "{{ .message | len }}" # Example: Add message length (verify 'len' function availability)
      drop_fields:
      - "temp_parsing_field"
      - "debug_info_v1"
      sid_replication_path: null # (optional) More relevant for Windows Event Log sources.
    # mappings: null # Deprecated or less common for field manipulation than client_options.mapping.transform
    indexing:
      enabled: true
      default_index: "syslog-data-{{ .identity.oid | substr 0 8 }}" # Example: index by org prefix
    is_compressed: false # (optional) Set to true if the incoming syslog data stream is compressed.
    sensor_seed_key: "SEED_KEY_SYSLOG_ADAPTER_001" # (required) A unique key for this sensor to register with LimaCharlie.
    dest_url: "https://input.limacharlie.io" # (optional) The destination URL. Usually defaults correctly.

Step 3: Configure syslog output to send messages to a local listener

This step will depend on the type of syslog daemon you are using (syslog, rsyslog, syslog-ng, etc.) Within the daemon configuration file, configure the desired facility(-ies) to direct to the local listener. In the following example, we configured auth and authpriv events to write to both /var/log/audit.log and 127.0.0.1:1514.

auth,authpriv.*			/var/log/auth.log
auth,authpriv.*			@@127.0.0.1:1514

After applying the appropriate configuration, restart the syslog daemon.

Step 4: Confirm that syslog messages are sent to the correct location

Utilizing a tool like netcat, you can listen on the appropriate port to confirm that messages are being sent. The following command will spawn a netcat listener on port 1514:

nc -l -p 1514

Step 5: Run the LimaCharlie Adapter

Execute the binary Adapter with the syslog configuration file in order to start the LimaCharlie listener. If started correctly, you should see the following messages in stdout:

DBG <date>: usp-client connecting
DBG <date>: usp-client connected
DBG <date>: listening for connections on :1514

Double-check the LimaCharlie Sensors list, and you should see the text adapter with the respective hostname sending Syslog events.