Outputs

Overview

LimaCharlie provides multiple output options, referred to as streams, for you to send data from LimaCharlie to other destination(s). We provide native support for output with multiple different providers, or "destinations". The diagram below provides some basic examples of where data is sourced from and where data can be sent to.

Outputs should be thought of in two capacities: Streams and Destinations. A stream is what you are sending, whereas a destination is where you are sending it to. We will look at both in detail.

Check out the following YouTube video for a walkthrough of configuring an output.

Streams

Streams define which events will be sent to an output destination.

Available streams include:

• event : The bulk of data events coming from sensors. Note: this will be very verbose

• detect : Alerts, as generated by the report action in detection and response rules.

• audit : Events generated by the LimaCharlie platform, such as access control.

• deployment : Events about your deployment, like Sensor enrollments or cloned sensors.

• artifact : Meta-events reporting on newly-ingested files through the Artifact Collection mechanism.

• tailored : Only events specifically flagged for outputs sent to this stream.

Destinations

LimaCharlie integrates with several providers, such as S3, Google Cloud, or Slack, as Output Destinations.

Destinations are the recipients of LimaCharlie streams. Oftentimes, users will rely on LimaCharlie for 365 data retention, while pushing high-fidelity alerts or other platform logs to another source for subsequent auditing or ticketing. As such, we have created native and/or easy-to-use destination options.

Configuring destinations

Every destination will have both general and specific parameters. Destinations can be configured via the LimaCharlie GUI, API, or command-line.

General Parameters

All destinations can be configured with the following options:

• is_flat: take the json output and flatten the whole thing to a flat structure.

• is_payload_as_string: converts the payload (event or detect components) of events and detections into a JSON string instead of a JSON object.

• inv_id: only send events matching the investigation id to this output (event stream only).

• tag: only send events from sensors with this Tag to this output (event stream only).

• cat: only send detections from this category to this output (detect stream only).

• cat_black_list: only send detections that do not match the prefixes in this list (newline-separated).

• event_white_list: only send event of the types in this list (newline-separated, event and audit streams only).

• event_black_list: only send event not of the types in this list (newline-separated, event and audit streams only).

• is_delete_on_failure: if an error occurs during output, delete the output automatically.

• is_prefix_data: wrap JSON events in a dictionary with the event_type as the key and original event as value.

• sample_rate: limits data sent to Output to be 1/sample_rate.

• custom_transform: a template and transforms to apply to the JSON data as a last output step.

Specific Parameters

If you are configuring destinations using the LimaCharlie UI, required options must be provided before the output can be created.

Output Destinations

See Output Destinations for a list of supported destinations.

Transforming Output Data

To learn how you can manipulate data prior to sending to your Output Destination, read about Transforming Output Data.

Testing Outputs

The easiest way to test if the outputs are configured correctly is to set the stream to Audit which will send auditing events about activity around the management of the platform in the cloud. You can then edit the same output or make any other change on the platform, which will trigger an audit event to be sent.

After you have confirmed that the output configurations works, you can switch the data stream from Audit to the one you are looking to use.

If you are running into an error configuring an output, the error details will be listed in the Platform Logs section under Errors, with the key that looks like outputs/OUTPUT_NAME.

If an output fails, it gets disabled temporarily to avoid spam. It will be re-enabled automatically after a while, or you can force it to be re-enabled by updating the configuration.

Use Cases

There are multiple use cases or integration strategies for shipping telemetry to and from the LimaCharlie platform. Some common approaches we have seen:

All data over batched files via SFTP, Splunk or ELK consumes the received files for ingestion.

Sensor ---> LC (All Streams) ---> SFTP ---> ( Splunk | ELK )

All data streamed in real-time via Syslog, Splunk or ELK receive directly via an open Syslog socket.

Sensor ---> LC (All Streams) ---> Syslog( TCP+SSL) ---> ( Splunk | ELK )

All data over batched files stored on Amazon S3, Splunk or ELK consumes the received files remotely for ingestion.

Sensor ---> LC (All Streams) ---> Amazon S3 ---> ( Splunk | ELK )

Bulk events are uploaded to Amazon S3 for archiving, while alerts and auditing events are sent in real-time to Splunk via Syslog. Note: This has the added benefit of reducing Splunk license cost while keeping the raw events available for analysis at a lower cost.

Sensor ---> LC (Event Stream) ---> Amazon S3
       +--> LC (Alert+Audit Streams) ---> Syslog (TCP+SSL) ---> Splunk

IP Sources

Outputs from the LimaCharlie cloud do not come from a single predictible IP address due to the highly distributed nature of the cloud.

An approximation can be made using the blocks of IP addresses published by Google Cloud Platform here.

The following LimaCharlie datacenters map to the following GCP regions:

• USA: us-central1

• Canada: northamerica-northeast1

• Europe: europe-west4

• UK: europe-west2

• India: asia-south1