Google Cloud Pubsub

Overview

This Adapter allows you to ingest events from a Google Cloud Pubsub subscription.

Configurations

Adapter Type: pubsub

• client_options: common configuration for adapter as defined here.

• sub_name: the name of the subscription to subscribe to.

• service_account_creds: the string version of the JSON credentials for a (Google) Service Account to use accessing the subscription.

• project_name: project name where the sub_name exists.

CLI Deployment

This example assumes that the Adapter is running from a host that has default credentials (via the GOOGLE_APPLICATION_CREDENTIALS environment variable) setup. If it's not the case you will need to use service_account_creds to provide the contents of the JSON credentials of the GCP Service Account to use.

./lc_adapter pubsub client_options.identity.installation_key=f5eaaaad-575a-498e-bfc2-5f83e249a646 \
    client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd \
    client_options.platform=gcp \
    sub_name=usp \
    project_name=monitored-proj \
    client_options.sensor_seed_key=gcplogs

Here's the breakdown of the above example:

• lc_adapter: simply the CLI Adapter.

• pubsub: the method the Adapter should use to collect data locally.

• client_options.identity.installation_key=....: the Installation Key value from LimaCharlie.

• client_options.identity.oid=....: the Organization ID from LimaCharlie the installation key above belongs to.

• client_options.platform=gcp: this indicates that the data read is logs from Google Cloud Platform.

• client_options.sensor_seed_key=....: this is the value that identifies this instance of the Adapter. Record it to re-use the Sensor ID generated for this Adapter later if you have to re-install the Adapter.

• sub_name=usp: the Subscription name to consume the logs from.

• project_name=monitored-proj: the GCP Project name this Subscription belongs to.

Infrastructure as Code Deployment

# Google Cloud Pub/Sub Specific Docs: https://docs.limacharlie.io/docs/adapter-types-google-cloud-pubsub

sensor_type: "pubsub"
pubsub:
  # This Adapter allows you to ingest events from a Google Cloud Pubsub subscription.
  sub_name: "your-pubsub-subscription-name" # (required) the name of the subscription to subscribe to.
  project_name: "your-gcp-project-id" # (required) project name where the sub_name exists.
  service_account_creds: "/opt/limacharlie_adapter/gcp_creds/pubsub_service_account.json" # (required) path to the JSON credentials file for a Service Account. Can also be the JSON string itself.
  max_ps_buffer: 1000 # (optional) Max messages to buffer. e.g., 1000. Default is 10.
  client_options:
    identity:
      oid: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # (required) Organization ID from LimaCharlie.
      installation_key: "YOUR_LC_INSTALLATION_KEY_PUBSUB" # (required) Installation key associated with the OID.
    hostname: "gcp-pubsub-adapter-prod-01.example.com" # (required if not using sensor_hostname_path)
    platform: "gcp_cloud_function" # (required) Platform where the adapter runs or logically represents. e.g., "gcp_cloud_function", "linux"
    architecture: "x86_64" # (optional)
    mapping:
      # Pub/Sub messages are often JSON. parsing_re might be null if events are already structured JSON.
      # If messages are plain text or need regex extraction from a JSON field, define parsing_re.
      parsing_re: null # e.g., if messages are JSON strings within a Pub/Sub message attribute, you might parse the attribute then apply regex, or use transform.
      sensor_key_path: "attributes.deviceId" # (optional) Path to uniquely ID the original sensor from Pub/Sub message attributes or JSON payload.
      sensor_hostname_path: null # (optional) Use if client_options.hostname is not set. e.g., "attributes.deviceHostname"
      event_type_path: "PUBSUB_{{ .attributes.eventType | token | upper | default \"GENERIC\" }}" # (optional) Example based on Pub/Sub message attributes.
      event_time_path: "publishTime" # (optional) Pub/Sub messages often have a 'publishTime'. Or path within the message data.
      investigation_id_path: "message.data.alert_uuid" # (optional)
      transform:
        "+data_source_type": "gcp_pubsub"
        "+message_id": "{{ .messageId }}" # Capture Pub/Sub messageId
        # If message data is base64 encoded JSON:
        # "+decoded_payload": "{{ .data | b64dec | json }}" # Assuming b64dec and json functions
        # Then further process 'decoded_payload' in subsequent rules or D&R rules.
      drop_fields:
      - "ackId" # Drop Pub/Sub specific fields not needed for telemetry
    # mappings: null
    indexing:
      enabled: true
      default_index: "gcp-pubsub-{{ .identity.oid | substr 0 8 }}"
    is_compressed: false # (optional) GCP Pub/Sub messages are typically not compressed at this layer.
    sensor_seed_key: "SEED_KEY_PUBSUB_ADAPTER_001" # (required)
    dest_url: "https://input.limacharlie.io" # (optional) The destination URL. Usually defaults correctly.

API Doc

See the official documentation.