- 12 Jun 2025
- 4 Minutes to read
- Print
- DarkLight
Google Cloud Pubsub
- Updated on 12 Jun 2025
- 4 Minutes to read
- Print
- DarkLight
Overview
This Adapter allows you to ingest events from a Google Cloud Pubsub subscription.
Configurations
Adapter Type: pubsub
client_options
: common configuration for adapter as defined here.sub_name
: the name of the subscription to subscribe to.service_account_creds
: the string version of the JSON credentials for a (Google) Service Account to use accessing the subscription.project_name
: project name where thesub_name
exists.
CLI Deployment
This example assumes that the Adapter is running from a host that has default credentials (via the GOOGLE_APPLICATION_CREDENTIALS
environment variable) setup. If it's not the case you will need to use service_account_creds
to provide the contents of the JSON credentials of the GCP Service Account to use.
./lc_adapter pubsub client_options.identity.installation_key=f5eaaaad-575a-498e-bfc2-5f83e249a646 \
client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd \
client_options.platform=gcp \
sub_name=usp \
project_name=monitored-proj \
client_options.sensor_seed_key=gcplogs
Here's the breakdown of the above example:
lc_adapter
: simply the CLI Adapter.pubsub
: the method the Adapter should use to collect data locally.client_options.identity.installation_key=....
: the Installation Key value from LimaCharlie.client_options.identity.oid=....
: the Organization ID from LimaCharlie the installation key above belongs to.client_options.platform=gcp
: this indicates that the data read is logs from Google Cloud Platform.client_options.sensor_seed_key=....
: this is the value that identifies this instance of the Adapter. Record it to re-use the Sensor ID generated for this Adapter later if you have to re-install the Adapter.sub_name=usp
: the Subscription name to consume the logs from.project_name=monitored-proj
: the GCP Project name this Subscription belongs to.
Infrastructure as Code Deployment
# Google Cloud Pub/Sub Specific Docs: https://docs.limacharlie.io/docs/adapter-types-google-cloud-pubsub
sensor_type: "pubsub"
pubsub:
# This Adapter allows you to ingest events from a Google Cloud Pubsub subscription.
sub_name: "your-pubsub-subscription-name" # (required) the name of the subscription to subscribe to.
project_name: "your-gcp-project-id" # (required) project name where the sub_name exists.
service_account_creds: "/opt/limacharlie_adapter/gcp_creds/pubsub_service_account.json" # (required) path to the JSON credentials file for a Service Account. Can also be the JSON string itself.
max_ps_buffer: 1000 # (optional) Max messages to buffer. e.g., 1000. Default is 10.
client_options:
identity:
oid: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # (required) Organization ID from LimaCharlie.
installation_key: "YOUR_LC_INSTALLATION_KEY_PUBSUB" # (required) Installation key associated with the OID.
hostname: "gcp-pubsub-adapter-prod-01.example.com" # (required if not using sensor_hostname_path)
platform: "gcp_cloud_function" # (required) Platform where the adapter runs or logically represents. e.g., "gcp_cloud_function", "linux"
architecture: "x86_64" # (optional)
mapping:
# Pub/Sub messages are often JSON. parsing_re might be null if events are already structured JSON.
# If messages are plain text or need regex extraction from a JSON field, define parsing_re.
parsing_re: null # e.g., if messages are JSON strings within a Pub/Sub message attribute, you might parse the attribute then apply regex, or use transform.
sensor_key_path: "attributes.deviceId" # (optional) Path to uniquely ID the original sensor from Pub/Sub message attributes or JSON payload.
sensor_hostname_path: null # (optional) Use if client_options.hostname is not set. e.g., "attributes.deviceHostname"
event_type_path: "PUBSUB_{{ .attributes.eventType | token | upper | default \"GENERIC\" }}" # (optional) Example based on Pub/Sub message attributes.
event_time_path: "publishTime" # (optional) Pub/Sub messages often have a 'publishTime'. Or path within the message data.
investigation_id_path: "message.data.alert_uuid" # (optional)
transform:
"+data_source_type": "gcp_pubsub"
"+message_id": "{{ .messageId }}" # Capture Pub/Sub messageId
# If message data is base64 encoded JSON:
# "+decoded_payload": "{{ .data | b64dec | json }}" # Assuming b64dec and json functions
# Then further process 'decoded_payload' in subsequent rules or D&R rules.
drop_fields:
- "ackId" # Drop Pub/Sub specific fields not needed for telemetry
# mappings: null
indexing:
enabled: true
default_index: "gcp-pubsub-{{ .identity.oid | substr 0 8 }}"
is_compressed: false # (optional) GCP Pub/Sub messages are typically not compressed at this layer.
sensor_seed_key: "SEED_KEY_PUBSUB_ADAPTER_001" # (required)
dest_url: "https://input.limacharlie.io" # (optional) The destination URL. Usually defaults correctly.
API Doc
See the official documentation.