MENU
    Google Cloud Pubsub
    • 12 Jun 2025
    • 4 Minutes to read
    • Dark

    Google Cloud Pubsub

    • Dark

    Article summary

    Overview

    This Adapter allows you to ingest events from a Google Cloud Pubsub subscription.

    Configurations

    Adapter Type: pubsub

    • client_options: common configuration for adapter as defined here.

    • sub_name: the name of the subscription to subscribe to.

    • service_account_creds: the string version of the JSON credentials for a (Google) Service Account to use accessing the subscription.

    • project_name: project name where the sub_name exists.

    CLI Deployment

    This example assumes that the Adapter is running from a host that has default credentials (via the GOOGLE_APPLICATION_CREDENTIALS environment variable) setup. If it's not the case you will need to use service_account_creds to provide the contents of the JSON credentials of the GCP Service Account to use.

    ./lc_adapter pubsub client_options.identity.installation_key=f5eaaaad-575a-498e-bfc2-5f83e249a646 \
        client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd \
        client_options.platform=gcp \
        sub_name=usp \
        project_name=monitored-proj \
        client_options.sensor_seed_key=gcplogs
    Bash

    Here's the breakdown of the above example:

    • lc_adapter: simply the CLI Adapter.

    • pubsub: the method the Adapter should use to collect data locally.

    • client_options.identity.installation_key=....: the Installation Key value from LimaCharlie.

    • client_options.identity.oid=....: the Organization ID from LimaCharlie the installation key above belongs to.

    • client_options.platform=gcp: this indicates that the data read is logs from Google Cloud Platform.

    • client_options.sensor_seed_key=....: this is the value that identifies this instance of the Adapter. Record it to re-use the Sensor ID generated for this Adapter later if you have to re-install the Adapter.

    • sub_name=usp: the Subscription name to consume the logs from.

    • project_name=monitored-proj: the GCP Project name this Subscription belongs to.

    Infrastructure as Code Deployment

    # Google Cloud Pub/Sub Specific Docs: https://docs.limacharlie.io/docs/adapter-types-google-cloud-pubsub
    
    sensor_type: "pubsub"
    pubsub:
      # This Adapter allows you to ingest events from a Google Cloud Pubsub subscription.
      sub_name: "your-pubsub-subscription-name" # (required) the name of the subscription to subscribe to.
      project_name: "your-gcp-project-id" # (required) project name where the sub_name exists.
      service_account_creds: "/opt/limacharlie_adapter/gcp_creds/pubsub_service_account.json" # (required) path to the JSON credentials file for a Service Account. Can also be the JSON string itself.
      max_ps_buffer: 1000 # (optional) Max messages to buffer. e.g., 1000. Default is 10.
      client_options:
        identity:
          oid: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" # (required) Organization ID from LimaCharlie.
          installation_key: "YOUR_LC_INSTALLATION_KEY_PUBSUB" # (required) Installation key associated with the OID.
        hostname: "gcp-pubsub-adapter-prod-01.example.com" # (required if not using sensor_hostname_path)
        platform: "gcp_cloud_function" # (required) Platform where the adapter runs or logically represents. e.g., "gcp_cloud_function", "linux"
        architecture: "x86_64" # (optional)
        mapping:
          # Pub/Sub messages are often JSON. parsing_re might be null if events are already structured JSON.
          # If messages are plain text or need regex extraction from a JSON field, define parsing_re.
          parsing_re: null # e.g., if messages are JSON strings within a Pub/Sub message attribute, you might parse the attribute then apply regex, or use transform.
          sensor_key_path: "attributes.deviceId" # (optional) Path to uniquely ID the original sensor from Pub/Sub message attributes or JSON payload.
          sensor_hostname_path: null # (optional) Use if client_options.hostname is not set. e.g., "attributes.deviceHostname"
          event_type_path: "PUBSUB_{{ .attributes.eventType | token | upper | default \"GENERIC\" }}" # (optional) Example based on Pub/Sub message attributes.
          event_time_path: "publishTime" # (optional) Pub/Sub messages often have a 'publishTime'. Or path within the message data.
          investigation_id_path: "message.data.alert_uuid" # (optional)
          transform:
            "+data_source_type": "gcp_pubsub"
            "+message_id": "{{ .messageId }}" # Capture Pub/Sub messageId
            # If message data is base64 encoded JSON:
            # "+decoded_payload": "{{ .data | b64dec | json }}" # Assuming b64dec and json functions
            # Then further process 'decoded_payload' in subsequent rules or D&R rules.
          drop_fields:
          - "ackId" # Drop Pub/Sub specific fields not needed for telemetry
        # mappings: null
        indexing:
          enabled: true
          default_index: "gcp-pubsub-{{ .identity.oid | substr 0 8 }}"
        is_compressed: false # (optional) GCP Pub/Sub messages are typically not compressed at this layer.
        sensor_seed_key: "SEED_KEY_PUBSUB_ADAPTER_001" # (required)
        dest_url: "https://input.limacharlie.io" # (optional) The destination URL. Usually defaults correctly.
    YAML

    API Doc

    See the official documentation.


    Was this article helpful?