Pipeline Architecture

A Monad pipeline is a directed graph of nodes connected by edges. Understanding this structure is essential for designing transforms and routing logic.

Pipeline Components

Node Types

Type Purpose Examples

Input

Ingest logs from sources

monad-syslog, monad-http, cisco-meraki-config-logs

Transform

Modify, filter, enrich logs

jq, drop_record_where_value_eq, rename_key

Output

Send logs to destinations

msft-sentinel, s3, splunk-hec

Edge Conditions

Edges connect nodes and control data flow:

Condition Behavior

always

All logs flow through this edge

conditional

Only logs matching condition pass

Pipeline Structure (JSON)

{
  "name": "ISE-to-Sentinel",
  "description": "Route ISE auth failures to Sentinel",
  "enabled": true,
  "nodes": [
    {
      "id": "uuid-input",
      "slug": "syslog-input",
      "component_id": "input-component-id",
      "component_type": "input",
      "enabled": true
    },
    {
      "id": "uuid-transform",
      "slug": "filter-transform",
      "component_id": "transform-component-id",
      "component_type": "transform",
      "enabled": true
    },
    {
      "id": "uuid-output",
      "slug": "sentinel-output",
      "component_id": "output-component-id",
      "component_type": "output",
      "enabled": true
    }
  ],
  "edges": [
    {
      "name": "ingest",
      "from_node_instance_id": "uuid-input",
      "to_node_instance_id": "uuid-transform",
      "conditions": { "operator": "always" }
    },
    {
      "name": "route-critical",
      "from_node_instance_id": "uuid-transform",
      "to_node_instance_id": "uuid-output",
      "conditions": { "operator": "always" }
    }
  ]
}

Data Flow Visualization

Simple Pipeline (Input → Output)

┌─────────────┐          ┌─────────────┐
│   Syslog    │ ──────▶  │  /dev/null  │
│   Input     │  always  │   Output    │
└─────────────┘          └─────────────┘

With Transform

┌─────────────┐          ┌─────────────┐          ┌─────────────┐
│   Syslog    │ ──────▶  │   Filter    │ ──────▶  │  Sentinel   │
│   Input     │  always  │  Transform  │  always  │   Output    │
└─────────────┘          └─────────────┘          └─────────────┘

Split Routing (Critical → Sentinel, Archive → S3)

                                       ┌─────────────┐
                         ┌── critical ─▶│  Sentinel   │
┌─────────────┐          │              └─────────────┘
│   Syslog    │ ──────▶  ├── transform ─┐
│   Input     │  always  │              │
└─────────────┘          │              ▼
                         │         ┌─────────────┐
                         └─ all ──▶│     S3      │
                                   └─────────────┘

Node Instance IDs

Every node in a pipeline has TWO IDs:

  1. component_id: The reusable component (input/output/transform definition)

  2. instance_id: This specific use in this pipeline

# Get node details
monad_pipeline_v2 <pipeline-id> | jq '.nodes[] | {
  instance_id: .id,
  component_id: .component_id,
  type: .component_type,
  slug: .slug
}'

Creating Pipelines

Via CLI

# Simple pipeline: Input → Output
monad_pipeline_create "test-pipeline" "Testing" \
  "input-component-id" \
  "output-component-id"

Via API (with Transform)

# Generate instance UUIDs
INPUT_ID=$(cat /proc/sys/kernel/random/uuid)
TRANSFORM_ID=$(cat /proc/sys/kernel/random/uuid)
OUTPUT_ID=$(cat /proc/sys/kernel/random/uuid)

# Create pipeline JSON
cat << EOF > /tmp/pipeline.json
{
  "name": "ISE-Auth-Filter",
  "description": "Filter ISE logs, route to Sentinel",
  "enabled": true,
  "nodes": [
    {
      "id": "$INPUT_ID",
      "slug": "ise-syslog",
      "component_id": "<syslog-input-id>",
      "component_type": "input",
      "enabled": true
    },
    {
      "id": "$TRANSFORM_ID",
      "slug": "auth-filter",
      "component_id": "<transform-id>",
      "component_type": "transform",
      "enabled": true
    },
    {
      "id": "$OUTPUT_ID",
      "slug": "sentinel-out",
      "component_id": "<sentinel-output-id>",
      "component_type": "output",
      "enabled": true
    }
  ],
  "edges": [
    {
      "name": "ingest",
      "from_node_instance_id": "$INPUT_ID",
      "to_node_instance_id": "$TRANSFORM_ID",
      "conditions": {"operator": "always"}
    },
    {
      "name": "route",
      "from_node_instance_id": "$TRANSFORM_ID",
      "to_node_instance_id": "$OUTPUT_ID",
      "conditions": {"operator": "always"}
    }
  ]
}
EOF

# Create pipeline
curl -s -X POST \
  -H "X-API-Key: $MONAD_API_KEY" \
  -H "Content-Type: application/json" \
  -d @/tmp/pipeline.json \
  "https://app.monad.com/api/v2/$MONAD_ORG_ID/pipelines" | jq .

Syslog Endpoint Pattern

Each syslog input generates a unique endpoint:

<pipeline-id>.l4.monad.com:6514

Configure ISE/FTD/ASA to forward syslog to this endpoint:

# ISE MnT Syslog Target
Target: d4b83eb2-cbfb-474a-8dda-67c4921d1bab.l4.monad.com
Port: 6514
Protocol: TCP+TLS

Inspecting Pipelines

List All Pipelines

monad_pipelines | jq '.pipelines[] | {name, id: .id[0:8], enabled}'

Get Full Pipeline Structure

monad_pipeline_v2 <pipeline-id> | jq '{
  name: .name,
  nodes: [.nodes[] | {type: .component_type, slug: .slug}],
  edges: [.edges[] | {from: .from_node_instance_id[0:8], to: .to_node_instance_id[0:8]}]
}'

Find Transform Details

# Get transform component from pipeline node
monad_pipeline_v2 <pipeline-id> | jq '.nodes[] | select(.component_type=="transform") | .component_id'

# Then get transform config
monad_transform <transform-component-id> | jq '.config'

Pipeline Lifecycle

State Meaning Action

enabled: true

Processing logs

Monitor logs, check metrics

enabled: false

Paused, not processing

Safe to modify

Draft (UI)

Not yet activated

Complete configuration

Enable/Disable

# Disable for maintenance
monad_pipeline_disable <pipeline-id>

# Re-enable
monad_pipeline_enable <pipeline-id>

Key Takeaways

  1. Pipelines are graphs - Nodes connected by edges

  2. Three node types - Input, Transform, Output

  3. Instance IDs are unique - Each node in a pipeline has its own UUID

  4. Edges control flow - always or conditional

  5. Syslog endpoints are per-pipeline - <pipeline-id>.l4.monad.com:6514

Next Module

Transform Types - All 12 transform operations with examples.