MSSP deployment guide

This guide covers running deadair across client SIEMs from an operator-owned host. A source is a concrete index, data stream, or Sentinel table visible to the tenant credential. See Read the findings for the evidence and triage model.

Run the lab first

make mssp-lab

The lab covers the Elastic and OpenSearch Docker path. It seeds two working tenants and three failure cases, runs check --fleet, runs redacted scans with schema and downtime enabled, scrapes serve --fleet --redact, and writes artifacts to integration/mssp-lab-out/.

The generated files show the operator surfaces you will use in a real deployment:

File What to inspect
check.txt fleet preflight, including bad credentials and an unreachable tenant
fleet-summary.txt terminal fleet scan output
fleet-redacted.json pseudonymized fleet report; review its findings and counts before sharing
metrics.txt Prometheus metrics with per-instance health
downtime.json expected downtime config
state.json.* per-instance state files

The lab covers the Docker workflow, config shape, redaction, partial failures, schema drift, downtime suppression, exporter labels, and per-instance state. Validate source cadence and hosted SIEM behavior in each customer environment.

Deployment shape

Run deadair from a hardened host with outbound HTTPS access to each client SIEM API. Nothing is installed in the client SIEM.

Keep every credential or workload identity scoped to the tenants it is meant to scan. Store only secret references in the fleet file.

/etc/deadair/
  fleet.json
  secrets/
    acme-api-key
    beta-opensearch-password
  ca/
    acme.pem

/var/lib/deadair/
  state/
    fleet-state.json.acme-prod
    fleet-state.json.beta-corp
    fleet-state.json.gamma-sentinel
  reports/
    latest-redacted.json
    latest-internal.json

Recommended permissions:

  • /etc/deadair/fleet.json: readable by the deadair service user only
  • /etc/deadair/secrets/*: 0600
  • /var/lib/deadair/state/*: 0600
  • /var/lib/deadair/reports/*: 0600

Fleet config

Instance names must be unique without regard to case. Use 1–64 ASCII letters, digits, dots, underscores or hyphens, starting with a letter or digit. Trailing dots and reserved device names such as CON are rejected. Names become state-file suffixes, so paths and Unicode names are not accepted. Fleet files reject unknown fields and extra JSON documents.

Choose one credential source for each secret: an environment variable or a file. Explicit secret references must resolve to a nonempty value. OpenSearch accepts either an API key or a complete username/password pair; omit both to connect without authentication.

{"instances": [
  {
    "name": "acme-prod",
    "backend": "elastic",
    "es_url": "https://es.acme.example:9200",
    "kibana_url": "https://kibana.acme.example:5601",
    "api_key_file": "/etc/deadair/secrets/acme-api-key",
    "ca_cert": "/etc/deadair/ca/acme.pem"
  },
  {
    "name": "beta-corp",
    "backend": "opensearch",
    "opensearch_url": "https://os.beta.example:9200",
    "username": "deadair",
    "password_file": "/etc/deadair/secrets/beta-opensearch-password"
  },
  {
    "name": "gamma-sentinel",
    "backend": "sentinel",
    "azure_subscription_id": "<subscription-id>",
    "azure_resource_group": "gamma-security",
    "sentinel_workspace": "gamma-sentinel"
  }
]}

Sentinel fleet entries use the process’s DefaultAzureCredential; they do not carry credentials in the fleet file. If customers require different Azure identities, run those fleet entries in separate deadair processes. Add sentinel_remote_workspaces only for literal workspace() targets you intend to assess. The Sentinel usage guide explains the mapping format and the cross-tenant execution boundary.

Keep name stable. It keys metrics, per-instance state files, redacted pseudonyms, and historical baselines. Renaming a tenant starts a new baseline unless you deliberately migrate the state file.

Check a candidate across workspaces

Keep the target workspaces in a Sentinel-only fleet file, then check the same candidate against each one before deployment:

deadair scan \
  --fleet /etc/deadair/sentinel-fleet.json \
  --rule detections/new-rule.yaml \
  --redact \
  --json-out candidate-report.json

Each workspace gets its own dependency evidence. Exit 1 means at least one candidate has gated findings; exit 2 means a target failed or a candidate could not be safely assessed. Nothing is installed. The Azure identity needs read access to every target; use separate processes when customers require different identities.

This checks inputs, not complete KQL compatibility. A fresh WebAccess_CL table can satisfy the dependency check even if a query names ClientIP and that workspace only has ClientIP_s. Run query and parser tests alongside this gate. Schema snapshots and drift do not infer the columns a candidate requires.

The GitHub Action example checks one workspace per invocation. Use a job matrix for separate workspace results, with a unique artifact name for each job. Keep the target list and gate policy under the same review controls as the deployment workflow.

Preflight

Run preflight after onboarding a tenant, rotating a credential, changing network paths, or upgrading a backend.

deadair check --fleet /etc/deadair/fleet.json --timeout 90s

check verifies required read paths and reports optional schema visibility separately.

One-shot reports

Use a pseudonymized JSON report when exact identifiers are unnecessary:

deadair scan \
  --fleet /etc/deadair/fleet.json \
  --state-file /var/lib/deadair/state/fleet-state.json \
  --schema \
  --redact \
  --json \
  --json-out /var/lib/deadair/reports/latest-redacted.json

Keep an unredacted internal report only where analysts need exact source and rule names:

deadair scan \
  --fleet /etc/deadair/fleet.json \
  --state-file /var/lib/deadair/state/fleet-state.json \
  --schema \
  --json \
  --json-out /var/lib/deadair/reports/latest-internal.json

With --fleet, one --state-file prefix becomes one state file per instance.

Review findings per tenant

Fleet rollups are useful for prioritization, but the evidence is tenant-local. The same enabled rule can be healthy for one customer, intentionally out of scope for another, and broken for a third.

For example, three tenants inherit a rule that queries netflow-*:

Tenant state Evidence Disposition
NetFlow onboarded pattern resolves to a live index healthy
NetFlow not in the customer’s service scope pattern resolves to no source accepted coverage scope or onboarding backlog; document it
NetFlow was migrated to a renamed data stream old pattern resolves to no source after prior coverage regression; update the rule or restore the expected name

For each actionable finding, keep enough context for the receiving team to start work:

Field Why it matters
instance identifies the customer or deployment with the gap
rule and severity identifies affected detection coverage and priority
configured patterns or dependencies shows what input the rule expects
matched sources and health shows whether resolution or telemetry delivery failed
first seen or diff state separates backlog from a new regression
disposition and owner records accepted scope, remediation, or false positive

The terminal fleet view is a summary. Keep the internal JSON report for exact patterns and sources; use the redacted report for authorized cross-team or client trend reporting.

Open one customer’s source evidence without rescanning the fleet:

deadair inspect --instance acme-prod --source logs-system.auth-default latest-internal.json

Use the pseudonym shown in the report when inspecting a redacted instance. A failed instance has no saved source assessment; fix its scan error before drawing conclusions about its telemetry.

Continuous monitoring

deadair serve \
  --fleet /etc/deadair/fleet.json \
  --state-file /var/lib/deadair/state/fleet-state.json \
  --schema \
  --redact \
  --interval 30m

Default bind address is 127.0.0.1:9317. Put it behind authenticated scraping if Prometheus is not local to the host.

Prometheus scrapes the cached last scan. Scrape volume does not hit customer SIEM APIs.

Redaction

Use --redact before authorized sharing through:

  • client-facing reports
  • shared Prometheus
  • screenshots
  • shared issue reports

Redaction covers tenant, source, rule, pattern, field, dependency, lineage, provenance, workspace, watchlist, template, and package identifiers with keyed HMAC pseudonyms. Generate a separate random key file for each correlation boundary and set DEADAIR_REDACT_KEY_FILE to its path. Supplying the key file also enables redaction. Reports record the non-secret key identifier, so a reporting job can reject inputs created with different keys. Without a key file, pseudonyms are stable only for the life of one process. Redacted fleet reports expose a fixed failure category rather than raw backend errors.

Redaction is not declassification. Backend versions, counts, severities, timing, lag, storage volume, and the kinds of blind spots remain visible. Review the result and share it only with an authorized recipient.

Keep unredacted reports restricted to analysts who need exact names for remediation.

Alert routing

Route by the type of work, not by the tool.

Signal Suggested owner
deadair_up == 0 platform owner for the deadair host
deadair_instance_up == 0 tenant onboarding, credential, or network owner
no matching source detection engineering or tenant onboarding, after checking credential scope
all matching sources stale or empty telemetry pipeline owner, with detection engineering copied
impaired detections detection engineering plus parser or pipeline owner
source plan incompatible detection engineering and the Sentinel workspace owner
lag blind window detection engineering and telemetry pipeline owner
stale, empty, or low-volume sources telemetry pipeline owner
schema drift parser, integration, or content owner
unused telemetry detection engineering and cost/platform owner

Alertmanager sketch:

route:
  group_by: ["alertname", "instance"]
  receiver: deadair-platform
  routes:
    - matchers: ['alertname=~"DeadairScanFailing|DeadairInstanceScanFailing"']
      receiver: deadair-platform
    - matchers: ['alertname=~"DeadairDeadDetections|DeadairImpairedDetections"']
      receiver: detection-engineering
    - matchers: ['alertname=~"DeadairSourceStale|DeadairVolumeLow"']
      receiver: telemetry-pipelines
    - matchers: ['alertname="DeadairSchemaDrift"']
      receiver: content-engineering

Start with tickets or chat notifications. Page only after the first manual scan has been reviewed and noisy findings have been fixed or suppressed correctly.

Schedule fleet scans

Fleet scans are sequential. Cycle time is roughly:

fleet scan time ~= sum(per-tenant scan time) + network/auth overhead

Measure your own p95:

time deadair scan --fleet /etc/deadair/fleet.json --json --json-out /tmp/deadair.json

Keep serve --interval comfortably above observed fleet scan time. If a fleet grows too large, split it by customer segment, region, or backend. Raise --concurrency only after checking SIEM API behavior in your environment.

Back up state files if baseline continuity matters. Losing state does not affect the backend, but it restarts warmup and schema history.

Failure modes

One tenant failure does not hide the rest. Fleet reports include successful instance reports plus an error entry for each failed instance. The process exits 2 to mark the scan incomplete.

Failure What happens Operator action
expired or revoked credential instance fails with 401/403 rotate the tenant secret, then run deadair check --fleet
credential excludes expected sources sources outside role scope appear absent and can produce no-match findings expand the tenant role to the intended sources, then verify a known-good rule/source pair
missing optional privilege scan works, but schema, lag, lineage, or provenance evidence may be unavailable add the optional read privilege only if you need that check
tenant SIEM unreachable instance is reported failed; other tenants still scan check DNS, proxy, allowlists, VPN, and client-side availability
API throttling or timeouts instance fails or sources become unknown increase interval, lower concurrency, or split the fleet
planned maintenance stale or empty findings may appear after the window declare downtime windows instead of excluding sources
tenant rename baselines restart keep names stable or migrate the matching state file