Injector Dev

injector-dev

injector-dev is a CLI that turns the manual loop of *build image → push → manage a cluster → deploy workloads by hand*

into a single declarative command. You describe a scenario in YAML — which

Agent components to deploy, how to configure them, and what test workloads to

run — and injector-dev apply brings the whole environment up on a local

Kubernetes cluster.

Scenarios are reproducible and shareable: tear an environment down and recreate

it identically at any time.

When to use this skill

• Iterating on local datadog-agent, datadog-operator, or Helm

chart changes and needing them running on a real cluster.

• Reproducing an APM auto-instrumentation / injection bug locally.
• Writing, editing, or debugging a scenario.yaml.
• Pinning a test environment against a specific released version, a CI pipeline

artifact, or the agent main branch.

Prerequisites

• A local Kubernetes platform. Supported drivers: kind (recommended),

colima, minikube, nvkind, and none (use an existing

cluster/context). kind is fast, reliable across machines, and the easiest to

reset. There is also a workspace driver that runs the cluster on a remote

Datadog Workspace VM — see Remote clusters.

• A Docker runtime installed and running.
• helm and kubectl on your PATH. kind must be installed to use the

recommended kind platform.

• Datadog API + App keys (a real API key is needed for the Agent to report).

Set them however you prefer:

  export DD_API_KEY="<your-api-key>"
  export DD_APP_KEY="<your-app-key>"

> If you already have colima and/or minikube running, shut them down before

> using injector-dev to avoid conflicts.

Installation

git clone https://github.com/DataDog/injector-dev
cd injector-dev
make install   # builds and installs the binary to /usr/local/bin (uses sudo)

Configuration (~/.injector-dev/config.yaml)

A practical starting config:

---
platform: kind              # kind (recommended) | colima | minikube | nvkind | workspace | none
builder:
  code_root: "/Users/<USER>/dd"   # parent dir containing datadog-agent/, auto_inject/, datadog-operator/, ...
  dev_container:
    name: "injector-dev-builder"
    enabled: true
    persist: true           # keep the build container alive between builds → much faster rebuilds
installer:
  repo_root: "/Users/<USER>/dd/injector-dev"
  # api_key / app_key are optional here; DD_API_KEY / DD_APP_KEY env vars are the fallback.

Key points:

• builder.code_root — parent directory holding your component repos. The

tool derives each repo path as code_root/<repo-name> (e.g.

code_root/datadog-agent), so all repos must be siblings under this dir.

• builder.dev_container.persist: true — leaves the build container running

between applies. First build takes several minutes (installing deps); later

builds drop to ~30 seconds.

• installer.repo_root — path to your local injector-dev checkout.
• Every config key can be overridden by env var: prefix with INJECTOR_DEV_

and replace dots with underscores, e.g.

INJECTOR_DEV_INSTALLER_API_KEY, INJECTOR_DEV_BUILDER_CODE_ROOT.

• API/App key resolution order: config.yaml (installer.api_key/app_key)

→ then DD_API_KEY / DD_APP_KEY env vars.

Profiles (multiple configs)

To manage several environments (staging, sandbox, org2, …), drop additional

config files at ~/.injector-dev/<profile>.yaml and select one per command:

injector-dev apply -f scenario.yaml --profile sandbox

Omitting --profile uses config.yaml.

Remote clusters: the workspace platform

Instead of a local cluster, injector-dev can run the kind cluster on a remote

Datadog Workspace

VM. Image builds still happen locally — only the cluster and workloads run on

the workspace. Useful when your laptop is resource-constrained or you want a

beefier, disposable environment.

How it works: docker build runs locally; the image is streamed to the workspace

(docker save | ssh … kind load); kind is installed on the workspace

automatically on first use (workspaces ship with docker but not kind); and the

remote cluster's API server is exposed to your machine through a persistent SSH

tunnel, with an injector-dev-ws-<name> context merged into ~/.kube/config and

made current — so local kubectl/helm/k9s work against it transparently.

Prerequisites

• A workspace reachable over SSH as ssh workspace-<name> (be connected to

Appgate). Create one with:

  workspaces create <name> --repo dd/datadog-agent

injector-dev does not create the workspace; if it's missing it fails fast

and prints this command.

• Local docker for building images, as usual. The workspace only needs docker

(kind is installed for you).

Selecting the workspace

The workspace name comes from the --workspace flag or a scenario's

platform.workspace (flag wins). It is intentionally not read from the global

config.yaml.

In a scenario (picked up by apply):

platform:
  type: workspace
  workspace: firstname-lastname   # SSH host = workspace-firstname-lastname
  name: my-cluster                # optional kind cluster name on the VM (default: "kind")
  reset: false
helm:
  # ... same as any other scenario ...

Or by flag — required for start/stop/reset, which don't read a scenario:

injector-dev apply -f scenario.yaml --workspace firstname-lastname --build
injector-dev stop  --platform=workspace --workspace firstname-lastname

Usage

# bring the remote cluster up + deploy (build local, load remote)
injector-dev apply -f scenario.yaml --workspace <name> --build

# local kubectl now targets the remote cluster through the tunnel
kubectl get pods -A

# tear down the remote cluster, tunnel, and kube-context
injector-dev stop --platform=workspace --workspace <name>

Notes

• start/apply are non-destructive when the cluster already exists: they reuse

it and just re-establish the tunnel and switch your kube-context (so

apply --reset=false still re-points kubectl at the workspace).

• The SSH tunnel is persistent (survives after the command exits) so local

kubectl keeps working. injector-dev stop closes it.

• Inspect open tunnels: ls ~/.injector-dev/*.sock, and check one with

ssh -O check -S ~/.injector-dev/workspace-<name>.sock workspace-<name>.

• You can also inspect the cluster on the workspace directly: ssh workspace-<name>

then kubectl (kind writes a kubeconfig there too).

The core loop: apply

apply is the primary command. It (optionally) resets/starts the cluster,

installs the Datadog stack via Helm or the Operator, deploys any apps/manifests,

and waits for health.

injector-dev apply -f workloads/my-feature/scenario.yaml            # deploy
injector-dev apply -f workloads/my-feature/scenario.yaml --build    # build local source first

apply flags

| Flag | Default | Purpose |

| --- | --- | --- |

| -f, --file | — | Path to the scenario file (required). |

| --build | false | Run build steps for any component with build: {}. |

| --reset | true | Reset the cluster before applying. Set --reset=false for fast iteration. |

| --hard | false | Hard reset (rebuilds the VM — colima only). |

| --wait | true | Wait for the install to become healthy. |

| --skip-agent-validation | false | Skip the "agent started successfully" check. |

| -t, --app-image-tag | — | Global image tag applied to all test apps. |

| --helm-skip-schema-validation | false | Pass --skip-schema-validation to Helm (useful with local chart changes). |

| --profile | config.yaml | Select a config profile. |

| --platform | from config | Override the driver. If you set it on start, you must set it on every apply. |

| --workspace | — | Remote workspace name for --platform=workspace (see Remote clusters). Global flag — also valid on start/stop/reset. |

| --debug | false | Verbose logging. |

Scenario files

A scenario is helm: or operator:, optionally preceded by a platform:

block. Keep each scenario in its own directory alongside its manifests:

workloads/
├── hello-world/
│   └── scenario.yaml
├── my-feature/
│   ├── scenario.yaml
│   └── redis.yaml

Generate a starter template with injector-dev new --type helm --output scenario.yaml

(add --edit to open it in $EDITOR).

platform block

platform:
  type: kind              # kind (recommended) | colima | minikube | nvkind | workspace | none
  name: my-dev-cluster    # unique cluster/profile name — give each scenario its own
  reset: false            # false → reuse the cluster if it exists (fast); true → recreate each apply

Precedence for both platform and reset: CLI flag > scenario platform: block > default.

Deploying a pre-built version (simplest case)

---
platform:
  type: kind              # recommended
  name: hello-world
  reset: false
helm:
  versions:
    agent:
      version: "7.81.0"       # use the latest available agent version
    cluster_agent:
      version: "7.81.0"       # use the latest available cluster-agent version
    injector: "0.60.0"        # use the latest available injector version
  config:
    datadog:
      kubelet:
        tlsVerify: false      # needed locally; the kubelet cert usually isn't trusted
    clusterAgent:
      enabled: true

Building from local source

Add build: {} to any component and pass --build:

---
platform:
  type: kind              # recommended
  name: my-dev-cluster
  reset: false
helm:
  versions:
    agent:
      version: "7.81.0"       # use the latest available version
      build: {}             # build agent from local source at code_root/datadog-agent
    cluster_agent:
      version: "7.81.0"       # use the latest available version
      build: {}
    injector:
      version: "0.60.0"       # use the latest available version
      build: {}             # build auto_inject from code_root/auto_inject
  config:
    datadog:
      kubelet:
        tlsVerify: false
    clusterAgent:
      enabled: true

injector-dev apply -f scenario.yaml --build

You can pin a build tag with build: { tag: "dev.1" } (defaults to a

git-derived tag otherwise).

Version / image field reference

Each of agent, cluster_agent, injector, csi, (and operator in operator

scenarios) accepts either a string or a map:

injector: "0.60.0"          # string → pull this tag from the default repo

agent:                       # map form
  tag: "7.81.0"
  repository: registry.ddbuild.io/ci/datadog-agent/agent   # override the image repo
  pullPolicy: IfNotPresent
  build:                     # presence of `build` → build locally (needs --build)
    tag: "dev.1"

Pin to a CI pipeline / branch artifact — reproduce a coworker's PR build (or

any pipeline build) without compiling locally:

helm:
  versions:
    agent:
      repository: registry.ddbuild.io/ci/datadog-agent/agent
      tag: v<PIPELINE>-<COMMIT>-7-amd64
    cluster_agent:
      repository: registry.ddbuild.io/ci/datadog-agent/cluster-agent
      tag: v<PIPELINE>-<COMMIT>-amd64

Full helm: schema

| Field | Description |

| --- | --- |

| versions | agent, cluster_agent, injector, csi image specs (see above). |

| config | YAML passed to Helm as the values file (the datadog / clusterAgent / agents tree). |

| configFile | Path to an external Helm values file instead of inline config. |

| localChartPath | Install from a local chart dir instead of the public repo (see below). |

| apps | List of test apps deployed via the base app chart (see Apps). |

| namespaces | Explicitly create namespaces with specific labels. |

| manifests | Raw Kubernetes YAML files applied after the agent + apps. |

| charts | Additional Helm charts to install alongside. |

Test apps

Apps are deployed through a shared base chart (schema in apps/base/values.yaml).

Sample apps live in apps/: c, dotnet, java, js, php, python, ruby.

helm:
  apps:
    - name: python
      namespace: application
      values:
        image:
          repository: registry.ddbuild.io/ci/injector-dev/python
          tag: "2cd78ded"
        service:
          port: "8080"
        podLabels:
          language: python
          tags.datadoghq.com/env: local
        env:
          - name: DD_TRACE_DEBUG
            value: "true"
          - name: DD_APM_INSTRUMENTATION_DEBUG
            value: "true"

App fields: name, namespace, values (or valuesFile), build (build the

app image locally), injector (override injector image per-app), wait.

> Kubernetes health checks hit each pod's endpoints, so a running sample app

> automatically produces traces once instrumentation is enabled — a quick way to

> confirm injection is working.

Raw manifests & namespaces

helm:
  namespaces:
    - name: cache
      labels:
        team: platform
  manifests:
    - path: "redis-with-password.yaml"   # relative to the scenario file
      namespace: cache                    # auto-created if missing

Local Helm chart

If you're also changing the Datadog Helm chart, point at a local copy.

injector-dev then skips the repo add/update and installs from the path:

helm:
  localChartPath: ~/dd/helm-charts/charts/datadog
  versions:
    agent: { version: "7.81.0", build: {} }         # use the latest available version
    cluster_agent: { version: "7.81.0", build: {} }
    injector: "0.60.0"
  config:
    datadog:
      kubelet: { tlsVerify: false }
    clusterAgent: { enabled: true }

Pair with --helm-skip-schema-validation if your local chart adds values the

published schema doesn't know about yet.

Operator scenarios

Switch the top-level key to operator:. config becomes the **DatadogAgent CRD

spec** rather than Helm values:

---
platform:
  type: kind              # recommended
  name: operator-example
  reset: false
operator:
  versions:
    operator: "1.28.0"        # use the latest available operator version
    agent: "7.81.0"           # use the latest available version
    cluster_agent: "7.81.0"
    injector: "0.60.0"
  config:
    apiVersion: datadoghq.com/v2alpha1
    kind: DatadogAgent
    metadata:
      name: datadog
    spec:
      features:
        apm:
          instrumentation:
            enabled: true

The operator itself can be built from local source too: set

versions.operator.build: {} and pass --build.

Other commands

| Command | Description |

| --- | --- |

| injector-dev start [--platform <p>] [--debug] | Start the k8s platform manually. |

| injector-dev stop | Tear everything down (end of day). |

| injector-dev reset | Reset the cluster to a clean state. |

| injector-dev reset --hard | Full reset including the VM (colima only). |

| injector-dev new --type helm\|operator --output scenario.yaml [--edit] | Scaffold a scenario. |

| injector-dev build --type <t> [...] | Build a single component without deploying. |

| injector-dev version | Print version / commit / build time. |

Standalone builds

build --type accepts: app, agent, cluster-agent, injector, operator, csi.

injector-dev build --type injector
injector-dev build --type cluster-agent
injector-dev build --type app --context ./apps/python

build flags: -t/--type, -n/--name, -c/--context, -f/--dockerfile,

-r/--repository, -g/--tag.

Fast-iteration tips

• Set platform.reset: false and a stable platform.name per scenario so

applies reuse the cluster instead of recreating it. Override with --reset=true

only when you need a clean slate.

• Keep dev_container.persist: true for ~30s rebuilds after the first build.
• Use --skip-agent-validation when the Agent intentionally won't fully start

(e.g. testing a failure path) so apply doesn't error out.

• --app-image-tag/-t sets one image tag across all apps at once.
• tlsVerify: false under datadog.kubelet is almost always needed locally.

Troubleshooting

• Agent won't report / auth errors → check DD_API_KEY/DD_APP_KEY (or the

active profile's config) and that you're pointed at the right org.

• Kubelet TLS errors → set datadog.kubelet.tlsVerify: false.
• Platform flag "sticks" → if you passed --platform to start, pass it to

every apply too, or set platform: in the scenario/config.

• Stale/unhealthy cluster → injector-dev reset (or reset --hard on colima).
• Helm schema rejects new values → --helm-skip-schema-validation.
• Colima/minikube conflicts → stop any pre-existing instances first.
• Add --debug to any command for verbose logs.