Deploying Go SDK Bundles Agent Skill

Deploying Go SDK Bundles

A Go SDK deployment has one artifact: a bundle, a single self-contained native executable that also carries its embedded source and a manifest (the AFBNDL01 format, "the executable *is* the bundle"). You build and pack it with go, place it where Airflow's ExecutableCoordinator scans, and the Python task runner forks it once per task instance. This skill is platform-neutral: it shows the build, the coordinator wiring, then how to get the bundle onto a worker.

> Experimental. The Go SDK is under active development and not production-ready. Everything resolves against the single module github.com/apache/airflow/go-sdk (Go 1.24+).

> Order of operations: write the tasks (authoring-go-sdk-tasks) -> build and pack the bundle (this skill) -> place it under executables_root and configure the coordinator -> deploy the matching Python stub DAG.

Build and pack the bundle

The coordinator only recognizes a packed bundle: it scans for the AFBNDL01 trailer and silently skips any file that lacks it, so a plain go build binary is not deployable on its own. Use the packer, shipped as a Go 1.24 tool directive in go.mod (no global install, version pinned per project):

go tool airflow-go-pack ./example/bundle                              # build + pack in one step
go tool airflow-go-pack --goos linux --goarch amd64 ./example/bundle -- -trimpath  # cross-compile; flags after -- pass to `go build`
go tool airflow-go-pack --executable ./bin/sample-dag-bundle --source main.go --airflow-metadata <airflow-metadata.yaml> # pack an existing binary
go tool airflow-go-pack inspect ./bin/sample-dag-bundle               # inspect a packed bundle

The packer builds the binary, execs it with --airflow-metadata to capture the manifest from RegisterDags, then appends source + manifest + a 64-byte trailer. The result is one runnable file.

• Build for the worker's OS/arch. The bundle is a native executable and is not portable; cross-compile with --goos/--goarch. A mismatched binary fails on the worker with exec format error.
• Re-pack after any change to the binary. Re-stripping, re-signing, or swapping in a debug build invalidates the trailer's binary_sha256, and the bundle is then rejected.

Wire up the coordinator

Python's ExecutableCoordinator scans executables_root, matches the incoming dag_id against each bundle's embedded manifest, verifies its integrity hash, then forks the bundle. No Go process runs on the host.

• Place the packed executable under a scanned directory:

   cp ./bundle /opt/airflow/executable-bundles/   # identified by the AFBNDL01 trailer, not by filename

• Register ExecutableCoordinator and route the queue to it (see configuring-airflow-language-sdks):

   [sdk]
   coordinators = {"go": {"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator", "kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}}}
   queue_to_coordinator = {"golang": "go"}

• Deploy the matching Python stub DAG; its queue= must equal the queue_to_coordinator key (golang here), and its dag_id/task_ids must match what the bundle registered.

Deployment paths

The SDK runs on any Airflow with the Task SDK; Astronomer tooling is not required.

Docker / Kubernetes

Cross-compile the bundle for the image's platform and bake it in. No Go runtime or worker process is needed in the image; the Python task runner forks the bundle.

FROM apache/airflow:3.3.0        # the language SDKs target Airflow 3.3+
COPY ./executable-bundles/ /opt/airflow/executable-bundles/
# set AIRFLOW__SDK__COORDINATORS and AIRFLOW__SDK__QUEUE_TO_COORDINATOR as env vars

On the Helm chart, bake the bundle into a custom image as above or mount it via a shared volume, and set the [sdk] config through environment variables on the worker/scheduler. See deploying-airflow for the broader Docker Compose and Helm workflow.

> The apache/airflow:3.3.0 tag above is illustrative: the language SDKs need Airflow 3.3 or newer. Pin whatever current 3.x you actually run rather than copying this tag from memory; read the base image's current tags or docs.

Astro (one option, not required)

• Build/pack the bundle, then stage it in the project: mkdir -p include/executable-bundles && cp ../go-bundle/<packed-bundle> include/executable-bundles/.
• In the project Dockerfile, copy the bundle to the coordinator's directory: COPY include/executable-bundles/ /opt/airflow/executable-bundles/.
• Put the coordinator config in the project .env (loaded automatically): the AIRFLOW__SDK__* JSON values (see configuring-airflow-language-sdks).
• astro dev start (or astro dev restart after changes); deploy with astro deploy.

> Don't pin Astro Runtime / Airflow versions from memory; read the generated Dockerfile or current docs. While the Go SDK is in preview, a beta/dev image may be required.

Versioning and preview installs

go-sdk/ is a single Go module, so its release tag takes the monorepo subdir form, go-sdk/vX.Y.Z (do not create per-cmd tags). Your bundle module depends on github.com/apache/airflow/go-sdk; pinning that version also pins airflow-go-pack, which is a package in the same module referenced through the tool directive. Pin against the release tag:

go get github.com/apache/airflow/[email protected]

To build against an unreleased commit or branch (for example, to try a fix ahead of the next tag), depend on it directly and Go fabricates a pseudo-version:

go get github.com/apache/airflow/go-sdk@<commit-or-branch>

Deploy checklist

• Bundle built and packed (go tool airflow-go-pack); registered dag_id/task_id match the Python stubs.
• Built for the worker's OS/arch (e.g. --goos linux --goarch amd64).
• Packed AFBNDL01 bundle placed under a directory in executables_root.
• ExecutableCoordinator + queue_to_coordinator configured (configuring-airflow-language-sdks).
• Python stub DAG deployed, its queue= routed to the Go coordinator.
• Re-packed after any rebuild/strip/sign (preserves binary_sha256).

Related Skills

• authoring-go-sdk-tasks: Write the Go task code and the matching Python stubs.
• configuring-airflow-language-sdks: Register ExecutableCoordinator and route the queue.
• deploying-airflow: General Airflow deployment (Astro, Docker Compose, Kubernetes).
• setting-up-astro-project: Initialize and configure an Astro project.