azure/acn-go-version-bump
Go version upgrade procedure for Azure Container Networking. Use when upgrading Go minor/patch versions, bumping MS Go toolchain, fixing FIPS/systemcrypto configuration, updating Dockerfile templates, or responding to Go CVE patches. Covers the 3-tier automation (digest refresh, patch bump, minor upgrade) and the manual steps for each tier.
npx skills add https://github.com/Azure/azure-container-networking --skill acn-go-version-bump
Persona: You are a Go platform engineer maintaining the Azure Container Networking build toolchain. You understand MS Go's FIPS requirements, MCR image tagging, and the multi-file version propagation needed for Go upgrades in this repo.
Modes:
go-version-check.yaml automation workflow.CRITICAL: Before making ANY code changes, you MUST fetch and analyze ALL relevant MS Go documentation for the target version. Do not rely on hardcoded rules — requirements change between versions.
These docs may be pre-cached in .github/ms-go-docs/ by copilot-setup-steps.yml (which runs before the agent firewall activates). Try reading the cached files first. If they don't exist (e.g., running locally or cache wasn't populated), fall back to gh api.
# 1. FIPS README — the AUTHORITATIVE source for crypto configuration
# PREFERRED: read pre-cached file (firewall blocks cross-repo gh api calls)
cat .github/ms-go-docs/README.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/fips/README.md?ref=microsoft/main" --jq '.content' | base64 -d
# 2. NocgoOpenSSL — detailed nocgo backend docs
cat .github/ms-go-docs/NocgoOpenSSL.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/NocgoOpenSSL.md?ref=microsoft/main" --jq '.content' | base64 -d
# 3. Installation guide
cat .github/ms-go-docs/Installation.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/Installation.md?ref=microsoft/main" --jq '.content' | base64 -d
# 4. Migration Guide — toolchain behavior, breaking changes
cat .github/ms-go-docs/MigrationGuide.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/MigrationGuide.md?ref=microsoft/main" --jq '.content' | base64 -d
# 5. FIPS User Guide — runtime requirements, crypto API behavior
cat .github/ms-go-docs/UserGuide.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/fips/UserGuide.md?ref=microsoft/main" --jq '.content' | base64 -d
# 6. Additional Features — all MS-specific patches
cat .github/ms-go-docs/AdditionalFeatures.md 2>/dev/null || \
gh api "repos/microsoft/go/contents/eng/doc/AdditionalFeatures.md?ref=microsoft/main" --jq '.content' | base64 -d
# 7. Upstream Go release notes (fetch from go.dev — may be blocked)
# curl -sL https://go.dev/doc/go1.<MINOR> || echo "Blocked by firewall — skip"
Image digests are also pre-resolved into .github/image-digests/:
cat .github/image-digests/go-image.txt # Go builder SHA
cat .github/image-digests/mariner-core.txt # Azure Linux base/core SHA
cat .github/image-digests/mariner-distroless.txt # Azure Linux distroless SHA
cat .github/image-digests/windows-hpc.txt # Windows HPC base image SHA
Use these cached values for Dockerfile updates when skopeo is blocked by the firewall.
This is the step the agent failed on previously. Follow it precisely.
After fetching the docs, you MUST determine the correct GOEXPERIMENT value for EVERY build in this repo. The answer depends on THREE things:
CGO_ENABLED=0 or CGO_ENABLED=1How to find the answer in the docs:
eng/doc/fips/README.md and find the section titled "Usage: Common configurations"systemcrypto (uses OpenSSL via dlopen)ms_nocgo_opensslcrypto)CRITICAL UNDERSTANDING: In MS Go, the crypto backend is NOT optional — it's mandatory for FIPS compliance. If a build requires CGO_ENABLED=0 (static binary) on Linux, and the default crypto backend requires CGO, then you MUST set a GOEXPERIMENT that provides a nocgo-compatible backend. Without it, the build will fail with linker errors or crypto initialization panics.
DO NOT assume that "no GOEXPERIMENT" is safe for CGO=0 builds. Read the docs and determine what backend is used by default and whether it requires CGO.
After determining the correct GOEXPERIMENT per (CGO, OS) pair, audit EVERY build path:
# Find all CGO settings in build scripts
grep -rn "CGO_ENABLED" .pipelines/build/scripts/ --include="*.sh"
# Find all CGO settings in Dockerfiles and templates
grep -rn "CGO_ENABLED" . --include="*.Dockerfile" --include="*.Dockerfile.tmpl" --include="Dockerfile.tmpl"
# Find CGO settings in Makefiles
grep -rn "CGO_ENABLED" Makefile */Makefile
# ALSO find implicit CGO=1 via -buildmode=c-shared (doesn't explicitly set CGO_ENABLED)
grep -rn "buildmode=c-shared" Makefile */Makefile .pipelines/build/scripts/
For EACH file that sets CGO_ENABLED OR uses -buildmode=c-shared, you MUST ensure the correct GOEXPERIMENT is set in the same scope.
GOEXPERIMENT values per (CGO, OS) combination — from fips/README.md table-buildmode, -ldflags, etc.)GOTOOLCHAIN behavior changesgo mod tidy behavior changes (stricter validation, new directives)Cross-reference:
MARINER_DISTROLESS_IMG in build/images.mkMS_GO_NOSYSTEMCRYPTO deprecated?Before making any code changes, produce a change plan:
## MS Go <VERSION> Upgrade — Requirements Analysis
### Source Documents Reviewed
- [ ] eng/doc/fips/README.md: <GOEXPERIMENT table findings>
- [ ] eng/doc/NocgoOpenSSL.md: <nocgo backend details>
- [ ] docs/go1.XX.md: <version-specific changes>
- [ ] eng/doc/MigrationGuide.md: <relevant migration steps>
- [ ] eng/doc/fips/UserGuide.md: <runtime requirements>
### GOEXPERIMENT Determination (from fips/README.md)
| Build Configuration | GOEXPERIMENT Required | Reason |
|---|---|---|
| Linux + CGO_ENABLED=1 | <value from docs> | <why> |
| Linux + CGO_ENABLED=0 | <value from docs> | <why — if blank, explain why safe> |
| Windows (any CGO) | <value from docs> | <why> |
### Files Requiring GOEXPERIMENT Changes
List EVERY file that needs modification:
| File | CGO Setting | Current GOEXPERIMENT | Required GOEXPERIMENT | Action |
|---|---|---|---|---|
| .pipelines/build/scripts/cni.sh | 0 | (none) | <value> | Add export |
| .pipelines/build/scripts/cns.sh | 0 | (none) | <value> | Add export |
| ... | ... | ... | ... | ... |
| Makefile (all CGO=0 targets) | 0 | (none) | <value> | Add inline |
### Risk Assessment
- Breaking changes affecting this codebase: ...
- FIPS compliance impact: ...
Only proceed with code changes after the analysis is complete and EVERY build path is accounted for.
ACN uses floating minor version tags for the Go build image (build/images.mk):
GO_IMG uses a 2-part minor version tag (e.g., golang:1.26-azurelinux3.0)make dockerfiles timego.mod version rule (minor upgrades only): When performing a minor version upgrade (Tier 3), the go directive in go.mod files MUST use 1.XX.1 (the .1 patch), NOT the latest patch (e.g., NOT 1.26.4). This is because:
.0 releases are pre-release/stabilization — avoid them.1 is the first stable patch — use this as the minimumNote: For patch bumps (Tier 2), the workflow correctly sets go.mod to the target patch version (e.g., 1.26.3 → 1.26.4). This rule only applies to the initial minor upgrade.
Example: If upgrading to Go 1.26, set go 1.26.1 in all go.mod files, even if the latest available patch is 1.26.4.
⚠️ IMPORTANT: The ROOT go.mod is the FIRST file to update. Do NOT only update sub-modules.
build/images.mk (GO_IMG=golang:1.XX-azurelinux3.0) ← primary image tag
│
├── ROOT MODULE (update FIRST):
│ └── → go.mod (go 1.XY.Z) ← must match (use .1+ not .0)
│
├── BUILD ENVIRONMENT:
│ ├── → tools-go/go.mod (go 1.XY.Z) ← must match root
│ ├── → .devcontainer/Dockerfile (VARIANT="1.XX") ← dev container version
│ ├── → .pipelines/build/scripts/install-go.sh (DEFAULT_IMAGE SHA)
│ ├── → bpf-prog/ipv6-hp-bpf/linux.Dockerfile (Go image SHA)
│ ├── → npm/linux.Dockerfile (tag 1.XX.Y)
│ ├── → npm/windows.Dockerfile (tag 1.XX.Y)
│ └── → All .tmpl Dockerfiles (via `make dockerfiles`)
│
└── INDEPENDENT MODULES (bump go directive in EACH):
├── → azure-ipam/go.mod
├── → azure-ip-masq-merger/go.mod
├── → azure-iptables-monitor/go.mod
├── → bpf-prog/ipv6-hp-bpf/go.mod
├── → cilium-log-collector/go.mod
├── → cni/go.mod
├── → crd/go.mod
├── → dropgz/go.mod
├── → npm/go.mod
├── → pkgerrlint/go.mod
├── → tools/azure-npm-to-cilium-validator/go.mod
└── → zapai/go.mod
go.mod (ROOT) — Update go directive FIRST. Use 1.XX.1 (NOT latest patch)go 1.26.1 (NOT go 1.26.4)build/images.mk — Update GO_IMG tag1.27-azurelinux3.0, never 1.27.0-azurelinux3.0tools-go/go.mod — Update go directive to match rootgo.mod files — Update go directive to match (see full list above)go mod tidy — it times out in the agent environmentgo.sum files remain valid for pure version bumps (deps don't change)tools-go/go.sum is handled by the migration step (copy from tools.go.sum).devcontainer/Dockerfile — Update VARIANT arg to "1.XX".pipelines/build/scripts/install-go.sh — Update DEFAULT_IMAGE to new Go image digest: # Read the pre-cached Go image digest (resolved during copilot-setup-steps)
NEW_GO_DIGEST=$(cat .github/image-digests/go-image.txt 2>/dev/null)
# If cache exists, use it to update install-go.sh:
if [ -n "$NEW_GO_DIGEST" ]; then
sed -i "s|DEFAULT_IMAGE=\".*\"|DEFAULT_IMAGE=\"${NEW_GO_DIGEST}\"|" .pipelines/build/scripts/install-go.sh
fi
Also update the skopeo inspect comment above it to reference the new tag.
bpf-prog/ipv6-hp-bpf/linux.Dockerfile — Update Go image SHA (use same digest from step 7)npm/linux.Dockerfile and npm/windows.Dockerfile — Update Go taggolang:1.26.4), NOT the -azurelinux3.0 suffixed tagnpm/windows.Dockerfile builds on a Linux builder (--platform=linux/amd64) cross-compiling with GOOS=windows — it still needs GOEXPERIMENT for CGO_ENABLED=0 on the Linux build stageMS_GO_NOSYSTEMCRYPTO=1 with appropriate GOEXPERIMENT=<value> in BOTH files10. Run make dockerfiles — Regenerate all template-based Dockerfiles
This step is where the previous agent failed. Do NOT skip any file.
Based on your GOEXPERIMENT determination from Step 0, you must update EVERY build path. Here is the COMPLETE list of locations that need GOEXPERIMENT:
.pipelines/build/scripts/*.sh)Each script that sets CGO_ENABLED MUST also export the correct GOEXPERIMENT:
# For CGO_ENABLED=0 scripts (Linux-only guard if script is multi-platform):
if [[ "$GOOS" == "linux" ]] || [[ -z "$GOOS" ]]; then
export GOEXPERIMENT=<value_for_cgo0>
fi
export CGO_ENABLED=0
# For CGO_ENABLED=1 scripts:
export GOEXPERIMENT=<value_for_cgo1>
export CGO_ENABLED=1
Scripts to update:
.pipelines/build/scripts/cni.sh.pipelines/build/scripts/cns.sh.pipelines/build/scripts/npm.sh.pipelines/build/scripts/dropgz.sh.pipelines/build/scripts/azure-ipam.sh.pipelines/build/scripts/azure-ip-masq-merger.sh.pipelines/build/scripts/azure-iptables-monitor.sh.pipelines/build/scripts/ipv6-hp-bpf.sh.pipelines/build/scripts/cilium-log-collector.sh*.Dockerfile.tmpl)Each template that sets CGO_ENABLED in a RUN go build must have ENV GOEXPERIMENT=<value> set BEFORE the build stage:
# For CGO_ENABLED=0 stages:
ENV GOEXPERIMENT=<value_for_cgo0>
RUN CGO_ENABLED=0 go build ...
# For CGO_ENABLED=1 stages:
ENV GOEXPERIMENT=<value_for_cgo1>
RUN CGO_ENABLED=1 go build ...
Templates to update:
cni/Dockerfile.tmplcns/Dockerfile.tmplazure-ipam/Dockerfile.tmplazure-ip-masq-merger/Dockerfile.tmplazure-iptables-monitor/Dockerfile.tmplcilium-log-collector/Dockerfile.tmplbpf-prog/ipv6-hp-bpf/linux.Dockerfilenpm/linux.Dockerfilenpm/windows.Dockerfile ← builds on Linux (--platform=linux/amd64), needs GOEXPERIMENT for CGO=0The root Makefile has CGO_ENABLED=0 build lines for local development. Add a variable and apply it inline:
# GOEXPERIMENT for CGO_ENABLED=0 Linux builds (MS Go FIPS requirement)
ACN_GOEXPERIMENT ?= <value_for_cgo0>
# Apply to each CGO_ENABLED=0 build line:
GOEXPERIMENT=$(ACN_GOEXPERIMENT) CGO_ENABLED=0 go build ...
Do NOT export GOEXPERIMENT globally — it breaks renderkit/tool builds that don't recognize the experiment.
cilium-log-collector/Makefile — The cilium-log-collector-binary target uses -buildmode=c-shared which implicitly requires CGO_ENABLED=1. You MUST add explicit CGO_ENABLED=1 GOEXPERIMENT=<value_for_cgo1> to the build command: # BEFORE (missing GOEXPERIMENT):
cd $(CILIUM_LOG_COLLECTOR_DIR) && go build -buildmode=c-shared ...
# AFTER (correct):
cd $(CILIUM_LOG_COLLECTOR_DIR) && CGO_ENABLED=1 GOEXPERIMENT=<value_for_cgo1> go build -buildmode=c-shared ...
Do NOT skip this file — -buildmode=c-shared implies CGO but doesn't explicitly set CGO_ENABLED=1, so grep for CGO_ENABLED won't find it. Search for -buildmode=c-shared as well.
Go 1.26's stricter go mod tidy rejects root-level modfiles (tools.go.mod) that share the same module path as go.mod. The tools module MUST live in its own directory.
If tools.go.mod exists at the repo root (not yet migrated):
tools-go/ directorytools.go.mod → tools-go/go.modtools.go.sum → tools-go/go.sumtools-go/go.mod: - module github.com/Azure/azure-container-networking
+ module github.com/Azure/azure-container-networking/tools-go
grep -rn "tools\.go\.mod" . --include="*.go" --include="Makefile" --include="*.sh" --include="*.yaml" --include="*.yml" | grep -v vendor
Common locations that reference -modfile=tools.go.mod:
crd/clustersubnetstate/Makefilecrd/multitenancy/Makefilecrd/multitenantnetworkcontainer/Makefilecrd/nodenetworkconfig/Makefilecrd/overlayextensionconfig/Makefilecns/multitenantcontroller/mockclients/Makefilenpm/pkg/dataplane/Makefileplatform/Makefilescripts/install-protoc.shMakefile (TOOLS_GO_MOD variable)Replace all: tools.go.mod → tools-go/go.mod
go mod tidy — just copy the sum file (deps don't change for version bumps)If tools-go/go.mod already exists (already migrated):
go directive to match rootgo mod tidy (times out in agent environment)-modfile references point to tools-go/go.mod (not old tools.go.mod)After making all changes:
go.mod has the correct version: head -5 go.mod # MUST show "go 1.XX.Y" — if still old version, the upgrade is INCOMPLETE
go build ./... — Verify compilation succeeds (all binaries)go vet ./... — Check for deprecated API usagedocker build (spot-check) — Verify at least one container image builds: # Quick validation that Dockerfiles + GOEXPERIMENT produce working images
docker build -f cni/Dockerfile -t acn-cni-test --build-arg VERSION=test .
make dockerfiles — Regenerate ALL template-based Dockerfiles. This resolves:{{.GO_PIN}} → current Go image as image:tag@sha{{.MARINER_CORE_PIN}} → current azurelinux/base/core as image:tag@sha{{.MARINER_DISTROLESS_PIN}} → current azurelinux/distroless/base as image:tag@shaPins MUST keep the tag (image:tag@sha, not image@sha) so Dependabot can update them.
The generated files live in TWO locations:
cni/Dockerfile, cns/Dockerfile, azure-ipam/Dockerfile, etc..pipelines/build/dockerfiles/*.DockerfileIf make dockerfiles fails (e.g., skopeo blocked by firewall or MCR auth issues), use the pre-cached digests:
# Read pre-resolved digests from setup steps (already image:tag@sha)
GO_PIN=$(cat .github/image-digests/go-image.txt 2>/dev/null)
MARINER_CORE_PIN=$(cat .github/image-digests/mariner-core.txt 2>/dev/null)
MARINER_DISTROLESS_PIN=$(cat .github/image-digests/mariner-distroless.txt 2>/dev/null)
WINDOWS_HPC_PIN=$(cat .github/image-digests/windows-hpc.txt 2>/dev/null)
# If cached files don't exist, try skopeo directly (may fail behind firewall)
if [ -z "$GO_PIN" ]; then
GO_IMG=mcr.microsoft.com/oss/go/microsoft/golang:1.XX-azurelinux3.0
GO_PIN="${GO_IMG}@$(skopeo inspect docker://${GO_IMG} --format "{{.Digest}}" 2>/dev/null)"
fi
Then use sed to update image pins in ALL generated .Dockerfile files (not .tmpl).
IMPORTANT: Both .pipelines/build/dockerfiles/*.Dockerfile AND component */Dockerfile files must be updated — they are ALL generated from templates.
go mod tidy — it times out. Existing go.sum files remain valid for version bumps.replace directives are needed grep "VARIANT" .devcontainer/Dockerfile # Must show target version
# Verify ALL CGO_ENABLED=0 scripts have the correct GOEXPERIMENT
for script in .pipelines/build/scripts/*.sh; do
if grep -q "CGO_ENABLED=0" "$script"; then
if ! grep -q "GOEXPERIMENT=<value_for_cgo0>" "$script"; then
echo "MISSING GOEXPERIMENT in: $script"
fi
fi
done
# Verify ALL CGO_ENABLED=0 Dockerfile templates have it
for tmpl in $(find . -name '*.Dockerfile.tmpl' -o -name 'Dockerfile.tmpl' | grep -v vendor); do
if grep -q "CGO_ENABLED=0" "$tmpl"; then
if ! grep -q "GOEXPERIMENT=<value_for_cgo0>" "$tmpl"; then
echo "MISSING GOEXPERIMENT in: $tmpl"
fi
fi
done
TARGET="1.XX" # Replace with actual target minor version
# Root go.mod MUST be updated
grep "^go " go.mod | grep -q "$TARGET" || echo "FAIL: root go.mod not updated!"
# .devcontainer must reference new version
grep -q "VARIANT=\"$TARGET\"" .devcontainer/Dockerfile || echo "FAIL: .devcontainer not updated!"
# All sub-module go.mod files
for mod in azure-ipam cni crd dropgz npm zapai azure-ip-masq-merger azure-iptables-monitor \
bpf-prog/ipv6-hp-bpf cilium-log-collector pkgerrlint tools/azure-npm-to-cilium-validator; do
if [ -f "$mod/go.mod" ]; then
grep "^go " "$mod/go.mod" | grep -q "$TARGET" || echo "FAIL: $mod/go.mod not updated!"
fi
done
# tools-go module
grep "^go " tools-go/go.mod | grep -q "$TARGET" || echo "FAIL: tools-go/go.mod not updated!"
# build/images.mk
grep "GO_IMG" build/images.mk | grep -q "$TARGET" || echo "FAIL: build/images.mk not updated!"
chore: upgrade Go <OLD> → <NEW>release/v1.7Every Go version change on master MUST be backported to release/v1.7.
release/v1.7go mod tidy — existing go.sum remains valid for version bumpsmake dockerfileschore(release/v1.7): upgrade Go <OLD> → <NEW>build/images.mk defines GO_IMG and MARINER_DISTROLESS_IMG.tmpl files are rendered into Dockerfiles by make dockerfilesrenderkit and skopeo to resolve image tags to image:tag@sha pins (tag required for Dependabot).pipelines/build/scripts/install-go.sh> GOEXPERIMENT values are version-dependent. Always determine the correct value from
> eng/doc/fips/README.md for the target version. The CGO settings below are fixed per component.
| Component | CGO_ENABLED | Platform | Build Mode | Notes |
|-----------|:-----------:|:--------:|:----------:|-------|
| cni | 0 | linux | static binary | Network plugin |
| cns | 0 | linux/windows | static binary | Node daemon |
| npm | 0 | linux/windows | static binary | Network policy |
| dropgz | 0 | linux | static binary | Installer wrapper |
| azure-ipam | 0 | linux | static binary | IP allocator |
| azure-ip-masq-merger | 0 | linux | static binary | IPtables helper |
| azure-iptables-monitor | 0 | linux | static binary | IPtables monitor |
| ipv6-hp-bpf | 0 | linux | static binary | BPF health probe |
| cilium-log-collector | 1 | linux | c-shared (.so) | Fluent Bit plugin, requires CGO |
When upgrading Go, verify compatibility with AKS supported Kubernetes versions:
release/v1.7, etc.) that map to AKS release trains1.XX.1 in go.mod — NOT the latest patch. The container image provides the actual binary version.npm/ component is released as npm-lite — ensure Dockerfiles build correctlygolang:1.26.4) without -azurelinux3.0 suffixnpm/windows.Dockerfile builds on a Linux builder (--platform=linux/amd64) — still needs GOEXPERIMENT for CGO=0baseimages.yaml CI workflow fails if make dockerfiles output doesn't match committed filesbuild/images.mkTake azure/acn-go-version-bump from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.