Azure Functions

Azure Functions Dev/Test Workflow

This skill helps tracer engineers test changes to the Datadog.AzureFunctions package: build a local dev version, deploy it to a test Azure Function App, trigger the function, and analyze traces/logs to verify instrumentation behavior.

Prerequisites

This skill requires the following tools (assume they are installed and only troubleshoot if errors occur):

• PowerShell: pwsh (PowerShell 7+) preferred, or powershell.exe (PowerShell 5.1+ on Windows)
• Always prefer pwsh over powershell.exe when available
• Scripts use PowerShell-specific cmdlets like Expand-Archive
• Azure CLI: az (must be authenticated)
• Azure Functions Core Tools: func
• .NET SDK: Matching target framework of sample app

Only if a tool fails, provide installation links:

• PowerShell: https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell
• Azure CLI: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli
• Azure Functions Core Tools: https://learn.microsoft.com/en-us/azure/azure-functions/functions-run-local
• .NET SDK: https://dotnet.microsoft.com/download

Commands

When invoked with an argument, perform the corresponding workflow:

• /azure-functions build-nuget - Build the Datadog.AzureFunctions NuGet package
• /azure-functions deploy [app-name] - Deploy to Azure Function App
• /azure-functions configure [app-name] - Configure environment variables for Datadog instrumentation
• /azure-functions test [app-name] - Trigger and verify function execution
• /azure-functions logs [app-name] - Download and analyze logs

If no argument is provided, guide the user through the full workflow interactively.

Context

Current repository: This skill assumes you are working from the root of the dd-trace-dotnet repository.

Prerequisites: Users provide their own Azure Function App name (-AppName), resource group (-ResourceGroup), and sample app path (-SampleAppPath). The sample app must:

• Reference the Datadog.AzureFunctions NuGet package
• Have a nuget.config file (in the app directory or a parent directory) that defines a local NuGet feed pointing to a directory on disk, for example:

   <?xml version="1.0" encoding="utf-8"?>
   <configuration>
     <packageSources>
       <clear />
       <add key="local" value="nuget/local-source" />
       <add key="nuget" value="https://api.nuget.org/v3/index.json" />
     </packageSources>
   </configuration>

The -CopyTo parameter of Build-AzureFunctionsNuget.ps1 must point to the same directory as the local feed (e.g. if the feed value is nuget/local-source relative to the nuget.config location, then -CopyTo should be the absolute path to that directory).

Workflow Steps

1. Build NuGet Package

CRITICAL: Before building, verify that tracer/src/Datadog.AzureFunctions/Datadog.AzureFunctions.csproj uses PackageReference (not ProjectReference) for Datadog.Trace and Datadog.Trace.Annotations. This ensures the locally-built package references the latest releases from nuget.org instead of building them from source.

Check the .csproj — it should contain:

<PackageReference Include="Datadog.Trace" Version="*"/>
<PackageReference Include="Datadog.Trace.Annotations" Version="*" />

If instead it contains ProjectReference lines like these, replace them with the PackageReference lines above:

<!-- These are the production references — replace for local testing: -->
<ProjectReference Include="$(MSBuildThisFileDirectory)..\Datadog.Trace.Manual\Datadog.Trace.Manual.csproj" />
<ProjectReference Include="$(MSBuildThisFileDirectory)..\Datadog.Trace.Annotations\Datadog.Trace.Annotations.csproj" />

IMPORTANT: The PackageReference change is for local testing only. Do NOT commit it. If it's already using PackageReference, no change is needed.

Build the Datadog.AzureFunctions NuGet package with your changes:

./tracer/tools/Build-AzureFunctionsNuget.ps1 -CopyTo <output-dir>

What this does:

• (If -BuildId specified and files not already downloaded) Downloads bundle from Azure DevOps build once
• Generates a unique prerelease version from a timestamp (e.g. 3.38.0-dev20260209143022)
• Cleans previous builds
• Builds Datadog.Trace (net6.0)
• Publishes to bundle folder
• Packages Datadog.AzureFunctions.nupkg with the generated version (referencing latest nuget.org releases)
• Copies to the directory specified by -CopyTo

Versioning: Each build gets a unique version, so NuGet caching is never an issue.

The sample app should use a floating version like 3.38.0-dev.* in its package reference

(or Directory.Packages.props) to always resolve the latest local dev build.

Options:

• -CopyTo <output-dir> - Copy the built package to the specified directory (typically your local NuGet feed)
• -Version '3.38.0-dev.custom' - Use a specific version instead of auto-generating
• -BuildId 12345 - One-time download of bundle files from Azure DevOps build (only needed once per dd-trace-dotnet release, then reused for subsequent local builds)

Examples:

# Typical local build (after bundle files already downloaded)
./tracer/tools/Build-AzureFunctionsNuget.ps1 -CopyTo <output-dir>

# First build after new dd-trace-dotnet release (download bundle files once)
./tracer/tools/Build-AzureFunctionsNuget.ps1 -BuildId 12345 -CopyTo <output-dir>

2. Deploy Function

IMPORTANT: Before deploying, verify prerequisites:

Verify nuget.config exists:

$nugetConfig = ./.claude/skills/azure-functions/Find-NuGetConfig.ps1 -StartPath "<path-to-sample-app>"
if (-not $nugetConfig) {
    Write-Error "nuget.config not found in sample app directory or parent directories"
    exit 1
}
Write-Host "Found nuget.config at: $nugetConfig"

Verify environment variables are configured (skip if already done on a previous deploy):

$envCheck = ./.claude/skills/azure-functions/Test-EnvVars.ps1 -AppName "<app-name>" -ResourceGroup "<resource-group>"
if (-not $envCheck.AllRequiredPresent) {
    Write-Warning "Required environment variables are missing. Run '/azure-functions configure' first, or proceed if you plan to configure after deploying."
}

Use the Deploy-AzureFunction.ps1 script to automate deployment, wait, and trigger:

./tracer/tools/Deploy-AzureFunction.ps1 `
  -AppName "<app-name>" `
  -ResourceGroup "<resource-group>" `
  -SampleAppPath "<path-to-sample-app>"

What this does:

• Runs dotnet restore in the sample app directory
• Publishes to Azure with func azure functionapp publish
• Waits 60 seconds (default) for worker process to restart
• Triggers the HTTP endpoint and captures execution timestamp
• Outputs a result object for pipeline usage

Options:

• -SkipBuild - Skip dotnet restore
• -SkipWait - Skip 2-minute wait (not recommended)
• -WaitSeconds 60 - Custom wait duration
• -SkipTrigger - Skip HTTP trigger
• -TriggerUrl "https://..." - Custom trigger URL

Note: -AppName and -ResourceGroup are required parameters.

Pipeline usage (save output for log analysis):

$deploy = ./tracer/tools/Deploy-AzureFunction.ps1 `
  -AppName "<app-name>" `
  -ResourceGroup "<resource-group>" `
  -SampleAppPath "<path-to-sample-app>"

3. Configure Environment Variables

Configure Datadog instrumentation environment variables for an Azure Function App using Set-EnvVars.ps1:

./.claude/skills/azure-functions/Set-EnvVars.ps1 `
  -AppName "<app-name>" `
  -ResourceGroup "<resource-group>" `
  -ApiKey "<api-key>" `
  -Tier recommended `
  -Env "dev-lucas"

What this does:

• Detects the OS/platform (Linux or Windows)
• Builds the correct set of variables for the chosen tier
• Sets platform-specific profiler paths automatically
• Applies all settings via Azure CLI
• Restarts the function app (unless -SkipRestart)

Tiers:

• required - Minimum variables for instrumentation (CORECLR_*, DD_API_KEY, DD_DOTNET_TRACER_HOME, DOTNET_STARTUP_HOOKS)
• recommended - required + feature disables (AppSec, CI Visibility, RCM, Agent Feature Polling, Process)
• debug - recommended + debug logging (DD_TRACE_DEBUG, DD_LOG_LEVEL, DD_TRACE_LOG_SINKS, direct log submission)

Options:

• -Tier required|recommended|debug - Configuration tier (default: required)
• -Env "dev-lucas" - Set DD_ENV
• -Service "my-service" - Set DD_SERVICE
• -Version "1.0.0" - Set DD_VERSION
• -SamplingRules '<json>' - Set DD_TRACE_SAMPLING_RULES
• -ExtraSettings @{"KEY"="value"} - Set additional variables
• -SkipRestart - Don't restart the app after applying
• -WhatIf - Preview changes without applying

Preview before applying:

./.claude/skills/azure-functions/Set-EnvVars.ps1 `
  -AppName "<app-name>" `
  -ResourceGroup "<resource-group>" `
  -ApiKey "<api-key>" `
  -Tier debug `
  -WhatIf

Complete reference: See environment-variables.md for all available variables.

4. Test Function

Trigger an already-deployed function and capture the execution timestamp. Useful for re-testing after a deploy, or testing an app that was deployed earlier.

Discover available triggers:

# List HTTP-triggered functions and their URLs
func azure functionapp list-functions <app-name> --show-keys

Or via Azure CLI:

az functionapp function list --name <app-name> --resource-group <resource-group> --query "[].{name:name, href:invokeUrlTemplate}" -o table

Trigger and capture timestamp:

$timestamp = (Get-Date).ToUniversalTime().ToString("yyyy-MM-dd HH:mm:ss")
$response = Invoke-WebRequest -Uri "<trigger-url>" -UseBasicParsing
Write-Host "HTTP Status: $($response.StatusCode)"
Write-Host "Execution timestamp (UTC): $timestamp"

Save the timestamp — you'll need it for log filtering in the next step.

Note: The Deploy script (step 2) already triggers and captures a timestamp. Use this step when you want to re-test without redeploying.

5. Download and Analyze Logs

Use the Get-AzureFunctionLogs.ps1 script to download, extract, and analyze logs:

./tracer/tools/Get-AzureFunctionLogs.ps1 `
  -AppName "<app-name>" `
  -ResourceGroup "<resource-group>" `
  -OutputPath $env:TEMP `
  -ExecutionTimestamp "<YYYY-MM-DD HH:MM:SS>" `
  -All

IMPORTANT: Always specify -OutputPath $env:TEMP to save logs to the system temp folder instead of cluttering the repository directory.

What this does:

• Downloads logs from Azure to a timestamped zip file
• Extracts the archive
• Identifies host and worker log files
• Analyzes tracer version, span count, and trace parenting

Analysis options:

• -ShowVersion - Display Datadog tracer version from worker logs
• -ShowSpans - Count spans at execution timestamp (split by host/worker)
• -CheckParenting - Validate trace parenting (detect root span duplication)
• -All - Enable all analysis (recommended)

Pipeline usage (with Deploy script):

$deploy = ./tracer/tools/Deploy-AzureFunction.ps1 `
  -AppName "<app-name>" `
  -ResourceGroup "<resource-group>" `
  -SampleAppPath "<path-to-sample-app>"

./tracer/tools/Get-AzureFunctionLogs.ps1 `
  -AppName $deploy.AppName `
  -ResourceGroup "<resource-group>" `
  -OutputPath $env:TEMP `
  -ExecutionTimestamp $deploy.ExecutionTimestamp `
  -All

Log file patterns:

• Host process: dotnet-tracer-managed-Microsoft.Azure.WebJobs.Script.WebHost-{pid}.log
• Worker process: dotnet-tracer-managed-dotnet-{pid}.log

CRITICAL: The script automatically filters logs by execution timestamp. Raw log files are append-only and contain entries from multiple deployments/restarts.

Parenting analysis:

• Span fields: s_id (span ID), p_id (parent ID), t_id (trace ID)
• Healthy trace: Worker spans have same t_id as host, p_id matching host span IDs
• Broken trace: Worker spans have different t_id or p_id: null (orphaned root)

Verification Checklist

After deployment and testing:

• [ ] Function responds successfully (HTTP 200)
• [ ] Worker loaded correct tracer version (check "Assembly metadata")
• [ ] Host and worker spans share same trace ID
• [ ] Span parent-child relationships are correct (check p_id → s_id links)
• [ ] Process tags are correct (aas.function.process:host or worker)
• [ ] No error logs at execution timestamp
• [ ] AsyncLocal context flows correctly (if debugging context issues)

Common Troubleshooting

General guidance: For environment variable configuration issues, see environment-variables.md for complete reference on required, recommended, and debugging variables.

Function Not Responding

First, check if the app is running:

$envCheck = ./.claude/skills/azure-functions/Test-EnvVars.ps1 -AppName "<app-name>" -ResourceGroup "<resource-group>"
if ($envCheck.State -ne "Running") {
    Write-Host "App is '$($envCheck.State)' — starting it..."
    az functionapp start --name <app-name> --resource-group <resource-group>
}

If the app is running but not responding:

# Restart function app
az functionapp restart --name <app-name> --resource-group <resource-group>

Deployment Fails (func azure functionapp publish)

If the publish command fails, diagnose with:

# Check recent deployment status
az functionapp deployment list \
  --name <app-name> \
  --resource-group <resource-group> \
  --query "[0].{status:status, message:message, startTime:startTime}" -o table

# Stream live logs to see startup errors
func azure functionapp logstream <app-name>

Common causes:

• Auth expired: Run az login and retry
• App not running: Start it first with az functionapp start --name <app-name> --resource-group <resource-group>
• Build errors: Check dotnet restore output in the sample app directory

Wrong Tracer Version After Deployment

# Check all worker initializations
grep "Assembly metadata" LogFiles/datadog/dotnet-tracer-managed-dotnet-*.log

# If old version, rebuild from the dd-trace-dotnet repo root (each build gets a unique version, no cache issues)
./tracer/tools/Build-AzureFunctionsNuget.ps1 -CopyTo <output-dir> -Verbose
# Then restore and redeploy the sample app

Traces Not Appearing in Datadog

Verify all required environment variables (including DD_API_KEY, profiler paths, etc.):

./.claude/skills/azure-functions/Test-EnvVars.ps1 -AppName "<app-name>" -ResourceGroup "<resource-group>" -IncludeRecommended

If all env vars pass, check worker initialization in logs:

grep "Datadog Tracer initialized" LogFiles/datadog/dotnet-tracer-managed-dotnet-*.log

Complete reference: See environment-variables.md for all available variables.

Separate Traces (Parenting Issue)

• Get trace ID from host logs at execution timestamp
• Search worker logs for same trace ID
• If not found → worker created separate trace
• Look for worker spans with p_id: null instead of parent IDs matching host spans
• Enable debug logging: az functionapp config appsettings set --name <app> --resource-group <resource-group> --settings DD_TRACE_DEBUG=1
• Re-test and analyze debug messages about AsyncLocal context flow

Query Datadog API

When traces or logs have reached Datadog (e.g., verifying spans look correct, correlating logs with a trace ID), use these scripts. Both require DD_API_KEY and DD_APPLICATION_KEY environment variables.

Retrieve all spans for a trace ID:

# Table view (default)
./tracer/tools/Get-DatadogTrace.ps1 -TraceId "<trace-id>"

# Hierarchy view — shows span parent-child tree with process tags
./tracer/tools/Get-DatadogTrace.ps1 -TraceId "<trace-id>" -OutputFormat hierarchy

# Search further back in time (default: 2h)
./tracer/tools/Get-DatadogTrace.ps1 -TraceId "<trace-id>" -TimeRange "1d"

Query logs from Datadog:

./tracer/tools/Get-DatadogLogs.ps1 -Query "service:<app-name>"
./tracer/tools/Get-DatadogLogs.ps1 -Query "service:<app-name> error" -TimeRange "2h" -Limit 100

See scripts-reference.md for full parameter reference.

Additional Resources

• Log analysis: log-analysis-guide.md - Manual log investigation patterns (when the automated -All analysis isn't sufficient)
• Scripts: scripts-reference.md - Reusable PowerShell scripts and one-liners
• Environment variables: environment-variables.md - Complete reference for Azure Functions configuration

References

• Detailed docs: docs/development/AzureFunctions.md
• Architecture deep dive: docs/development/for-ai/AzureFunctions-Architecture.md