Migrate to Copilot SDK for azd agent implementation

[wbreza]

Replace langchaingo with GitHub Copilot SDK for agent mode

Resolves #6871 #6872 #6873 #6874 #6875 | Epic #6870

Replaces langchaingo agent orchestration with GitHub Copilot SDK (copilot-sdk/go v0.1.32).

Agent API

agent, _ := factory.Create(ctx, agent.WithMode(agent.AgentModePlan))
initResult, _ := agent.Initialize(ctx)
result, _ := agent.SendMessage(ctx, prompt)
agent.Stop()

Screenshots

Startup

Mid Flow

Completed

Features

• Copilot SDK client lifecycle, session create/resume, permission hooks
• Initialize() with model/reasoning prompts, dynamic model list
• SelectSession() UX picker, resume via WithSessionID()
• AgentDisplay: 15+ event types, reasoning window, intent spinner, nested subagents
• Azure plugin auto-install, skills + MCP servers from plugin
• Usage metrics: tokens, billing rate, premium requests, duration
• Init flow: single prompt with azure-prepare/azure-validate skills

Config Namespace Migration

• All config keys moved from ai.agent.* to copilot.* namespace
• Composable constants in internal/agent/copilot/config_keys.go with ConfigRoot prefix
• mcp.consent to copilot.consent, mcp.errorHandling.* to copilot.errorHandling.*
• Clean break - no backward compatibility for old keys

Package Restructure

• Deleted entire pkg/llm/ package (azure_openai, ollama, github_copilot, model_factory, manager, langchaingo deps)
• Created internal/agent/copilot/ subpackage (copilot_client, session_config, feature flag, config keys)
• FeatureLlm to FeatureCopilot (alpha key string stays "llm" for existing user config compat)
• Consent commands moved: azd mcp consent to azd copilot consent
• WithSystemMessage AgentOption for system prompt overrides

Error Middleware Streamlining

• Replaced 4-prompt, 5-agent-call orchestration with single consent prompt + single agent interaction in plan mode
• Troubleshooting workflow in embedded Go text templates
• Agent-driven flow: diagnose, explain, ask user via ask_user tool, fix or show steps
• "Always allow" and "always skip" preference persistence via copilot.errorHandling.* config keys
• Retry-after-fix prompt before re-running failed command

UX Rendering Fixes

• Fix WaitForIdle hang: SessionIdle events arriving before AssistantMessage are now deferred via pendingIdle flag and flushed when the message arrives, preventing indefinite hangs
• Fix ticker vs Pause() race: Added renderGuard sync.RWMutex so Pause() write-locks to block ticker renders and wait for any in-flight render to complete, closing the TOCTOU gap that caused spinner to render over consent prompts
• Fix consent grant persistence: PromptAndGrantConsent no longer returns config save errors as tool denials - user approval is honored immediately, persistence errors are logged but don't block execution
• Add consent error logging: Silent error swallowing in checkUnifiedRules and permission handler now logs failures for [consent] diagnostics

Agent Display Improvements

• Contextual verbs: Read, Edit, Create, Search, Find, Ran, Fetched, Queried instead of generic "Ran toolname with"
• Colored diff stats: Edit shows (+3 -1) with green/red ANSI, Create shows (+25) in green
• Tree-style sub-detail: Shell commands show description on main line, actual command on indented sub-line; MCP tools show args as tree
• Error display: Failed tool calls show error message on red sub-line
• Skill/subagent display: Skills show plugin name and version, subagents show description tree and completion summary
• Consistent whitespace: All section transitions use printSeparated for uniform blank-line spacing
• Spinner matches completion: In-progress spinner uses same contextual verb format; description/intent preferred over raw command
• AssistantMessage rendered in real-time by AgentDisplay (was silently cached, never printed)
• Guard against empty/whitespace-only assistant messages and reasoning causing extra blank lines

Copilot CLI Distribution

• On-demand download following the Bicep tool pattern (internal/agent/copilot/cli.go)
• Downloads version-pinned CLI binary from npm registry on first agent use (CLI 1.0.2 / SDK v0.1.32)
• Caches at ~/.azd/bin/copilot-cli-{version}[.exe], override via AZD_COPILOT_CLI_PATH
• Implements tools.ExternalTool interface (Name, InstallUrl, CheckInstalled)
• Thread-safe via sync.Once, progress spinner during download, 200MB decompression limit
• Deleted fragile npm node_modules path scanning and SDK bundler (~106MB binary size reduction)

CI Pipeline Cleanup

• Removed ghCopilot build tag, GhCopilotClientId/GhCopilotIntegrationId from ci-build.ps1 and 5 pipeline YAMLs
• Removed langchaingo from go.mod

Testing

• 80+ unit tests covering display helpers, consent persistence, usage metrics
• Consent persistence tests verify approve-once to approve-always to auto-approve flow
• Consent reload test verifies rules survive config file round-trip
• E2E test, golangci-lint 0 issues

[Copilot]

Pull request overview

This PR adds the GitHub Copilot SDK (v0.1.25) as a foundational dependency and creates new agent infrastructure that will eventually replace the existing langchaingo-based implementation. This is Phase 1 of a 3-phase migration, where all new code coexists alongside existing code without any deletions or modifications to current functionality.

Changes:

• Adds GitHub Copilot SDK dependency and creates wrapper types (CopilotClientManager, SessionConfigBuilder) that bridge azd config to SDK types
• Implements CopilotAgent and CopilotAgentFactory that satisfy the existing Agent interface for seamless future switchover
• Creates event handling infrastructure (SessionEventLogger, SessionFileLogger) that maps SDK events to azd's existing UX patterns
• Adds 8 new config options (ai.agent.*) for customizing agent behavior (model, tools, MCP servers, system message)

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
go.mod / go.sum	Adds copilot-sdk/go v0.1.25 and transitive dependency jsonschema-go v0.4.2 (both marked indirect)
resources/config_options.yaml	Adds 8 new config keys for Copilot SDK agent customization
pkg/llm/copilot_client.go	Manager wrapping SDK client lifecycle with azd-specific error messages
pkg/llm/copilot_client_test.go	Tests for client manager instantiation (2 cases)
pkg/llm/session_config.go	Bridge converting azd config → SDK SessionConfig with MCP server merging
pkg/llm/session_config_test.go	Tests for config reading, tool control, MCP merging (8 cases)
internal/agent/copilot_agent.go	Agent implementation using SDK Session.SendAndWait, reuses existing UX patterns
internal/agent/copilot_agent_factory.go	Factory creating agents with SDK client, session, hooks, event handlers
internal/agent/logging/session_event_handler.go	Event handlers mapping SDK events to thought channel + file logging
internal/agent/logging/session_event_handler_test.go	Tests for event handling, tool input extraction, composite handler (10 cases)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[spboyer]

Combined Code Review — GPT-5.4 + Claude Opus 4.6

Reviewed independently by two models, findings deduplicated and combined below.

Summary

Severity	Count	Details
🔴 Critical/Bug	7	3x nil deref in error middleware, 1x error swallowed, 2x security (fail-open consent, blanket permissions), 1x WaitForIdle hang
🟡 Warning	5	Always-on content logging, swallowed errors, cleanup ordering, singleton lifecycle, config fallback
🔵 Suggestion	1	Ticker/canvas race condition

Cross-Model Consensus (flagged by BOTH models independently)

These are the highest-confidence findings:

• Nil deref on agentResult.Content in error middleware (3 locations)
• Original error silently swallowed when user declines fix
• Blanket SDK permission approval bypasses safety boundary
• Consent check fails open on error

Dead Config Keys (flagged by GPT-5.4)

ai.agent.mode in config_options.yaml is never read (factory hard-codes "interactive"), and ai.agent.copilot.logLevel is never read (container constructs NewCopilotClientManager(nil) which defaults to "debug"). Users can set these keys, but they have no runtime effect. Either wire them into construction or remove from config_options.yaml until they work.

[vhvb1989]

Heads up: orphaned ghCopilot build tag infrastructure

This PR removes the ghCopilot build-gated registration files (github_copilot_registration.go and github_copilot_registration_stub.go), but several related pieces are still in place and are now effectively dead code:

Still present (untouched by this PR):

• cli/azd/pkg/llm/github_copilot.go — Still has //go:build ghCopilot, still imports langchaingo, still defines the full GitHubCopilotModelProvider with device code auth flow and the clientID/copilotIntegrationID ldflags. Since the registration call site is deleted, this provider is compiled but never wired into IoC.
• cli/azd/ci-build.ps1 — Still passes -tags ghCopilot and the ldflags (-X ...clientID, -X ...copilotIntegrationID) when GitHubCopilotClientId is provided.
• CI pipeline YAML files — build-cli.yml, cross-build-cli.yml, and build-and-test.yml still pass ghCopilotClientId / ghCopilotIntegrationId parameters through.

The build won't break (the old code compiles but is just unused), but it's worth tracking cleanup of these artifacts — either in this PR or as a follow-up:

1. Remove pkg/llm/github_copilot.go (and its langchaingo dependency)
2. Remove the ghCopilot tag logic from ci-build.ps1
3. Clean up the ghCopilotClientId/ghCopilotIntegrationId params from CI YAML files

Just flagging since the list is fresh — totally fine if this is planned for a subsequent iteration.

[vhvb1989]

No blockers from my side. The migration to the Copilot SDK looks solid overall. Left a couple of comments for consideration:

1. Silent plugin install — ensurePlugins() installs the Azure plugin system-wide without user awareness. Worth surfacing a message or prompting.
2. Old config migration — Users with ai.agent.model.type set to "github-copilot" will get a hard failure. A small auto-migration or friendly error would help early adopters.
3. Orphaned ghCopilot build tag — github_copilot.go, ci-build.ps1, and CI YAML params still reference the old build tag. Dead code that can be cleaned up in a follow-up.

None of these are blocking — just flagging for author consideration.