Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions cli/azd/extensions/azure.ai.agents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,20 @@ effect: activity-protocol agents open the Microsoft 365 Agents Playground rather
than the Agent Inspector, and `--port 8087` on its own collides with the
inspector's own default UI port.

### Local client route telemetry

When installed from the official registry, the extension reports the
`local_client.route.selected` usage event after `azd ai agent run` resolves the
service and protocol profile. Its `ext.route` attribute is exactly one of:

- `inspector` for a non-activity agent;
- `playground` for an activity-protocol agent; or
- `suppressed` when `--no-client` or the deprecated `--no-inspector` is set.

The event is emitted before checking client availability, starting the local
agent, or launching a client. It records route selection, not successful client
launch.

## Migrating Legacy Agent Configuration

New Foundry agent projects keep the agent definition directly on the
Expand Down
2 changes: 1 addition & 1 deletion cli/azd/extensions/azure.ai.agents/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerregistry/armcontainerregistry v1.3.0-beta.3
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armsubscriptions v1.3.0
github.com/azure/azure-dev/cli/azd v1.28.0
github.com/azure/azure-dev/cli/azd v1.31.0
github.com/braydonk/yaml v0.9.0
github.com/drone/envsubst v1.0.3 // indirect
github.com/fatih/color v1.18.0
Expand Down
4 changes: 2 additions & 2 deletions cli/azd/extensions/azure.ai.agents/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ github.com/aymanbagabas/go-udiff v0.2.0 h1:TK0fH4MteXUDspT88n8CKzvK0X9O2xu9yQjWp
github.com/aymanbagabas/go-udiff v0.2.0/go.mod h1:RE4Ex0qsGkTAJoQdQQCA0uG+nAzJO/pI/QwceO5fgrA=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/azure/azure-dev/cli/azd v1.28.0 h1:mqqyV85m7A1XfWJFjV/Ut0QoIEImFeF++1Ruq/cRp0s=
github.com/azure/azure-dev/cli/azd v1.28.0/go.mod h1:Ge7QaU9PoJM7i6J0xArDoQCf2tUn6O7OIKkoItxFTA8=
github.com/azure/azure-dev/cli/azd v1.31.0 h1:p0U4F6w2bPrdzmzavksqfJCnlXoQu9GTQogy+6KXMmM=
github.com/azure/azure-dev/cli/azd v1.31.0/go.mod h1:HFBGeWRWhNsOoYaUcyToqaowibqcbSCfkfJfnIfI4nU=
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
github.com/benbjohnson/clock v1.3.5 h1:VvXlSJBzZpA/zum6Sj74hxwYI2DIxRWuNIoXAzHZz5o=
Expand Down
31 changes: 30 additions & 1 deletion cli/azd/extensions/azure.ai.agents/internal/cmd/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@ import (
const (
agentInspectorExtensionID = "azure.ai.inspector"
agentInspectorReadyPollPeriod = 250 * time.Millisecond
localClientRouteSelectedEvent = "local_client.route.selected"
localClientRouteAttribute = "route"
localClientRouteInspector = "inspector"
localClientRoutePlayground = "playground"
localClientRouteSuppressed = "suppressed"
// defaultInspectorUIPort mirrors the default UI port of the
// azure.ai.inspector extension. The inspector extension remains the source
// of truth for the actual default: when --inspector-port is unset we do not
Expand Down Expand Up @@ -152,6 +157,8 @@ func runRun(ctx context.Context, flags *runFlags, noPrompt bool) error {
// validation can fail without starting a process, and such a failure must
// not clear a session belonging to an already-running agent.
activityProfile := resolveActivityRunProfile(runCtx.Definition)
suppressClient := flags.noInspector || flags.noClient
reportLocalClientRouteSelected(ctx, azdClient.Telemetry(), activityProfile, suppressClient)
if err := validateInspectorPortForProfile(flags, activityProfile.IsActivity); err != nil {
return err
}
Expand All @@ -171,7 +178,6 @@ func runRun(ctx context.Context, flags *runFlags, noPrompt bool) error {
// Resolve local-client availability before the agent starts so advisory
// port warnings can account for whether an inspector will actually launch.
// Reuse the result after proc.Start rather than issuing a second RPC.
suppressClient := flags.noInspector || flags.noClient
inspectorInstalled := false
var inspectorInstallErr error
if !activityProfile.IsActivity && !suppressClient {
Expand Down Expand Up @@ -379,6 +385,29 @@ func runRun(ctx context.Context, flags *runFlags, noPrompt bool) error {
return nil
}

func reportLocalClientRouteSelected(
ctx context.Context,
telemetry azdext.TelemetryServiceClient,
activityProfile activityRunProfile,
suppressClient bool,
) {
route := localClientRouteInspector
if suppressClient {
route = localClientRouteSuppressed
} else if activityProfile.IsActivity {
route = localClientRoutePlayground
}

if _, err := telemetry.ReportUsage(ctx, &azdext.ReportUsageRequest{
EventName: localClientRouteSelectedEvent,
Attributes: map[string]string{
localClientRouteAttribute: route,
},
Comment thread
dooriya marked this conversation as resolved.
}); err != nil {
log.Printf("run: failed to report local client route selection: %v", err)
}
}

func handleInspectorAutoLaunch(
ctx context.Context,
workflow azdext.WorkflowServiceClient,
Expand Down
85 changes: 85 additions & 0 deletions cli/azd/extensions/azure.ai.agents/internal/cmd/run_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import (
"errors"
"fmt"
"io"
"maps"
"net"
"net/http"
"net/http/httptest"
Expand Down Expand Up @@ -238,6 +239,73 @@ func TestWaitForLocalPort(t *testing.T) {
})
}

func TestReportLocalClientRouteSelected(t *testing.T) {
t.Parallel()

tests := []struct {
name string
activityProfile activityRunProfile
suppressClient bool
reportErr error
wantRoute string
}{
{
name: "selects Inspector for non-activity agent",
wantRoute: localClientRouteInspector,
},
{
name: "selects Playground for activity agent",
activityProfile: activityRunProfile{IsActivity: true},
wantRoute: localClientRoutePlayground,
},
{
name: "selects suppressed for non-activity agent",
suppressClient: true,
wantRoute: localClientRouteSuppressed,
},
{
name: "suppression overrides activity route",
activityProfile: activityRunProfile{IsActivity: true},
suppressClient: true,
wantRoute: localClientRouteSuppressed,
},
{
name: "reporting failure is best effort",
reportErr: errors.New("telemetry unavailable"),
wantRoute: localClientRouteInspector,
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()

telemetry := &recordingTelemetryClient{err: tt.reportErr}
reportLocalClientRouteSelected(
t.Context(),
telemetry,
tt.activityProfile,
tt.suppressClient,
)

if telemetry.request == nil {
t.Fatal("expected telemetry request")
}
if telemetry.request.EventName != localClientRouteSelectedEvent {
t.Fatalf(
"event name = %q, want %q",
telemetry.request.EventName,
localClientRouteSelectedEvent,
)
}
wantAttributes := map[string]string{localClientRouteAttribute: tt.wantRoute}
if !maps.Equal(telemetry.request.Attributes, wantAttributes) {
t.Fatalf("attributes = %v, want %v", telemetry.request.Attributes, wantAttributes)
}
})
}
}

func TestLaunchInspectorUsesWorkflowCommand(t *testing.T) {
t.Parallel()

Expand Down Expand Up @@ -743,6 +811,11 @@ type recordingWorkflowClient struct {
called chan struct{}
}

type recordingTelemetryClient struct {
request *azdext.ReportUsageRequest
err error
}

type lockedBuffer struct {
mu sync.Mutex
bytes.Buffer
Expand Down Expand Up @@ -772,6 +845,18 @@ func (c *recordingWorkflowClient) Run(
return &azdext.EmptyResponse{}, c.err
}

func (c *recordingTelemetryClient) ReportUsage(
_ context.Context,
request *azdext.ReportUsageRequest,
_ ...grpc.CallOption,
) (*azdext.ReportUsageResponse, error) {
c.request = request
if c.err != nil {
return nil, c.err
}
return &azdext.ReportUsageResponse{Accepted: true}, nil
}

// createVenv sets up a minimal .venv directory structure for testing.
// Returns the path to the .venv directory.
func createVenv(t *testing.T, projectDir string) string {
Expand Down
2 changes: 1 addition & 1 deletion cli/azd/extensions/azure.ai.projects/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.0-beta.3
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/cognitiveservices/armcognitiveservices/v2 v2.0.0
github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources v1.2.0
github.com/azure/azure-dev/cli/azd v1.28.0
github.com/azure/azure-dev/cli/azd v1.31.0
github.com/drone/envsubst v1.0.3
github.com/fatih/color v1.18.0
github.com/spf13/cobra v1.10.1
Expand Down
4 changes: 2 additions & 2 deletions cli/azd/extensions/azure.ai.projects/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ github.com/aymanbagabas/go-udiff v0.2.0 h1:TK0fH4MteXUDspT88n8CKzvK0X9O2xu9yQjWp
github.com/aymanbagabas/go-udiff v0.2.0/go.mod h1:RE4Ex0qsGkTAJoQdQQCA0uG+nAzJO/pI/QwceO5fgrA=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/azure/azure-dev/cli/azd v1.28.0 h1:mqqyV85m7A1XfWJFjV/Ut0QoIEImFeF++1Ruq/cRp0s=
github.com/azure/azure-dev/cli/azd v1.28.0/go.mod h1:Ge7QaU9PoJM7i6J0xArDoQCf2tUn6O7OIKkoItxFTA8=
github.com/azure/azure-dev/cli/azd v1.31.0 h1:p0U4F6w2bPrdzmzavksqfJCnlXoQu9GTQogy+6KXMmM=
github.com/azure/azure-dev/cli/azd v1.31.0/go.mod h1:HFBGeWRWhNsOoYaUcyToqaowibqcbSCfkfJfnIfI4nU=
github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
github.com/benbjohnson/clock v1.3.5 h1:VvXlSJBzZpA/zum6Sj74hxwYI2DIxRWuNIoXAzHZz5o=
Expand Down
7 changes: 7 additions & 0 deletions docs/reference/telemetry-data.md
Original file line number Diff line number Diff line change
Expand Up @@ -460,6 +460,7 @@ Emitted at provision start by the `microsoft.foundry` provisioning provider (the
| `extension.version` | string | Extension version |
| `extension.event` | string | Extension-chosen event name on an `ext.usage` span |
| `ext.<key>` | string | One extension-supplied attribute on an `ext.usage` span. The key after the `ext.` prefix and the value are chosen by the extension |
| `ext.route` | string | Local-client route selected by `azure.ai.agents`: `inspector`, `playground`, or `suppressed` (`local_client.route.selected`) |
| `extension.installed` | string[] | List of installed extensions (`id@version`) |
| `extension.installed.source.category` | string[] | Installed extension source categories (`id@category`) |
| `extension.version.from` | string | Version before an update or promotion (`ext.update`, `ext.promote`) |
Expand Down Expand Up @@ -489,6 +490,12 @@ source succeeds but records nothing, as does any report past the limit of 100
spans per `azd` invocation. This is a configuration-based admission check, not
a cryptographic provenance guarantee.

Reviewed first-party extension usage events currently include:

| Extension | `extension.event` | Trigger | Dynamic attributes |
|-----------|-------------------|---------|--------------------|
| `azure.ai.agents` | `local_client.route.selected` | `azd ai agent run` resolves the service and protocol profile; emitted before client availability, agent startup, and client launch | `ext.route`: `inspector`, `playground`, or `suppressed`; suppression takes precedence |

Source-category fields are classified from the configured source type and location, not the user-defined source name.
Raw source names, URLs, paths, and hosts are not emitted in those fields.
</details>
Expand Down
1 change: 1 addition & 0 deletions docs/specs/metrics-audit/feature-telemetry-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,4 +168,5 @@ reserved field contracts.
| **Up-graph performance** | `up` (graph execution) | (none — enriches the `up` command span) | `perf.provision_duration_ms`, `perf.deploy_duration_ms`, `perf.total_duration_ms` | Emitted from `internal/cmd/up_graph.go` after the graph completes; provision/deploy durations set only when those phases run |
| **VS RPC** | `vs-server` long-running session | `vsrpc.*` (event prefix) | Per-RPC attributes documented in `telemetry-schema.md` | Long-running RPC server for VS integration |
| **Extension telemetry service** | Extension calls `ReportUsage` over the extension gRPC API | `ext.usage` | `extension.id`, `extension.version`, `extension.source`, `extension.event`, plus one `ext.<key>` attribute per entry in the caller's attribute map | Telemetry requires no separate capability or declaration. Only extensions whose configured source matches the verified official `azd` registry name, type, and normalized URL are recorded — a call from any other source succeeds but is dropped, as is any call past 100 recorded events per `azd` invocation. Identity fields are derived from host-signed claims and the installed record, never from the request. Every caller key is prefixed with `ext.` so it cannot overwrite a host field, and the host bounds count and length only — it does not enumerate or pattern-check values |
| **Azure AI Agents local-client routing** | `azd ai agent run` resolves the service and protocol profile | `ext.usage` with `extension.event=local_client.route.selected` | `ext.route` (`inspector`, `playground`, or `suppressed`) | Records one mutually exclusive route before client availability, agent startup, and client launch; suppression takes precedence and the event does not indicate launch success |
| **App detection** | `init`, `up` (fresh projects without `azure.yaml`, via `appdetect.Detect`) | `aspire.apphost.unsupported` | `aspire.apphost.language` (fixed enum — `typescript` / `python` / `go` / `java` / `rust`; not hashed) | Emitted from `internal/appdetect/dotnet_apphost.go` when an Aspire polyglot (non-C#) AppHost is detected; azd surfaces an actionable error referencing [#7138](https://github.com/Azure/azure-dev/issues/7138) instead of falling through to a generic source build |
6 changes: 6 additions & 0 deletions docs/specs/metrics-audit/telemetry-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,12 @@ guarantees about the whole class:
| Trust | `extension.id` and `extension.version` are derived from host-signed claims; `extension.source` and eligibility are checked against the installed record and verified source config, never from the request |
| Review | Extension telemetry is reviewed when the extension is admitted to the official registry, under the same documentation, classification, and privacy rules as core fields. The eligibility rule above is what ties recording to that review |

Reviewed first-party event contracts:

| Extension | `extension.event` | Trigger | Extension attributes |
|-----------|-------------------|---------|----------------------|
| `azure.ai.agents` | `local_client.route.selected` | `azd ai agent run` resolves the service and protocol profile; this precedes client availability, agent startup, and client launch | `ext.route`: fixed enum `inspector`, `playground`, or `suppressed`; suppression takes precedence |

Because `ext.usage` spans share the command's trace, they join the originating
command in Kusto on `operation_Id`. See
[ADR-001](../../architecture/adr-001-extension-telemetry-events.md) for
Expand Down
Loading