Skip to content

aitools: raise the plugin probe timeout from 5s to 20s - #7472

Open
anton-107 wants to merge 1 commit into
aitools/version-check-install-hintfrom
aitools/plugin-probe-timeout
Open

aitools: raise the plugin probe timeout from 5s to 20s#7472
anton-107 wants to merge 1 commit into
aitools/version-check-install-hintfrom
aitools/plugin-probe-timeout

Conversation

@anton-107

@anton-107 anton-107 commented Nov 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Stacked on #7471.

Stacked on: aitools: mention databricks aitools install in version --check

Summary

databricks aitools install runs <agent> plugin --help to check that the agent's CLI can install plugins. Agent CLIs are large Node/Rust binaries, and a cold start (first run, antivirus scan, slow container disk) can take several seconds. When the probe times out, the agent is reported as unavailable.

 // libs/aitools/installer/plugin.go
-pluginProbeTimeout = 5 * time.Second
+pluginProbeTimeout = 20 * time.Second

The same constant also bounds <agent> plugin marketplace list. Changelog fragment: .nextchanges/cli/aitools-plugin-probe-timeout.md.

Evidence

A temporary test (not committed) points probePluginCLI at a fake claude script that sleeps 7s and then prints usage. Linux, at this PR's head:

  • Before (constant reverted to 5s): 
    --- FAIL: TestProbePluginCLISlowColdStart (7.13s)
    claude-code: cli-not-on-path: .../claude plugin --help: signal: killed
    After (20s): 
    --- PASS: TestProbePluginCLISlowColdStart (7.06s)

go test ./libs/aitools/... passes. No committed test pins the timeout value, because such a test would have to wait out the timeout.

Merge Danger

Door: two-way

Reverting is a one-line change.

Blast Radius: small

  • An agent CLI that hangs now blocks aitools install for up to 20s at the probe instead of 5s. The same applies to plugin marketplace list, where a timeout makes install assume the marketplace is already registered.
  • On Windows, an npm-installed agent CLI runs as a .cmd shim, and runAgentCmd kills only the direct child on timeout. A hung grandchild that keeps stdout/stderr open can make the probe wait past 20s. This predates the PR; bounding it with process.WithProcessGroup() is left to a follow-up.
  • No exit codes or output change. A probe that still times out reports the same cli-not-on-path reason as before.
  • agents.ProbePlugin in libs/aitools/agents/detect.go has its own copy of the constant, also raised to 20s. It has no non-test callers.
  • Not tested on Windows or macOS.

This pull request and its description were written by Isaac.

@anton-107 anton-107 added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
@anton-107
anton-107 requested review from a team as code owners November 11, 2026 14:25
@anton-107 anton-107 added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
@anton-107 anton-107 mentioned this pull request Nov 11, 2026
Open
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 2e276b6 to 848f252 Compare November 11, 2026 14:32
@anton-107
anton-107 requested a review from a team as a code owner November 11, 2026 14:32
@anton-107
anton-107 requested a review from rauchy November 11, 2026 14:32
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from cc3a109 to 3e0bf18 Compare November 11, 2026 14:33
@github-actions

github-actions Bot commented Nov 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: d82eb6be3 | Workflow run

Regression test report

Selected 12 changed tests (limit=20, 0 not selected)

12 tests: regression=8 unreleased=0 coverage=4 failing=0
test branch main (8b4f052) latest (v1.26.0)
TestAccept/cmd/aitools/install-host-metadata ✅ ❌ ❌
TestAccept/cmd/aitools/list/explicit-ref ✅ ✅ ✅
TestAccept/cmd/aitools/list/latest ✅ ✅ ✅
TestAccept/cmd/aitools/list/latest-lookup-fails ✅ ✅ ✅
TestAccept/cmd/aitools/list/pinned ✅ ✅ ✅
TestAccept/cmd/aitools/telemetry/apps-init-tip ✅ ❌ ❌
TestAccept/cmd/aitools/telemetry/hint ✅ ❌ ❌
TestAccept/cmd/aitools/telemetry/install ✅ ❌ ❌
TestAccept/cmd/aitools/telemetry/install-failure ✅ ❌ ❌
TestAccept/cmd/aitools/telemetry/update-uninstall ✅ ❌ ❌
TestAccept/cmd/aitools/telemetry/workspace-unreachable ✅ ❌ ❌
TestAccept/cmd/version ✅ ❌ ❌
TestAccept/cmd/aitools/install-host-metadata: branch ✅ | main (8b4f052) ❌ | latest (v1.26.0) ❌

main (8b4f052):

acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:35107" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== workspace unreachable: the background auth resolution prints nothing\n\n>>> [CLI] aitools install --skills-only --scope=global --agents=claude-code\nInstalling Databricks skills for Claude Code...\nUsing skills version test-ref\nFetching skills manifest...\nInstalled 1 skill.\n\n=== --debug still shows the hostmetadata line, at debug level\nDebug: [hostmetadata] failed to fetch host metadata for https://unreachable.test\n"
        	            	actual  : "\n=== workspace unreachable: the background auth resolution prints nothing\n\n>>> [CLI] aitools install --skills-only --scope=global --agents=claude-code\nWarn: [hostmetadata] failed to fetch host metadata for https://unreachable.test, will skip for 1m0s\nInstalling Databricks skills for Claude Code...\nUsing skills version test-ref\nFetching skills manifest...\nInstalled 1 skill.\n\n=== --debug still shows the hostmetadata line, at debug level\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,2 +4,3 @@
        	            	 >>> [CLI] aitools install --skills-only --scope=global --agents=claude-code
        	            	+Warn: [hostmetadata] failed to fetch host metadata for https://unreachable.test, will skip for 1m0s
        	            	 Installing Databricks skills for Claude Code...
        	            	@@ -10,3 +11,4 @@
        	            	 === --debug still shows the hostmetadata line, at debug level
        	            	-Debug: [hostmetadata] failed to fetch host metadata for https://unreachable.test
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/cmd/aitools/install-host-metadata
        	Messages:   	cmd/aitools/install-host-metadata/output.txt vs /tmp/TestAcceptcmdaitoolsinstall-host-metadata230522714/002/output.txt

latest (v1.26.0):

acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:38429" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== workspace unreachable: the background auth resolution prints nothing\n\n>>> [CLI] aitools install --skills-only --scope=global --agents=claude-code\nInstalling Databricks skills for Claude Code...\nUsing skills version test-ref\nFetching skills manifest...\nInstalled 1 skill.\n\n=== --debug still shows the hostmetadata line, at debug level\nDebug: [hostmetadata] failed to fetch host metadata for https://unreachable.test\n"
        	            	actual  : "\n=== workspace unreachable: the background auth resolution prints nothing\n\n>>> [CLI] aitools install --skills-only --scope=global --agents=claude-code\nWarn: [hostmetadata] failed to fetch host metadata for https://unreachable.test, will skip for 1m0s\nInstalling Databricks skills for Claude Code...\nUsing skills version test-ref\nFetching skills manifest...\nInstalled 1 skill.\n\n=== --debug still shows the hostmetadata line, at debug level\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,2 +4,3 @@
        	            	 >>> [CLI] aitools install --skills-only --scope=global --agents=claude-code
        	            	+Warn: [hostmetadata] failed to fetch host metadata for https://unreachable.test, will skip for 1m0s
        	            	 Installing Databricks skills for Claude Code...
        	            	@@ -10,3 +11,4 @@
        	            	 === --debug still shows the hostmetadata line, at debug level
        	            	-Debug: [hostmetadata] failed to fetch host metadata for https://unreachable.test
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/cmd/aitools/install-host-metadata
        	Messages:   	cmd/aitools/install-host-metadata/output.txt vs /tmp/TestAcceptcmdaitoolsinstall-host-metadata1292266145/002/output.txt
TestAccept/cmd/aitools/telemetry/apps-init-tip: branch ✅ | main (8b4f052) ❌ | latest (v1.26.0) ❌

main (8b4f052):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:39467" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:39467" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== apps init with flags prints the tip: one hint-shown event per hint printed\n\n>>> [CLI] apps init --template ./template --skip-install --name app-tip\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\n✔ Creating project\n\n✔ Project created successfully!\n\n  Location: [TEST_TMP_DIR]/app-tip\n  Files: 3\n\nTip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.\n\n>>> print_telemetry\n{\"command\":\"apps_init\",\"exit_code\":0,\"aitools_hint_event\":{\"hint_kind\":\"INSTALL_TEXT\"}}\n{\"command\":\"apps_init\",\"exit_code\":0,\"aitools_hint_event\":{\"hint_kind\":\"APPS_INIT_TIP\"}}\n\n=== plugin installed: no hint, no tip, no event\n\n>>> [CLI] apps init --template ./template --skip-install --name app-plugin\n✔ Creating project\n\n✔ Project created successfully!\n\n  Location: [TEST_TMP_DIR]/app-plugin\n  Files: 3\n\n\n>>> print_telemetry\n"
        	            	actual  : "\n=== apps init with flags prints the tip: one hint-shown event per hint printed\n\n>>> [CLI] apps init --template ./template --skip-install --name app-tip\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\n✔ Creating project\n\n✔ Project created successfully!\n\n  Location: [TEST_TMP_DIR]/app-tip\n  Files: 3\n\nTip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.\n\n>>> print_telemetry\n\n=== plugin installed: no hint, no tip, no event\n\n>>> [CLI] apps init --template ./template --skip-install --name app-plugin\n✔ Creating project\n\n✔ Project created successfully!\n\n  Location: [TEST_TMP_DIR]/app-plugin\n  Files: 3\n\nTip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.\n\n>>> print_telemetry\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -15,4 +15,2 @@
        	            	 >>> print_telemetry
        	            	-{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
        	            	-{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"APPS_INIT_TIP"}}
        	            	 
        	            	@@ -28,2 +26,3 @@
        	            	 
        	            	+Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.
        	            	 
        	Test:       	TestAccept/cmd/aitools/telemetry/apps-init-tip
        	Messages:   	cmd/aitools/telemetry/apps-init-tip/output.txt vs /tmp/TestAcceptcmdaitoolstelemetryapps-init-tip2113746331/002/output.txt
    acceptance_test.go:1123: LOG.apps-init: 
    acceptance_test.go:1123: LOG.apps-init-plugin:

latest (v1.26.0):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:42193" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:42193" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== apps init with flags prints the tip: one hint-shown event per hint printed\n\n>>> [CLI] apps init --template ./template --skip-install --name app-tip\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\n✔ Creating project\n\n✔ Project created successfully!\n\n  Location: [TEST_TMP_DIR]/app-tip\n  Files: 3\n\nTip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.\n\n>>> print_telemetry\n{\"command\":\"apps_init\",\"exit_code\":0,\"aitools_hint_event\":{\"hint_kind\":\"INSTALL_TEXT\"}}\n{\"command\":\"apps_init\",\"exit_code\":0,\"aitools_hint_event\":{\"hint_kind\":\"APPS_INIT_TIP\"}}\n\n=== plugin installed: no hint, no tip, no event\n\n>>> [CLI] apps init --template ./template --skip-install --name app-plugin\n✔ Creating project\n\n✔ Project created successfully!\n\n  Location: [TEST_TMP_DIR]/app-plugin\n  Files: 3\n\n\n>>> print_telemetry\n"
        	            	actual  : "\n=== apps init with flags prints the tip: one hint-shown event per hint printed\n\n>>> [CLI] apps init --template ./template --skip-install --name app-tip\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\n✔ Creating project\n\n✔ Project created successfully!\n\n  Location: [TEST_TMP_DIR]/app-tip\n  Files: 3\n\nTip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.\n\n>>> print_telemetry\n\n=== plugin installed: no hint, no tip, no event\n\n>>> [CLI] apps init --template ./template --skip-install --name app-plugin\n✔ Creating project\n\n✔ Project created successfully!\n\n  Location: [TEST_TMP_DIR]/app-plugin\n  Files: 3\n\nTip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.\n\n>>> print_telemetry\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -15,4 +15,2 @@
        	            	 >>> print_telemetry
        	            	-{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
        	            	-{"command":"apps_init","exit_code":0,"aitools_hint_event":{"hint_kind":"APPS_INIT_TIP"}}
        	            	 
        	            	@@ -28,2 +26,3 @@
        	            	 
        	            	+Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.
        	            	 
        	Test:       	TestAccept/cmd/aitools/telemetry/apps-init-tip
        	Messages:   	cmd/aitools/telemetry/apps-init-tip/output.txt vs /tmp/TestAcceptcmdaitoolstelemetryapps-init-tip964412651/002/output.txt
    acceptance_test.go:1123: LOG.apps-init: 
    acceptance_test.go:1123: LOG.apps-init-plugin:
TestAccept/cmd/aitools/telemetry/hint: branch ✅ | main (8b4f052) ❌ | latest (v1.26.0) ❌

main (8b4f052):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:39931" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:39931" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== hint printed on an authenticated command: one hint-shown event is uploaded\n\n>>> [CLI] jobs list\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\n\n>>> print_telemetry\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_hint_event\":{\"hint_kind\":\"INSTALL_TEXT\"}}\n\n=== hint printed on a command that resolves no auth: nothing is uploaded\n\n>>> [CLI] version\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\nDatabricks CLI v[CLI_VERSION]\n\n>>> print_telemetry\n\n=== plugin installed: no hint, no event\n\n>>> [CLI] jobs list\n\n>>> print_telemetry\n"
        	            	actual  : "\n=== hint printed on an authenticated command: one hint-shown event is uploaded\n\n>>> [CLI] jobs list\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\n\n>>> print_telemetry\n\n=== hint printed on a command that resolves no auth: nothing is uploaded\n\n>>> [CLI] version\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\nDatabricks CLI v[CLI_VERSION]\n\n>>> print_telemetry\n\n=== plugin installed: no hint, no event\n\n>>> [CLI] jobs list\n\n>>> print_telemetry\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,3 +7,2 @@
        	            	 >>> print_telemetry
        	            	-{"command":"jobs_list","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
        	            	 
        	Test:       	TestAccept/cmd/aitools/telemetry/hint
        	Messages:   	cmd/aitools/telemetry/hint/output.txt vs /tmp/TestAcceptcmdaitoolstelemetryhint4212086372/002/output.txt

latest (v1.26.0):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:35469" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:35469" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== hint printed on an authenticated command: one hint-shown event is uploaded\n\n>>> [CLI] jobs list\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\n\n>>> print_telemetry\n{\"command\":\"jobs_list\",\"exit_code\":0,\"aitools_hint_event\":{\"hint_kind\":\"INSTALL_TEXT\"}}\n\n=== hint printed on a command that resolves no auth: nothing is uploaded\n\n>>> [CLI] version\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\nDatabricks CLI v[CLI_VERSION]\n\n>>> print_telemetry\n\n=== plugin installed: no hint, no event\n\n>>> [CLI] jobs list\n\n>>> print_telemetry\n"
        	            	actual  : "\n=== hint printed on an authenticated command: one hint-shown event is uploaded\n\n>>> [CLI] jobs list\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\n\n>>> print_telemetry\n\n=== hint printed on a command that resolves no auth: nothing is uploaded\n\n>>> [CLI] version\nDatabricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install\nDatabricks CLI v[CLI_VERSION]\n\n>>> print_telemetry\n\n=== plugin installed: no hint, no event\n\n>>> [CLI] jobs list\n\n>>> print_telemetry\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,3 +7,2 @@
        	            	 >>> print_telemetry
        	            	-{"command":"jobs_list","exit_code":0,"aitools_hint_event":{"hint_kind":"INSTALL_TEXT"}}
        	            	 
        	Test:       	TestAccept/cmd/aitools/telemetry/hint
        	Messages:   	cmd/aitools/telemetry/hint/output.txt vs /tmp/TestAcceptcmdaitoolstelemetryhint3129971922/002/output.txt
TestAccept/cmd/aitools/telemetry/install: branch ✅ | main (8b4f052) ❌ | latest (v1.26.0) ❌

main (8b4f052):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:41707" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:41707" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: Diff:
        --- cmd/aitools/telemetry/install/output.txt
        +++ /tmp/TestAcceptcmdaitoolstelemetryinstall2544466978/002/output.txt
        @@ -8,14 +8,7 @@
         Installed 1 skill.
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SKILLS_INSTALLED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":0,"aitools_install_event":{"agents":["CLAUDE_CODE"],"scope":"GLOBAL","error_category":"TYPE_UNSPECIFIED"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CLAUDE_CODE"],"scope":"GLOBAL"}}
         
         === databricks-cli auth with a valid cached token: the event is uploaded
         
        @@ -26,14 +18,7 @@
         Installed 1 skill.
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SKILLS_INSTALLED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":0,"aitools_install_event":{"agents":["CLAUDE_CODE"],"scope":"GLOBAL","error_category":"TYPE_UNSPECIFIED"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CLAUDE_CODE"],"scope":"GLOBAL"}}
         
         === databricks-cli auth with an expired cached token: no refresh, nothing is uploaded
         
        @@ -44,6 +28,22 @@
         Installed 1 skill.
         
         >>> print_requests.py //oidc/v1/token //telemetry-ext
        +{
        +  "method": "POST",
        +  "path": "/oidc/v1/token",
        +  "raw_body": "client_id=databricks-cli&grant_type=refresh_token&refresh_token=test-refresh-token"
        +}
        +{
        +  "method": "POST",
        +  "path": "/telemetry-ext",
        +  "body": {
        +    "uploadTime": [UNIX_TIME_MILLIS],
        +    "items": [],
        +    "protoLogs": [
        +      "{\"frontend_log_event_id\":\"[UUID][0]\",\"entry\":{\"databricks_cli_log\":{\"execution_context\":{\"cmd_exec_id\":\"[UUID][1]\",\"version\":\"[CLI_VERSION]\",\"command\":\"aitools_install\",\"operating_system\":\"linux\",\"execution_time_ms\":11,\"exit_code\":0},\"aitools_install_event\":{\"agents\":[\"CLAUDE_CODE\"],\"scope\":\"GLOBAL\",\"error_category\":\"TYPE_UNSPECIFIED\"}}}}"
        +    ]
        +  }
        +}
         
         >>> gron.py [TEST_TMP_DIR]/home/.databricks/token-cache.json
        -json.tokens.test-profile.access_token = "expired-access-token";
        +json.tokens.test-profile.access_token = "oauth-token";

latest (v1.26.0):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:41363" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:41363" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: Diff:
        --- cmd/aitools/telemetry/install/output.txt
        +++ /tmp/TestAcceptcmdaitoolstelemetryinstall3945041025/002/output.txt
        @@ -8,14 +8,7 @@
         Installed 1 skill.
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SKILLS_INSTALLED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":0,"aitools_install_event":{"agents":["CLAUDE_CODE"],"scope":"GLOBAL","error_category":"TYPE_UNSPECIFIED"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CLAUDE_CODE"],"scope":"GLOBAL"}}
         
         === databricks-cli auth with a valid cached token: the event is uploaded
         
        @@ -26,14 +18,7 @@
         Installed 1 skill.
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SKILLS_INSTALLED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":0,"aitools_install_event":{"agents":["CLAUDE_CODE"],"scope":"GLOBAL","error_category":"TYPE_UNSPECIFIED"}}
        -{"command":"aitools_install","exit_code":0,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CLAUDE_CODE"],"scope":"GLOBAL"}}
         
         === databricks-cli auth with an expired cached token: no refresh, nothing is uploaded
         
        @@ -44,6 +28,22 @@
         Installed 1 skill.
         
         >>> print_requests.py //oidc/v1/token //telemetry-ext
        +{
        +  "method": "POST",
        +  "path": "/oidc/v1/token",
        +  "raw_body": "client_id=databricks-cli&grant_type=refresh_token&refresh_token=test-refresh-token"
        +}
        +{
        +  "method": "POST",
        +  "path": "/telemetry-ext",
        +  "body": {
        +    "uploadTime": [UNIX_TIME_MILLIS],
        +    "items": [],
        +    "protoLogs": [
        +      "{\"frontend_log_event_id\":\"[UUID][0]\",\"entry\":{\"databricks_cli_log\":{\"execution_context\":{\"cmd_exec_id\":\"[UUID][1]\",\"version\":\"[CLI_VERSION]\",\"command\":\"aitools_install\",\"operating_system\":\"linux\",\"execution_time_ms\":9,\"exit_code\":0},\"aitools_install_event\":{\"agents\":[\"CLAUDE_CODE\"],\"scope\":\"GLOBAL\",\"error_category\":\"TYPE_UNSPECIFIED\"}}}}"
        +    ]
        +  }
        +}
         
         >>> gron.py [TEST_TMP_DIR]/home/.databricks/token-cache.json
        -json.tokens.test-profile.access_token = "expired-access-token";
        +json.tokens.test-profile.access_token = "oauth-token";
TestAccept/cmd/aitools/telemetry/install-failure: branch ✅ | main (8b4f052) ❌ | latest (v1.26.0) ❌

main (8b4f052):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:44185" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:44185" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: Diff:
        --- cmd/aitools/telemetry/install-failure/output.txt
        +++ /tmp/TestAcceptcmdaitoolstelemetryinstall-failure2833339152/002/output.txt
        @@ -8,12 +8,6 @@
         Error: skill "nonexistent" not found
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"SKILL_NOT_FOUND","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"SKILL_NOT_FOUND"}}
         
         === a manifest the server does not have is MANIFEST_NOT_FOUND
        @@ -25,12 +19,6 @@
         Error: skills manifest at databricks-agent-skills@missing-ref: not found
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"MANIFEST_NOT_FOUND","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"MANIFEST_NOT_FOUND"}}
         
         === any other HTTP error status from the manifest fetch is not classified yet (the error is untyped)
        @@ -42,12 +30,6 @@
         Error: failed to fetch manifest: HTTP 500
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"UNCATEGORIZED","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"UNCATEGORIZED"}}
         
         === a named agent skipped for the scope takes its category from the agent's outcome
        @@ -57,10 +39,4 @@
         Error: Cursor: does not support project-scoped skills
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"PROJECT"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"PROJECT"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"PROJECT"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"UNSUPPORTED_SCOPE","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"PROJECT"}}
         {"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"PROJECT","error_category":"TYPE_UNSPECIFIED","agent_results":[{"agent":"CURSOR","error_category":"UNSUPPORTED_SCOPE"}]}}
        
FAIL

latest (v1.26.0):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:45525" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:45525" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: Diff:
        --- cmd/aitools/telemetry/install-failure/output.txt
        +++ /tmp/TestAcceptcmdaitoolstelemetryinstall-failure1790280583/002/output.txt
        @@ -8,12 +8,6 @@
         Error: skill "nonexistent" not found
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"SKILL_NOT_FOUND","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"SKILL_NOT_FOUND"}}
         
         === a manifest the server does not have is MANIFEST_NOT_FOUND
        @@ -25,12 +19,6 @@
         Error: skills manifest at databricks-agent-skills@missing-ref: not found
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"MANIFEST_NOT_FOUND","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"MANIFEST_NOT_FOUND"}}
         
         === any other HTTP error status from the manifest fetch is not classified yet (the error is untyped)
        @@ -42,12 +30,6 @@
         Error: failed to fetch manifest: HTTP 500
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"GLOBAL"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"UNCATEGORIZED","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"GLOBAL"}}
         {"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"UNCATEGORIZED"}}
         
         === a named agent skipped for the scope takes its category from the agent's outcome
        @@ -57,10 +39,4 @@
         Error: Cursor: does not support project-scoped skills
         
         >>> print_telemetry
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STARTED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"FLAGS_VALIDATED","interactive":false,"install_mode":"SKILLS_ONLY"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"SCOPE_RESOLVED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"PROJECT"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"AGENTS_SELECTED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"PROJECT"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"STEP_COMPLETED","step":"PLAN_CONFIRMED","interactive":false,"install_mode":"SKILLS_ONLY","scope":"PROJECT"}}
        -{"command":"aitools_install","exit_code":1,"aitools_command_event":{"command":"INSTALL","stage":"FINISHED","outcome":"FAILURE","error_category":"UNSUPPORTED_SCOPE","interactive":false,"install_mode":"SKILLS_ONLY","agents":["CURSOR"],"scope":"PROJECT"}}
         {"command":"aitools_install","exit_code":1,"aitools_install_event":{"agents":["CURSOR"],"scope":"PROJECT","error_category":"TYPE_UNSPECIFIED","agent_results":[{"agent":"CURSOR","error_category":"UNSUPPORTED_SCOPE"}]}}
TestAccept/cmd/aitools/telemetry/update-uninstall: branch ✅ | main (8b4f052) ❌ | latest (v1.26.0) ❌

main (8b4f052):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:46703" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:46703" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== update: STARTED, FINISHED with SUCCESS\n\n>>> [CLI] aitools update --scope=global\nAlready up to date.\n\n>>> print_telemetry\n{\"command\":\"aitools_update\",\"exit_code\":0,\"aitools_command_event\":{\"command\":\"UPDATE\",\"stage\":\"STARTED\",\"interactive\":false}}\n{\"command\":\"aitools_update\",\"exit_code\":0,\"aitools_command_event\":{\"command\":\"UPDATE\",\"stage\":\"FINISHED\",\"outcome\":\"SUCCESS\",\"interactive\":false,\"scope\":\"GLOBAL\"}}\n\n=== uninstall: STARTED, FINISHED with SUCCESS\n\n>>> [CLI] aitools uninstall --scope=global\nUninstalled 1 skill.\n\n>>> print_telemetry\n{\"command\":\"aitools_uninstall\",\"exit_code\":0,\"aitools_command_event\":{\"command\":\"UNINSTALL\",\"stage\":\"STARTED\",\"interactive\":false}}\n{\"command\":\"aitools_uninstall\",\"exit_code\":0,\"aitools_command_event\":{\"command\":\"UNINSTALL\",\"stage\":\"FINISHED\",\"outcome\":\"SUCCESS\",\"interactive\":false,\"scope\":\"GLOBAL\"}}\n"
        	            	actual  : "\n=== update: STARTED, FINISHED with SUCCESS\n\n>>> [CLI] aitools update --scope=global\nAlready up to date.\n\n>>> print_telemetry\n\n=== uninstall: STARTED, FINISHED with SUCCESS\n\n>>> [CLI] aitools uninstall --scope=global\nUninstalled 1 skill.\n\n>>> print_telemetry\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,4 +7,2 @@
        	            	 >>> print_telemetry
        	            	-{"command":"aitools_update","exit_code":0,"aitools_command_event":{"command":"UPDATE","stage":"STARTED","interactive":false}}
        	            	-{"command":"aitools_update","exit_code":0,"aitools_command_event":{"command":"UPDATE","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"scope":"GLOBAL"}}
        	            	 
        	            	@@ -16,4 +14,2 @@
        	            	 >>> print_telemetry
        	            	-{"command":"aitools_uninstall","exit_code":0,"aitools_command_event":{"command":"UNINSTALL","stage":"STARTED","interactive":false}}
        	            	-{"command":"aitools_uninstall","exit_code":0,"aitools_command_event":{"command":"UNINSTALL","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"scope":"GLOBAL"}}
        	            	 
        	Test:       	TestAccept/cmd/aitools/telemetry/update-uninstall
        	Messages:   	cmd/aitools/telemetry/update-uninstall/output.txt vs /tmp/TestAcceptcmdaitoolstelemetryupdate-uninstall4027156834/002/output.txt
    acceptance_test.go:1123: 
        LOG.install: Installing Databricks skills for Cursor...
        LOG.install: Using skills version test-ref
        LOG.install: Fetching skills manifest...
        LOG.install: Installed 1 skill.
    acceptance_test.go:1123: LOG.install-telemetry: {"command":"aitools_install","exit_code":0,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"TYPE_UNSPECIFIED"}}

latest (v1.26.0):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:33171" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:33171" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== update: STARTED, FINISHED with SUCCESS\n\n>>> [CLI] aitools update --scope=global\nAlready up to date.\n\n>>> print_telemetry\n{\"command\":\"aitools_update\",\"exit_code\":0,\"aitools_command_event\":{\"command\":\"UPDATE\",\"stage\":\"STARTED\",\"interactive\":false}}\n{\"command\":\"aitools_update\",\"exit_code\":0,\"aitools_command_event\":{\"command\":\"UPDATE\",\"stage\":\"FINISHED\",\"outcome\":\"SUCCESS\",\"interactive\":false,\"scope\":\"GLOBAL\"}}\n\n=== uninstall: STARTED, FINISHED with SUCCESS\n\n>>> [CLI] aitools uninstall --scope=global\nUninstalled 1 skill.\n\n>>> print_telemetry\n{\"command\":\"aitools_uninstall\",\"exit_code\":0,\"aitools_command_event\":{\"command\":\"UNINSTALL\",\"stage\":\"STARTED\",\"interactive\":false}}\n{\"command\":\"aitools_uninstall\",\"exit_code\":0,\"aitools_command_event\":{\"command\":\"UNINSTALL\",\"stage\":\"FINISHED\",\"outcome\":\"SUCCESS\",\"interactive\":false,\"scope\":\"GLOBAL\"}}\n"
        	            	actual  : "\n=== update: STARTED, FINISHED with SUCCESS\n\n>>> [CLI] aitools update --scope=global\nAlready up to date.\n\n>>> print_telemetry\n\n=== uninstall: STARTED, FINISHED with SUCCESS\n\n>>> [CLI] aitools uninstall --scope=global\nUninstalled 1 skill.\n\n>>> print_telemetry\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,4 +7,2 @@
        	            	 >>> print_telemetry
        	            	-{"command":"aitools_update","exit_code":0,"aitools_command_event":{"command":"UPDATE","stage":"STARTED","interactive":false}}
        	            	-{"command":"aitools_update","exit_code":0,"aitools_command_event":{"command":"UPDATE","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"scope":"GLOBAL"}}
        	            	 
        	            	@@ -16,4 +14,2 @@
        	            	 >>> print_telemetry
        	            	-{"command":"aitools_uninstall","exit_code":0,"aitools_command_event":{"command":"UNINSTALL","stage":"STARTED","interactive":false}}
        	            	-{"command":"aitools_uninstall","exit_code":0,"aitools_command_event":{"command":"UNINSTALL","stage":"FINISHED","outcome":"SUCCESS","interactive":false,"scope":"GLOBAL"}}
        	            	 
        	Test:       	TestAccept/cmd/aitools/telemetry/update-uninstall
        	Messages:   	cmd/aitools/telemetry/update-uninstall/output.txt vs /tmp/TestAcceptcmdaitoolstelemetryupdate-uninstall1372017359/002/output.txt
    acceptance_test.go:1123: 
        LOG.install: Installing Databricks skills for Cursor...
        LOG.install: Using skills version test-ref
        LOG.install: Fetching skills manifest...
        LOG.install: Installed 1 skill.
    acceptance_test.go:1123: LOG.install-telemetry: {"command":"aitools_install","exit_code":0,"aitools_install_event":{"agents":["CURSOR"],"scope":"GLOBAL","error_category":"TYPE_UNSPECIFIED"}}
TestAccept/cmd/aitools/telemetry/workspace-unreachable: branch ✅ | main (8b4f052) ❌ | latest (v1.26.0) ❌

main (8b4f052):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:35675" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:35675" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== install: the background auth resolution prints nothing\n\n>>> [CLI] aitools install --skills-only --scope=global --agents=cursor\nInstalling Databricks skills for Cursor...\nUsing skills version test-ref\nFetching skills manifest...\nInstalled 1 skill.\n\n=== update: the background auth resolution prints nothing\n\n>>> [CLI] aitools update --scope=global\nAlready up to date.\n\n=== uninstall: the background auth resolution prints nothing\n\n>>> [CLI] aitools uninstall --scope=global\nUninstalled 1 skill.\n"
        	            	actual  : "\n=== install: the background auth resolution prints nothing\n\n>>> [CLI] aitools install --skills-only --scope=global --agents=cursor\nWarn: [hostmetadata] failed to fetch host metadata for https://unreachable.test, will skip for 1m0s\nInstalling Databricks skills for Cursor...\nUsing skills version test-ref\nFetching skills manifest...\nInstalled 1 skill.\n\n=== update: the background auth resolution prints nothing\n\n>>> [CLI] aitools update --scope=global\nAlready up to date.\n\n=== uninstall: the background auth resolution prints nothing\n\n>>> [CLI] aitools uninstall --scope=global\nUninstalled 1 skill.\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,2 +4,3 @@
        	            	 >>> [CLI] aitools install --skills-only --scope=global --agents=cursor
        	            	+Warn: [hostmetadata] failed to fetch host metadata for https://unreachable.test, will skip for 1m0s
        	            	 Installing Databricks skills for Cursor...
        	Test:       	TestAccept/cmd/aitools/telemetry/workspace-unreachable
        	Messages:   	cmd/aitools/telemetry/workspace-unreachable/output.txt vs /tmp/TestAcceptcmdaitoolstelemetryworkspace-unreachable560255472/002/output.txt

latest (v1.26.0):

acceptance_test.go:1213: Substituted DATABRICKS_CLI_GITHUB_API_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:38405" ("[DATABRICKS_HOST]")
    acceptance_test.go:1213: Substituted DATABRICKS_SKILLS_BASE_URL "$DATABRICKS_HOST" -> "http://127.0.0.1:38405" ("[DATABRICKS_HOST]")
    acceptance_test.go:1345: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1345
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1086
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:642
        	Error:      	Not equal: 
        	            	expected: "\n=== install: the background auth resolution prints nothing\n\n>>> [CLI] aitools install --skills-only --scope=global --agents=cursor\nInstalling Databricks skills for Cursor...\nUsing skills version test-ref\nFetching skills manifest...\nInstalled 1 skill.\n\n=== update: the background auth resolution prints nothing\n\n>>> [CLI] aitools update --scope=global\nAlready up to date.\n\n=== uninstall: the background auth resolution prints nothing\n\n>>> [CLI] aitools uninstall --scope=global\nUninstalled 1 skill.\n"
        	            	actual  : "\n=== install: the background auth resolution prints nothing\n\n>>> [CLI] aitools install --skills-only --scope=global --agents=cursor\nWarn: [hostmetadata] failed to fetch host metadata for https://unreachable.test, will skip for 1m0s\nInstalling Databricks skills for Cursor...\nUsing skills version test-ref\nFetching skills manifest...\nInstalled 1 skill.\n\n=== update: the background auth resolution prints nothing\n\n>>> [CLI] aitools update --scope=global\nAlready up to date.\n\n=== uninstall: the background auth resolution prints nothing\n\n>>> [CLI] aitools uninstall --scope=global\nUninstalled 1 skill.\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,2 +4,3 @@
        	            	 >>> [CLI] aitools install --skills-only --scope=global --agents=cursor
        	            	+Warn: [hostmetadata] failed to fetch host metadata for https://unreachable.test, will skip for 1m0s
        	            	 Installing Databricks skills for Cursor...
        	Test:       	TestAccept/cmd/aitools/telemetry/workspace-unreachable
        	Messages:   	cmd/aitools/telemetry/workspace-unreachable/output.txt vs /tmp/TestAcceptcmdaitoolstelemetryworkspace-unreachable789053627/002/output.txt
TestAccept/cmd/version: branch ✅ | main (8b4f052) ❌ | latest (v1.26.0) ❌

main (8b4f052):

acceptance_test.go:1345: Diff:
        --- cmd/version/output.txt
        +++ /tmp/TestAcceptcmdversion3706040/002/output.txt
        @@ -24,7 +24,6 @@
         >>> [CLI] version --check
         Databricks CLI v[CLI_VERSION]
         This is a development build; skipping the update check.
        -Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.
         
         === version --check --output json with a coding agent prints no tip

latest (v1.26.0):

acceptance_test.go:1345: Diff:
        --- cmd/version/output.txt
        +++ /tmp/TestAcceptcmdversion786925269/002/output.txt
        @@ -8,7 +8,7 @@
         
         >>> [CLI] version --check
         Databricks CLI v[CLI_VERSION]
        -This is a development build; skipping the update check.
        +Could not reach GitHub to check for a newer version. See https://github.com/databricks/cli/releases for the latest release.
         
         === version --check --output json
         
        @@ -16,15 +16,14 @@
         {
           "current_version": "[CLI_VERSION]",
           "update_available": false,
        -  "development_build": true
        +  "check_failed": true
         }
         
         === version --check with a coding agent and no Databricks tooling
         
         >>> [CLI] version --check
         Databricks CLI v[CLI_VERSION]
        -This is a development build; skipping the update check.
        -Tip: coding agents detected without Databricks skills. Run 'databricks aitools install' to install them.
        +Could not reach GitHub to check for a newer version. See https://github.com/databricks/cli/releases for the latest release.
         
         === version --check --output json with a coding agent prints no tip
         
        @@ -32,7 +25,7 @@
         {
           "current_version": "[CLI_VERSION]",
           "update_available": false,
        -  "development_build": true
        +  "check_failed": true
         }
         
         === version --check from Claude Code prints only the root hint
        @@ -40,4 +33,4 @@
         >>> [CLI] version --check
         Databricks skills are not installed. To work with Databricks reliably, first run: databricks aitools install
         Databricks CLI v[CLI_VERSION]
        -This is a development build; skipping the update check.
        +Could not reach GitHub to check for a newer version. See https://github.com/databricks/cli/releases for the latest release.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: d82eb6b

Run: 43200323139

Env 🔄​flaky 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 27 12:02
💚​ aws-windows-latest-4core-16gb 1 272 25 9:29
💚​ azure linux-2core-8gb 1 269 27 10:05
💚​ azure-windows-latest-4core-16gb 1 271 25 8:23
💚​ gcp linux-2core-8gb 1 270 27 15:17
🔄​ gcp-windows-latest-4core-16gb 1 1 271 25 13:02
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🔄​ TestSyncIncrementalSyncPythonNotebookDelete ✅​p ✅​p ✅​p ✅​p ✅​p 🔄​f
Top 18 slowest tests (at least 2 minutes):
duration env testname
4:00 gcp-windows-latest-4core-16gb TestAccept
3:57 azure linux-2core-8gb TestAccept
3:42 azure-windows-latest-4core-16gb TestAccept
3:41 aws linux-2core-8gb TestAccept
3:35 gcp linux-2core-8gb TestFsCpDir/uc-volumes_to_dbfs
3:30 aws-windows-latest-4core-16gb TestAccept
3:16 gcp linux-2core-8gb TestAccept
2:43 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsDelete
2:40 gcp-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:31 gcp linux-2core-8gb TestFsCpDirToDirFileNotOverwritten/local_to_dbfs
2:08 gcp-windows-latest-4core-16gb TestFsCpDirToDirFileNotOverwritten/dbfs_to_dbfs
2:06 aws linux-2core-8gb TestExport
2:06 gcp linux-2core-8gb TestFsCpDir/dbfs_to_dbfs
2:06 gcp linux-2core-8gb TestFsCpDir/local_to_dbfs
2:02 gcp-windows-latest-4core-16gb TestFilerReadDir/dbfs
2:02 aws-windows-latest-4core-16gb TestExportWithFileFlag
2:01 gcp-windows-latest-4core-16gb TestFsCpDir/dbfs_to_uc-volumes
2:00 gcp-windows-latest-4core-16gb TestFilerReadWrite/dbfs

simonfaltum
simonfaltum reviewed Nov 11, 2026
View reviewed changes

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 3e0bf18. Each finding was verified against the code; simonfaltum may follow up.

Not approving yet: 1 open design question.

This PR raises the aitools plugin probe timeout from 5s to 20s so that a slow agent CLI cold start isn't reported as unavailable, and adds a changelog fragment. The change is correct and easy to revert, CI is green, and the description lays out the trade-offs. One question is open: whether a timed-out probe should get its own reason, since it is still reported as cli-not-on-path and telemetry can't confirm the fix. Beyond that, there are a few non-blocking cleanups: the stale 5s constant in agents/detect.go, the scope of the constant's comment, the changelog wording, and the PR description template.

Questions

  1. A probe that times out is still reported as cli-not-on-path. Every plugin --help error maps to ReasonCLINotOnPath in plugin.go:228-229, and the killed process returns signal: killed rather than context.DeadlineExceeded, so telemetry records CLI_NOT_ON_PATH instead of the existing TIMEOUT category. Raising the limit moves the cutoff, but nothing will show whether 20s is enough, and a hung agent CLI now costs 20s per plugin agent instead of 5s. Do you want to keep the constant bump as is, add a distinct timeout reason (here or in a follow-up) so the funnel telemetry can confirm the fix, or treat a probe timeout as inconclusive and let the 60s install step decide?

Comments outside the changed lines

  • [nit] .github/PULL_REQUEST_TEMPLATE.md: The description uses ## Summary / ## Evidence / ## Merge Danger. .agents/skills/pr-checklist/SKILL.md asks agent-authored PRs to follow .github/PULL_REQUEST_TEMPLATE.md exactly (## Changes, ## Why, ## Tests) and not to add sections such as ## Summary. Most recently merged PRs use the template. Could you move Summary into Changes/Why and Evidence into Tests?

Comment thread libs/aitools/installer/plugin.go
// Agent CLIs are large Node/Rust binaries whose cold start (first run,
// antivirus scan, slow container disk) can take several seconds, and a probe
// that times out reports the agent as unavailable, so leave headroom.
pluginProbeTimeout = 20 * time.Second

@simonfaltum simonfaltum Nov 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] libs/aitools/agents/detect.go:20 still has its own pluginProbeTimeout = 5 * time.Second for the same <agent> plugin --help probe in (*Agent).ProbePlugin. After this PR, two constants with the same name disagree (5s vs 20s). Only tests call ProbePlugin today. Its doc, though, says it is "meant to be called once, for a selected agent, right before install", so wiring it in later would bring back the 5s false "unavailable" result this PR fixes. Please either raise that constant too, or delete ProbePlugin with its constant and tests so only one probe is left.

@anton-107 anton-107 Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Raised pluginProbeTimeout in libs/aitools/agents/detect.go to 20s as well, with a comment saying it matches the installer probe, so ProbePlugin no longer disagrees if it gets wired in. I kept ProbePlugin rather than deleting it, to keep this PR to the timeout change. (410f869)

Comment thread libs/aitools/installer/plugin.go
Comment on lines 22 to +26
// pluginProbeTimeout bounds the `<agent> plugin --help` capability check.
pluginProbeTimeout = 5 * time.Second
// Agent CLIs are large Node/Rust binaries whose cold start (first run,
// antivirus scan, slow container disk) can take several seconds, and a probe
// that times out reports the agent as unavailable, so leave headroom.
pluginProbeTimeout = 20 * time.Second

@simonfaltum simonfaltum Nov 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This comment explains 20s by the cold start of plugin --help. The constant also bounds plugin marketplace list (line 124), where a timeout doesn't mark the agent unavailable. Instead marketplaceRegistered returns true, install doesn't record InstalledMarketplace, and a later uninstall leaves the marketplace registered. Anyone who tunes the probe timeout from this comment would change that behavior without knowing it. Please mention the second use here, or give marketplace list its own constant.

@anton-107 anton-107 Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment now also covers plugin marketplace list in marketplaceRegistered: on a timeout the marketplace is assumed already registered, so install does not record it and uninstall leaves it registered. (410f869)

Comment thread .nextchanges/cli/aitools-plugin-probe-timeout.md Outdated
@@ -0,0 +1 @@
* `databricks aitools install` gives a coding agent's CLI 20 seconds instead of 5 to answer the plugin capability check, so a slow first start no longer reports the agent as unavailable. ([#7472](https://github.com/databricks/cli/pull/7472))

@simonfaltum simonfaltum Nov 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Three small things in this fragment. (1) The same probe runs in databricks aitools update when it migrates raw skills to the plugin (cmd/aitools/update.go:253 → InstallPluginForAgent), but the entry names only install. (2) "a slow first start no longer reports the agent as unavailable" is true only for starts under 20s. (3) .agents/skills/pr-checklist/SKILL.md asks for an imperative verb or "<command> now …". Suggested wording: "* databricks aitools install and databricks aitools update now wait up to 20 seconds instead of 5 for a coding agent's CLI to answer the plugin capability check, so a slow first start is less likely to report the agent as unavailable. (#7472)"

@anton-107 anton-107 Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switched the fragment to your suggested wording. It names install and update, says "less likely", and keeps the PR link. (410f869)

@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 3e0bf18 to 410f869 Compare November 11, 2026 16:52
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 848f252 to 90c8726 Compare November 11, 2026 16:53
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 410f869 to 88362b5 Compare November 11, 2026 16:54
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 90c8726 to 125c8ca Compare November 11, 2026 17:28
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 88362b5 to 679dd3e Compare November 11, 2026 17:28
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 125c8ca to 0148734 Compare November 11, 2026 19:28
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 679dd3e to ca64d39 Compare November 11, 2026 19:28
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 0148734 to 7d1dfc8 Compare November 11, 2026 19:59
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from ca64d39 to 9d4b058 Compare November 11, 2026 19:59
@anton-107

anton-107 commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Replies to the two review-body items:

Question 1 (timed-out probe reported as cli-not-on-path): this PR keeps cli-not-on-path for a probe that times out. It only raises the timeout, and I'd rather not change how failures are classified in the same change. A separate reason for a timed-out probe is a reasonable follow-up. Later in this stack, #7475 adds BlockedError.TimedOut, which is set when the agent CLI is killed for exceeding its timeout, and telemetry maps it to PLUGIN_CLI_TIMEOUT. The user-facing reason can be split there so it matches that classification.

PR description template: I've kept the current format, because every PR in this stack uses the same Summary / Evidence / Merge Danger layout and that keeps them easy to compare.

simonfaltum
simonfaltum reviewed Nov 11, 2026
View reviewed changes

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at 9d4b058. Each finding was verified against the code; simonfaltum may follow up.

Not approving yet: 1 open design question.

This PR raises the aitools plugin probe timeout from 5s to 20s, so a slow agent CLI cold start is less likely to be reported as unavailable. It also updates the unused copy in agents/detect.go and adds a changelog fragment. The change is correct, CI is green, and every point from the earlier review has been addressed or answered. Nothing is blocking. One new non-blocking item: on Windows, npm wrapper CLIs aren't actually bounded by this timeout, because runAgentCmd doesn't use process.WithProcessGroup(). One question remains about the leftover ProbePlugin constant.

Questions

  1. (*Agent).ProbePlugin in libs/aitools/agents/detect.go still has only test callers, so its pluginProbeTimeout is a second copy kept equal to the installer's value only by a comment. Do you expect ProbePlugin to get a production caller? If not, would you delete it with its constant in a follow-up, or have installer and agents share one exported constant, so there's a single probe timeout to tune? Keeping both copies as they are is also fine if you'd rather.

Agreeing with existing threads

  • [nice to have] A timed-out probe is still reported as cli-not-on-path: #7472 (review)
  • [nit] The PR description doesn't use the repo's PR template: #7472 (review)

Comment thread libs/aitools/installer/plugin.go
// It also bounds `<agent> plugin marketplace list` in marketplaceRegistered,
// where a timeout means the marketplace is assumed already registered, so
// install does not record it and uninstall leaves it registered.
pluginProbeTimeout = 20 * time.Second

@simonfaltum simonfaltum Nov 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] runAgentCmd (lines 66-70) calls process.Background without process.WithProcessGroup(). That leaves Cmd.WaitDelay at zero, and only the direct child is killed when this timeout fires. On Windows, an npm-installed Codex or Copilot runs as codex.cmd → cmd.exe → node → the native binary, with inherited stdio. If plugin --help hangs, the grandchild keeps the stdout/stderr pipes open and Wait blocks past 20s, possibly indefinitely, so "blocks for up to 20s" doesn't hold there. This predates the PR. Since #7475 already touches runAgentCmd, consider passing process.WithProcessGroup() there, as libs/localenv/uv.go does. It sets WaitDelay and signals the whole process group.

@anton-107 anton-107 Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it's worth doing, but I'd keep it out of this PR, which only changes the timeout value. Two reasons:

  • aitools: typed fetch, state and agent CLI timeout errors #7475 rewrites runAgentCmd, including this exact process.Background call (it wraps the result in a timeout error). The option belongs there, or in a follow-up on top of it, so the stack doesn't edit the same line twice.
  • It changes behavior on both platforms, so it deserves its own test. On Unix the timeout would send SIGTERM to the whole group and allow a 10s grace period before SIGKILL, so a hung probe could take up to 30s instead of 20s. On Windows, WithProcessGroup in libs/process/group_other.go only sets WaitDelay. That bounds Wait at timeout + 10s, but grandchildren such as node under codex.cmd are not killed, so the "blocks for up to 20s" bound still won't hold exactly there.

I'll call out the Windows caveat in the Merge Danger section of the description.

@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 7d1dfc8 to b53927b Compare November 11, 2026 23:37
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 9d4b058 to 149db0e Compare November 11, 2026 23:37
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from b53927b to 14f43cd Compare November 11, 2026 23:50
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 149db0e to 3edb030 Compare November 11, 2026 23:50
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 14f43cd to 8e754f4 Compare November 11, 2026 23:58
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 3edb030 to 239e195 Compare November 11, 2026 23:58
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 8e754f4 to 14cb376 Compare November 12, 2026 00:05
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 239e195 to 9cf349f Compare November 12, 2026 00:06
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 14cb376 to d03528e Compare November 12, 2026 00:15
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 9cf349f to 005fc18 Compare November 12, 2026 00:15
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from d03528e to 0aed2cb Compare November 12, 2026 00:27
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 005fc18 to f251254 Compare November 12, 2026 00:27
@anton-107

anton-107 commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

On Question 1: I'll keep ProbePlugin and its constant as they are. Giving it a production caller or removing it is out of scope for this timeout change, so that can be a follow-up. Until then, the comment on the constant notes that it matches the installer's probe timeout.

simonfaltum
simonfaltum approved these changes Nov 12, 2026
View reviewed changes

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review from simonfaltum's review agent (pr-review skill) at f251254. Each finding was verified against the code; simonfaltum may follow up.

No blocking issues found, approving.

This PR raises the aitools plugin probe timeout from 5s to 20s, in the installer and in the unused agents.ProbePlugin copy, and adds a changelog fragment. The commit hasn't changed since the last review round. CI is green, and the earlier fixes (both constants, the comment on marketplace list, the changelog wording) are correct. Nothing is blocking. The open items were all discussed before: the hung-grandchild case in runAgentCmd, which also affects npm-installed Codex on Linux and macOS and has no fix in the stack yet; the cli-not-on-path reason for a timeout, which #7475 fixes; and the PR template.

Agreeing with existing threads

  • [nice to have] runAgentCmd doesn't bound a hung grandchild, on Linux and macOS too: #7472 (comment)
  • [nice to have] A timed-out probe is still reported as cli-not-on-path: #7472 (review)
  • [nit] The PR description doesn't follow the repo's PR template: #7472 (review)

@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 0aed2cb to 9dd6d21 Compare November 12, 2026 09:34
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from f251254 to 3e979d0 Compare November 12, 2026 09:34
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 9dd6d21 to 7e2cf0e Compare November 12, 2026 09:57
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 3e979d0 to 99db1f7 Compare November 12, 2026 09:57
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 7e2cf0e to 2920789 Compare November 12, 2026 10:20
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 99db1f7 to 07cf845 Compare November 12, 2026 10:20
@anton-107
anton-107 force-pushed the aitools/version-check-install-hint branch from 2920789 to 66114a4 Compare November 12, 2026 10:24
@anton-107 Isaac
aitools: raise the plugin probe timeout from 5s to 20s
d82eb6b 
`databricks aitools install` runs `<agent> plugin --help` to check that the agent's CLI can install plugins. Agent CLIs are large Node/Rust binaries, and a cold start (first run, antivirus scan, slow container disk) can take several seconds. When the probe times out, the agent is reported as unavailable.

```diff
 // libs/aitools/installer/plugin.go
-pluginProbeTimeout = 5 * time.Second
+pluginProbeTimeout = 20 * time.Second
```

The same constant also bounds `<agent> plugin marketplace list`. Changelog fragment: `.nextchanges/cli/aitools-plugin-probe-timeout.md`.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 force-pushed the aitools/plugin-probe-timeout branch from 07cf845 to d82eb6b Compare November 12, 2026 10:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@simonfaltum simonfaltum simonfaltum approved these changes
@lennartkats-db lennartkats-db Awaiting requested review from lennartkats-db lennartkats-db is a code owner
@abdelrahmanomar1515 abdelrahmanomar1515 Awaiting requested review from abdelrahmanomar1515 abdelrahmanomar1515 is a code owner automatically assigned from databricks/eng-deco-ax
@rauchy rauchy Awaiting requested review from rauchy rauchy was automatically assigned from databricks/eng-deco-platform

Assignees

No one assigned

Labels

ai-auto PRs made by AI loops & factories,

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@anton-107 @eng-dev-ecosystem-bot @simonfaltum