Repository navigation
test-companion: fixes from running every job as the GitHub App - #96
Open
naimish-browse wants to merge 9 commits into
Open
naimish-browse wants to merge 9 commits into
naimish-browse wants to merge 9 commits into
Conversation
- Ignore comments from bots, our own included. The App's own comments quote "@browserstack go", so they started runs, got a "write access" reply, and could push a real request out of the queue. The starter workflow also skips bot comments, so no runner starts. - Jobs whose output isn't code (generate, rca, verdict, validate) never commit to the branch, and BrowserStack tooling's by-products such as browserstack.err are never committed by any job. - Work handed back as a pull request after a request on a pull request now targets that pull request's branch, as a draft, titled for it. Before, it targeted the default branch and carried the whole PR. - A run that broke no longer offers "go" or leaves its check waiting for approval; it is reported like any other run, and the comment says why (the CLI's error line, e.g. a missing project or a credit limit). - Acting as the App, commits are made as its bot account, so they show its name and logo, and the loop guard still recognises them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
 |
A fix or automate branch that also carried a file Test Companion never commits, such as browserstack.err written by BrowserStack tooling, was refused outright, so a real fix could be blocked by a by-product. Now only files under .github/ and symlinks refuse the branch. Other never-committed files are left out: one commit on top of the agent's branch puts them back as they were, built in a scratch index so the checkout is untouched, and the comment says what was left out. If they were the branch's only changes, no pull request is opened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s literally - A handed-back branch with files Test Companion never commits now gets one squashed commit on the base, so those files are not in anything pushed. Before, the clean commit sat on top of the agent's commit, which still held them. - git reads every path as a plain file name (GIT_LITERAL_PATHSPECS), so a file named like ":!x" can't make git add or reset match other files. The left-out paths go to git on stdin. - The "jobs without code output never commit" rule is checked again with the job the pipeline chose, so event runs are covered too. - The App's commits count as ours even when a later run can't act as the App, so it doesn't push in reaction to them. - A pull request that can't be a draft is opened as a normal one; "already done" is only said when its pull request really is open. - The pull request targets the requesting pull request's branch only when the run started on that pull request's commit, and its title names the job from the hand-off. - File names in comments are capped at ten and rendered so a backtick in a name can't break out. - Tests now call the real functions, against real git repositories, and the starter's comment filter is evaluated rather than searched for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mments - The reason shown for a run that broke is the CLI's own error event first, then an "Error:" line from its stderr. "Error:" lines in the agent's test output on stdout are no longer mistaken for it. - The reason and the answer are redacted before they reach a pull request comment: credential-named environment values (tokens, access keys) and user:password in URLs. GitHub masks secrets in logs, not in comments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…closed work A handed-back fix or automate branch is now always pushed as one commit on the base, under our name, even when nothing is left out. The agent's own commits never reach GitHub, so a file it added and later deleted is not in the pushed history, and what was checked is exactly what is pushed. Pushing the branch: only an ordinary rejection means an earlier run already pushed it. A remote rejection (a protection rule, a missing permission) is reported as "could not push", never "pushed". When the branch is already there, an open pull request is reported as done, a closed or merged one is reported and not reopened, and with none a pull request is opened and said to be from the earlier run's branch. Credentials are now hidden in everything sent to GitHub (comments, replies, checks, pull requests), including the workflow token captured before the App token replaces it. The squash commit message keeps agent-chosen names on one line, so they can't add trailers, and the scratch reset no longer re-reads the checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…after a failed lookup A refused push now reports git's actual reason (the [remote rejected] line, a remote error such as a ruleset, or a fatal line) instead of the generic "failed to push some refs". A failed lookup for an earlier run's pull request now stops with a plain message instead of being read as "no pull request" and opening one, which could have reopened closed work. leaveOut retries its reset without --no-refresh on a git too old to know the option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h build was read - Each request (an @browserstack comment, or a `go`) now gets its own reply with that run's report, instead of rewriting one shared comment. Automatic runs, which answer no request, keep one status comment. - A spent plan no longer blocks a newer plan for the same job on the same commit: spent markers now name the plan they spent. Older markers still spend the plans posted before them. `go #<id>` on a spent plan says it was already approved or dropped. - A run a person asked for or approved may save on top of an earlier Test Companion commit; the loop guard now applies to automatic runs. - explain, fix and find-the-bad-PR say which failed run they read and list the other failures. A comment can name a run: its link, `run <id>`, or a word from the workflow's name. A named run must be in this repository, on one of the pull request's commits, and red. - When nothing has failed yet but CI is still running, the reply says so and lists what is running, instead of "Nothing has failed". - Read-only jobs (explain, find the bad PR) no longer show the preview banner, which read as "nothing happened"; they say they're read-only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d on lookups - Approving a plan now also spends every older plan of that job on the commit. Before, approving the newer of two plans left the older one waiting: a second "go" would run it, and the nightly sweep would drop it and turn the commit's check to "action required" over a green run. - Naming a failed workflow matches its whole name first (the longest wins, so "Lint Checks" beats "Lint"), then a name whose every significant word appears; an incidental word no longer picks a run. The "also failed" hint now asks by run id, which is unambiguous. - Workflow names in replies are code spans, so a name can't mention a person or team, link or format. - A manual run may build on our own commit only when a human started it, not automation. - Run lookups fail closed and say so: a GitHub failure is no longer reported as "nothing failed" or "no such run"; a run with no commit id never matches; a pull request too long to list fully says why. - The read-only line is chosen from the job asked for, never from the agent-written summary, and says nothing the job does is saved. - Removed latestFailedRun, unused since run selection moved to ciOnCommit; its tests now cover ciOnCommit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s don't name a workflow Approving a plan now spends every plan of the same job on that commit that already existed when the approval was recorded, older or newer. Approving an older plan therefore also settles a newer one that was waiting, so the nightly sweep can no longer drop it and flip the check over the approved run. A plan posted after the approval stays approvable. Dropping an expired plan still spends only it and older plans, never a newer one that may still be waiting. A marker that names a plan but no job spends only that plan. Old markers without a plan id keep their meaning. "go #<id>" on a plan that another approval settled now says a different plan for this job was approved, instead of claiming this one ran. Picking a failed workflow by name: a name made only of everyday words (build, Test, CI, explain, fix) no longer matches, because comments use those words anyway; such a run is asked for by run id, which is what the bot's own hint suggests. Matrix suffixes in parentheses are ignored, so "explain e2e" finds "E2E Tests (linux)" again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from running every Test Companion job as the BrowserStack GitHub App on a pilot repository.
The App and its own comments
What gets committed
generate,rca,verdict,validate) never commit to the branch, including when the pipeline file picks the job for an event.browserstack.err,.browserstack*, logs,.env, report folders) are never committed. A handed-backfixorautomatebranch is always pushed as one squashed commit on the base, under our name, so the agent's own commits never reach GitHub; by-products are left out of it. Only.github/files and symlinks still refuse a branch.:!xcan't widengit addorresetto other files.Pull requests handed back
Runs that break
go, and its check completes.Replies and approvals
@browserstackcomment, or ago) gets its own reply with that run's report. Results no longer overwrite each other in one shared comment. Automatic runs, which answer no request, keep one status comment.goapproves the newest plan waiting. One approval settles every plan of that job on that commit that existed at the time, so neither a secondgonor the nightly sweep acts on them; a plan asked for afterwards stays approvable.go #<id>on a settled plan says whether it was approved, dropped, or replaced by another approved plan.Which failure is read
explain,fixandfind the bad PRsay which failed run they read, and list any other failures. A comment can name the run: its link,run <id>, or the workflow's name (@browserstack explain e2e; matrix suffixes like(linux)are ignored, and names made only of everyday words such asbuildorCIare asked for by run id). A named run must be in this repository, on one of the pull request's commits, and red. The "also failed" hint asks by run id. A failed GitHub lookup is reported as such, never as "nothing failed".explainandfind the bad PRsay they're read-only, instead of showing the preview banner.Tests: all 845 pass. The new ones call the real functions, including against real git repositories, and the starter's comment filter is evaluated rather than matched as text.
🤖 Generated with Claude Code