Repository navigation
ci(oracles): run the framework self-oracle and the e2e fixed seed as two jobs each - #38
Merged
Merged
Conversation
…two jobs each The framework self-oracle (299 tests since #35) and the end-to-end oracle's fixed-seed units each set the workflow's critical path at about 10 to 15 minutes in CI. Each now runs as two jobs, and Vitest 5's self-oracle job as two too. scripts/pr-shards.mjs names the shards: one by a pattern over test names, the other the rest. Its check, which the Test job runs, lists each suite's tests and fails unless the shards together select exactly the tests one job selected before (for the e2e, the tests that -t "fixed seed" selects). Locally, at load average 4, the shards take 287 s and 240 s (framework, 589 s as one job at load 10) and 150 s and 145 s (e2e, 325 s as one). Co-authored-by: Isaac <no-reply@databricks.com>
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
 |
commit: |
Vitest 5 joins a suite's name and a test's with ` > `. The pattern's separator accepted a space or ` > `, so in a Vitest 5 name the space before `> ` matched, and the rest shard's negative lookahead then passed on `> determinism: …`: the second Vitest 5 framework job ran all 299 tests (570 s). A space followed by `> ` is now no separator. The check missed it because it required a shard to select both joins; it now takes any shard that selects either join as an owner and requires exactly one owner that selects both, which reports the old separator. A probe of every listed name under Vitest 4.1.12 and 5.1.0 selects 149 and 150 framework tests and 5 and 26 fixed-seed units under both. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The workflow's critical path grew from about 8 minutes to 14 to 20 minutes in CI over the last three merges. The maintainer's limit is 10 minutes. This change splits the two slowest jobs, the framework self-oracle and the end-to-end oracle's fixed seed, into two parallel jobs each. Vitest 5's self-oracle job splits the same way. No test is dropped: a check proves that the shards together run exactly the tests that one job ran before.
Where the time went
CI job times on
mainand on pull requests, in minutes:main)The same code gives different CI times. #36 did not touch the framework's code or the self-oracle, and a rerun of its self-oracle job took 11:44 against 14:41. The fast gap job took 7:55 on 908b1f9 against 4:35 on its pull request. So part of the growth is runner variance. To separate real cost from variance, I measured each suite locally, under the e2e lock, at each commit:
The shards
packages/oracles/scripts/pr-shards.mjsnames two partitions:oracle-framework: the first shard takes the determinism units, except the slowest (61 s). The second shard takes the rest. The measured costs are 322 s and 266 s.oracle-gaps-fixed-seed: the first shard takes the five fixed-seed units that cost most at 908b1f9. The second shard takes the rest. The measured costs are 168 s and 157 s.node scripts/pr-shards.mjs checklists each suite's tests withvitest list, and the Test job runs it. It fails in these cases:-t "fixed seed"A new test goes to the
restshard, so the shards' balance can drift, but CI never leaves a test out.vitest list -t "<pattern>"selects 149 and 150 framework tests, 299 in all. It selects 5 and 26 e2e units, 31 in all, the same as-t "fixed seed". The check rejected three wrong partitions. In the first, the old pattern selected a test that the shards do not. In the second, two shards overlap. The third names a missing exception.Measured shard times
Locally, at a load average of about 4, each shard ran as the workflow runs it:
CI took 1.0 to 1.5 times the local time for the self-oracle and 1.7 to 2.5 times for the e2e. With about 20 s of setup, the framework shards would take 5 to 7.5 minutes. The e2e shards would take 4.5 to 6.5 minutes. The real numbers come from this pull request's run.
The first CI run, and a Vitest 5 defect it found
The first run (43170624412) passed every job, with these times:
The second Vitest 5 framework job ran all 299 tests, not its 150. Vitest 5 joins a suite's name and a test's name with
>. The pattern's separator accepted a space or>. In a Vitest 5 name, the space before>matched, and the rest shard's negative lookahead then passed on> determinism: ….The check missed this. It counted a shard as an owner only when the shard selected both joins. The fix has two parts:
>is no separator now.A probe file holds every listed test name with an empty body. Under Vitest 4.1.12 and 5.1.0 alike, the patterns select 149 and 150 framework tests and 5 and 26 fixed-seed units.
scripts/e2e-shards.mjsgets the same check and separator.The run after the fix
The second run (43172279755) passed every job. Its critical path was 8:35, the first framework shard:
Runner variance is large. The same shard took 6:00 in the first run and 8:35 in the second. The fast gap job, which this change leaves as it is, took 6:42 and then 8:14. So on a slow runner it is now as long as the longest shard.
The workflow gains three jobs: one framework shard, one e2e shard, and one Vitest 5 framework shard. Each matrix job passes its shard through an environment variable.
This pull request and its description were written by Isaac.