This change adds the secret to the environment, which should trigger the
filtering intermittents script to actually upload new results to the
intermittent dashboard instead of simply querying.
This makes it easier to run `update-wpt` based on results from the bots.
A future version of this could aggregate all unexpected results that
were not filtered as intermittents.
After filtering intermittents, output the results as JSON. Update the
GitHub workflow to aggregate this JSON data into an artifact and use the
aggregated data to generate a GitHub comment with details about the try
run. The idea here is that this comment will make it easier to track
intermittent tests and notice when a change affects a test marked as
intermittent -- either causing it to permanently fail or fixing it.
GitHub supports adding files to an artifact in parallel, as long as the
filenames are unique. This makes it easier to download build results
when more than a single builder fails.
Integrate filter-intermittents into test-wpt
This change integrates the `filter-intermittents` command into `test-wpt`. This is in preparation for future work on tracking intermittent failures. This change also:
- Removes `ServoJsonFormatter` and replaces it with a generic WPT log handler which tracks unexpected results.
- The intermittent filter is now controlled via environment variables and the GitHub version requires a token instead of credentials.
- Output is saved to a single file and is always text.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they are changes to build infrastructure.
This reverts commit 7a885032e6, reversing
changes made to f70857906d.
Since we have now uploaded the LLVM 15.0.5
binaries to S3 so that `mach boostrap` can use
them directly, we no longer need to downgrade the
LLVM in GitHub runner image.
This change integrates the filter-intermittents command into test-wpt.
This is in preparation for future work on tracking intermittent
failures. This change also:
- Removes the SrvoJson logger and replaces it with a generic WPT log
handler which tracks unexpected results.
- The intermittent filter is now controlled via environment variables
and the GitHub version requires a token instead of credentials.
- Output is saved to a single file and is always text.
CI: clean up workflow names
The workflow names on [the actions page](https://github.com/servo/servo/actions) could be a bit clearer and more consistent.
This patch gives main.yml and pull-request.yml more meaningful names, clarifies the nightly builds, and groups the WPT-related builds in lexicographic order.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [ ] ~~`./mach build -d` does not report any errors~~
- [ ] ~~`./mach test-tidy` does not report any errors~~
- [ ] These changes fix #___ (GitHub issue number if applicable)
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they affect the CI configuration
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Fail faster on CI when Linux bootstrap fails
Raise an exception when dependencies fail to install. Also split the run phase of the Linux bootstrap so that either of these failing commands will cause the job to fail.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they just change minor script issues.
Raise an exception when dependencies fail to install. Also split the run
phase of the Linux bootstrap so that either of these failing commands
will cause the job to fail.
CI: run WPT tests on macOS
This patch uncomments and fixes the mac-wpt job in the main workflow, with a few changes:
* we downgrade python3 and pip3 to 3.9, because inspect.getargspec was [removed in 3.11](https://docs.python.org/3/whatsnew/3.11.html#removed)
* we should apparently move to [inspect.getfullargspec](https://docs.python.org/3/library/inspect.html#inspect.getfullargspec)
* we download the real Servo artifact, rather than the fake Servo for testing
We also align the rest of the job (other than `--timeout-multiplier`) to be as close as possible to linux-wpt, based on the assumption that linux-wpt is more up to date. This includes:
* setting strategy.fail-fast to false
* adding a failure() condition to the “Archive logs” step
* adding the chunk id to log filenames
There are some failing WPT tests, so for now the job is not required to succeed.
Example run: <https://github.com/delan/servo/actions/runs/3837530905>
---
- [ ] ~~`./mach build -d` does not report any errors~~
- [ ] ~~`./mach test-tidy` does not report any errors~~
- [x] These changes fix#28284 (GitHub issue number if applicable)
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they affect the CI configuration
CI: remove branch check steps that don’t actually do anything
f165e17ce2 (#28778) adds a step to each job of the main build that ostensibly bails out of the job, if the context branch is not `auto` or one of the `try` branches relevant for that platform. But a step that runs `exit 0` doesn’t really have any effect.
Since the main workflow already has `branches` limited to `auto` and the `try` branches, we can safely remove those steps without worrying about the main build running on dependabot branches etc. For example:
* [dependabot/cargo/nom-7.1.2](https://github.com/servo/servo/actions?query=branch%3Adependabot%2Fcargo%2Fnom-7.1.2) only has a run for the pull request workflow
* [this test build](https://github.com/delan/servo/actions/runs/3830596812/jobs/6518672884) shows that even if the branch check fails, the build continues
---
- [ ] ~~`./mach build -d` does not report any errors~~
- [ ] ~~`./mach test-tidy` does not report any errors~~
- [ ] ~~These changes fix #___ (GitHub issue number if applicable)~~
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they affect the CI configuration
Update actions/upload-artifact in GitHub Actions workflows to v3
Updates the `actions/upload-artifact` action used in the GitHub Actions workflow to its newest major version.
Changes in [actions/upload-artifact](https://github.com/actions/upload-artifact):
> ## v3.1.1
> - Update actions/core package to latest version to remove `set-output` deprecation warning
>
> ## v3.1.0
> - Bump @actions/artifact to v1.1.0
> - Adds checksum headers on artifact upload
>
> ## v3.0.0
>
> - Update default runtime to node16
> - Update package-lock.json file version to 2
Still using v2 of `actions/upload-artifact` will generate some warning like in this run: https://github.com/servo/servo/actions/runs/3718806495
> Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout@v2, actions/upload-artifact@v2
The PR will get rid of those warnings for `actions/upload-artifact`, because v3 uses Node.js 16.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [ ] `./mach build -d` does not report any errors
- [ ] `./mach test-tidy` does not report any errors
- [ ] These changes fix #___ (GitHub issue number if applicable)
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they only update components used in the CI workflows and those workflows still passing should be enough tests.
Update actions/checkout in GitHub Actions workflows to v3
Updates the `actions/checkout` action used in the GitHub Actions workflow to its newest major version.
Changes in [actions/checkout](https://github.com/actions/checkout):
> ## v3.1.0
> - Use @actions/core `saveState` and `getState`
> - Add `github-server-url` input
>
> ## v3.0.2
> - Add input `set-safe-directory`
>
> ## v3.0.1
> - Fixed an issue where checkout failed to run in container jobs due to the new git setting `safe.directory`
> - Bumped various npm package versions
>
> ## v3.0.0
>
> - Update to node 16
Still using v2 of `actions/checkout` will generate some warning like in this run: https://github.com/servo/servo/actions/runs/3718806495
> Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout@v2, actions/upload-artifact@v2
The PR will get rid of those warnings for `actions/checkout`, because v3 uses Node.js 16.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [ ] `./mach build -d` does not report any errors
- [ ] `./mach test-tidy` does not report any errors
- [ ] These changes fix #___ (GitHub issue number if applicable)
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they only update components used in the CI workflows and those workflows still passing is enough test.
