Add try command to mach & try build partitioned
Adds `./mach try` command that enables anybody to easily test their changes without opening PR and requesting try from bors-servo, by force pushing HEAD to appropriate branch. Command accepts branches names to select only partial runs of CI (same like bors try command). So if you only want to test mac build (that would be `@bors-servo try=mac`) you run `./mach try mac`. If no job is specified, try branch is used.
As partitioned CI jobs were not working after migration to GitHub Actions I remade them by using if guards.
Also WPT jobs were failing due to empty `INTERMITTENT_TRACKER_DASHBOARD_SECRET` on my fork, so I added additional check to prevent failed run.
And that concludes my work on #29379🎉
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#29379
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because it's CI
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Currently `test-tidy` only runs during the quick check. This can mean
that when a change is approved, the quick check can be skipped and a
change can land without `test-tidy` ever being run on it. This change
adds a `test-tidy` run to the main CI run to avoid this happening.
The layout-2020 variant will be consumed by the
internal WPT trend [dashboard][1] only and won't
be made available on servo.org.
This change will be removed once the layout
engine we will use going forward is finalized.
[1]: https://servo.github.io/internal-wpt-dashboard/
Signed-off-by: Mukilan Thiyagarajan <me@mukilan.in>
Do not run specialized workflow on forks
Per #29379 to make GitHub actions more fork-friendly.
I have not decided yet on how to handle main workflow, as I started to like my `git push origin HEAD:try --force` workflow, but using CI on all forks branches would also help enforce the [rule](https://github.com/servo/servo/blob/master/CONTRIBUTING.md#pull-request-checklist):
> Commits should be as small as possible, while ensuring that each commit is correct independently (i.e., each commit should compile and pass tests).
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes partially fix#29379
<!-- Either: -->
- [ ] There are tests for these changes OR
- [X] These changes do not require tests because it is only CI stuff
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
This change adds the secret to the environment, which should trigger the
filtering intermittents script to actually upload new results to the
intermittent dashboard instead of simply querying.
This makes it easier to run `update-wpt` based on results from the bots.
A future version of this could aggregate all unexpected results that
were not filtered as intermittents.
After filtering intermittents, output the results as JSON. Update the
GitHub workflow to aggregate this JSON data into an artifact and use the
aggregated data to generate a GitHub comment with details about the try
run. The idea here is that this comment will make it easier to track
intermittent tests and notice when a change affects a test marked as
intermittent -- either causing it to permanently fail or fixing it.
ci: Produce a single WPT log artifact
GitHub supports adding files to an artifact in parallel, as long as the filenames are unique. This makes it easier to download build results when more than a single builder fails.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because this is a small change to the CI.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
GitHub supports adding files to an artifact in parallel, as long as the
filenames are unique. This makes it easier to download build results
when more than a single builder fails.
Integrate filter-intermittents into test-wpt
This change integrates the `filter-intermittents` command into `test-wpt`. This is in preparation for future work on tracking intermittent failures. This change also:
- Removes `ServoJsonFormatter` and replaces it with a generic WPT log handler which tracks unexpected results.
- The intermittent filter is now controlled via environment variables and the GitHub version requires a token instead of credentials.
- Output is saved to a single file and is always text.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they are changes to build infrastructure.
Fix upstreaming of WPT changes
The GitHub Action needs access to Servo repository secrets, so switch to using the 'pull_request_target' event. Since these PRs have more complete access to the Servo repository, do not execute the version of the upstream script that comes with the PR. Instead, simply fetch the changes. To make this work, the script no longer expects the PR commit to be checked out, merely that they exist in the repository somewhere.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] There are tests for these changes
The GitHub Action needs access to Servo repository secrets, so switch to
using the 'pull_request_target' event. Since these PRs have more
complete access to the Servo repository, do not execute the version of
the upstream script that comes with the PR. Instead, simply fetch the
changes. To make this work, the script no longer expects the PR commit
to be checked out, merely that they exist in the repository somewhere.
Revert LLVM downgrade & fix nighly build
<!-- Please describe your changes on the following line: -->
Windows MSCV nightly builds have been failing
since Jan 18. The failure is similar to the
one @delan identified & fixed, where CI build started
to fail after the version of LLVM included in
Github Actions runner images was bumped from
14 to 15.
As discussed on Zulip, we have existing support
in ./mach bootstrap to download the LLVM binaries
from our s3 bucket. Following the documented
[process][1], new LLVM 15.0.5 binaries have now been
uploaded to S3.
I've validated that the windows msvc build no longer fails:
https://github.com/mukilan/servo/actions/runs/3996453221/jobs/6856528077
[1]: https://github.com/servo/servo/wiki/Upgrading-the-windows-LLVM-binaries
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix#29269 (GitHub issue number if applicable)
<!-- Either: -->
- [ ] There are tests for these changes OR
- [X] These changes do not require tests because they modify only CI steps that have been validated by manual run.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
This reverts commit 7a885032e6, reversing
changes made to f70857906d.
Since we have now uploaded the LLVM 15.0.5
binaries to S3 so that `mach boostrap` can use
them directly, we no longer need to downgrade the
LLVM in GitHub runner image.
This change integrates the filter-intermittents command into test-wpt.
This is in preparation for future work on tracking intermittent
failures. This change also:
- Removes the SrvoJson logger and replaces it with a generic WPT log
handler which tracks unexpected results.
- The intermittent filter is now controlled via environment variables
and the GitHub version requires a token instead of credentials.
- Output is saved to a single file and is always text.
CI: clean up workflow names
The workflow names on [the actions page](https://github.com/servo/servo/actions) could be a bit clearer and more consistent.
This patch gives main.yml and pull-request.yml more meaningful names, clarifies the nightly builds, and groups the WPT-related builds in lexicographic order.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [ ] ~~`./mach build -d` does not report any errors~~
- [ ] ~~`./mach test-tidy` does not report any errors~~
- [ ] These changes fix #___ (GitHub issue number if applicable)
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they affect the CI configuration
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Add a script to upstream WPT changes via a GitHub Action
This is a modified version of the webhook found at https://github.com/servo/upstream_wpt_webhook and deployed via SaltStack. It is updated to use modern Python and to assume that GitHub Actions will fetch the appropriate source code locally before the script is run.
---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#29206 and #23798
- [x] There are tests for these changes
This is a modified version of the webhook found at
servo/upstream_wpt_webhook and deployed via SaltStack. It is updated to
use modern Python and to assume that GitHub Actions will fetch the
appropriate source code locally before the script is run.
Fixes#29206.
Fixes#23798.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Fail faster on CI when Linux bootstrap fails
Raise an exception when dependencies fail to install. Also split the run phase of the Linux bootstrap so that either of these failing commands will cause the job to fail.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they just change minor script issues.
Raise an exception when dependencies fail to install. Also split the run
phase of the Linux bootstrap so that either of these failing commands
will cause the job to fail.
CI: run WPT tests on macOS
This patch uncomments and fixes the mac-wpt job in the main workflow, with a few changes:
* we downgrade python3 and pip3 to 3.9, because inspect.getargspec was [removed in 3.11](https://docs.python.org/3/whatsnew/3.11.html#removed)
* we should apparently move to [inspect.getfullargspec](https://docs.python.org/3/library/inspect.html#inspect.getfullargspec)
* we download the real Servo artifact, rather than the fake Servo for testing
We also align the rest of the job (other than `--timeout-multiplier`) to be as close as possible to linux-wpt, based on the assumption that linux-wpt is more up to date. This includes:
* setting strategy.fail-fast to false
* adding a failure() condition to the “Archive logs” step
* adding the chunk id to log filenames
There are some failing WPT tests, so for now the job is not required to succeed.
Example run: <https://github.com/delan/servo/actions/runs/3837530905>
---
- [ ] ~~`./mach build -d` does not report any errors~~
- [ ] ~~`./mach test-tidy` does not report any errors~~
- [x] These changes fix#28284 (GitHub issue number if applicable)
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they affect the CI configuration
CI: remove branch check steps that don’t actually do anything
f165e17ce2 (#28778) adds a step to each job of the main build that ostensibly bails out of the job, if the context branch is not `auto` or one of the `try` branches relevant for that platform. But a step that runs `exit 0` doesn’t really have any effect.
Since the main workflow already has `branches` limited to `auto` and the `try` branches, we can safely remove those steps without worrying about the main build running on dependabot branches etc. For example:
* [dependabot/cargo/nom-7.1.2](https://github.com/servo/servo/actions?query=branch%3Adependabot%2Fcargo%2Fnom-7.1.2) only has a run for the pull request workflow
* [this test build](https://github.com/delan/servo/actions/runs/3830596812/jobs/6518672884) shows that even if the branch check fails, the build continues
---
- [ ] ~~`./mach build -d` does not report any errors~~
- [ ] ~~`./mach test-tidy` does not report any errors~~
- [ ] ~~These changes fix #___ (GitHub issue number if applicable)~~
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they affect the CI configuration
Update actions/upload-artifact in GitHub Actions workflows to v3
Updates the `actions/upload-artifact` action used in the GitHub Actions workflow to its newest major version.
Changes in [actions/upload-artifact](https://github.com/actions/upload-artifact):
> ## v3.1.1
> - Update actions/core package to latest version to remove `set-output` deprecation warning
>
> ## v3.1.0
> - Bump @actions/artifact to v1.1.0
> - Adds checksum headers on artifact upload
>
> ## v3.0.0
>
> - Update default runtime to node16
> - Update package-lock.json file version to 2
Still using v2 of `actions/upload-artifact` will generate some warning like in this run: https://github.com/servo/servo/actions/runs/3718806495
> Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout@v2, actions/upload-artifact@v2
The PR will get rid of those warnings for `actions/upload-artifact`, because v3 uses Node.js 16.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [ ] `./mach build -d` does not report any errors
- [ ] `./mach test-tidy` does not report any errors
- [ ] These changes fix #___ (GitHub issue number if applicable)
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they only update components used in the CI workflows and those workflows still passing should be enough tests.
Update actions/checkout in GitHub Actions workflows to v3
Updates the `actions/checkout` action used in the GitHub Actions workflow to its newest major version.
Changes in [actions/checkout](https://github.com/actions/checkout):
> ## v3.1.0
> - Use @actions/core `saveState` and `getState`
> - Add `github-server-url` input
>
> ## v3.0.2
> - Add input `set-safe-directory`
>
> ## v3.0.1
> - Fixed an issue where checkout failed to run in container jobs due to the new git setting `safe.directory`
> - Bumped various npm package versions
>
> ## v3.0.0
>
> - Update to node 16
Still using v2 of `actions/checkout` will generate some warning like in this run: https://github.com/servo/servo/actions/runs/3718806495
> Node.js 12 actions are deprecated. For more information see: https://github.blog/changelog/2022-09-22-github-actions-all-actions-will-begin-running-on-node16-instead-of-node12/. Please update the following actions to use Node.js 16: actions/checkout@v2, actions/upload-artifact@v2
The PR will get rid of those warnings for `actions/checkout`, because v3 uses Node.js 16.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [ ] `./mach build -d` does not report any errors
- [ ] `./mach test-tidy` does not report any errors
- [ ] These changes fix #___ (GitHub issue number if applicable)
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they only update components used in the CI workflows and those workflows still passing is enough test.
