The -Zgcc-ld=lld flag makes rust use the rust-lld
linker that is distributed as part of rust toolchain.
However, this flag doesn't work on nixos correctly
as
1) rust-lld needs to be patched to have the correct rpath
to find libz.so
2) the bin/gcc-ld/ld.lld wrapper which calls rust-lld also
needs to be patched to use the correct dynamic loader
3) rust-lld doesn't respect NIX_LDFLAGS which contains
the additional search path derived from buildInputs.
The system linkers on nixos are wrapped so that
NIX_LDFLAGS is added as the rpath to the final binary.
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
This change replaces OpenSSL with rustls and also the manually curated
CA certs file with webpki-roots (effectively the same thing, but as a
crate).
Generally speaking the design of the network stack is the same. Changes:
- Code around certificate overrides needed to be refactored to work with
rustls so the various thread-safe list of certificates is refactored
into `CertificateErrorOverrideManager`
- hyper-rustls takes care of setting ALPN protocols for HTTP requests,
so for WebSockets this is moved to the WebSocket code.
- The safe set of cypher suites is chosen, which seem to correspond to
the "Modern" configuration from [1]. This can be adjusted later.
- Instead of passing a string of PEM CA certificates around, an enum is
used that includes parsed Certificates (or the default which reads
them from webpki-roots).
- Code for starting up an SSL server for testing is cleaned up a little,
due to the fact that the certificates need to be overriden explicitly
now. This is due to the fact that the `webpki` crate is more stringent
with self-signed certificates than SSL (CA certificates cannot used as
end-entity certificates). [2]
1. https://wiki.mozilla.org/Security/Server_Side_TLS
2. https://github.com/briansmith/webpki/issues/114Fixes#7888.
Fixes#13749.
Fixes#26835.
Fixes#29291.
Now that the new version of GStreamer fixes this issue, we can remove
the workarounds for this problem as well as all of the homebrew
bootstrapping logic.
This is the last piece of the puzzle to turning off bors. This makes
functionality provided by bors to understand "@bors-servo try" a GitHub
Action. For now the syntax is more or less the same, but we can modify
it in the future and even add support for custom configuration options
(more specific build combinations or even passing compiler flags).
The big difference between this and what bors does is that there is no
merge commit. GitHub simply runs tests on the version of the branch that
is on a pull request. There is always the risk that tests might start
failing when a branch is rebased, but this offers a bit more control
because you can easily rebase from the PR and the merge queue will check
this as well.
Much of the code used to import WPT tests from upstream has been moved
to the WPT repository itself, so this can be reused. In addition,
simplify the workflows by merging the entire process into mach and also
directly into the GitHub workflow. This should fix WPT imports after
combining compilation of layout and legacy layout.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
1. The options specified in `rustdoc-with-private` are essentially the
default now so we can remove this script.
2. `./mach browse-doc` is redundant with `./mach doc --open` which uses
the underlying cargo functionality to browse the documentation.
Fixes#29888.
Also fix report_aggregated_expected_results.py which was reporting an
error when there were no failing tests. This is more commonly an issue
with Layout 2020 because if runs fewer tests and was causing builds to
show up as failing even when they were not.
The official gstreamer .pkg distribution should contain
'relocatable' dylibs, but as discovered in #29732,
some dylibs have absolute links to liblzma.5.dylibs
Since /opt/homebrew is the default install location
on Apple Silicion, this will cause the packaging &
build steps to fail, even if 'xz' package is installed
via homebrew.
This is a temporary fix until upstream fixes the bug
and makes the package truly 'relocatable'.
Signed-off-by: Mukilan Thiyagarajan <me@mukilan.in>
This PR re-enables support for the gstreamer mediastack
in macOS by consuming the official binary '.pkg' files
from gstreamer.freedesktop.org
To maintain symmetry with other platforms, the '.pkg'
files are uploaded to servo-build-deps and fetched from
there using the new script 'etc/install_macos_gstreamer.sh'.
Unlike the Homebrew version, the official GStreamer is
distributed as a 'relocatable' framework i.e the dylibs all
have @rpath-relative install names and also link to other
dylibs using @rpath relative path. To address this difference
the 'servo' binary needs to be patched with 'install_name_tool'
to add an LC_RPATH command that sets the relative paths
that the dynamic linker should search when trying to satify
dependencies. In Servo's case, this will be a path relative to
the 'servo' binary itself i.e '@executable_path/lib/'
The additional 'lib' is due to a flaw in the gstreamer
packaging where the install names of some of the dylibs
have the prefix '@rpath/lib' and some of them just have '@rpath'.
This PR also fixes a couple of issues present in the
`mach build` process on MacOS:
1. `mach build` process was not copying transitive dependencies
of servo binary but only the first level dylibs
2. `mach build` process didn't patch the links to dylibs
in servo binary (and dependencies). This meant though
(some) dylibs were copied to local path, the binary
still loaded the dylibs from system GStreamer installation
i.e homebrew instead of the copieds dylibs
The build and runtime dependencies in etc/homebrew/Brewfile
and etc/homebrew/Brewfile-build have also been removed in This
PR.
Signed-off-by: Mukilan Thiyagarajan <me@mukilan.in>
When doing a try run, bors will often push the last closed merge onto
the branch before pushing the change to try. This means that test
results get reported on closed PRs. There are two issues with this:
1. Doing too much work on the bots.
2. Extra results on closed PRs.
This changes fixes the second issue.
Fixes#29583.
Make WPT results output more useful
Before when a subtest failed, the text of the failed assertion was not printed. This changes makes sure that it is printed in both the console and the aggregated test output.
Also fix a couple typing errors.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because these are improvements to build tools.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Before when a subtest failed, the text of the failed assertion was not
printed. This changes makes sure that it is printed in both the console
and the aggregated test output.
Also fix a couple typing errors.
This change moves all of Servo's WPT Python support scripts into one
directory as they were previously scattered throughout the directory
structure. This should allow more code reuse and make it easier to
understand how everything fits together.
The changes:
- `tests/wpt/update` → `python/wpt/importer`
- `etc/ci/upstream-wpt-changes/wptupstreamer` → `python/wpt/exporter`
- `etc/ci/upstream-wpt-changes/test.py` → `python/wpt/test.py`
- `etc/ci/upstream-wpt-changes/tests` → `python/wpt/tests`
- `tests/wpt/servowpt.py` →
- `python/wpt/update.py`
- `python/wpt/run.py`
- `tests/wpt/manifestupdate.py` → `python/wpt/manifestupdate.py`
This change also removes
- The ability to run the `update-wpt` and `test-wpt` commands without
using `mach`. These didn't work very well, because it was difficult
to get all of the wptrunner and mach dependencies installed outside
of the Python virtualenv. It's simpler if they are always run through
`mach`.
- The old WPT change upstreaming script that was no longer used.
Import WPT tests every week instead of every day
We often have multiple imports stacked up which makes it more confusing to land each one. In addition, giving a week to do the import will give us more margin to triage failing tests and hopefully to be able to use intermittent expectations for flaky tests.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because this just change the frequency of a scheduled action.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
We often have multiple imports stacked up which makes it more confusing
to land each one. In addition, giving a week to do the import will give
us more margin to triage failing tests and hopefully to be able to use
intermittent expectations for flaky tests.
- Have the WPT exporter script use the WPT_SYNC_TOKEN for checking out
wpt.
- Make sure the local WPT repository is unshallow when pushing.
- When searching for existing PRs use the main GitHub search API,
as the pull request search does not seem to properly process
the "head" parameter.
- When deleting branches in the downstream WPT repository, use
the full URL to avoid trying to modify the upstream repository.
This cleans up the GitHub actions yaml a bit and ensures that developers
are running this check locally before submitting changes. In addition,
it allows adding tests for this check. Finally, this change fixes the
tidy tests by upgrading voluptuous for Python 3.10 as well as by
reverting an inadvertent change for NixOS compatibility on one of the
dummy testing files.
There are two kinds of flaky/intermittent tests in Servo. The
traditional kind is the test that fails on the CI, but has an associated
bug indicating that the test is an intermittent failure. Many of these
tests have completely unstable results, for instance those where an
unpredictable set of subtests fail. It's impossible to generate stable
results for these, so we have traditionally simply discard these
unexpected results.
Another kind of intermittent test is one that will produce an expected
result when rerun (ie will flake). Some of these are also labeled with
bugs, while some are not. In some cases, there is flakiness in some core
Servo functionality that can lead to *any* test flaking, such as a race
condition that can lead to an early screenshot for reftests. When these
kinds of tests do not have associated bugs, they cause the CI to fail.
In this case, it is impossible to label these tests as intermittent
because it can literally be any test.
This change, reruns failed tests in order to detect unlabeled tests in
the second category. Instead of blocking the CI when the second run
leads to expected results, the CI will now pass, but the flake will be
reported to the new flakiness dashboard. This prevents unrelated flakes
from slowing down the merge queue.
Use the new intermittent dashboard to report intermittents and get
information about open bugs. This is now used to filter out
known-intermittents from results. In addition, this also allows the
scripts to report bug information to the GitHub. Display that in all
output.
After filtering intermittents, output the results as JSON. Update the
GitHub workflow to aggregate this JSON data into an artifact and use the
aggregated data to generate a GitHub comment with details about the try
run. The idea here is that this comment will make it easier to track
intermittent tests and notice when a change affects a test marked as
intermittent -- either causing it to permanently fail or fixing it.
GitHub supports adding files to an artifact in parallel, as long as the
filenames are unique. This makes it easier to download build results
when more than a single builder fails.
The GitHub Action needs access to Servo repository secrets, so switch to
using the 'pull_request_target' event. Since these PRs have more
complete access to the Servo repository, do not execute the version of
the upstream script that comes with the PR. Instead, simply fetch the
changes. To make this work, the script no longer expects the PR commit
to be checked out, merely that they exist in the repository somewhere.
This is a modified version of the webhook found at
servo/upstream_wpt_webhook and deployed via SaltStack. It is updated to
use modern Python and to assume that GitHub Actions will fetch the
appropriate source code locally before the script is run.
Fixes#29206.
Fixes#23798.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
