1. Make the tidy output easier to follow
2. Integrate the WPT manifest cleanliness step into tidy
itself and don't run it if nothing has changed in the WPT
directory.
3. Fix an issue where Python test requirements were not installed,
which could cause issues with some modules not being found.
Fixes#30002.
Upstream wpt script has bug where 'product' defaults to
'firefox' and this causes import of uninstalled python
modules specific to firefox runners.
Fixes#30452
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* Fix `mach test-wpt` to make crash tests work
There are two issues related to crash tests:
1. test-wpt is unable to find existing crash tests even when
called with --test-types=crashtests. The fix here is to
add crashtests to the default test suite types to python/wpt/run.py
2. When running in headless mode, crashes in style threads
don't cause servo to crash because the logic in constellation.rs
currently calls handle_panic only when the top-level browsing
context id is some value. Since style pool threads are shared,
they always generate Panic messages with None as top-level
browsing context id.
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* Send bactrace to stderr and capture it in test runner
Servo's panic hook writes backtraces to stdout. This
patch changes it so they are written to stderr.
The crash test executor for servo in WPT grouping formatter
was also not capturing the output correctly for crashtests
as the log events were being aggregated based on thread name
which doesn't seem to match correctly in case of crashtests.
This patch also fixes the log grouping logic to be based on
test name.
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* crashtests: update expectations for layout 2020
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* crashtests: update expectations for layout 2013
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* remove outdated & intemittent test expectations
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
---------
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Most tests were only being run for layout-2013, not for layout-2020.
This wasn't great since layout-2020 is now the default.
So this patch unifies the lists of included tests for both layouts.
For layout-2013 this implies adding css/css-content/, css/css-logical/
and css/css-masking/clip/.
For layout-2020 this implies adding several additional css tests, and
also tests like dom/, js/, html/, etc.
There were some issues with the way that the `--release` and `--dev`
arguments were handled in mach commands.
- Not all commands accepted them in the same way. For instance `./mach
test-wpt` didn't really accept them at all.
- If you did not pass either of them, mach would try to guess which
build you meant. This guess was often quite surprising as it wasn't
printed and it depended on the state of the your target directory,
which is difficult to remember.
- The `dev` profile is colloquially called a "debug" profile and some
commands accepted `-d` or `--debug...` like arguments, but `--debug`
with `./mach run` meant run in a debugger. It was easy to mix this
up.
This change:
- Centralizes where build type argument processing happens. Now it the
same shared decorator in CommandBase.
- Uses a `BuildType` enum instead of passing around two different
booleans. This reduces the error checking for situations where both
are true.
- Be much less clever about guessing what build to use. Now if you
don't specify a build type, `--dev` is chosen. I think this behavior
matches cargo.
- Makes it so that `./mach test-wpt` accepts the exact same arguments
and has the same behavior as other commands. In addition, the suite
correct for `test-wpt` is removed. There are only two suites now and
it's quite unlikely that people will confuse WPT tests for rust unit
tests.
Currently, `./mach test-wpt` family of commands (`test-wpt`,
`test-wpt-android`, and `test-wpt-failure`) and `./mach update-wpt`
default to using the legacy-layout option `--layout-2013` unless
`--layout-2020` is specified.
Given that we are now using layout-2020 by default, this change updates
these commands to default to using the `--layout-2020` option instead of
`--layout-2013`.
Much of the code used to import WPT tests from upstream has been moved
to the WPT repository itself, so this can be reused. In addition,
simplify the workflows by merging the entire process into mach and also
directly into the GitHub workflow. This should fix WPT imports after
combining compilation of layout and legacy layout.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Combine all script tests into `test-scripts`
Remove:
- tidy self test support from `./mach test`
- `./mach test-idl`
Adds a `./mach test-scripts` command that is responsible for running all
Python script tests. Run this during the CI to catch regressions in
changes to scripts. The WebIDL tests are still *very* slow and there are
from Gecko, so only run them when "-a" is passed meaning all tests.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] There are tests for these changes
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Remove:
- tidy self test support from `./mach test`
- `./mach test-idl`
Adds a `./mach test-scripts` command that is responsible for running all
Python script tests. Run this during the CI to catch regressions in
changes to scripts. The WebIDL tests are still *very* slow and there are
from Gecko, so only run them when "-a" is passed meaning all tests.
Before when a subtest failed, the text of the failed assertion was not
printed. This changes makes sure that it is printed in both the console
and the aggregated test output.
Also fix a couple typing errors.
This change moves all of Servo's WPT Python support scripts into one
directory as they were previously scattered throughout the directory
structure. This should allow more code reuse and make it easier to
understand how everything fits together.
The changes:
- `tests/wpt/update` → `python/wpt/importer`
- `etc/ci/upstream-wpt-changes/wptupstreamer` → `python/wpt/exporter`
- `etc/ci/upstream-wpt-changes/test.py` → `python/wpt/test.py`
- `etc/ci/upstream-wpt-changes/tests` → `python/wpt/tests`
- `tests/wpt/servowpt.py` →
- `python/wpt/update.py`
- `python/wpt/run.py`
- `tests/wpt/manifestupdate.py` → `python/wpt/manifestupdate.py`
This change also removes
- The ability to run the `update-wpt` and `test-wpt` commands without
using `mach`. These didn't work very well, because it was difficult
to get all of the wptrunner and mach dependencies installed outside
of the Python virtualenv. It's simpler if they are always run through
`mach`.
- The old WPT change upstreaming script that was no longer used.
