https://github.com/web-platform-tests/wpt/pull/50041 allows us to start
running the webdriver conformance tests in Servo, which will make it
easier for us to track regressions/improvements in our webdriver server
implementation.
---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes are part of #15274
- [x] There are tests for these changes
---------
Signed-off-by: Josh Matthews <josh@joshmatthews.net>
We were already not compiling it and not running tests on it by default.
So it's simpler to just completely remove it.
Signed-off-by: Oriol Brufau <obrufau@igalia.com>
The previous code was simply reporting the number of categories,
instead of the sum of the number of expected tests in each category.
Signed-off-by: Simon Wülker <simon.wuelker@arcor.de>
Before we appended all subdirs of unused_dir to unused_dirs, which caused errors on removing because root dir was already removed.
Fix#35026, tested locally.
Signed-off-by: Samson <16504129+sagudev@users.noreply.github.com>
wptrunner internally derives the path to the MANIFEST.json file from the
`metadata_path` passed via `test_paths`. The current logic overrides
only the `metadata_path`, leaving the manifest_path pointing at the
wrong (non-legacy) layout's MANIFEST.json.
In #34436 we observed that the recent WPT imports create a transient
`.cache` diretory that wptrunner logic uses for optimization. This has
not been observed until #34436 because a [recent bump in MANIFEST.json's
schema version][1] triggered the creation of the cache. Because of the
above issue with incorrect path and the fact that we *first* trigger
`mach update-wpt` for legacy layout during WPT import, the MANIFEST.json
of non-legacy layout gets incorrectly migrated during the invocation of
`update-wpt` for legacy layout but the cache is still created under
legacy-layout's path as it is not based on `manifest_path`. The
subsequent invocation of `mach update-wpt` for non-legacy finds the
MANIFEST.json already migrated so the `.cache` directory is not
constucted.
This change simply replaces the whole object using the wptrunner's
`TestRoot` class constructor so that all derived paths are calculated
correctly. We also add the `.cache` folders to gitignore as it seems
like they are expected to be created during such version migrations.
[1]: https://github.com/web-platform-tests/wpt/pull/49406
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
The GitHub search API is a bit sensitive. There isn't a great way to
search for the repository organization and the branch name when looking
for open PRs. Instead use the bot username as the author name, which
should likely have been having before. This fixes the WPT export job.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This switches the WPT export script to use Servo's fork the `wpt`
upstream repository. The reasoning is that Servo developers's can fix
issues with branches there. This isn't possible because no one has
permissions to the servo-wpt-sync fork of the repository.
I have given `servo-wpt-sync` write persmissions to `servo/wpt`.
This also updates the tests a bit to reflect the new default repository.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
This is a speculative fix for #31845. Instead of sending all of the
output to the dashboard, send just the first 1024 characters. This value
can be adjusted in the future if it is too large or too small.
Fixes#31845.
1. Make the tidy output easier to follow
2. Integrate the WPT manifest cleanliness step into tidy
itself and don't run it if nothing has changed in the WPT
directory.
3. Fix an issue where Python test requirements were not installed,
which could cause issues with some modules not being found.
Fixes#30002.
Upstream wpt script has bug where 'product' defaults to
'firefox' and this causes import of uninstalled python
modules specific to firefox runners.
Fixes#30452
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* Fix `mach test-wpt` to make crash tests work
There are two issues related to crash tests:
1. test-wpt is unable to find existing crash tests even when
called with --test-types=crashtests. The fix here is to
add crashtests to the default test suite types to python/wpt/run.py
2. When running in headless mode, crashes in style threads
don't cause servo to crash because the logic in constellation.rs
currently calls handle_panic only when the top-level browsing
context id is some value. Since style pool threads are shared,
they always generate Panic messages with None as top-level
browsing context id.
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* Send bactrace to stderr and capture it in test runner
Servo's panic hook writes backtraces to stdout. This
patch changes it so they are written to stderr.
The crash test executor for servo in WPT grouping formatter
was also not capturing the output correctly for crashtests
as the log events were being aggregated based on thread name
which doesn't seem to match correctly in case of crashtests.
This patch also fixes the log grouping logic to be based on
test name.
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* crashtests: update expectations for layout 2020
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* crashtests: update expectations for layout 2013
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* remove outdated & intemittent test expectations
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
---------
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Most tests were only being run for layout-2013, not for layout-2020.
This wasn't great since layout-2020 is now the default.
So this patch unifies the lists of included tests for both layouts.
For layout-2013 this implies adding css/css-content/, css/css-logical/
and css/css-masking/clip/.
For layout-2020 this implies adding several additional css tests, and
also tests like dom/, js/, html/, etc.
There were some issues with the way that the `--release` and `--dev`
arguments were handled in mach commands.
- Not all commands accepted them in the same way. For instance `./mach
test-wpt` didn't really accept them at all.
- If you did not pass either of them, mach would try to guess which
build you meant. This guess was often quite surprising as it wasn't
printed and it depended on the state of the your target directory,
which is difficult to remember.
- The `dev` profile is colloquially called a "debug" profile and some
commands accepted `-d` or `--debug...` like arguments, but `--debug`
with `./mach run` meant run in a debugger. It was easy to mix this
up.
This change:
- Centralizes where build type argument processing happens. Now it the
same shared decorator in CommandBase.
- Uses a `BuildType` enum instead of passing around two different
booleans. This reduces the error checking for situations where both
are true.
- Be much less clever about guessing what build to use. Now if you
don't specify a build type, `--dev` is chosen. I think this behavior
matches cargo.
- Makes it so that `./mach test-wpt` accepts the exact same arguments
and has the same behavior as other commands. In addition, the suite
correct for `test-wpt` is removed. There are only two suites now and
it's quite unlikely that people will confuse WPT tests for rust unit
tests.
Currently, `./mach test-wpt` family of commands (`test-wpt`,
`test-wpt-android`, and `test-wpt-failure`) and `./mach update-wpt`
default to using the legacy-layout option `--layout-2013` unless
`--layout-2020` is specified.
Given that we are now using layout-2020 by default, this change updates
these commands to default to using the `--layout-2020` option instead of
`--layout-2013`.
Much of the code used to import WPT tests from upstream has been moved
to the WPT repository itself, so this can be reused. In addition,
simplify the workflows by merging the entire process into mach and also
directly into the GitHub workflow. This should fix WPT imports after
combining compilation of layout and legacy layout.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
Combine all script tests into `test-scripts`
Remove:
- tidy self test support from `./mach test`
- `./mach test-idl`
Adds a `./mach test-scripts` command that is responsible for running all
Python script tests. Run this during the CI to catch regressions in
changes to scripts. The WebIDL tests are still *very* slow and there are
from Gecko, so only run them when "-a" is passed meaning all tests.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] There are tests for these changes
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Remove:
- tidy self test support from `./mach test`
- `./mach test-idl`
Adds a `./mach test-scripts` command that is responsible for running all
Python script tests. Run this during the CI to catch regressions in
changes to scripts. The WebIDL tests are still *very* slow and there are
from Gecko, so only run them when "-a" is passed meaning all tests.
Before when a subtest failed, the text of the failed assertion was not
printed. This changes makes sure that it is printed in both the console
and the aggregated test output.
Also fix a couple typing errors.
This change moves all of Servo's WPT Python support scripts into one
directory as they were previously scattered throughout the directory
structure. This should allow more code reuse and make it easier to
understand how everything fits together.
The changes:
- `tests/wpt/update` → `python/wpt/importer`
- `etc/ci/upstream-wpt-changes/wptupstreamer` → `python/wpt/exporter`
- `etc/ci/upstream-wpt-changes/test.py` → `python/wpt/test.py`
- `etc/ci/upstream-wpt-changes/tests` → `python/wpt/tests`
- `tests/wpt/servowpt.py` →
- `python/wpt/update.py`
- `python/wpt/run.py`
- `tests/wpt/manifestupdate.py` → `python/wpt/manifestupdate.py`
This change also removes
- The ability to run the `update-wpt` and `test-wpt` commands without
using `mach`. These didn't work very well, because it was difficult
to get all of the wptrunner and mach dependencies installed outside
of the Python virtualenv. It's simpler if they are always run through
`mach`.
- The old WPT change upstreaming script that was no longer used.
