mirror of
https://github.com/servo/servo.git
synced 2025-06-06 16:45:39 +00:00
Update in-tree docs to point to the new book (#32743)
* Update in-tree docs to point to the new book * Revive build setup section in README as quickstart guide * Apply feedback about titles
This commit is contained in:
Delan Azabani
GitHub
parent
72e6a1f007
commit
34d9be70f9
12 changed files with 63 additions and 1413 deletions
248
tests/wpt/README.md
vendored
248
tests/wpt/README.md
vendored
|
|
@ -1,246 +1,6 @@
|
|||
This folder contains the web platform tests and the
|
||||
code required to integrate them with Servo.
|
||||
To learn how to write tests, go [here](http://web-platform-tests.org/writing-tests/index.html).
|
||||
# Web tests (including [Web Platform Tests](https://web-platform-tests.org))
|
||||
|
||||
Contents
|
||||
========
|
||||
Moved to the Servo book:
|
||||
|
||||
In particular, this folder contains:
|
||||
|
||||
* `config.ini`: some configuration for the web-platform-tests.
|
||||
* `include.ini`: the subset of web-platform-tests we currently run.
|
||||
* `tests`: copy of the web-platform-tests.
|
||||
* `meta`: expected failures for the web-platform-tests we run.
|
||||
* `mozilla`: web-platform-tests that cannot be upstreamed.
|
||||
|
||||
Running the tests
|
||||
=================
|
||||
|
||||
The simplest way to run the web-platform-tests in Servo is `./mach
|
||||
test-wpt` in the root directory. This will run the subset of
|
||||
JavaScript tests defined in `include.ini` and log the output to
|
||||
stdout.
|
||||
|
||||
A subset of tests may be run by providing positional arguments to the
|
||||
mach command, either as filesystem paths or as test urls e.g.
|
||||
|
||||
./mach test-wpt tests/wpt/tests/dom/historical.html
|
||||
|
||||
to run the dom/historical.html test, or
|
||||
|
||||
./mach test-wpt dom
|
||||
|
||||
to run all the DOM tests.
|
||||
|
||||
There are also a large number of command line options accepted by the
|
||||
test harness; these are documented by running with `--help`.
|
||||
|
||||
Running all tests
|
||||
------------------------------
|
||||
|
||||
Running all the WPT tests with debug mode results in a lot of timeout.
|
||||
If one wants to run all the tests,
|
||||
build with `mach build -r`
|
||||
and
|
||||
test with `mach test-wpt --release`
|
||||
|
||||
Running the tests with an external WPT server
|
||||
---------------------------------------------
|
||||
|
||||
Normally wptrunner starts its own WPT server, but occasionally you
|
||||
might want to run multiple instances of `mach test-wpt`, such as when
|
||||
debugging one test while running the full suite in the background, or
|
||||
when running a single test many times in parallel (--processes only
|
||||
works across different tests).
|
||||
|
||||
This would lead to a “Failed to start HTTP server” errors, because you
|
||||
can only run one WPT server at a time. To fix this:
|
||||
|
||||
1. Follow the steps in [**Running the tests manually**](#running-the-tests-manually)
|
||||
2. Add a `break` to [start_servers in serve.py](https://github.com/servo/servo/blob/ce92b7bfbd5855aac18cb4f8a8ec59048041712e/tests/wpt/web-platform-tests/tools/serve/serve.py#L745-L783) as follows:
|
||||
```
|
||||
--- a/tests/wpt/tests/tools/serve/serve.py
|
||||
+++ b/tests/wpt/tests/tools/serve/serve.py
|
||||
@@ -746,6 +746,7 @@ def start_servers(logger, host, ports, paths, routes, bind_address, config,
|
||||
mp_context, log_handlers, **kwargs):
|
||||
servers = defaultdict(list)
|
||||
for scheme, ports in ports.items():
|
||||
+ break
|
||||
assert len(ports) == {"http": 2, "https": 2}.get(scheme, 1)
|
||||
|
||||
# If trying to start HTTP/2.0 server, check compatibility
|
||||
```
|
||||
3. Run `mach test-wpt` as many times as needed
|
||||
|
||||
If you get unexpected TIMEOUT in testharness tests, then the custom
|
||||
testharnessreport.js may have been installed incorrectly (see
|
||||
[**Running the tests manually**](#running-the-tests-manually)
|
||||
for more details).
|
||||
|
||||
|
||||
Running the tests manually
|
||||
--------------------------
|
||||
|
||||
(See also [the relevant section of the upstream README][upstream-running].)
|
||||
|
||||
It can be useful to run a test without the interference of the test runner, for
|
||||
example when using a debugger such as `gdb`. To do this, we need to start the
|
||||
WPT server manually, which requires some extra configuration.
|
||||
|
||||
To do this, first add the following to the system's hosts file:
|
||||
|
||||
127.0.0.1 www.web-platform.test
|
||||
127.0.0.1 www1.web-platform.test
|
||||
127.0.0.1 www2.web-platform.test
|
||||
127.0.0.1 web-platform.test
|
||||
127.0.0.1 xn--n8j6ds53lwwkrqhv28a.web-platform.test
|
||||
127.0.0.1 xn--lve-6lad.web-platform.test
|
||||
|
||||
Navigate to `tests/wpt/web-platform-tests` for the remainder of this section.
|
||||
|
||||
Normally wptrunner [installs Servo’s version of testharnessreport.js][environment],
|
||||
but when starting the WPT server manually, we get the default version, which
|
||||
won’t report test results correctly. To fix this:
|
||||
|
||||
1. Create a directory `local-resources`
|
||||
2. Copy `tools/wptrunner/wptrunner/testharnessreport-servo.js` to `local-resources/testharnessreport.js`
|
||||
3. Edit `local-resources/testharnessreport.js` to substitute the variables as follows:
|
||||
* `%(output)d`
|
||||
* → `1` if you want to play with the test interactively (≈ pause-after-test)
|
||||
* → `0` if you don’t care about that (though it’s also ok to use `1` always)
|
||||
* `%(debug)s` → `true`
|
||||
4. Create a `./config.json` as follows (see `tools/wave/config.default.json` for defaults):
|
||||
```
|
||||
{"aliases": [{
|
||||
"url-path": "/resources/testharnessreport.js",
|
||||
"local-dir": "local-resources"
|
||||
}]}
|
||||
```
|
||||
|
||||
[environment]: https://github.com/servo/servo/blob/ce92b7bfbd5855aac18cb4f8a8ec59048041712e/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/environment.py#L231-L237
|
||||
|
||||
Then start the server with `./wpt serve`. To check if `testharnessreport.js` was installed correctly:
|
||||
|
||||
* The standard output of `curl http://web-platform.test:8000/resources/testharnessreport.js`
|
||||
should look like [testharnessreport-servo.js], not like [the default testharnessreport.js]
|
||||
* The standard output of `target/release/servo http://web-platform.test:8000/css/css-pseudo/highlight-pseudos-computed.html`
|
||||
(or any [testharness test]) should contain lines starting with:
|
||||
* `TEST START`
|
||||
* `TEST STEP`
|
||||
* `TEST DONE`
|
||||
* `ALERT: RESULT:`
|
||||
|
||||
[testharnessreport-servo.js]: https://github.com/servo/servo/blob/ce92b7bfbd5855aac18cb4f8a8ec59048041712e/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/testharnessreport-servo.js
|
||||
[the default testharnessreport.js]: https://github.com/servo/servo/blob/ce92b7bfbd5855aac18cb4f8a8ec59048041712e/tests/wpt/web-platform-tests/resources/testharnessreport.js
|
||||
[testharness test]: http://web-platform-tests.org/writing-tests/testharness.html
|
||||
|
||||
To prevent browser SSL warnings when running HTTPS tests locally,
|
||||
you will need to run Servo with `--certificate-path resources/cert-wpt-only`.
|
||||
|
||||
[upstream-running]: https://github.com/w3c/web-platform-tests#running-the-tests
|
||||
|
||||
Running the tests in Firefox
|
||||
----------------------------
|
||||
|
||||
When working with tests, you may want to compare Servo's result with Firefox.
|
||||
You can supply `--product firefox` along with the path to a Firefox binary (as
|
||||
well as few more odds and ends) to run tests in Firefox from your Servo
|
||||
checkout:
|
||||
|
||||
GECKO="$HOME/projects/mozilla/gecko"
|
||||
GECKO_BINS="$GECKO/obj-firefox-release-artifact/dist/Nightly.app/Contents/MacOS"
|
||||
./mach test-wpt dom --product firefox --binary $GECKO_BINS/firefox --certutil-binary $GECKO_BINS/certutil --prefs-root $GECKO/testing/profiles
|
||||
|
||||
Updating test expectations
|
||||
==========================
|
||||
|
||||
When fixing a bug that causes the result of a test to change, the expected
|
||||
results for that test need to be changed. This can be done manually, by editing
|
||||
the `.ini` file under the `meta` folder that corresponds to the test. In
|
||||
this case, remove the references to tests whose expectation is now `PASS`, and
|
||||
remove `.ini` files that no longer contain any expectations.
|
||||
|
||||
When a larger number of changes is required, this process can be automated.
|
||||
This first requires saving the raw, unformatted log from a test run, for
|
||||
example by running:
|
||||
|
||||
./mach test-wpt --log-raw /tmp/servo.log
|
||||
|
||||
Once the log is saved, run from the root directory:
|
||||
|
||||
./mach update-wpt /tmp/servo.log
|
||||
|
||||
The same process can be done for the legacy layout engine:
|
||||
|
||||
./mach test-wpt --legacy-layout --log-raw /tmp/servo-legacy.log
|
||||
./mach update-wpt --legacy-layout /tmp/servo-legacy.log
|
||||
|
||||
The updated expectations will be in `tests/wpt/meta/` and
|
||||
`tests/wpt/meta-legacy-layout/` respectively.
|
||||
|
||||
Writing new tests
|
||||
=================
|
||||
|
||||
The simplest way to create a new test is to use the following command:
|
||||
|
||||
./mach create-wpt tests/wpt/path/to/new/test.html
|
||||
|
||||
This will create test.html in the appropriate directory using the WPT
|
||||
template for JavaScript tests. Tests are written using [testharness.js](https://github.com/w3c/testharness.js/). Documentation can be found [here](http://testthewebforward.org/docs/testharness-library.html).
|
||||
To create a new reference test instead, use the following:
|
||||
|
||||
./mach create-wpt --reftest tests/wpt/path/to/new/reftest.html --reference tests/wpt/path/to/reference.html
|
||||
|
||||
`reference.html` will be created if it does not exist, and `reftest.html`
|
||||
will be created using the WPT reftest template. To know more about reftests, check [this](http://web-platform-tests.org/writing-tests/reftests.html).
|
||||
These new tests can then be run in the following manner like any other WPT test:
|
||||
|
||||
./mach test-wpt tests/wpt/path/to/new/test.html
|
||||
./mach test-wpt tests/wpt/path/to/new/reftest.html
|
||||
|
||||
|
||||
|
||||
Editing tests
|
||||
=============
|
||||
|
||||
web-platform-tests may be edited in-place and the changes committed to
|
||||
the servo tree. These changes will be upstreamed when the tests are
|
||||
next synced.
|
||||
|
||||
Servo-specific tests
|
||||
====================
|
||||
|
||||
The `mozilla` directory contains tests that cannot be upstreamed for some
|
||||
reason (e.g. because they depend on Servo-specific APIs), as well as some
|
||||
legacy tests that should be upstreamed at some point. When run they are
|
||||
mounted on the server under `/_mozilla/`.
|
||||
|
||||
Analyzing reftest results
|
||||
=========================
|
||||
|
||||
Reftest results can be analyzed from a raw log file. To generate this run
|
||||
with the `--log-raw` option e.g.
|
||||
|
||||
./mach test-wpt --log-raw wpt.log
|
||||
|
||||
This file can then be fed into the
|
||||
[reftest analyzer](https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer-structured.xhtml)
|
||||
which will show all failing tests (not just those with unexpected results).
|
||||
Note that this ingests logs in a different format to [original version of the
|
||||
tool](https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml)
|
||||
written for gecko reftests.
|
||||
|
||||
The reftest analyzer allows pixel-level comparison of the test and reference
|
||||
screenshots. Tests that both fail and have an unexpected result are marked
|
||||
with a `!`.
|
||||
|
||||
Updating the WPT manifest
|
||||
=========================
|
||||
|
||||
MANIFEST.json can be regenerated automatically with the mach command `update-manifest` e.g.
|
||||
|
||||
./mach update-manifest
|
||||
|
||||
This is equivalent to running
|
||||
|
||||
./mach test-wpt --manifest-update SKIP_TESTS
|
||||
- Hacking on Servo
|
||||
- [Automated testing](https://book.servo.org/hacking/testing.html)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue