Mirrors/servo
1
0
Fork 0
mirror of https://github.com/servo/servo.git synced 2025-08-12 00:45:33 +01:00
Code Issues Projects Releases Packages Wiki Activity

Update web-platform-tests to revision 097043b336e46876e281ddec3bb014fe9c480128

Browse source
This commit is contained in:
WPT Sync Bot 2019-08-03 10:25:42 +00:00
parent ecd32570c0
commit b68253eac0
405 changed files with 9164 additions and 3050 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
tests/wpt/web-platform-tests/docs/assets/reftest-tutorial-test-screenshot.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 20 KiB

16
tests/wpt/web-platform-tests/docs/running-tests/chrome.md
View file

@ -3,12 +3,16 @@
When running Chrome, there are some useful command line arguments.
You can inform `wpt` of the release channel of Chrome using `--channel`.
However, `wpt` currently does not support installing Chrome or finding the
Chrome binary of a specific channel, so you would also need to specify the path
to the Chrome binary with `--binary`. For example, to run Chrome Dev on Linux:
`wpt` is able to find the correct binary in the following cases:
* On Linux for stable, beta and dev channels if
`google-chrome-{stable,beta,unstable}` are in `PATH`;
* On Mac for stable and canary channels if the official DMGs are installed.
```
./wpt run --channel dev --binary `which google-chrome-unstable` chrome
In other cases, you will need to specify the path to the Chrome binary with
`--binary`. For example:
```bash
./wpt run --channel dev --binary /path/to/non-default/google-chrome chrome
```
Note: when the channel is "dev", `wpt` will *automatically* enable all
@ -19,7 +23,7 @@ Note: when the channel is "dev", `wpt` will *automatically* enable all
If you want to enable a specific [runtime enabled feature][1], use
`--binary-arg` to specify the flag(s) that you want to pass to Chrome:
```
```bash
./wpt run --binary-arg=--enable-blink-features=AsyncClipboard chrome clipboard-apis/
```

38
tests/wpt/web-platform-tests/docs/running-tests/chrome_android.md
View file

@ -2,35 +2,21 @@
To run WPT on Chrome on an Android device, some additional set up is required.
First of all, as usual Android development, we need to have `adb` and be able to
connect to the device.
As with usual Android development, you need to have `adb` and be able to
connect to the device. Run `adb devices` to verify.
## Hosts
Currently, Android support is a prototype with some known issues:
Until we find a better way, we need to root the Android device and update the
/etc/hosts file to include the entries printed by `./wpt make-hosts-file`.
* If you have previously run `./wpt run` against Chrome, you might need to
remove `_venv/bin/chromedriver` so that we can install the correct
ChromeDriver corresponding to your Chrome for Android version.
* We do not support reftests at the moment.
* You will need to manually kill Chrome (all channels) before running tests.
## CA certificate
Note: rooting the device or installing a root CA is no longer required.
In order to run HTTPS tests, we need to add
[WPT's CA](https://github.com/web-platform-tests/wpt/blob/master/tools/certs/cacert.pem)
to the phone. First, convert the certificate from PEM to CRT:
Example (assuming you have Chrome Canary installed on your phone):
```
openssl x509 -outform der -in tools/certs/cacert.pem -out cacert.crt
```
Then copy `cacert.crt` to your phone's external storage (preferably to
Downloads/ as it'll be easier to find). Open Settings -> Security & location ->
Encryption & credentials -> Install from storage. Find and install `cacert.crt`.
(The setting entries might be slightly different based your Android version.)
Note that having this CA installed on your device outside of a test
environment represents a security risk.
Finally, we may run wpt with the `chrome_android` product
```
./wpt run chrome_android [test_list]
```bash
./wpt run --test-type=testharness --channel=canary chrome_android TESTS
```

1
tests/wpt/web-platform-tests/docs/writing-tests/index.md
View file

@ -18,6 +18,7 @@ There's also a load of [general guidelines](general-guidelines) that apply to al
h2tests
lint-tool
manual
reftest-tutorial
reftests
rendering
server-features

276
tests/wpt/web-platform-tests/docs/writing-tests/reftest-tutorial.md Normal file
View file

@ -0,0 +1,276 @@
# Writing a reftest
<!--
Note to maintainers:
This tutorial is designed to be an authentic depiction of the WPT contribution
experience. It is not intended to be comprehensive; its scope is intentionally
limited in order to demonstrate authoring a complete test without overwhelming
the reader with features. Because typical WPT usage patterns change over time,
this should be updated periodically; please weigh extensions against the
demotivating effect that a lengthy guide can have on new contributors.
-->
Let's say you've discovered that WPT doesn't have any tests for the `dir`
attribute of [the `<bdo>`
element](https://developer.mozilla.org/en-US/docs/Web/HTML/Element/bdo). This
tutorial will guide you through the process of writing and submitting a test.
You'll need to [configure your system to use WPT's
tools](../running-tests/from-local-system), but you won't need them until
towards the end of this tutorial. Although it includes some very brief
instructions on using git, you can find more guidance in [the tutorial for git
and GitHub](../appendix/github-intro).
WPT's reftests are great for testing web-platform features that have some
visual effect. [The reftests reference page](reftests) describes them in the
abstract, but for the purposes of this guide, we'll only consider the features
we need to test the `<bdo>` element.
```eval_rst
.. contents::
:local:
```
## Setting up your workspace
To make sure you have the latest code, first type the following into a terminal
located in the root of the WPT git repository:
$ git fetch git@github.com:web-platform-tests/wpt.git
Next, we need a place to store the change set we're about to author. Here's how
to create a new git branch named `reftest-for-bdo` from the revision of WPT we
just downloaded:
$ git checkout -b reftest-for-bdo FETCH_HEAD
Now you're ready to create your patch.
## Writing the test file
First, we'll create a file that demonstrates the "feature under test." That is:
we'll write an HTML document that displays some text using a `<bdo>` element.
WPT has thousands of tests, so it can be daunting to decide where to put a new
one. Generally speaking, [test files should be placed in directories
corresponding to the specification text they are
verifying](../test-suite-design). `<bdo>` is defined in [the "text-level
semantics" chapter of the HTML
specification](https://html.spec.whatwg.org/multipage/text-level-semantics.html),
so we'll want to create our new test in the directory
`html/semantics/text-level-semantics/the-bdo-element/`. Create a file named
`rtl.html` and open it in your text editor.
Here's one way to demonstrate the feature:
```html
<!DOCTYPE html>
<meta charset="utf-8">
<title>BDO element dir=rtl</title>
<link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element">
<meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'.">
<p>Test passes if WAS is displayed below.</p>
<bdo dir="rtl">SAW</bdo>
```
That's pretty dense! Let's break it down:
- ```html
<!DOCTYPE html>
<meta charset="utf-8">
```
We explicitly set the DOCTYPE and character set to be sure that browsers
don't infer them to be something we aren't expecting. We're omitting the
`<html>` and `<head>` tags. That's a common practice in WPT, preferred
because it makes tests more concise.
- ```html
<title>BDO element dir=rtl</title>
```
The document's title should succinctly describe the feature under test.
- ```html
<link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element">
```
The "help" metadata should reference the specification under test so that
everyone understands the motivation. This is so helpful that [the CSS Working
Group requires it for CSS tests](css-metadata)! If you're writing a reftest
for a feature outside of CSS, feel free to omit this tag.
- ```html
<meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'.">
```
The "assert" metadata is a structured way for you to describe exactly what
you want your reftest to verify. For a direct test like the one we're writing
here, it might seem a little superfluous. It's much more helpful for
more-involved tests where reviewers might need some help understanding your
intentions.
This tag is optional, so you can skip it if you think it's unnecessary. We
recommend using it for your first few tests since it may let reviewers give
you more helpful feedback. As you get more familiar with WPT and the
specifications, you'll get a sense for when and where it's better to leave it
out.
- ```html
<p>Test passes if WAS is displayed below.</p>
```
We're communicating the "pass" condition in plain English to make the test
self-describing.
- ```html
<bdo dir="rtl">SAW</bdo>
```
This is the real focus of the test. We're including some text inside a
`<bdo>` element in order to demonstrate the feature under test.
Since this page doesn't rely on any [special WPT server
features](server-features), we can view it by loading the HTML file directly.
There are a bunch of ways to do this; one is to navigate to the
`html/semantics/text-level-semantics/the-bdo-element/` directory in a file
browser and drag the new `rtl.html` file into an open web browser window.
![](/assets/reftest-tutorial-test-screenshot.png "screen shot of the new test")
Sighted people can open that document and verify whether or not the stated
expectation is satisfied. If we were writing a [manual test](manual), we'd be
done. However, it's time-consuming for a human to run tests, so we should
prefer making tests automatic whenever possible. Remember that we set out to
write a "reference test." Now it's time to write the reference file.
## Writing a "match" reference
The "match" reference file describes what the test file is supposed to look
like. Critically, it *must not* use the technology that we are testing. The
reference file is what allows the test to be run by a computer--the computer
can verify that each pixel in the test document exactly matches the
corresponding pixel in the reference document.
Make a new file in the same
`html/semantics/text-level-semantics/the-bdo-element/` directory named
`rtl-ref.html`, and save the following markup into it:
```html
<!DOCTYPE html>
<meta charset="utf-8">
<title>BDO element dir=rtl reference</title>
<p>Test passes if WAS is displayed below.</p>
<p>WAS</p>
```
This is like a stripped-down version of the test file. In order to produce a
visual rendering which is the same as the expected rendering, it uses a `<p>`
element whose contents is the characters in right-to-left order. That way, if
the browser doesn't support the `<bdo>` element, this file will still show text
in the correct sequence.
This file is also completely functional without the WPT server, so you can open
it in a browser directly from your hard drive.
Currently, there's no way for a human operator or an automated script to know
that the two files we've created are supposed to match visually. We'll need to
add one more piece of metadata to the test file we created earlier. Open
`html/semantics/text-level-semantics/the-bdo-element/rtl.html` in your text
editor and add another `<link>` tag as described by the following change
summary:
```diff
<!DOCTYPE html>
<meta charset="utf-8">
<title>BDO element dir=rtl</title>
<link rel="author" title="Sam Smith" href="mailto:sam@example.com">
<link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element">
+<link rel="match" href="rtl-ref.html">
<meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'.">
<p>Test passes if WAS is displayed below.</p>
<bdo dir="rtl">SAW</bdo>
```
Now, anyone (human or computer) reviewing the test file will know where to find
the associated reference file.
## Verifying our work
We're done writing the test, but we should make sure it fits in with the rest
of WPT before we submit it. This involves using some of the project's tools, so
this is the point you'll need to [configure your system to run
WPT](../running-tests/from-local-system).
[The lint tool](lint-tool) can detect some of the common mistakes people make
when contributing to WPT. To run it, open a command-line terminal, navigate to
the root of the WPT repository, and enter the following command:
python ./wpt lint html/semantics/text-level-semantics/the-bdo-element
If this recognizes any of those common mistakes in the new files, it will tell
you where they are and how to fix them. If you do have changes to make, you can
run the command again to make sure you got them right.
Now, we'll run the test using the automated pixel-by-pixel comparison approach
mentioned earlier. This is important for reftests because the test and the
reference may differ in very subtle ways that are hard to catch with the naked
eye. That's not to say your test has to pass in all browsers (or even in *any*
browser). But if we expect the test to pass, then running it this way will help
us catch other kinds of mistakes.
The tools support running the tests in many different browsers. We'll use
Firefox this time:
python ./wpt run firefox html/semantics/text-level-semantics/the-bdo-element/rtl.html
We expect this test to pass, so if it does, we're ready to submit it. If we
were testing a web platform feature that Firefox didn't support, we would
expect the test to fail instead.
There are a few problems to look out for in addition to passing/failing status.
The report will describe fewer tests than we expect if the test isn't run at
all. That's usually a sign of a formatting mistake, so you'll want to make sure
you've used the right file names and metadata. Separately, the web browser
might crash. That's often a sign of a browser bug, so you should consider
[reporting it to the browser's
maintainers](https://rachelandrew.co.uk/archives/2017/01/30/reporting-browser-bugs/)!
## Submitting the test
First, let's stage the new files for committing:
$ git add html/semantics/text-level-semantics/the-bdo-element/rtl.html
$ git add html/semantics/text-level-semantics/the-bdo-element/rtl-ref.html
We can make sure the commit has everything we want to submit (and nothing we
don't) by using `git diff`:
$ git diff --staged
On most systems, you can use the arrow keys to navigate through the changes,
and you can press the `q` key when you're done reviewing.
Next, we'll create a commit with the staged changes:
$ git commit -m '[html] Add test for the `<bdo>` element'
And now we can push the commit to our fork of WPT:
$ git push origin reftest-for-bdo
The last step is to submit the test for review. WPT doesn't actually need the
test we wrote in this tutorial, but if we wanted to submit it for inclusion in
the repository, we would create a pull request on GitHub. [The guide on git and
GitHub](../appendix/github-intro) has all the details on how to do that.
## More practice
Here are some ways you can keep experimenting with WPT using this test:
- Improve coverage by adding more tests for related behaviors (e.g. nested
`<bdo>` elements)
- Add another reference document which describes what the test should *not*
look like using [`rel=mismatch`](reftests)

45
tests/wpt/web-platform-tests/docs/writing-tests/reftests.md
View file

@ -3,7 +3,9 @@
Reftests are one of the primary tools for testing things relating to
rendering; they are made up of the test and one or more other pages
("references") with assertions as to whether they render identically
or not.
or not. This page describes their aspects exhaustively; [the tutorial
on writing a reftest](reftest-tutorial) offers a more limited but
grounded guide to the process.
## How to Run Reftests
@ -180,46 +182,5 @@ element works, it's possible to construct a reftest for underlining
a block element, by constructing a reference using underlines on a
```<span>``` that wraps all the content inside the block.
## Example Reftests
This example follows the recommended approach in being a
self-describing test as it has a simple statement on the page
describing how it should render to pass the tests.
### Test File
This test verifies that a right-to-left rendering of **SAW** within a
```<bdo>``` element displays as **WAS**.
```html
<!DOCTYPE html>
<meta charset="utf-8">
<title>BDO element dir=rtl</title>
<link rel="help" href="https://html.spec.whatwg.org/#the-bdo-element">
<meta name="assert" content="BDO element's DIR content attribute renders corrently given value of 'rtl'.">
<link rel="match" href="test-bdo-001.html">
<p>Pass if you see WAS displayed below.</p>
<bdo dir="rtl">SAW</bdo>
```
### Reference File
The reference file must look exactly like the test file,
except that the code behind it is different.
* All metadata is removed.
* The ```title``` need not match.
* The markup that created the actual test data is
different: here, the same effect is created with
very mundane, dependable technology.
```html
<!DOCTYPE html>
<meta charset="utf-8">
<title>HTML Reference File</title>
<p>Pass if you see WAS displayed below.</p>
<p>WAS</p>
```
[general guidelines]: general-guidelines
[rendering]: rendering

21
tests/wpt/web-platform-tests/docs/writing-tests/submission-process.md
View file

@ -38,6 +38,27 @@ Hop on to the [mailing list][public-test-infra] or [IRC][]
no need to announce your review request, as soon as you make a Pull Request
GitHub will inform interested parties.
## Previews
The website [wpt-submissions.live](http://wpt-submissions.live) exists to help
contributors demonstrate their proposed changes to others. If your pull request
is open and has the GitHub label `pull-request-has-preview`, then it will be
available at `http://wpt-submissions.live/{{pull request ID}}`, where "pull
request ID" is the numeric identifier for the pull request.
For example, a pull request at https://github.com/web-platform-tests/wpt/pull/3
has a pull request ID `3`. Once that has been assigned the
`pull-request-has-preview` label, then its contents can be viewed at
http://wpt-submissions.live/3.
If you are [a GitHub
collaborator](https://help.github.com/en/articles/permission-levels-for-a-user-account-repository)
on WPT, the label and the preview will be created automatically. Because the
WPT server will execute Python code in the mirrored submissions, previews are
not created automatically for non-collaborators. Collaborators are encouraged
to enable the preview by adding the label, provided they trust the authors not
to submit malicious code.
[repo]: https://github.com/web-platform-tests/wpt/
[github flow]: https://guides.github.com/introduction/flow/
[public-test-infra]: https://lists.w3.org/Archives/Public/public-test-infra/

9
tests/wpt/web-platform-tests/docs/writing-tests/testharness-api.md
View file

@ -158,7 +158,7 @@ promise_test(test_function, name, properties)
```
`test_function` is a function that receives a test as an argument. It must
return a promise. The test completes when the returned promise resolves. The
return a promise. The test completes when the returned promise settles. The
test fails if the returned promise rejects.
E.g.:
@ -184,7 +184,12 @@ Note that in the promise chain constructed in `test_function` assertions don't
need to be wrapped in `step` or `step_func` calls.
Unlike Asynchronous Tests, Promise Tests don't start running until after the
previous Promise Test finishes.
previous Promise Test finishes. [Under rare
circumstances](https://github.com/web-platform-tests/wpt/pull/17924), the next
test may begin to execute before the returned promise has settled. Use
[add_cleanup](#cleanup) to register any necessary cleanup actions such as
resetting global state that need to happen consistently before the next test
starts.
`promise_rejects` can be used to test Promises that need to reject: