Update web-platform-tests to revision e45156b5e558c062a609356905c83a0258c516e3

This commit is contained in:
WPT Sync Bot 2019-05-02 21:47:51 -04:00
parent 9f6005be16
commit 5fcf52d946
199 changed files with 4430 additions and 530 deletions

View file

@ -11,43 +11,48 @@ There's also a load of [general guidelines][] that apply to all tests.
## Test Type
There are four main test types:
Tests in this project use a few different approaches to verify expected
behavior. The tests can be classified based on the way they express
expectations:
* [Reftests][] should be used to test rendering and layout. They
consist of two or more pages with assertions as to whether they
render identically or not.
* Rendering tests should be used to verify that the browser graphically
displays pages as expected. See the [rendering test guidelines][rendering]
for tips on how to write great rendering tests. There are a few different
ways to write rendering tests:
* [testharness.js][] tests should be used (where
possible!) for testing everything else. They are built with the
testharness.js unit testing framework, and consist of assertions
written in JavaScript.
* [Reftests][] should be used to test rendering and layout. They consist of
two or more pages with assertions as to whether they render identically or
not.
* [Visual tests][visual] should be used for checking rendering where
there is a large number of conforming renderings such that reftests
are impractical. They consist of a page that renders to final state
at which point a screenshot can be taken and compared to an expected
rendering for that user agent on that platform.
* [Visual tests][visual] should be used for checking rendering where there is
a large number of conforming renderings such that reftests are impractical.
They consist of a page that renders to final state at which point a
screenshot can be taken and compared to an expected rendering for that user
agent on that platform.
* [Manual tests][manual] are used as a last resort for anything
that can't be tested using any of the above. They consist of a page
that needs manual interaction or verification of the final result.
* [testharness.js][] tests should be used (where possible!) for testing
everything else. They are built with the testharness.js unit testing
framework, and consist of assertions written in JavaScript.
In general, there is a strong preference towards the first two test
types (as they can be easily run without human interaction), so they
* [wdspec][] tests are written in Python using
[pytest](https://docs.pytest.org/en/latest/) and test [the WebDriver browser
automation protocol](https://w3c.github.io/webdriver/)
* [Manual tests][manual] are used as a last resort for anything that can't be
tested using any of the above. They consist of a page that needs manual
interaction or verification of the final result.
In general, there is a strong preference towards reftests and testharness.js
tests types (as they can be easily run without human interaction), so they
should be used in preference to the others even if it results in a
somewhat cumbersome test; there is a far weaker preference between the
first two, and it is at times advisable to use testharness.js tests
two test types, and it is at times advisable to use testharness.js tests
for things which would typically be tested using reftests but for
which it would be overly cumbersome.
See [file names][] for test types and features determined by the file names,
and [server features][] for advanced testing features.
In addition to the four main test types, there are also WebDriver
tests, which are used exclusively for testing the WebDriver protocol
itself. There is currently no documentation about these tests,
however.
## Submitting Tests
Once you've written tests, please submit them using
@ -58,9 +63,11 @@ make sure you run the [`lint` script][lint-tool] before opening a pull request!
[file names]: {{ site.baseurl }}{% link _writing-tests/file-names.md %}
[general guidelines]: {{ site.baseurl }}{% link _writing-tests/general-guidelines.md %}
[reftests]: {{ site.baseurl }}{% link _writing-tests/reftests.md %}
[rendering]: {{ site.baseurl }}{% link _writing-tests/rendering.md %}
[server features]: {{ site.baseurl }}{% link _writing-tests/server-features.md %}
[testharness.js]: {{ site.baseurl }}{% link _writing-tests/testharness.md %}
[visual]: {{ site.baseurl }}{% link _writing-tests/visual.md %}
[manual]: {{ site.baseurl }}{% link _writing-tests/manual.md %}
[submission-process]: {{ site.baseurl }}{% link _writing-tests/submission-process.md %}
[lint-tool]: {{ site.baseurl }}{% link _writing-tests/lint-tool.md %}
[wdspec]: {{ site.baseurl }}{% link _writing-tests/wdspec.md %}

View file

@ -52,7 +52,7 @@ below to fix all errors reported.
the `<meta name="timeout"...>` element to precede the `script` element.
* **LAYOUTTESTS APIS**: Test file uses `eventSender`, `testRunner`, or
`window.internals` which are LayoutTests-specific APIs used in WebKit/Blink.
`internals` which are LayoutTests-specific APIs used in WebKit/Blink.
* **MALFORMED-VARIANT**: Test file with a `<meta name='variant'...>`
element whose `content` attribute has a malformed value; **fix**: ensure

View file

@ -0,0 +1,75 @@
---
layout: page
title: wdspec tests
order: -1
---
The term "wdspec" describes a type of test in WPT which verifies some aspect of
[the WebDriver protocol](https://w3c.github.io/webdriver/). These tests are
written in [the Python programming language](https://www.python.org/) and
structured with [the pytest testing
framework](https://docs.pytest.org/en/latest/).
The test files are organized into subdirectories based on the WebDriver
command under test. For example, tests for [the Close Window
command](https://w3c.github.io/webdriver/#close-window) are located in then
`close_window` directory.
Similar to [testharness.js][] tests, wdspec tests contain within them any
number of "sub-tests." Sub-tests are defined as Python functions whose name
begins with `test_`, e.g. `test_stale_element`.
## The `webdriver` client library
web-platform-tests maintains a WebDriver client library called `webdriver`
located in the `tools/webdriver/` directory. Like other client libraries, it
makes it easier to write code which interfaces with a browser using the
protocol.
Many tests require some "set up" code--logic intended to bring the browser to a
known state from which the expected behavior can be verified. The convenience
methods in the `webdriver` library **should** be used to perform this task
because they reduce duplication.
However, the same methods **should not** be used to issue the command under
test. Instead, the HTTP request describing the command should be sent directly.
This practice promotes the descriptive quality of the tests and limits
indirection that tends to obfuscate test failures.
Here is an example of a test for [the Element Click
command](https://w3c.github.io/webdriver/#element-click):
```python
from tests.support.asserts import assert_success
from tests.support.inline import inline
def test_null_response_value(session):
# The high-level API is used to set up a document and locate a click target
session.url = inline("<p>foo")
element = session.find.css("p", all=False)
# An HTTP request is explicitly constructed for the "click" command itself
response = session.transport.send(
"POST", "session/{session_id}/element/{element_id}/click".format(
session_id=session.session_id,
element_id=element.id))
assert_success(response)
```
## Utility functions
The `wedbdriver` library is minimal by design. It mimics the structure of the
WebDriver specification. Many conformance tests perform similar operations
(e.g. calculating the center point of an element or creating a document), but
the library does not expose methods to facilitate them. Instead, wdspec tests
define shared functionality in the form of "support" files.
Many of these functions are intended to be used directly from the tests using
Python's built-in `import` keyword. Others (particularly those that operate on
a WebDriver session) are defined in terms of Pytest "fixtures" and must be
loaded accordingly. For more detail on how to define and use test fixtures,
please refer to [the pytest project's documentation on the
topic](https://docs.pytest.org/en/latest/fixture.html).
[testharness.js]: {{ site.baseurl }}{% link _writing-tests/testharness.md %}

View file

@ -58,27 +58,32 @@ the filesystem, and is preferred for larger specifications.
## Test Types
The testsuite has a few types of tests, outlined below:
Tests in this project use a few different approaches to verify expected
behavior. The tests can be classified based on the way they express
expectations:
* [testharness.js][] tests, which are run
through a JS harness and report their result back with JS.
* Rendering tests ensure that the browser graphically displays pages as
expected. There are a few different ways this is done:
* [Reftests][], which render two (or more) web
pages and combine them with equality assertions about their
rendering (e.g., `A.html` and `B.html` must render identically), run
either by the user switching between tabs/windows and trying to
observe differences or through automated scripts.
* [Reftests][] render two (or more) web pages and combine them with equality
assertions about their rendering (e.g., `A.html` and `B.html` must render
identically), run either by the user switching between tabs/windows and
trying to observe differences or through [automated
scripts][running-from-local-system].
* [Visual tests][visual] which display a page where the
result is determined either by a human looking at it or by comparing
it with a saved screenshot for that user agent on that platform.
* [Visual tests][visual] display a page where the result is determined either
by a human looking at it or by comparing it with a saved screenshot for
that user agent on that platform.
* [Manual tests][manual], which rely on a human to run
them and determine their result.
* [testharness.js][] tests verify that JavaScript interfaces behave as
expected. They get their name from the JavaScript harness that's used to
execute them.
* WebDriver tests, which are used for testing the WebDriver protocol
itself.
* [wdspec]() tests are written in Python and test [the WebDriver browser
automation protocol](https://w3c.github.io/webdriver/)
* [Manual tests][manual] rely on a human to run them and determine their
result.
## GitHub
@ -108,3 +113,5 @@ free to add yourself to the META.yml file!
[visual]: {{ site.baseurl }}{% link _writing-tests/visual.md %}
[manual]: {{ site.baseurl }}{% link _writing-tests/manual.md %}
[github-intro]: {{ site.baseurl }}{% link _appendix/github-intro.md %}
[running-from-local-system]: {{ site.baseurl}}{% link _running-tests/from-local-system.md %}
[wdspec]: {{ site.baseurl }}{% link _writing-tests/wdspec.md %}