mirror of
https://github.com/servo/servo.git
synced 2025-08-07 06:25:32 +01:00
Update web-platform-tests to revision e45156b5e558c062a609356905c83a0258c516e3
This commit is contained in:
parent
9f6005be16
commit
5fcf52d946
199 changed files with 4430 additions and 530 deletions
|
@ -11,43 +11,48 @@ There's also a load of [general guidelines][] that apply to all tests.
|
|||
|
||||
## Test Type
|
||||
|
||||
There are four main test types:
|
||||
Tests in this project use a few different approaches to verify expected
|
||||
behavior. The tests can be classified based on the way they express
|
||||
expectations:
|
||||
|
||||
* [Reftests][] should be used to test rendering and layout. They
|
||||
consist of two or more pages with assertions as to whether they
|
||||
render identically or not.
|
||||
* Rendering tests should be used to verify that the browser graphically
|
||||
displays pages as expected. See the [rendering test guidelines][rendering]
|
||||
for tips on how to write great rendering tests. There are a few different
|
||||
ways to write rendering tests:
|
||||
|
||||
* [testharness.js][] tests should be used (where
|
||||
possible!) for testing everything else. They are built with the
|
||||
testharness.js unit testing framework, and consist of assertions
|
||||
written in JavaScript.
|
||||
* [Reftests][] should be used to test rendering and layout. They consist of
|
||||
two or more pages with assertions as to whether they render identically or
|
||||
not.
|
||||
|
||||
* [Visual tests][visual] should be used for checking rendering where
|
||||
there is a large number of conforming renderings such that reftests
|
||||
are impractical. They consist of a page that renders to final state
|
||||
at which point a screenshot can be taken and compared to an expected
|
||||
rendering for that user agent on that platform.
|
||||
* [Visual tests][visual] should be used for checking rendering where there is
|
||||
a large number of conforming renderings such that reftests are impractical.
|
||||
They consist of a page that renders to final state at which point a
|
||||
screenshot can be taken and compared to an expected rendering for that user
|
||||
agent on that platform.
|
||||
|
||||
* [Manual tests][manual] are used as a last resort for anything
|
||||
that can't be tested using any of the above. They consist of a page
|
||||
that needs manual interaction or verification of the final result.
|
||||
* [testharness.js][] tests should be used (where possible!) for testing
|
||||
everything else. They are built with the testharness.js unit testing
|
||||
framework, and consist of assertions written in JavaScript.
|
||||
|
||||
In general, there is a strong preference towards the first two test
|
||||
types (as they can be easily run without human interaction), so they
|
||||
* [wdspec][] tests are written in Python using
|
||||
[pytest](https://docs.pytest.org/en/latest/) and test [the WebDriver browser
|
||||
automation protocol](https://w3c.github.io/webdriver/)
|
||||
|
||||
* [Manual tests][manual] are used as a last resort for anything that can't be
|
||||
tested using any of the above. They consist of a page that needs manual
|
||||
interaction or verification of the final result.
|
||||
|
||||
In general, there is a strong preference towards reftests and testharness.js
|
||||
tests types (as they can be easily run without human interaction), so they
|
||||
should be used in preference to the others even if it results in a
|
||||
somewhat cumbersome test; there is a far weaker preference between the
|
||||
first two, and it is at times advisable to use testharness.js tests
|
||||
two test types, and it is at times advisable to use testharness.js tests
|
||||
for things which would typically be tested using reftests but for
|
||||
which it would be overly cumbersome.
|
||||
|
||||
See [file names][] for test types and features determined by the file names,
|
||||
and [server features][] for advanced testing features.
|
||||
|
||||
In addition to the four main test types, there are also WebDriver
|
||||
tests, which are used exclusively for testing the WebDriver protocol
|
||||
itself. There is currently no documentation about these tests,
|
||||
however.
|
||||
|
||||
## Submitting Tests
|
||||
|
||||
Once you've written tests, please submit them using
|
||||
|
@ -58,9 +63,11 @@ make sure you run the [`lint` script][lint-tool] before opening a pull request!
|
|||
[file names]: {{ site.baseurl }}{% link _writing-tests/file-names.md %}
|
||||
[general guidelines]: {{ site.baseurl }}{% link _writing-tests/general-guidelines.md %}
|
||||
[reftests]: {{ site.baseurl }}{% link _writing-tests/reftests.md %}
|
||||
[rendering]: {{ site.baseurl }}{% link _writing-tests/rendering.md %}
|
||||
[server features]: {{ site.baseurl }}{% link _writing-tests/server-features.md %}
|
||||
[testharness.js]: {{ site.baseurl }}{% link _writing-tests/testharness.md %}
|
||||
[visual]: {{ site.baseurl }}{% link _writing-tests/visual.md %}
|
||||
[manual]: {{ site.baseurl }}{% link _writing-tests/manual.md %}
|
||||
[submission-process]: {{ site.baseurl }}{% link _writing-tests/submission-process.md %}
|
||||
[lint-tool]: {{ site.baseurl }}{% link _writing-tests/lint-tool.md %}
|
||||
[wdspec]: {{ site.baseurl }}{% link _writing-tests/wdspec.md %}
|
||||
|
|
|
@ -52,7 +52,7 @@ below to fix all errors reported.
|
|||
the `<meta name="timeout"...>` element to precede the `script` element.
|
||||
|
||||
* **LAYOUTTESTS APIS**: Test file uses `eventSender`, `testRunner`, or
|
||||
`window.internals` which are LayoutTests-specific APIs used in WebKit/Blink.
|
||||
`internals` which are LayoutTests-specific APIs used in WebKit/Blink.
|
||||
|
||||
* **MALFORMED-VARIANT**: Test file with a `<meta name='variant'...>`
|
||||
element whose `content` attribute has a malformed value; **fix**: ensure
|
||||
|
|
75
tests/wpt/web-platform-tests/docs/_writing-tests/wdspec.md
Normal file
75
tests/wpt/web-platform-tests/docs/_writing-tests/wdspec.md
Normal file
|
@ -0,0 +1,75 @@
|
|||
---
|
||||
layout: page
|
||||
title: wdspec tests
|
||||
order: -1
|
||||
---
|
||||
|
||||
The term "wdspec" describes a type of test in WPT which verifies some aspect of
|
||||
[the WebDriver protocol](https://w3c.github.io/webdriver/). These tests are
|
||||
written in [the Python programming language](https://www.python.org/) and
|
||||
structured with [the pytest testing
|
||||
framework](https://docs.pytest.org/en/latest/).
|
||||
|
||||
The test files are organized into subdirectories based on the WebDriver
|
||||
command under test. For example, tests for [the Close Window
|
||||
command](https://w3c.github.io/webdriver/#close-window) are located in then
|
||||
`close_window` directory.
|
||||
|
||||
Similar to [testharness.js][] tests, wdspec tests contain within them any
|
||||
number of "sub-tests." Sub-tests are defined as Python functions whose name
|
||||
begins with `test_`, e.g. `test_stale_element`.
|
||||
|
||||
## The `webdriver` client library
|
||||
|
||||
web-platform-tests maintains a WebDriver client library called `webdriver`
|
||||
located in the `tools/webdriver/` directory. Like other client libraries, it
|
||||
makes it easier to write code which interfaces with a browser using the
|
||||
protocol.
|
||||
|
||||
Many tests require some "set up" code--logic intended to bring the browser to a
|
||||
known state from which the expected behavior can be verified. The convenience
|
||||
methods in the `webdriver` library **should** be used to perform this task
|
||||
because they reduce duplication.
|
||||
|
||||
However, the same methods **should not** be used to issue the command under
|
||||
test. Instead, the HTTP request describing the command should be sent directly.
|
||||
This practice promotes the descriptive quality of the tests and limits
|
||||
indirection that tends to obfuscate test failures.
|
||||
|
||||
Here is an example of a test for [the Element Click
|
||||
command](https://w3c.github.io/webdriver/#element-click):
|
||||
|
||||
```python
|
||||
from tests.support.asserts import assert_success
|
||||
from tests.support.inline import inline
|
||||
|
||||
def test_null_response_value(session):
|
||||
# The high-level API is used to set up a document and locate a click target
|
||||
session.url = inline("<p>foo")
|
||||
element = session.find.css("p", all=False)
|
||||
|
||||
# An HTTP request is explicitly constructed for the "click" command itself
|
||||
response = session.transport.send(
|
||||
"POST", "session/{session_id}/element/{element_id}/click".format(
|
||||
session_id=session.session_id,
|
||||
element_id=element.id))
|
||||
|
||||
assert_success(response)
|
||||
```
|
||||
|
||||
## Utility functions
|
||||
|
||||
The `wedbdriver` library is minimal by design. It mimics the structure of the
|
||||
WebDriver specification. Many conformance tests perform similar operations
|
||||
(e.g. calculating the center point of an element or creating a document), but
|
||||
the library does not expose methods to facilitate them. Instead, wdspec tests
|
||||
define shared functionality in the form of "support" files.
|
||||
|
||||
Many of these functions are intended to be used directly from the tests using
|
||||
Python's built-in `import` keyword. Others (particularly those that operate on
|
||||
a WebDriver session) are defined in terms of Pytest "fixtures" and must be
|
||||
loaded accordingly. For more detail on how to define and use test fixtures,
|
||||
please refer to [the pytest project's documentation on the
|
||||
topic](https://docs.pytest.org/en/latest/fixture.html).
|
||||
|
||||
[testharness.js]: {{ site.baseurl }}{% link _writing-tests/testharness.md %}
|
|
@ -58,27 +58,32 @@ the filesystem, and is preferred for larger specifications.
|
|||
|
||||
## Test Types
|
||||
|
||||
The testsuite has a few types of tests, outlined below:
|
||||
Tests in this project use a few different approaches to verify expected
|
||||
behavior. The tests can be classified based on the way they express
|
||||
expectations:
|
||||
|
||||
* [testharness.js][] tests, which are run
|
||||
through a JS harness and report their result back with JS.
|
||||
* Rendering tests ensure that the browser graphically displays pages as
|
||||
expected. There are a few different ways this is done:
|
||||
|
||||
* [Reftests][], which render two (or more) web
|
||||
pages and combine them with equality assertions about their
|
||||
rendering (e.g., `A.html` and `B.html` must render identically), run
|
||||
either by the user switching between tabs/windows and trying to
|
||||
observe differences or through automated scripts.
|
||||
* [Reftests][] render two (or more) web pages and combine them with equality
|
||||
assertions about their rendering (e.g., `A.html` and `B.html` must render
|
||||
identically), run either by the user switching between tabs/windows and
|
||||
trying to observe differences or through [automated
|
||||
scripts][running-from-local-system].
|
||||
|
||||
* [Visual tests][visual] which display a page where the
|
||||
result is determined either by a human looking at it or by comparing
|
||||
it with a saved screenshot for that user agent on that platform.
|
||||
* [Visual tests][visual] display a page where the result is determined either
|
||||
by a human looking at it or by comparing it with a saved screenshot for
|
||||
that user agent on that platform.
|
||||
|
||||
* [Manual tests][manual], which rely on a human to run
|
||||
them and determine their result.
|
||||
* [testharness.js][] tests verify that JavaScript interfaces behave as
|
||||
expected. They get their name from the JavaScript harness that's used to
|
||||
execute them.
|
||||
|
||||
* WebDriver tests, which are used for testing the WebDriver protocol
|
||||
itself.
|
||||
* [wdspec]() tests are written in Python and test [the WebDriver browser
|
||||
automation protocol](https://w3c.github.io/webdriver/)
|
||||
|
||||
* [Manual tests][manual] rely on a human to run them and determine their
|
||||
result.
|
||||
|
||||
## GitHub
|
||||
|
||||
|
@ -108,3 +113,5 @@ free to add yourself to the META.yml file!
|
|||
[visual]: {{ site.baseurl }}{% link _writing-tests/visual.md %}
|
||||
[manual]: {{ site.baseurl }}{% link _writing-tests/manual.md %}
|
||||
[github-intro]: {{ site.baseurl }}{% link _appendix/github-intro.md %}
|
||||
[running-from-local-system]: {{ site.baseurl}}{% link _running-tests/from-local-system.md %}
|
||||
[wdspec]: {{ site.baseurl }}{% link _writing-tests/wdspec.md %}
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue