Mirrors/servo
1
0
Fork 0
mirror of https://github.com/servo/servo.git synced 2025-08-09 23:45:35 +01:00
Code Issues Projects Releases Packages Wiki Activity

Update web-platform-tests to revision 0d318188757a9c996e20b82db201fd04de5aa255

Browse source
This commit is contained in:
James Graham 2015-03-27 09:15:38 +00:00
parent b2a5225831
commit 1a81b18b9f
12321 changed files with 544385 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

548
tests/wpt/web-platform-tests/resources/docs/api.md Normal file
View file

@ -0,0 +1,548 @@
## Introduction ##
testharness.js provides a framework for writing testcases. It is intended to
provide a convenient API for making common assertions, and to work both
for testing synchronous and asynchronous DOM features in a way that
promotes clear, robust, tests.
## Basic Usage ##
The test harness script can be used from HTML or SVG documents and web worker
scripts.
From an HTML or SVG document, start by importing both `testharness.js` and
`testharnessreport.js` scripts into the document:
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
Refer to the [Web Workers](#web-workers) section for details and an example on
testing within a web worker.
Within each file one may define one or more tests. Each test is atomic in the
sense that a single test has a single result (`PASS`/`FAIL`/`TIMEOUT`/`NOTRUN`).
Within each test one may have a number of asserts. The test fails at the first
failing assert, and the remainder of the test is (typically) not run.
If the file containing the tests is a HTML file, a table containing the test
results will be added to the document after all tests have run. By default this
will be added to a `div` element with `id=log` if it exists, or a new `div`
element appended to `document.body` if it does not.
NOTE: By default tests must be created before the load event fires. For ways
to create tests after the load event, see "Determining when all tests
are complete", below.
## Synchronous Tests ##
To create a synchronous test use the test() function:
test(test_function, name, properties)
`test_function` is a function that contains the code to test. For example a
trivial passing test would be:
test(function() {assert_true(true)}, "assert_true with true")
The function passed in is run in the `test()` call.
`properties` is a javascript object for passing extra options to the
test. Currently it is only used to provide test-specific
metadata, as described in the [metadata](#metadata) section below.
## Asynchronous Tests ##
Testing asynchronous features is somewhat more complex since the result of
a test may depend on one or more events or other callbacks. The API provided
for testing these features is indended to be rather low-level but hopefully
applicable to many situations.
To create a test, one starts by getting a Test object using async_test:
async_test(name, properties)
e.g.
var t = async_test("Simple async test")
Assertions can be added to the test by calling the step method of the test
object with a function containing the test assertions:
t.step(function() {assert_true(true)});
When all the steps are complete, the done() method must be called:
t.done();
As a convenience, async_test can also takes a function as first argument.
This function is called with the test object as both its `this` object and
first argument. The above example can be rewritten as:
async_test(function(t) {
object.some_event = function() {
t.step(function (){assert_true(true); t.done();});
};
}, "Simple async test");
which avoids cluttering the global scope with references to async
tests instances.
The properties argument is identical to that for `test()`.
In many cases it is convenient to run a step in response to an event or a
callback. A convenient method of doing this is through the step_func method
which returns a function that, when called runs a test step. For example
object.some_event = t.step_func(function(e) {assert_true(e.a)});
For asynchronous callbacks that should never execute, `unreached_func` can
be used. For example:
object.some_event = t.unreached_func("some_event should not fire");
## Promise Tests ##
`promise_test` can be used to test APIs that are based on Promises:
promise_test(test_function, name, properties)
`test_function` is a function that receives a test as an argument and returns a
promise. The test completes when the returned promise resolves. The test fails
if the returned promise rejects.
E.g.:
function foo() {
return Promise.resolve("foo");
}
promise_test(function() {
return foo()
.then(function(result) {
assert_equals(result, "foo", "foo should return 'foo'");
});
}, "Simple example");
In the example above, `foo()` returns a Promise that resolves with the string
"foo". The `test_function` passed into `promise_test` invokes `foo` and attaches
a resolve reaction that verifies the returned value.
Note that in the promise chain constructed in `test_function` assertions don't
need to wrapped in `step` or `step_func` calls.
`promise_rejects` can be used to test Promises that need to reject:
promise_rejects(test_object, code, promise)
The `code` argument is equivalent to the same argument to the `assert_throws`
function.
Here's an example where the `bar()` function returns a Promise that rejects
with a TypeError:
function bar() {
return Promise.reject(new TypeError());
}
promise_test(function(t) {
return promise_rejects(t, new TypeError(), bar);
}, "Another example");
## Single Page Tests ##
Sometimes, particularly when dealing with asynchronous behaviour,
having exactly one test per page is desirable, and the overhead of
wrapping everything in functions for isolation becomes
burdensome. For these cases `testharness.js` support "single page
tests".
In order for a test to be interpreted as a single page test, the
it must simply not call `test()` or `async_test()` anywhere on the page, and
must call the `done()` function to indicate that the test is complete. All
the `assert_*` functions are avaliable as normal, but are called without
the normal step function wrapper. For example:
<!doctype html>
<title>Example single-page test</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<body>
<script>
assert_equals(document.body, document.getElementsByTagName("body")[0])
done()
</script>
The test title for sinple page tests is always taken from `document.title`.
## Making assertions ##
Functions for making assertions start `assert_`. The full list of
asserts avaliable is documented in the [asserts](#asserts) section
below.. The general signature is
assert_something(actual, expected, description)
although not all assertions precisely match this pattern e.g. `assert_true`
only takes `actual` and `description` as arguments.
The description parameter is used to present more useful error messages when
a test fails
NOTE: All asserts must be located in a `test()` or a step of an
`async_test()`, unless the test is a single page test. Asserts outside
these places won't be detected correctly by the harness and may cause
unexpected exceptions that will lead to an error in the harness.
## Cleanup ##
Occasionally tests may create state that will persist beyond the test itself.
In order to ensure that tests are independent, such state should be cleaned
up once the test has a result. This can be achieved by adding cleanup
callbacks to the test. Such callbacks are registered using the `add_cleanup`
function on the test object. All registered callbacks will be run as soon as
the test result is known. For example
test(function() {
window.some_global = "example";
this.add_cleanup(function() {delete window.some_global});
assert_true(false);
});
## Harness Timeout ##
The overall harness admits two timeout values `"normal"` (the
default) and `"long"`, used for tests which have an unusually long
runtime. After the timeout is reached, the harness will stop
waiting for further async tests to complete. By default the
timeouts are set to 10s and 60s, respectively, but may be changed
when the test is run on hardware with different performance
characteristics to a common desktop computer. In order to opt-in
to the longer test timeout, the test must specify a meta element:
<meta name="timeout" content="long">
Occasionally tests may have a race between the harness timing out and
a particular test failing; typically when the test waits for some event
that never occurs. In this case it is possible to use `test.force_timeout()`
in place of `assert_unreached()`, to immediately fail the test but with a
status of `TIMEOUT`. This should only be used as a last resort when it is
not possible to make the test reliable in some other way.
## Setup ##
Sometimes tests require non-trivial setup that may fail. For this purpose
there is a `setup()` function, that may be called with one or two arguments.
The two argument version is:
setup(func, properties)
The one argument versions may omit either argument.
func is a function to be run synchronously. `setup()` becomes a no-op once
any tests have returned results. Properties are global properties of the test
harness. Currently recognised properties are:
`explicit_done` - Wait for an explicit call to done() before declaring all
tests complete (see below; implicitly true for single page tests)
`output_document` - The document to which results should be logged. By default
this is the current document but could be an ancestor document in some cases
e.g. a SVG test loaded in an HTML wrapper
`explicit_timeout` - disable file timeout; only stop waiting for results
when the `timeout()` function is called (typically for use when integrating
with some existing test framework that has its own timeout mechanism).
`allow_uncaught_exception` - don't treat an uncaught exception as an error;
needed when e.g. testing the `window.onerror` handler.
`timeout_multiplier` - Multiplier to apply to per-test timeouts.
## Determining when all tests are complete ##
By default the test harness will assume there are no more results to come
when:
1. There are no `Test` objects that have been created but not completed
2. The load event on the document has fired
This behaviour can be overridden by setting the `explicit_done` property to
true in a call to `setup()`. If `explicit_done` is true, the test harness will
not assume it is done until the global `done()` function is called. Once `done()`
is called, the two conditions above apply like normal.
Dedicated and shared workers don't have an event that corresponds to the `load`
event in a document. Therefore these worker tests always behave as if the
`explicit_done` property is set to true. Service workers depend on the
[install](https://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#service-worker-global-scope-install-event)
event which is fired following the completion of [running the
worker](https://html.spec.whatwg.org/multipage/workers.html#run-a-worker).
## Generating tests ##
There are scenarios in which is is desirable to create a large number of
(synchronous) tests that are internally similar but vary in the parameters
used. To make this easier, the `generate_tests` function allows a single
function to be called with each set of parameters in a list:
generate_tests(test_function, parameter_lists, properties)
For example:
generate_tests(assert_equals, [
["Sum one and one", 1+1, 2],
["Sum one and zero", 1+0, 1]
])
Is equivalent to:
test(function() {assert_equals(1+1, 2)}, "Sum one and one")
test(function() {assert_equals(1+0, 1)}, "Sum one and zero")
Note that the first item in each parameter list corresponds to the name of
the test.
The properties argument is identical to that for `test()`. This may be a
single object (used for all generated tests) or an array.
## Callback API ##
The framework provides callbacks corresponding to 4 events:
* `start` - triggered when the first Test is created
* `test_state` - triggered when a test state changes
* `result` - triggered when a test result is recieved
* `complete` - triggered when all results are recieved
The page defining the tests may add callbacks for these events by calling
the following methods:
`add_start_callback(callback)` - callback called with no arguments
`add_test_state_callback(callback)` - callback called with a test argument
`add_result_callback(callback)` - callback called with a test argument
`add_completion_callback(callback)` - callback called with an array of tests
and an status object
tests have the following properties:
* `status` - A status code. This can be compared to the `PASS`, `FAIL`,
`TIMEOUT` and `NOTRUN` properties on the test object
* `message` - A message indicating the reason for failure. In the future this
will always be a string
The status object gives the overall status of the harness. It has the
following properties:
* `status` - Can be compared to the `OK`, `ERROR` and `TIMEOUT` properties
* `message` - An error message set when the status is `ERROR`
## External API ##
In order to collect the results of multiple pages containing tests, the test
harness will, when loaded in a nested browsing context, attempt to call
certain functions in each ancestor and opener browsing context:
* start - `start_callback`
* test\_state - `test_state_callback`
* result - `result_callback`
* complete - `completion_callback`
These are given the same arguments as the corresponding internal callbacks
described above.
## External API through cross-document messaging ##
Where supported, the test harness will also send messages using cross-document
messaging to each ancestor and opener browsing context. Since it uses the
wildcard keyword (\*), cross-origin communication is enabled and script on
different origins can collect the results.
This API follows similar conventions as those described above only slightly
modified to accommodate message event API. Each message is sent by the harness
is passed a single vanilla object, available as the `data` property of the event
object. These objects are structures as follows:
* start - `{ type: "start" }`
* test\_state - `{ type: "test_state", test: Test }`
* result - `{ type: "result", test: Test }`
* complete - `{ type: "complete", tests: [Test, ...], status: TestsStatus }`
## Web Workers ##
The `testharness.js` script can be used from within [dedicated workers, shared
workers](https://html.spec.whatwg.org/multipage/workers.html) and [service
workers](https://slightlyoff.github.io/ServiceWorker/spec/service_worker/).
Testing from a worker script is different from testing from an HTML document in
several ways:
* Workers have no reporting capability since they are runing in the background.
Hence they rely on `testharness.js` running in a companion client HTML document
for reporting.
* Shared and service workers do not have a unique client document since there
could be more than one document that communicates with these workers. So a
client document needs to explicitly connect to a worker and fetch test results
from it using `fetch_tests_from_worker`. This is true even for a dedicated
worker. Once connected, the individual tests running in the worker (or those
that have already run to completion) will be automatically reflected in the
client document.
* The client document controls the timeout of the tests. All worker scripts act
as if they were started with the `explicit_timeout` option (see the [Harness
timeout](#harness-timeout) section).
* Dedicated and shared workers don't have an equivalent of an `onload` event.
Thus the test harness has no way to know when all tests have completed (see
[Determining when all tests are
complete](#determining-when-all-tests-are-complete)). So these worker tests
behave as if they were started with the `explicit_done` option. Service
workers depend on the
[oninstall](https://slightlyoff.github.io/ServiceWorker/spec/service_worker/index.html#service-worker-global-scope-install-event)
event and don't require an explicit `done` call.
Here's an example that uses a dedicated worker.
`worker.js`:
importScripts("/resources/testharness.js");
test(function(t) {
assert_true(true, "true is true");
}, "Simple test");
// done() is needed because the testharness is running as if explicit_done
// was specified.
done();
`test.html`:
<!DOCTYPE html>
<title>Simple test</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<div id="log"></div>
<script>
fetch_tests_from_worker(new Worker("worker.js"));
</script>
The argument to the `fetch_tests_from_worker` function can be a
[`Worker`](https://html.spec.whatwg.org/multipage/workers.html#dedicated-workers-and-the-worker-interface),
a [`SharedWorker`](https://html.spec.whatwg.org/multipage/workers.html#shared-workers-and-the-sharedworker-interface)
or a [`ServiceWorker`](https://slightlyoff.github.io/ServiceWorker/spec/service_worker/#service-worker-obj).
Once called, the containing document fetches all the tests from the worker and
behaves as if those tests were running in the containing document itself.
## List of Assertions ##
### `assert_true(actual, description)`
asserts that `actual` is strictly true
### `assert_false(actual, description)`
asserts that `actual` is strictly false
### `assert_equals(actual, expected, description)`
asserts that `actual` is the same value as `expected`
### `assert_not_equals(actual, expected, description)`
asserts that `actual` is a different value to `expected`.
This means that `expected` is a misnomer.
### `assert_in_array(actual, expected, description)`
asserts that `expected` is an Array, and `actual` is equal to one of the
members i.e. `expected.indexOf(actual) != -1`
### `assert_array_equals(actual, expected, description)`
asserts that `actual` and `expected` have the same
length and the value of each indexed property in `actual` is the strictly equal
to the corresponding property value in `expected`
### `assert_approx_equals(actual, expected, epsilon, description)`
asserts that `actual` is a number within +`- `epsilon` of `expected`
### `assert_less_than(actual, expected, description)`
asserts that `actual` is a number less than `expected`
### `assert_greater_than(actual, expected, description)`
asserts that `actual` is a number greater than `expected`
### `assert_between_exclusive(actual, lower, upper, description`
asserts that `actual` is a number between `lower` and `upper` but not
equal to either of them
### `assert_less_than_equal(actual, expected, description)`
asserts that `actual` is a number less than or equal to `expected`
### `assert_greater_than_equal(actual, expected, description)`
asserts that `actual` is a number greater than or equal to `expected`
### `assert_between_inclusive(actual, lower, upper, description`
asserts that `actual` is a number between `lower` and `upper` or
equal to either of them
### `assert_regexp_match(actual, expected, description)`
asserts that `actual` matches the regexp `expected`
### `assert_class_string(object, class_name, description)`
asserts that the class string of `object` as returned in
`Object.prototype.toString` is equal to `class_name`.
### `assert_own_property(object, property_name, description)`
assert that object has own property `property_name`
### `assert_inherits(object, property_name, description)`
assert that object does not have an own property named
`property_name` but that `property_name` is present in the prototype
chain for object
### `assert_idl_attribute(object, attribute_name, description)`
assert that an object that is an instance of some interface has the
attribute attribute_name following the conditions specified by WebIDL
### `assert_readonly(object, property_name, description)`
assert that property `property_name` on object is readonly
### `assert_throws(code, func, description)`
`code` - the expected exception. This can take several forms:
* string - the thrown exception must be a DOMException with the given
name, e.g., "TimeoutError" (for compatibility with existing
tests, a constant is also supported, e.g., "TIMEOUT_ERR")
* object - the thrown exception must have a property called "name" that
matches code.name
* null - allow any exception (in general, one of the options above
should be used)
`func` - a function that should throw
### `assert_unreached(description)`
asserts if called. Used to ensure that some codepath is *not* taken e.g.
an event does not fire.
### `assert_any(assert_func, actual, expected_array, extra_arg_1, ... extra_arg_N)`
asserts that one `assert_func(actual, expected_array_N, extra_arg1, ..., extra_arg_N)`
is true for some `expected_array_N` in `expected_array`. This only works for `assert_func`
with signature `assert_func(actual, expected, args_1, ..., args_N)`. Note that tests
with multiple allowed pass conditions are bad practice unless the spec specifically
allows multiple behaviours. Test authors should not use this method simply to hide
UA bugs.
### `assert_exists(object, property_name, description)`
**deprecated**
asserts that object has an own property `property_name`
### `assert_not_exists(object, property_name, description)`
**deprecated**
assert that object does not have own property `property_name`
## Metadata ##
It is possible to add optional metadata to tests; this can be done in
one of two ways; either by adding `<meta>` elements to the head of the
document containing the tests, or by adding the metadata to individual
`[async_]test` calls, as properties.

118
tests/wpt/web-platform-tests/resources/docs/idlharness.md Normal file
View file

@ -0,0 +1,118 @@
## Introduction ##
`idlharness.js` automatically generates browser tests for WebIDL interfaces, using
the testharness.js framework. To use, first include the following:
<script src=/resources/testharness.js></script>
<script src=/resources/testharnessreport.js></script>
<script src=/resources/WebIDLParser.js></script>
<script src=/resources/idlharness.js></script>
Then you'll need some type of IDLs. Here's some script that can be run on a
spec written in HTML, which will grab all the elements with `class="idl"`,
concatenate them, and replace the body so you can copy-paste:
var s = "";
[].forEach.call(document.getElementsByClassName("idl"), function(idl) {
//https://www.w3.org/Bugs/Public/show_bug.cgi?id=14914
if (!idl.classList.contains("extract"))
{
s += idl.textContent + "\n\n";
}
});
document.body.innerHTML = '<pre></pre>';
document.body.firstChild.textContent = s;
Once you have that, put it in your script somehow. The easiest way is to
embed it literally in an HTML file with `<script type=text/plain>` or similar,
so that you don't have to do any escaping. Another possibility is to put it
in a separate .idl file that's fetched via XHR or similar. Sample usage:
var idl_array = new IdlArray();
idl_array.add_untested_idls("interface Node { readonly attribute DOMString nodeName; };");
idl_array.add_idls("interface Document : Node { readonly attribute DOMString URL; };");
idl_array.add_objects({Document: ["document"]});
idl_array.test();
This tests that `window.Document` exists and meets all the requirements of
WebIDL. It also tests that window.document (the result of evaluating the
string "document") has URL and nodeName properties that behave as they
should, and otherwise meets WebIDL's requirements for an object whose
primary interface is Document. It does not test that window.Node exists,
which is what you want if the Node interface is already tested in some other
specification's suite and your specification only extends or refers to it.
Of course, each IDL string can define many different things, and calls to
add_objects() can register many different objects for different interfaces:
this is a very simple example.
## Public methods of IdlArray ##
IdlArray objects can be obtained with `new IdlArray()`. Anything not
documented in this section should be considered an implementation detail,
and outside callers should not use it.
### `add_idls(idl_string)`
Parses `idl_string` (throwing on parse error) and adds the results to the
IdlArray. All the definitions will be tested when you run test(). If
some of the definitions refer to other definitions, those must be present
too. For instance, if `idl_string` says that `Document` inherits from `Node`,
the `Node` interface must also have been provided in some call to `add_idls()`
or `add_untested_idls()`.
### `add_untested_idls(idl_string)`
Like `add_idls()`, but the definitions will not be tested. If an untested
interface is added and then extended with a tested partial interface, the
members of the partial interface will still be tested. Also, all the
members will still be tested for objects added with `add_objects()`, because
you probably want to test that (for instance) window.document has all the
properties from `Node`, not just `Document`, even if the `Node` interface itself
is tested in a different test suite.
### `add_objects(dict)`
`dict` should be an object whose keys are the names of interfaces or
exceptions, and whose values are arrays of strings. When an interface or
exception is tested, every string registered for it with `add_objects()`
will be evaluated, and tests will be run on the result to verify that it
correctly implements that interface or exception. This is the only way to
test anything about `[NoInterfaceObject]` interfaces, and there are many
tests that can't be run on any interface without an object to fiddle with.
The interface has to be the *primary* interface of all the objects
provided. For example, don't pass `{Node: ["document"]}`, but rather
`{Document: ["document"]}`. Assuming the `Document` interface was declared to
inherit from `Node`, this will automatically test that document implements
the `Node` interface too.
Warning: methods will be called on any provided objects, in a manner that
WebIDL requires be safe. For instance, if a method has mandatory
arguments, the test suite will try calling it with too few arguments to
see if it throws an exception. If an implementation incorrectly runs the
function instead of throwing, this might have side effects, possibly even
preventing the test suite from running correctly.
### `prevent_multiple_testing(name)`
This is a niche method for use in case you're testing many objects that
implement the same interfaces, and don't want to retest the same
interfaces every single time. For instance, HTML defines many interfaces
that all inherit from `HTMLElement`, so the HTML test suite has something
like
`.add_objects({
HTMLHtmlElement: ['document.documentElement'],
HTMLHeadElement: ['document.head'],
HTMLBodyElement: ['document.body'],
...
})`
and so on for dozens of element types. This would mean that it would
retest that each and every one of those elements implements `HTMLElement`,
`Element`, and `Node`, which would be thousands of basically redundant tests.
The test suite therefore calls `prevent_multiple_testing("HTMLElement")`.
This means that once one object has been tested to implement `HTMLElement`
and its ancestors, no other object will be. Thus in the example code
above, the harness would test that `document.documentElement` correctly
implements `HTMLHtmlElement`, `HTMLElement`, `Element`, and `Node`; but
`document.head` would only be tested for `HTMLHeadElement`, and so on for
further objects.
### `test()`
Run all tests. This should be called after you've called all other
methods to add IDLs and objects.