Update wptrunner.

tests/wpt/harness/README.rst

@@ -25,9 +25,15 @@ following are most significant:
 ``--product`` (defaults to `firefox`)
   The product to test against: `b2g`, `chrome`, `firefox`, or `servo`.
 
-``--binary`` (required)
+``--binary`` (required if product is `firefox` or `servo`)
   The path to a binary file for the product (browser) to test against.
 
+``--webdriver-binary`` (required if product is `chrome`)
+  The path to a `*driver` binary; e.g., a `chromedriver` binary.
+
+``--certutil-binary`` (required if product is `firefox` [#]_)
+  The path to a `certutil` binary (for tests that must be run over https).
+
 ``--metadata`` (required)
   The path to a directory containing test metadata. [#]_
 
@@ -37,6 +43,9 @@ following are most significant:
 ``--prefs-root`` (required only when testing a Firefox binary)
   The path to a directory containing Firefox test-harness preferences. [#]_
 
+.. [#] The ``--certutil-binary`` option is required when the product is
+   ``firefox`` unless ``--ssl-type=none`` is specified.
+
 .. [#] The ``--metadata`` path is to a directory that contains:
 
   * a ``MANIFEST.json`` file (the web-platform-tests documentation has
@@ -56,26 +65,29 @@ To test a Firefox Nightly build in an OS X environment, you might start
 wptrunner using something similar to the following example::
 
   wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
-  --binary=~/mozilla-central/obj-x86_64-apple-darwin14.0.0/dist/Nightly.app/Contents/MacOS/firefox \
+    --binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/dist/Nightly.app/Contents/MacOS/firefox \
-  --prefs-root=~/mozilla-central/testing/profiles
+    --certutil-binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/security/nss/cmd/certutil/certutil \
+    --prefs-root=~/mozilla-central/testing/profiles
 
 And to test a Chromium build in an OS X environment, you might start
 wptrunner using something similar to the following example::
 
   wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
-  --binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
+    --binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
-  --product=chrome
+    --webdriver-binary=/usr/local/bin/chromedriver --product=chrome
 
 -------------------------------------
 Example: How to run a subset of tests
 -------------------------------------
 
 To restrict a test run just to tests in a particular web-platform-tests
-subdirectory, use ``--include`` with the directory name; for example::
+subdirectory, specify the directory name in the positional arguments after
+the options; for example, run just the tests in the `dom` subdirectory::
 
   wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
-  --binary=/path/to/firefox --prefs-root=/path/to/testing/profiles \
+    --binary=/path/to/firefox --certutil-binary=/path/to/certutil \
-  --include=dom
+    --prefs-root=/path/to/testing/profiles \
+    dom
 
 Output
 ~~~~~~
@@ -95,8 +107,9 @@ log to a file and a human-readable summary to stdout, you might start
 wptrunner using something similar to the following example::
 
   wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
-  --binary=/path/to/firefox --prefs-root=/path/to/testing/profiles
+    --binary=/path/to/firefox --certutil-binary=/path/to/certutil \
-  --log-raw=output.log --log-mach=-
+    --prefs-root=/path/to/testing/profiles \
+    --log-raw=output.log --log-mach=-
 
 Expectation Data
 ~~~~~~~~~~~~~~~~

tests/wpt/harness/docs/usage.rst

@@ -56,9 +56,15 @@ takes multiple options, of which the following are most significant:
 ``--product`` (defaults to `firefox`)
   The product to test against: `b2g`, `chrome`, `firefox`, or `servo`.
 
-``--binary`` (required)
+``--binary`` (required if product is `firefox` or `servo`)
   The path to a binary file for the product (browser) to test against.
 
+``--webdriver-binary`` (required if product is `chrome`)
+  The path to a `*driver` binary; e.g., a `chromedriver` binary.
+
+``--certutil-binary`` (required if product is `firefox` [#]_)
+  The path to a `certutil` binary (for tests that must be run over https).
+
 ``--metadata`` (required only when not `using default paths`_)
   The path to a directory containing test metadata. [#]_
 
@@ -68,6 +74,9 @@ takes multiple options, of which the following are most significant:
 ``--prefs-root`` (required only when testing a Firefox binary)
   The path to a directory containing Firefox test-harness preferences. [#]_
 
+.. [#] The ``--certutil-binary`` option is required when the product is
+   ``firefox`` unless ``--ssl-type=none`` is specified.
+
 .. [#] The ``--metadata`` path is to a directory that contains:
 
   * a ``MANIFEST.json`` file (the web-platform-tests documentation has
@@ -89,26 +98,30 @@ To test a Firefox Nightly build in an OS X environment, you might start
 wptrunner using something similar to the following example::
 
   wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
-    --binary=~/mozilla-central/obj-x86_64-apple-darwin14.0.0/dist/Nightly.app/Contents/MacOS/firefox \
+    --binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/dist/Nightly.app/Contents/MacOS/firefox \
+    --certutil-binary=~/mozilla-central/obj-x86_64-apple-darwin14.3.0/security/nss/cmd/certutil/certutil \
     --prefs-root=~/mozilla-central/testing/profiles
 
+
 And to test a Chromium build in an OS X environment, you might start
 wptrunner using something similar to the following example::
 
   wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
     --binary=~/chromium/src/out/Release/Chromium.app/Contents/MacOS/Chromium \
-    --product=chrome
+    --webdriver-binary=/usr/local/bin/chromedriver --product=chrome
 
 --------------------
 Running test subsets
 --------------------
 
 To restrict a test run just to tests in a particular web-platform-tests
-subdirectory, use ``--include`` with the directory name; for example::
+subdirectory, specify the directory name in the positional arguments after
+the options; for example, run just the tests in the `dom` subdirectory::
 
   wptrunner --metadata=~/web-platform-tests/ --tests=~/web-platform-tests/ \
-    --binary=/path/to/firefox --prefs-root=/path/to/testing/profiles \
+    --binary=/path/to/firefox --certutil-binary=/path/to/certutil \
-    --include=dom
+    --prefs-root=/path/to/testing/profiles \
+    dom
 
 -------------------
 Running in parallel

tests/wpt/harness/test/metadata/testharness/firefox/__dir__.ini

@@ -0,0 +1,2 @@
+prefs: ["browser.display.foreground_color:#FF0000",
+        "browser.display.background_color:#000000"]

tests/wpt/harness/test/metadata/testharness/firefox/subdir/test_pref_reset.html.ini

@@ -0,0 +1,2 @@
+[test_pref_reset.html]
+  prefs: [@Reset]

tests/wpt/harness/test/metadata/testharness/subdir/__dir__.ini

@@ -0,0 +1 @@
+disabled: true

tests/wpt/harness/test/metadata/testharness/subdir/testharness_1.html.ini

@@ -0,0 +1,2 @@
+[testharness_1.html]
+  disabled: @False

tests/wpt/harness/test/metadata/testharness/tags/__dir__.ini

@@ -0,0 +1 @@
+tags: [dir-tag-1, dir-tag-2]

tests/wpt/harness/test/metadata/testharness/tags/testharness_0.html.ini

@@ -0,0 +1,4 @@
+tags: [file-tag]
+
+[testharness_0.html]
+  tags: [test-tag]

tests/wpt/harness/test/metadata/testharness/tags/testharness_1.html.ini

@@ -0,0 +1,2 @@
+[testharness_0.html]
+  tags: [test-1-tag]

tests/wpt/harness/test/metadata/testharness/tags/testharness_2.html.ini

@@ -0,0 +1,4 @@
+tags: [file-tag]
+
+[testharness_2.html]
+  tags: [test-2-tag, @Reset]

tests/wpt/harness/test/test.py

@@ -101,6 +101,8 @@ def settings_to_argv(settings):
 def set_from_args(settings, args):
     if args.test:
         settings["include"] = args.test
+    if args.tags:
+        settings["tags"] = args.tags
 
 def run(config, args):
     logger = structuredlog.StructuredLogger("web-platform-tests")
@@ -139,6 +141,8 @@ def get_parser():
                         help="Specific product to include in test run")
     parser.add_argument("--pdb", action="store_true",
                         help="Invoke pdb on uncaught exception")
+    parser.add_argument("--tag", action="append", dest="tags",
+                        help="tags to select tests")
     parser.add_argument("test", nargs="*",
                         help="Specific tests to include in test run")
     return parser

tests/wpt/harness/test/testdata/testharness/firefox/subdir/test_pref_inherit.html

@@ -0,0 +1,10 @@
+<!doctype html>
+<title>Example pref test</title>
+<script src="/resources/testharness.js"></script>
+<script src="/resources/testharnessreport.js"></script>
+<p>Test requires the pref browser.display.foreground_color to be set to #00FF00</p>
+<script>
+test(function() {
+  assert_equals(getComputedStyle(document.body).color, "rgb(255, 0, 0)");
+}, "Test that pref was set");
+</script>

tests/wpt/harness/test/testdata/testharness/firefox/subdir/test_pref_reset.html

@@ -0,0 +1,10 @@
+<!doctype html>
+<title>Example pref test</title>
+<script src="/resources/testharness.js"></script>
+<script src="/resources/testharnessreport.js"></script>
+<p>Test requires the pref browser.display.foreground_color to be set to #00FF00</p>
+<script>
+test(function() {
+  assert_equals(getComputedStyle(document.body).color, "rgb(0, 0, 0)");
+}, "Test that pref was reset");
+</script>

tests/wpt/harness/test/testdata/testharness/firefox/test_pref_dir.html

@@ -0,0 +1,10 @@
+<!doctype html>
+<title>Example pref test</title>
+<script src="/resources/testharness.js"></script>
+<script src="/resources/testharnessreport.js"></script>
+<p>Test requires the pref browser.display.foreground_color to be set to #FF0000</p>
+<script>
+test(function() {
+  assert_equals(getComputedStyle(document.body).color, "rgb(255, 0, 0)");
+}, "Test that pref was set");
+</script>

tests/wpt/harness/test/testdata/testharness/firefox/test_pref_set.html

@@ -1,5 +1,5 @@
 <!doctype html>
-<title>Example https test</title>
+<title>Example pref test</title>
 <script src="/resources/testharness.js"></script>
 <script src="/resources/testharnessreport.js"></script>
 <p>Test requires the pref browser.display.foreground_color to be set to #00FF00</p>
@@ -7,4 +7,4 @@
 test(function() {
   assert_equals(getComputedStyle(document.body).color, "rgb(0, 255, 0)");
 }, "Test that pref was set");
-</script>
+</script>

tests/wpt/harness/test/testdata/testharness/subdir/testharness_1.html

@@ -0,0 +1,9 @@
+<!doctype html>
+<title>Test should be enabled</title>
+<script src="/resources/testharness.js"></script>
+<script src="/resources/testharnessreport.js"></script>
+<script>
+test(function() {
+  assert_true(true);
+}, "Test that should pass");
+</script>

tests/wpt/harness/test/testdata/testharness/tags/testharness_0.html

@@ -0,0 +1,9 @@
+<!doctype html>
+<title>Test</title>
+<script src="/resources/testharness.js"></script>
+<script src="/resources/testharnessreport.js"></script>
+<script>
+test(function() {
+  assert_true(true);
+}, "Test that should pass");
+</script>

tests/wpt/harness/test/testdata/testharness/tags/testharness_1.html

@@ -0,0 +1,9 @@
+<!doctype html>
+<title>Test</title>
+<script src="/resources/testharness.js"></script>
+<script src="/resources/testharnessreport.js"></script>
+<script>
+test(function() {
+  assert_true(true);
+}, "Test that should pass");
+</script>

tests/wpt/harness/test/testdata/testharness/tags/testharness_2.html

@@ -0,0 +1,9 @@
+<!doctype html>
+<title>Test</title>
+<script src="/resources/testharness.js"></script>
+<script src="/resources/testharnessreport.js"></script>
+<script>
+test(function() {
+  assert_true(true);
+}, "Test that should pass");
+</script>

tests/wpt/harness/test/testdata/testharness/testharness_0.html

@@ -1,13 +1,9 @@
 <!doctype html>
-<title>Simple testharness.js usage</title>
+<title>Test should be disabled</title>
 <script src="/resources/testharness.js"></script>
 <script src="/resources/testharnessreport.js"></script>
 <script>
-test(function() {
-  assert_true(true);
-}, "Test that should pass");
-
 test(function() {
   assert_true(false);
 }, "Test that should fail");
-</script>
+</script>

tests/wpt/harness/wptrunner/browsers/b2g.py

@@ -42,7 +42,8 @@ def browser_kwargs(test_environment, **kwargs):
             "no_backup": kwargs.get("b2g_no_backup", False)}
 
 
-def executor_kwargs(test_type, server_config, cache_manager, **kwargs):
+def executor_kwargs(test_type, server_config, cache_manager, run_info_data,
+                    **kwargs):
     timeout_multiplier = kwargs["timeout_multiplier"]
     if timeout_multiplier is None:
         timeout_multiplier = 2

tests/wpt/harness/wptrunner/browsers/chrome.py

@@ -20,7 +20,7 @@ __wptrunner__ = {"product": "chrome",
 
 
 def check_args(**kwargs):
-    require_arg(kwargs, "binary")
+    require_arg(kwargs, "webdriver_binary")
 
 
 def browser_kwargs(**kwargs):
@@ -28,15 +28,16 @@ def browser_kwargs(**kwargs):
             "webdriver_binary": kwargs["webdriver_binary"]}
 
 
-def executor_kwargs(test_type, server_config, cache_manager, **kwargs):
+def executor_kwargs(test_type, server_config, cache_manager, run_info_data,
+                    **kwargs):
     from selenium.webdriver import DesiredCapabilities
 
     executor_kwargs = base_executor_kwargs(test_type, server_config,
                                            cache_manager, **kwargs)
     executor_kwargs["close_after_done"] = True
-    executor_kwargs["capabilities"] = dict(DesiredCapabilities.CHROME.items() +
+    executor_kwargs["capabilities"] = dict(DesiredCapabilities.CHROME.items())
-                                           {"chromeOptions":
+    if kwargs["binary"] is not None:
-                                            {"binary": kwargs["binary"]}}.items())
+        executor_kwargs["capabilities"]["chromeOptions"] = {"binary": kwargs["binary"]}
 
     return executor_kwargs
 

tests/wpt/harness/wptrunner/browsers/firefox.py

@@ -46,10 +46,13 @@ def browser_kwargs(**kwargs):
             "ca_certificate_path": kwargs["ssl_env"].ca_cert_path()}
 
 
-def executor_kwargs(test_type, server_config, cache_manager, **kwargs):
+def executor_kwargs(test_type, server_config, cache_manager, run_info_data,
+                    **kwargs):
     executor_kwargs = base_executor_kwargs(test_type, server_config,
                                            cache_manager, **kwargs)
     executor_kwargs["close_after_done"] = True
+    if run_info_data["debug"] and kwargs["timeout_multiplier"] is None:
+        executor_kwargs["timeout_multiplier"] = 3
     return executor_kwargs
 
 

tests/wpt/harness/wptrunner/browsers/servo.py

@@ -29,7 +29,8 @@ def browser_kwargs(**kwargs):
             "debug_info": kwargs["debug_info"]}
 
 
-def executor_kwargs(test_type, server_config, cache_manager, **kwargs):
+def executor_kwargs(test_type, server_config, cache_manager, run_info_data,
+                    **kwargs):
     rv = base_executor_kwargs(test_type, server_config,
                               cache_manager, **kwargs)
     rv["pause_after_test"] = kwargs["pause_after_test"]

tests/wpt/harness/wptrunner/executors/base.py

@@ -99,7 +99,7 @@ class TestExecutor(object):
         self.timeout_multiplier = timeout_multiplier
         self.debug_info = debug_info
         self.last_environment = {"protocol": "http",
-                                 "prefs": []}
+                                 "prefs": {}}
         self.protocol = None # This must be set in subclasses
 
     @property

tests/wpt/harness/wptrunner/executors/executormarionette.py

@@ -139,55 +139,62 @@ class MarionetteProtocol(Protocol):
 
     def on_environment_change(self, old_environment, new_environment):
         #Unset all the old prefs
-        for name, _ in old_environment.get("prefs", []):
+        for name in old_environment.get("prefs", {}).iterkeys():
             value = self.executor.original_pref_values[name]
             if value is None:
                 self.clear_user_pref(name)
             else:
                 self.set_pref(name, value)
 
-        for name, value in new_environment.get("prefs", []):
+        for name, value in new_environment.get("prefs", {}).iteritems():
             self.executor.original_pref_values[name] = self.get_pref(name)
             self.set_pref(name, value)
 
     def set_pref(self, name, value):
+        if value.lower() not in ("true", "false"):
+            try:
+                int(value)
+            except ValueError:
+                value = "'%s'" % value
+        else:
+            value = value.lower()
+
         self.logger.info("Setting pref %s (%s)" % (name, value))
-        self.marionette.set_context(self.marionette.CONTEXT_CHROME)
+
         script = """
             let prefInterface = Components.classes["@mozilla.org/preferences-service;1"]
                                           .getService(Components.interfaces.nsIPrefBranch);
             let pref = '%s';
             let type = prefInterface.getPrefType(pref);
+            let value = %s;
             switch(type) {
                 case prefInterface.PREF_STRING:
-                    prefInterface.setCharPref(pref, '%s');
+                    prefInterface.setCharPref(pref, value);
                     break;
                 case prefInterface.PREF_BOOL:
-                    prefInterface.setBoolPref(pref, %s);
+                    prefInterface.setBoolPref(pref, value);
                     break;
                 case prefInterface.PREF_INT:
-                    prefInterface.setIntPref(pref, %s);
+                    prefInterface.setIntPref(pref, value);
                     break;
             }
-            """ % (name, value, value, value)
+            """ % (name, value)
-        self.marionette.execute_script(script)
+        with self.marionette.using_context(self.marionette.CONTEXT_CHROME):
-        self.marionette.set_context(self.marionette.CONTEXT_CONTENT)
+            self.marionette.execute_script(script)
 
     def clear_user_pref(self, name):
         self.logger.info("Clearing pref %s" % (name))
-        self.marionette.set_context(self.marionette.CONTEXT_CHROME)
         script = """
             let prefInterface = Components.classes["@mozilla.org/preferences-service;1"]
                                           .getService(Components.interfaces.nsIPrefBranch);
             let pref = '%s';
             prefInterface.clearUserPref(pref);
             """ % name
-        self.marionette.execute_script(script)
+        with self.marionette.using_context(self.marionette.CONTEXT_CHROME):
-        self.marionette.set_context(self.marionette.CONTEXT_CONTENT)
+            self.marionette.execute_script(script)
 
     def get_pref(self, name):
-        self.marionette.set_context(self.marionette.CONTEXT_CHROME)
+        script = """
-        self.marionette.execute_script("""
             let prefInterface = Components.classes["@mozilla.org/preferences-service;1"]
                                           .getService(Components.interfaces.nsIPrefBranch);
             let pref = '%s';
@@ -202,8 +209,9 @@ class MarionetteProtocol(Protocol):
                 case prefInterface.PREF_INVALID:
                     return null;
             }
-            """ % (name))
+            """ % name
-        self.marionette.set_context(self.marionette.CONTEXT_CONTENT)
+        with self.marionette.using_context(self.marionette.CONTEXT_CHROME):
+            self.marionette.execute_script(script)
 
 class MarionetteRun(object):
     def __init__(self, logger, func, marionette, url, timeout):
@@ -383,10 +391,7 @@ class MarionetteRefTestExecutor(RefTestExecutor):
                              timeout).run()
 
     def _screenshot(self, marionette, url, timeout):
-        try:
+        marionette.navigate(url)
-            marionette.navigate(url)
-        except errors.MarionetteException:
-            raise ExecutorException("ERROR", "Failed to load url %s" % (url,))
 
         marionette.execute_async_script(self.wait_script)
 

tests/wpt/harness/wptrunner/executors/executorservo.py

@@ -62,10 +62,9 @@ class ServoTestharnessExecutor(ProcessTestExecutor):
         self.result_data = None
         self.result_flag = threading.Event()
 
-        debug_args, command = browser_command(self.binary, ["--cpu", "--hard-fail", "-z",
+        debug_args, command = browser_command(self.binary,
-                                                            "-u", "Servo/wptrunner",
+            ["--cpu", "--hard-fail", "-u", "Servo/wptrunner", "-z", self.test_url(test)],
-                                                            self.test_url(test)],
+            self.debug_info)
-                                              self.debug_info)
 
         self.command = command
 
@@ -101,15 +100,18 @@ class ServoTestharnessExecutor(ProcessTestExecutor):
                 self.proc.wait()
 
             proc_is_running = True
-            if self.result_flag.is_set() and self.result_data is not None:
+
-                self.result_data["test"] = test.url
+            if self.result_flag.is_set():
-                result = self.convert_result(test, self.result_data)
+                if self.result_data is not None:
-            else:
+                    self.result_data["test"] = test.url
-                if self.proc.poll() is not None:
+                    result = self.convert_result(test, self.result_data)
+                else:
+                    self.proc.wait()
                     result = (test.result_cls("CRASH", None), [])
                     proc_is_running = False
-                else:
+            else:
-                    result = (test.result_cls("TIMEOUT", None), [])
+                result = (test.result_cls("TIMEOUT", None), [])
+
 
             if proc_is_running:
                 if self.pause_after_test:
@@ -188,8 +190,8 @@ class ServoRefTestExecutor(ProcessTestExecutor):
 
         with TempFilename(self.tempdir) as output_path:
             self.command = [self.binary, "--cpu", "--hard-fail", "--exit",
-                            "-Z", "disable-text-aa,disable-canvas-aa", "--output=%s" % output_path,
+                            "-u", "Servo/wptrunner", "-Z", "disable-text-aa",
-                            full_url]
+                            "--output=%s" % output_path, full_url]
 
             env = os.environ.copy()
             env["HOST_FILE"] = self.hosts_path
@@ -200,7 +202,8 @@ class ServoRefTestExecutor(ProcessTestExecutor):
 
             try:
                 self.proc.run()
-                rv = self.proc.wait(timeout=test.timeout)
+                timeout = test.timeout * self.timeout_multiplier + 5
+                rv = self.proc.wait(timeout=timeout)
             except KeyboardInterrupt:
                 self.proc.kill()
                 raise

tests/wpt/harness/wptrunner/executors/executorservodriver.py

@@ -84,7 +84,7 @@ class ServoWebDriverProtocol(Protocol):
 
 
 class ServoWebDriverRun(object):
-    def __init__(self, func, session, url, timeout):
+    def __init__(self, func, session, url, timeout, current_timeout=None):
         self.func = func
         self.result = None
         self.session = session
@@ -93,18 +93,10 @@ class ServoWebDriverRun(object):
         self.result_flag = threading.Event()
 
     def run(self):
-        timeout = self.timeout
-
-        try:
-            self.session.timeouts.script = timeout + extra_timeout
-        except IOError:
-            self.logger.error("Lost webdriver connection")
-            return Stop
-
         executor = threading.Thread(target=self._run)
         executor.start()
 
-        flag = self.result_flag.wait(timeout + 2 * extra_timeout)
+        flag = self.result_flag.wait(self.timeout + extra_timeout)
         if self.result is None:
             assert not flag
             self.result = False, ("EXTERNAL-TIMEOUT", None)
@@ -144,6 +136,7 @@ class ServoWebDriverTestharnessExecutor(TestharnessExecutor):
         self.protocol = ServoWebDriverProtocol(self, browser, capabilities=capabilities)
         with open(os.path.join(here, "testharness_servodriver.js")) as f:
             self.script = f.read()
+        self.timeout = None
 
     def on_protocol_change(self, new_protocol):
         pass
@@ -154,10 +147,20 @@ class ServoWebDriverTestharnessExecutor(TestharnessExecutor):
     def do_test(self, test):
         url = self.test_url(test)
 
+        timeout = test.timeout * self.timeout_multiplier + extra_timeout
+
+        if timeout != self.timeout:
+            try:
+                self.protocol.session.timeouts.script = timeout
+                self.timeout = timeout
+            except IOError:
+                self.logger.error("Lost webdriver connection")
+                return Stop
+
         success, data = ServoWebDriverRun(self.do_testharness,
                                           self.protocol.session,
                                           url,
-                                          test.timeout * self.timeout_multiplier).run()
+                                          timeout).run()
 
         if success:
             return self.convert_result(test, data)
@@ -172,8 +175,9 @@ class ServoWebDriverTestharnessExecutor(TestharnessExecutor):
                                "url": strip_server(url),
                                "timeout_multiplier": self.timeout_multiplier,
                                "timeout": timeout * 1000}))
-        if "test" not in result:
+        # Prevent leaking every page in history until Servo develops a more sane
-            result["test"] = strip_server(url)
+        # page cache
+        session.back()
         return result
 
 
@@ -194,7 +198,7 @@ class ServoWebDriverRefTestExecutor(RefTestExecutor):
         self.protocol = ServoWebDriverProtocol(self, browser,
                                                capabilities=capabilities)
         self.implementation = RefTestImplementation(self)
-
+        self.timeout = None
         with open(os.path.join(here, "reftest-wait_servodriver.js")) as f:
             self.wait_script = f.read()
 
@@ -217,7 +221,17 @@ class ServoWebDriverRefTestExecutor(RefTestExecutor):
             return test.result_cls("ERROR", message), []
 
     def screenshot(self, test):
-        timeout = test.timeout * self.timeout_multiplier if self.debug_info is None else None
+        timeout = (test.timeout * self.timeout_multiplier + extra_timeout
+                   if self.debug_info is None else None)
+
+        if self.timeout != timeout:
+            try:
+                self.protocol.session.timeouts.script = timeout
+                self.timeout = timeout
+            except IOError:
+                self.logger.error("Lost webdriver connection")
+                return Stop
+
         return ServoWebDriverRun(self._screenshot,
                                  self.protocol.session,
                                  self.test_url(test),

tests/wpt/harness/wptrunner/executors/testharness_marionette.js

@@ -5,24 +5,27 @@
 window.wrappedJSObject.timeout_multiplier = %(timeout_multiplier)d;
 window.wrappedJSObject.explicit_timeout = %(explicit_timeout)d;
 
-window.wrappedJSObject.done = function(tests, status) {
+window.wrappedJSObject.addEventListener("message", function listener(event) {
-  clearTimeout(timer);
+    if (event.data.type != "complete") {
-  var test_results = tests.map(function(x) {
+        return;
-    return {name:x.name, status:x.status, message:x.message, stack:x.stack}
+    }
-  });
+    window.wrappedJSObject.removeEventListener("message", listener);
-  marionetteScriptFinished({test:"%(url)s",
+    clearTimeout(timer);
-                            tests:test_results,
+    var tests = event.data.tests;
-                            status: status.status,
+    var status = event.data.status;
-                            message: status.message,
+    marionetteScriptFinished({test:"%(url)s",
-                            stack: status.stack});
+                              tests: tests,
-}
+                              status: status.status,
+                              message: status.message,
+                              stack: status.stack});
+}, false);
 
 window.wrappedJSObject.win = window.open("%(abs_url)s", "%(window_id)s");
 
 var timer = null;
 if (%(timeout)s) {
-  timer = setTimeout(function() {
+    timer = setTimeout(function() {
-      log("Timeout fired");
+        log("Timeout fired");
-      window.wrappedJSObject.win.timeout();
+        window.wrappedJSObject.win.timeout();
-  }, %(timeout)s);
+    }, %(timeout)s);
 }

tests/wpt/harness/wptrunner/executors/testharness_webdriver.js

@@ -5,17 +5,16 @@
 var callback = arguments[arguments.length - 1];
 window.timeout_multiplier = %(timeout_multiplier)d;
 
-window.done = function(tests, status) {
+window.addEventListener("message", function(event) {
+  var tests = event.data[0];
+  var status = event.data[1];
   clearTimeout(timer);
-  var test_results = tests.map(function(x) {
-    return {name:x.name, status:x.status, message:x.message, stack:x.stack}
-  });
   callback({test:"%(url)s",
-            tests:test_results,
+            tests: tests,
             status: status.status,
             message: status.message,
             stack: status.stack});
-}
+}, false);
 
 window.win = window.open("%(abs_url)s", "%(window_id)s");
 

tests/wpt/harness/wptrunner/executors/webdriver.py

@@ -444,7 +444,7 @@ class Session(object):
                 body = {"id": frame.json()}
             else:
                 body = {"id": frame}
-        print body
+
         return self.send_command("POST", url, body)
 
     @command

tests/wpt/harness/wptrunner/manifestexpected.py

@@ -29,6 +29,42 @@ def data_cls_getter(output_node, visited_node):
     raise ValueError
 
 
+def disabled(node):
+    """Boolean indicating whether the test is disabled"""
+    try:
+        return node.get("disabled")
+    except KeyError:
+        return None
+
+
+def tags(node):
+    """Set of tags that have been applied to the test"""
+    try:
+        value = node.get("tags")
+        if isinstance(value, (str, unicode)):
+            return {value}
+        return set(value)
+    except KeyError:
+        return set()
+
+
+def prefs(node):
+    def value(ini_value):
+        if isinstance(ini_value, (str, unicode)):
+            return tuple(ini_value.split(":", 1))
+        else:
+            return (ini_value, None)
+
+    try:
+        node_prefs = node.get("prefs")
+        if type(node_prefs) in (str, unicode):
+            prefs = {value(node_prefs)}
+        rv = dict(value(item) for item in node_prefs)
+    except KeyError:
+        rv = {}
+    return rv
+
+
 class ExpectedManifest(ManifestItem):
     def __init__(self, name, test_path, url_base):
         """Object representing all the tests in a particular manifest
@@ -71,6 +107,32 @@ class ExpectedManifest(ManifestItem):
         return urlparse.urljoin(self.url_base,
                                 "/".join(self.test_path.split(os.path.sep)))
 
+    @property
+    def disabled(self):
+        return disabled(self)
+
+    @property
+    def tags(self):
+        return tags(self)
+
+    @property
+    def prefs(self):
+        return prefs(self)
+
+
+class DirectoryManifest(ManifestItem):
+    @property
+    def disabled(self):
+        return disabled(self)
+
+    @property
+    def tags(self):
+        return tags(self)
+
+    @property
+    def prefs(self):
+        return prefs(self)
+
 
 class TestNode(ManifestItem):
     def __init__(self, name):
@@ -100,21 +162,17 @@ class TestNode(ManifestItem):
     def id(self):
         return urlparse.urljoin(self.parent.url, self.name)
 
+    @property
     def disabled(self):
-        """Boolean indicating whether the test is disabled"""
+        return disabled(self)
-        try:
-            return self.get("disabled")
-        except KeyError:
-            return False
 
+    @property
+    def tags(self):
+        return tags(self)
+
+    @property
     def prefs(self):
-        try:
+        return prefs(self)
-            prefs = self.get("prefs")
-            if type(prefs) in (str, unicode):
-                prefs = [prefs]
-            return [item.split(":", 1) for item in prefs]
-        except KeyError:
-            return []
 
     def append(self, node):
         """Add a subtest to the current test
@@ -159,9 +217,28 @@ def get_manifest(metadata_root, test_path, url_base, run_info):
     manifest_path = expected.expected_path(metadata_root, test_path)
     try:
         with open(manifest_path) as f:
-            return static.compile(f, run_info,
+            return static.compile(f,
+                                  run_info,
                                   data_cls_getter=data_cls_getter,
                                   test_path=test_path,
                                   url_base=url_base)
     except IOError:
         return None
+
+def get_dir_manifest(metadata_root, path, run_info):
+    """Get the ExpectedManifest for a particular test path, or None if there is no
+    metadata stored for that test path.
+
+    :param metadata_root: Absolute path to the root of the metadata directory
+    :param path: Path to the ini file relative to the metadata root
+    :param run_info: Dictionary of properties of the test run for which the expectation
+                     values should be computed.
+    """
+    full_path = os.path.join(metadata_root, path)
+    try:
+        with open(full_path) as f:
+            return static.compile(f,
+                                  run_info,
+                                  data_cls_getter=lambda x,y: DirectoryManifest)
+    except IOError:
+        return None

tests/wpt/harness/wptrunner/metadata.py

@@ -153,17 +153,32 @@ def update_from_logs(manifests, *log_filenames, **kwargs):
 
     return expected_map
 
+def directory_manifests(metadata_path):
+    rv = []
+    for dirpath, dirname, filenames in os.walk(metadata_path):
+        if "__dir__.ini" in filenames:
+            rel_path = os.path.relpath(dirpath, metadata_path)
+            rv.append(os.path.join(rel_path, "__dir__.ini"))
+    return rv
 
 def write_changes(metadata_path, expected_map):
     # First write the new manifest files to a temporary directory
     temp_path = tempfile.mkdtemp(dir=os.path.split(metadata_path)[0])
     write_new_expected(temp_path, expected_map)
 
+    # Keep all __dir__.ini files (these are not in expected_map because they
+    # aren't associated with a specific test)
+    keep_files = directory_manifests(metadata_path)
+
     # Copy all files in the root to the temporary location since
     # these cannot be ini files
-    keep_files = [item for item in os.listdir(metadata_path) if
+    keep_files.extend(item for item in os.listdir(metadata_path) if
-                  not os.path.isdir(os.path.join(metadata_path, item))]
+                      not os.path.isdir(os.path.join(metadata_path, item)))
+
     for item in keep_files:
+        dest_dir = os.path.dirname(os.path.join(temp_path, item))
+        if not os.path.exists(dest_dir):
+            os.makedirs(dest_dir)
         shutil.copyfile(os.path.join(metadata_path, item),
                         os.path.join(temp_path, item))
 

tests/wpt/harness/wptrunner/reduce.py

@@ -56,8 +56,8 @@ class Reducer(object):
         self.test_loader = wptrunner.TestLoader(kwargs["tests_root"],
                                                 kwargs["metadata_root"],
                                                 [self.test_type],
-                                                test_filter,
+                                                run_info,
-                                                run_info)
+                                                manifest_filer=test_filter)
         if kwargs["repeat"] == 1:
             logger.critical("Need to specify --repeat with more than one repetition")
             sys.exit(1)

tests/wpt/harness/wptrunner/testharnessreport-servodriver.js

@@ -9,7 +9,9 @@ add_completion_callback(function() {
         var test_results = tests.map(function(x) {
             return {name:x.name, status:x.status, message:x.message, stack:x.stack}
         });
-        var results = JSON.stringify({tests:test_results,
+        var id = location.pathname + location.search + location.hash;
+        var results = JSON.stringify({test: id,
+                                      tests:test_results,
                                       status: status.status,
                                       message: status.message,
                                       stack: status.stack});

tests/wpt/harness/wptrunner/testharnessreport.js

@@ -3,7 +3,8 @@
  * file, You can obtain one at http://mozilla.org/MPL/2.0/. */
 
 var props = {output:%(output)d,
-             explicit_timeout: true};
+             explicit_timeout: true,
+             message_events: ["completion"]};
 
 if (window.opener && "timeout_multiplier" in window.opener) {
     props["timeout_multiplier"] = window.opener.timeout_multiplier;
@@ -16,6 +17,14 @@ if (window.opener && window.opener.explicit_timeout) {
 setup(props);
 add_completion_callback(function() {
     add_completion_callback(function(tests, status) {
-        window.opener.done(tests, status)
+        var harness_status = {
+            "status": status.status,
+            "message": status.message,
+            "stack": status.stack
+        };
+        var test_results = tests.map(function(x) {
+            return {name:x.name, status:x.status, message:x.message, stack:x.stack}
+        });
+        window.opener.postMessage([test_results, harness_status], "*");
     })
 });

tests/wpt/harness/wptrunner/testloader.py

@@ -1,9 +1,10 @@
 import json
 import os
+import sys
 import urlparse
 from abc import ABCMeta, abstractmethod
 from Queue import Empty
-from collections import defaultdict, OrderedDict
+from collections import defaultdict, OrderedDict, deque
 from multiprocessing import Queue
 
 import manifestinclude
@@ -25,6 +26,7 @@ class TestChunker(object):
         self.total_chunks = total_chunks
         self.chunk_number = chunk_number
         assert self.chunk_number <= self.total_chunks
+        self.logger = structured.get_default_logger()
 
     def __call__(self, manifest):
         raise NotImplementedError
@@ -47,18 +49,15 @@ class HashChunker(TestChunker):
             if hash(test_path) % self.total_chunks == chunk_index:
                 yield test_path, tests
 
-
 class EqualTimeChunker(TestChunker):
-    """Chunker that uses the test timeout as a proxy for the running time of the test"""
+    def _group_by_directory(self, manifest_items):
+        """Split the list of manifest items into a ordered dict that groups tests in
+        so that anything in the same subdirectory beyond a depth of 3 is in the same
+        group. So all tests in a/b/c, a/b/c/d and a/b/c/e will be grouped together
+        and separate to tests in a/b/f
 
-    def _get_chunk(self, manifest_items):
+        Returns: tuple (ordered dict of {test_dir: PathData}, total estimated runtime)
-        # For each directory containing tests, calculate the maximum execution time after running all
+        """
-        # the tests in that directory. Then work out the index into the manifest corresponding to the
-        # directories at fractions of m/N of the running time where m=1..N-1 and N is the total number
-        # of chunks. Return an array of these indicies
-
-        total_time = 0
-        by_dir = OrderedDict()
 
         class PathData(object):
             def __init__(self, path):
@@ -66,73 +65,8 @@ class EqualTimeChunker(TestChunker):
                 self.time = 0
                 self.tests = []
 
-        class Chunk(object):
+        by_dir = OrderedDict()
-            def __init__(self):
+        total_time = 0
-                self.paths = []
-                self.tests = []
-                self.time = 0
-
-            def append(self, path_data):
-                self.paths.append(path_data.path)
-                self.tests.extend(path_data.tests)
-                self.time += path_data.time
-
-        class ChunkList(object):
-            def __init__(self, total_time, n_chunks):
-                self.total_time = total_time
-                self.n_chunks = n_chunks
-
-                self.remaining_chunks = n_chunks
-
-                self.chunks = []
-
-                self.update_time_per_chunk()
-
-            def __iter__(self):
-                for item in self.chunks:
-                    yield item
-
-            def __getitem__(self, i):
-                return self.chunks[i]
-
-            def sort_chunks(self):
-                self.chunks = sorted(self.chunks, key=lambda x:x.paths[0])
-
-            def get_tests(self, chunk_number):
-                return self[chunk_number - 1].tests
-
-            def append(self, chunk):
-                if len(self.chunks) == self.n_chunks:
-                    raise ValueError("Tried to create more than %n chunks" % self.n_chunks)
-                self.chunks.append(chunk)
-                self.remaining_chunks -= 1
-
-            @property
-            def current_chunk(self):
-                if self.chunks:
-                    return self.chunks[-1]
-
-            def update_time_per_chunk(self):
-                self.time_per_chunk = (self.total_time - sum(item.time for item in self)) / self.remaining_chunks
-
-            def create(self):
-                rv = Chunk()
-                self.append(rv)
-                return rv
-
-            def add_path(self, path_data):
-                sum_time = self.current_chunk.time + path_data.time
-                if sum_time > self.time_per_chunk and self.remaining_chunks > 0:
-                    overshoot = sum_time - self.time_per_chunk
-                    undershoot = self.time_per_chunk - self.current_chunk.time
-                    if overshoot < undershoot:
-                        self.create()
-                        self.current_chunk.append(path_data)
-                    else:
-                        self.current_chunk.append(path_data)
-                        self.create()
-                else:
-                    self.current_chunk.append(path_data)
 
         for i, (test_path, tests) in enumerate(manifest_items):
             test_dir = tuple(os.path.split(test_path)[0].split(os.path.sep)[:3])
@@ -144,42 +78,238 @@ class EqualTimeChunker(TestChunker):
             time = sum(wpttest.DEFAULT_TIMEOUT if test.timeout !=
                        "long" else wpttest.LONG_TIMEOUT for test in tests)
             data.time += time
+            total_time += time
             data.tests.append((test_path, tests))
 
-            total_time += time
+        return by_dir, total_time
 
-        chunk_list = ChunkList(total_time, self.total_chunks)
+    def _maybe_remove(self, chunks, i, direction):
+        """Trial removing a chunk from one chunk to an adjacent one.
+
+        :param chunks: - the list of all chunks
+        :param i: - the chunk index in the list of chunks to try removing from
+        :param direction: either "next" if we are going to move from the end to
+                          the subsequent chunk, or "prev" if we are going to move
+                          from the start into the previous chunk.
+
+        :returns bool: Did a chunk get moved?"""
+        source_chunk = chunks[i]
+        if direction == "next":
+            target_chunk = chunks[i+1]
+            path_index = -1
+            move_func = lambda: target_chunk.appendleft(source_chunk.pop())
+        elif direction == "prev":
+            target_chunk = chunks[i-1]
+            path_index = 0
+            move_func = lambda: target_chunk.append(source_chunk.popleft())
+        else:
+            raise ValueError("Unexpected move direction %s" % direction)
+
+        return self._maybe_move(source_chunk, target_chunk, path_index, move_func)
+
+    def _maybe_add(self, chunks, i, direction):
+        """Trial adding a chunk from one chunk to an adjacent one.
+
+        :param chunks: - the list of all chunks
+        :param i: - the chunk index in the list of chunks to try adding to
+        :param direction: either "next" if we are going to remove from the
+                          the subsequent chunk, or "prev" if we are going to remove
+                          from the the previous chunk.
+
+        :returns bool: Did a chunk get moved?"""
+        target_chunk = chunks[i]
+        if direction == "next":
+            source_chunk = chunks[i+1]
+            path_index = 0
+            move_func = lambda: target_chunk.append(source_chunk.popleft())
+        elif direction == "prev":
+            source_chunk = chunks[i-1]
+            path_index = -1
+            move_func = lambda: target_chunk.appendleft(source_chunk.pop())
+        else:
+            raise ValueError("Unexpected move direction %s" % direction)
+
+        return self._maybe_move(source_chunk, target_chunk, path_index, move_func)
+
+    def _maybe_move(self, source_chunk, target_chunk, path_index, move_func):
+        """Move from one chunk to another, assess the change in badness,
+        and keep the move iff it decreases the badness score.
+
+        :param source_chunk: chunk to move from
+        :param target_chunk: chunk to move to
+        :param path_index: 0 if we are moving from the start or -1 if we are moving from the
+                           end
+        :param move_func: Function that actually moves between chunks"""
+        if len(source_chunk.paths) <= 1:
+            return False
+
+        move_time = source_chunk.paths[path_index].time
+
+        new_source_badness = self._badness(source_chunk.time - move_time)
+        new_target_badness = self._badness(target_chunk.time + move_time)
+
+        delta_badness = ((new_source_badness + new_target_badness) -
+                         (source_chunk.badness + target_chunk.badness))
+        if delta_badness < 0:
+            move_func()
+            return True
+
+        return False
+
+    def _badness(self, time):
+        """Metric of badness for a specific chunk
+
+        :param time: the time for a specific chunk"""
+        return (time - self.expected_time)**2
+
+    def _get_chunk(self, manifest_items):
+        by_dir, total_time = self._group_by_directory(manifest_items)
 
         if len(by_dir) < self.total_chunks:
             raise ValueError("Tried to split into %i chunks, but only %i subdirectories included" % (
                 self.total_chunks, len(by_dir)))
 
-        # Put any individual dirs with a time greater than the time per chunk into their own
+        self.expected_time = float(total_time) / self.total_chunks
-        # chunk
+
+        chunks = self._create_initial_chunks(by_dir)
+
         while True:
-            to_remove = []
+            # Move a test from one chunk to the next until doing so no longer
-            for path_data in by_dir.itervalues():
+            # reduces the badness
-                if path_data.time > chunk_list.time_per_chunk:
+            got_improvement = self._update_chunks(chunks)
-                    to_remove.append(path_data)
+            if not got_improvement:
-            if to_remove:
-                for path_data in to_remove:
-                    chunk = chunk_list.create()
-                    chunk.append(path_data)
-                    del by_dir[path_data.path]
-                chunk_list.update_time_per_chunk()
-            else:
                 break
 
-        chunk = chunk_list.create()
+        self.logger.debug(self.expected_time)
-        for path_data in by_dir.itervalues():
+        for i, chunk in chunks.iteritems():
-            chunk_list.add_path(path_data)
+            self.logger.debug("%i: %i, %i" % (i + 1, chunk.time, chunk.badness))
 
-        assert len(chunk_list.chunks) == self.total_chunks, len(chunk_list.chunks)
+        assert self._all_tests(by_dir) == self._chunked_tests(chunks)
-        assert sum(item.time for item in chunk_list) == chunk_list.total_time
 
-        chunk_list.sort_chunks()
+        return self._get_tests(chunks)
 
-        return chunk_list.get_tests(self.chunk_number)
+    @staticmethod
+    def _all_tests(by_dir):
+        """Return a set of all tests in the manifest from a grouping by directory"""
+        return set(x[0] for item in by_dir.itervalues()
+                   for x in item.tests)
+
+    @staticmethod
+    def _chunked_tests(chunks):
+        """Return a set of all tests in the manifest from the chunk list"""
+        return set(x[0] for chunk in chunks.itervalues()
+                   for path in chunk.paths
+                   for x in path.tests)
+
+
+    def _create_initial_chunks(self, by_dir):
+        """Create an initial unbalanced list of chunks.
+
+        :param by_dir: All tests in the manifest grouped by subdirectory
+        :returns list: A list of Chunk objects"""
+
+        class Chunk(object):
+            def __init__(self, paths, index):
+                """List of PathData objects that together form a single chunk of
+                tests"""
+                self.paths = deque(paths)
+                self.time = sum(item.time for item in paths)
+                self.index = index
+
+            def appendleft(self, path):
+                """Add a PathData object to the start of the chunk"""
+                self.paths.appendleft(path)
+                self.time += path.time
+
+            def append(self, path):
+                """Add a PathData object to the end of the chunk"""
+                self.paths.append(path)
+                self.time += path.time
+
+            def pop(self):
+                """Remove PathData object from the end of the chunk"""
+                assert len(self.paths) > 1
+                self.time -= self.paths[-1].time
+                return self.paths.pop()
+
+            def popleft(self):
+                """Remove PathData object from the start of the chunk"""
+                assert len(self.paths) > 1
+                self.time -= self.paths[0].time
+                return self.paths.popleft()
+
+            @property
+            def badness(self_):
+                """Badness metric for this chunk"""
+                return self._badness(self_.time)
+
+        initial_size = len(by_dir) / self.total_chunks
+        chunk_boundaries = [initial_size * i
+                            for i in xrange(self.total_chunks)] + [len(by_dir)]
+
+        chunks = OrderedDict()
+        for i, lower in enumerate(chunk_boundaries[:-1]):
+            upper = chunk_boundaries[i + 1]
+            paths = by_dir.values()[lower:upper]
+            chunks[i] = Chunk(paths, i)
+
+        assert self._all_tests(by_dir) == self._chunked_tests(chunks)
+
+        return chunks
+
+    def _update_chunks(self, chunks):
+        """Run a single iteration of the chunk update algorithm.
+
+        :param chunks: - List of chunks
+        """
+        #TODO: consider replacing this with a heap
+        sorted_chunks = sorted(chunks.values(), key=lambda x:-x.badness)
+        got_improvement = False
+        for chunk in sorted_chunks:
+            if chunk.time < self.expected_time:
+                f = self._maybe_add
+            else:
+                f = self._maybe_remove
+
+            if chunk.index == 0:
+                order = ["next"]
+            elif chunk.index == self.total_chunks - 1:
+                order = ["prev"]
+            else:
+                if chunk.time < self.expected_time:
+                    # First try to add a test from the neighboring chunk with the
+                    # greatest total time
+                    if chunks[chunk.index + 1].time > chunks[chunk.index - 1].time:
+                        order = ["next", "prev"]
+                    else:
+                        order = ["prev", "next"]
+                else:
+                    # First try to remove a test and add to the neighboring chunk with the
+                    # lowest total time
+                    if chunks[chunk.index + 1].time > chunks[chunk.index - 1].time:
+                        order = ["prev", "next"]
+                    else:
+                        order = ["next", "prev"]
+
+            for direction in order:
+                if f(chunks, chunk.index, direction):
+                    got_improvement = True
+                    break
+
+            if got_improvement:
+                break
+
+        return got_improvement
+
+    def _get_tests(self, chunks):
+        """Return the list of tests corresponding to the chunk number we are running.
+
+        :param chunks: List of chunks"""
+        tests = []
+        for path in chunks[self.chunk_number - 1].paths:
+            tests.extend(path.tests)
+
+        return tests
 
     def __call__(self, manifest_iter):
         manifest = list(manifest_iter)
@@ -214,6 +344,14 @@ class TestFilter(object):
             if include_tests:
                 yield test_path, include_tests
 
+class TagFilter(object):
+    def __init__(self, tags):
+        self.tags = set(tags)
+
+    def __call__(self, test_iter):
+        for test in test_iter:
+            if test.tags & self.tags:
+                yield test
 
 class ManifestLoader(object):
     def __init__(self, test_paths, force_manifest_update=False):
@@ -276,20 +414,30 @@ class ManifestLoader(object):
 
         return manifest_file
 
+def iterfilter(filters, iter):
+    for f in filters:
+        iter = f(iter)
+    for item in iter:
+        yield item
+
 class TestLoader(object):
     def __init__(self,
                  test_manifests,
                  test_types,
-                 test_filter,
                  run_info,
+                 manifest_filters=None,
+                 meta_filters=None,
                  chunk_type="none",
                  total_chunks=1,
                  chunk_number=1,
                  include_https=True):
 
         self.test_types = test_types
-        self.test_filter = test_filter
         self.run_info = run_info
+
+        self.manifest_filters = manifest_filters if manifest_filters is not None else []
+        self.meta_filters = meta_filters if meta_filters is not None else []
+
         self.manifests = test_manifests
         self.tests = None
         self.disabled_tests = None
@@ -305,6 +453,9 @@ class TestLoader(object):
                                                                     chunk_number)
 
         self._test_ids = None
+
+        self.directory_manifests = {}
+
         self._load_tests()
 
     @property
@@ -316,22 +467,39 @@ class TestLoader(object):
                     self._test_ids += [item.id for item in test_dict[test_type]]
         return self._test_ids
 
-    def get_test(self, manifest_test, expected_file):
+    def get_test(self, manifest_test, inherit_metadata, test_metadata):
-        if expected_file is not None:
+        if test_metadata is not None:
-            expected = expected_file.get_test(manifest_test.id)
+            inherit_metadata.append(test_metadata)
-        else:
+            test_metadata = test_metadata.get_test(manifest_test.id)
-            expected = None
 
-        return wpttest.from_manifest(manifest_test, expected)
+        return wpttest.from_manifest(manifest_test, inherit_metadata, test_metadata)
 
-    def load_expected_manifest(self, test_manifest, metadata_path, test_path):
+    def load_dir_metadata(self, test_manifest, metadata_path, test_path):
-        return manifestexpected.get_manifest(metadata_path, test_path, test_manifest.url_base, self.run_info)
+        rv = []
+        path_parts = os.path.dirname(test_path).split(os.path.sep)
+        for i in xrange(1,len(path_parts) + 1):
+            path = os.path.join(os.path.sep.join(path_parts[:i]), "__dir__.ini")
+            if path not in self.directory_manifests:
+                self.directory_manifests[path] = manifestexpected.get_dir_manifest(
+                    metadata_path, path, self.run_info)
+            manifest = self.directory_manifests[path]
+            if manifest is not None:
+                rv.append(manifest)
+        return rv
+
+    def load_metadata(self, test_manifest, metadata_path, test_path):
+        inherit_metadata = self.load_dir_metadata(test_manifest, metadata_path, test_path)
+        test_metadata = manifestexpected.get_manifest(
+            metadata_path, test_path, test_manifest.url_base, self.run_info)
+        return inherit_metadata, test_metadata
 
     def iter_tests(self):
         manifest_items = []
 
         for manifest in self.manifests.keys():
-            manifest_items.extend(self.test_filter(manifest.itertypes(*self.test_types)))
+            manifest_iter = iterfilter(self.manifest_filters,
+                                       manifest.itertypes(*self.test_types))
+            manifest_items.extend(manifest_iter)
 
         if self.chunker is not None:
             manifest_items = self.chunker(manifest_items)
@@ -339,12 +507,15 @@ class TestLoader(object):
         for test_path, tests in manifest_items:
             manifest_file = iter(tests).next().manifest
             metadata_path = self.manifests[manifest_file]["metadata_path"]
-            expected_file = self.load_expected_manifest(manifest_file, metadata_path, test_path)
+            inherit_metadata, test_metadata = self.load_metadata(manifest_file, metadata_path, test_path)
 
-            for manifest_test in tests:
+            for test in iterfilter(self.meta_filters,
-                test = self.get_test(manifest_test, expected_file)
+                                   self.iter_wpttest(inherit_metadata, test_metadata, tests)):
-                test_type = manifest_test.item_type
+                yield test_path, test.test_type, test
-                yield test_path, test_type, test
+
+    def iter_wpttest(self, inherit_metadata, test_metadata, tests):
+        for manifest_test in tests:
+            yield self.get_test(manifest_test, inherit_metadata, test_metadata)
 
     def _load_tests(self):
         """Read in the tests from the manifest file and add them to a queue"""

tests/wpt/harness/wptrunner/testrunner.py

@@ -293,8 +293,8 @@ class TestRunnerManager(threading.Thread):
                                 # reason
                                 # Need to consider the unlikely case where one test causes the
                                 # runner process to repeatedly die
-                                self.logger.info("Last test did not complete, requeueing")
+                                self.logger.critical("Last test did not complete")
-                                self.requeue_test()
+                                break
                             self.logger.warning(
                                 "More tests found, but runner process died, restarting")
                             self.restart_count += 1
@@ -466,10 +466,6 @@ class TestRunnerManager(threading.Thread):
     def start_next_test(self):
         self.send_message("run_test")
 
-    def requeue_test(self):
-        self.test_source.requeue(self.test)
-        self.test = None
-
     def test_start(self, test):
         self.test = test
         self.logger.test_start(test.id)

tests/wpt/harness/wptrunner/wptcommandline.py

@@ -86,6 +86,14 @@ def create_parser(product_choices=None):
                         default=False,
                         help="List the tests that are disabled on the current platform")
 
+    build_type = parser.add_mutually_exclusive_group()
+    build_type.add_argument("--debug-build", dest="debug", action="store_true",
+                            default=None,
+                            help="Build is a debug build (overrides any mozinfo file)")
+    build_type.add_argument("--release-build", dest="debug", action="store_false",
+                            default=None,
+                            help="Build is a release (overrides any mozinfo file)")
+
     test_selection_group = parser.add_argument_group("Test Selection")
     test_selection_group.add_argument("--test-types", action="store",
                                       nargs="*", default=["testharness", "reftest"],
@@ -97,6 +105,8 @@ def create_parser(product_choices=None):
                                       help="URL prefix to exclude")
     test_selection_group.add_argument("--include-manifest", type=abs_path,
                                       help="Path to manifest listing tests to include")
+    test_selection_group.add_argument("--tag", action="append", dest="tags",
+                                      help="Labels applied to tests to include in the run. Labels starting dir: are equivalent to top-level directories.")
 
     debugging_group = parser.add_argument_group("Debugging")
     debugging_group.add_argument('--debugger', const="__default__", nargs="?",

tests/wpt/harness/wptrunner/wptmanifest/backends/conditional.py

@@ -115,6 +115,9 @@ class Compiler(NodeVisitor):
     def visit_ValueNode(self, node):
         return (lambda x: True, node.data)
 
+    def visit_AtomNode(self, node):
+        return (lambda x: True, node.data)
+
     def visit_ConditionalNode(self, node):
         return self.visit(node.children[0]), self.visit(node.children[1])
 

tests/wpt/harness/wptrunner/wptmanifest/backends/static.py

@@ -68,6 +68,9 @@ class Compiler(NodeVisitor):
     def visit_ValueNode(self, node):
         return node.data
 
+    def visit_AtomNode(self, node):
+        return node.data
+
     def visit_ListNode(self, node):
         return [self.visit(child) for child in node.children]
 

tests/wpt/harness/wptrunner/wptmanifest/node.py

@@ -93,6 +93,10 @@ class ValueNode(Node):
         raise TypeError
 
 
+class AtomNode(ValueNode):
+    pass
+
+
 class ConditionalNode(Node):
     pass
 

tests/wpt/harness/wptrunner/wptmanifest/parser.py

@@ -44,6 +44,9 @@ binary_operators = ["==", "!=", "and", "or"]
 
 operators = ["==", "!=", "not", "and", "or"]
 
+atoms = {"True": True,
+         "False": False,
+         "Reset": object()}
 
 def decode(byte_str):
     return byte_str.decode("utf8")
@@ -55,7 +58,7 @@ def precedence(operator_node):
 
 class TokenTypes(object):
     def __init__(self):
-        for type in ["group_start", "group_end", "paren", "list_start", "list_end", "separator", "ident", "string", "number", "eof"]:
+        for type in ["group_start", "group_end", "paren", "list_start", "list_end", "separator", "ident", "string", "number", "atom", "eof"]:
             setattr(self, type, type)
 
 token_types = TokenTypes()
@@ -232,6 +235,8 @@ class Tokenizer(object):
             self.state = self.eol_state
         elif self.char() == ",":
             raise ParseError(self.filename, self.line_number, "List item started with separator")
+        elif self.char() == "@":
+            self.state = self.list_value_atom_state
         else:
             self.state = self.list_value_state
 
@@ -267,6 +272,11 @@ class Tokenizer(object):
         if rv:
             yield (token_types.string, decode(rv))
 
+    def list_value_atom_state(self):
+        self.consume()
+        for _, value in self.list_value_state():
+            yield token_types.atom, value
+
     def list_end_state(self):
         self.consume()
         yield (token_types.list_end, "]")
@@ -282,29 +292,36 @@ class Tokenizer(object):
                 self.state = self.comment_state
             else:
                 self.state = self.line_end_state
+        elif self.char() == "@":
+            self.consume()
+            for _, value in self.value_inner_state():
+                yield token_types.atom, value
         else:
-            rv = ""
+            self.state = self.value_inner_state
-            spaces = 0
+
-            while True:
+    def value_inner_state(self):
-                c = self.char()
+        rv = ""
-                if c == "\\":
+        spaces = 0
-                    rv += self.consume_escape()
+        while True:
-                elif c == "#":
+            c = self.char()
-                    self.state = self.comment_state
+            if c == "\\":
-                    break
+                rv += self.consume_escape()
-                elif c == " ":
+            elif c == "#":
-                    # prevent whitespace before comments from being included in the value
+                self.state = self.comment_state
-                    spaces += 1
+                break
-                    self.consume()
+            elif c == " ":
-                elif c == eol:
+                # prevent whitespace before comments from being included in the value
-                    self.state = self.line_end_state
+                spaces += 1
-                    break
+                self.consume()
-                else:
+            elif c == eol:
-                    rv += " " * spaces
+                self.state = self.line_end_state
-                    spaces = 0
+                break
-                    rv += c
+            else:
-                    self.consume()
+                rv += " " * spaces
-            yield (token_types.string, decode(rv))
+                spaces = 0
+                rv += c
+                self.consume()
+        yield (token_types.string, decode(rv))
 
     def comment_state(self):
         while self.char() is not eol:
@@ -544,13 +561,18 @@ class Parser(object):
             if self.token[0] == token_types.string:
                 self.value()
             self.eof_or_end_group()
+        elif self.token[0] == token_types.atom:
+            self.atom()
         else:
             raise ParseError
 
     def list_value(self):
         self.tree.append(ListNode())
-        while self.token[0] == token_types.string:
+        while self.token[0] in (token_types.atom, token_types.string):
-            self.value()
+            if self.token[0] == token_types.atom:
+                self.atom()
+            else:
+                self.value()
         self.expect(token_types.list_end)
         self.tree.pop()
 
@@ -571,6 +593,13 @@ class Parser(object):
         self.consume()
         self.tree.pop()
 
+    def atom(self):
+        if self.token[1] not in atoms:
+            raise ParseError(self.tokenizer.filename, self.tokenizer.line_number, "Unrecognised symbol @%s" % self.token[1])
+        self.tree.append(AtomNode(atoms[self.token[1]]))
+        self.consume()
+        self.tree.pop()
+
     def expr_start(self):
         self.expr_builder = ExpressionBuilder(self.tokenizer)
         self.expr_builders.append(self.expr_builder)
@@ -605,21 +634,21 @@ class Parser(object):
         elif self.token[0] == token_types.number:
             self.expr_number()
         else:
-            raise ParseError
+            raise ParseError(self.tokenizer.filename, self.tokenizer.line_number, "Unrecognised operand")
 
     def expr_unary_op(self):
         if self.token[1] in unary_operators:
             self.expr_builder.push_operator(UnaryOperatorNode(self.token[1]))
             self.consume()
         else:
-            raise ParseError(self.filename, self.tokenizer.line_number, "Expected unary operator")
+            raise ParseError(self.tokenizer.filename, self.tokenizer.line_number, "Expected unary operator")
 
     def expr_bin_op(self):
         if self.token[1] in binary_operators:
             self.expr_builder.push_operator(BinaryOperatorNode(self.token[1]))
             self.consume()
         else:
-            raise ParseError(self.filename, self.tokenizer.line_number, "Expected binary operator")
+            raise ParseError(self.tokenizer.filename, self.tokenizer.line_number, "Expected binary operator")
 
     def expr_value(self):
         node_type = {token_types.string: StringNode,

tests/wpt/harness/wptrunner/wptmanifest/serializer.py

@@ -3,7 +3,9 @@
 # You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from node import NodeVisitor, ValueNode, ListNode, BinaryExpressionNode
-from parser import precedence
+from parser import atoms, precedence
+
+atom_names = {v:"@%s" % k for (k,v) in atoms.iteritems()}
 
 named_escapes = set(["\a", "\b", "\f", "\n", "\r", "\t", "\v"])
 
@@ -80,6 +82,9 @@ class ManifestSerializer(NodeVisitor):
             quote = ""
         return [quote + escape(node.data, extras=quote) + quote]
 
+    def visit_AtomNode(self, node):
+        return [atom_names[node.data]]
+
     def visit_ConditionalNode(self, node):
         return ["if %s: %s" % tuple(self.visit(item)[0] for item in node.children)]
 

tests/wpt/harness/wptrunner/wptmanifest/tests/test_parser.py

@@ -67,5 +67,13 @@ key:
                   ]]]]]]
         )
 
+    def test_atom_0(self):
+        with self.assertRaises(parser.ParseError):
+            self.parse("key: @Unknown")
+
+    def test_atom_1(self):
+        with self.assertRaises(parser.ParseError):
+            self.parse("key: @true")
+
 if __name__ == "__main__":
     unittest.main()

tests/wpt/harness/wptrunner/wptmanifest/tests/test_serializer.py

@@ -209,3 +209,19 @@ class TokenizerTest(unittest.TestCase):
     def test_escape_11(self):
         self.compare(r"""key: \\ab
 """)
+
+    def test_atom_1(self):
+            self.compare(r"""key: @True
+""")
+
+    def test_atom_2(self):
+            self.compare(r"""key: @False
+""")
+
+    def test_atom_3(self):
+            self.compare(r"""key: @Reset
+""")
+
+    def test_atom_4(self):
+        self.compare(r"""key: [a, @Reset, b]
+""")

tests/wpt/harness/wptrunner/wptrunner.py

@@ -40,20 +40,27 @@ def setup_logging(*args, **kwargs):
     global logger
     logger = wptlogging.setup(*args, **kwargs)
 
-def get_loader(test_paths, product, ssl_env, debug=False, **kwargs):
+def get_loader(test_paths, product, ssl_env, debug=None, **kwargs):
     run_info = wpttest.get_run_info(kwargs["run_info"], product, debug=debug)
 
     test_manifests = testloader.ManifestLoader(test_paths, force_manifest_update=kwargs["manifest_update"]).load()
 
-    test_filter = testloader.TestFilter(include=kwargs["include"],
+    manifest_filters = []
-                                        exclude=kwargs["exclude"],
+    meta_filters = []
-                                        manifest_path=kwargs["include_manifest"],
+
-                                        test_manifests=test_manifests)
+    if kwargs["include"] or kwargs["exclude"] or kwargs["include_manifest"]:
+        manifest_filters.append(testloader.TestFilter(include=kwargs["include"],
+                                                      exclude=kwargs["exclude"],
+                                                      manifest_path=kwargs["include_manifest"],
+                                                      test_manifests=test_manifests))
+    if kwargs["tags"]:
+        meta_filters.append(testloader.TagFilter(tags=kwargs["tags"]))
 
     test_loader = testloader.TestLoader(test_manifests,
                                         kwargs["test_types"],
-                                        test_filter,
                                         run_info,
+                                        manifest_filters=manifest_filters,
+                                        meta_filters=meta_filters,
                                         chunk_type=kwargs["chunk_type"],
                                         total_chunks=kwargs["total_chunks"],
                                         chunk_number=kwargs["this_chunk"],
@@ -111,7 +118,7 @@ def run_tests(config, test_paths, product, **kwargs):
         check_args(**kwargs)
 
         if "test_loader" in kwargs:
-            run_info = wpttest.get_run_info(kwargs["run_info"], product, debug=False)
+            run_info = wpttest.get_run_info(kwargs["run_info"], product, debug=None)
             test_loader = kwargs["test_loader"]
         else:
             run_info, test_loader = get_loader(test_paths, product, ssl_env,
@@ -163,6 +170,7 @@ def run_tests(config, test_paths, product, **kwargs):
                     executor_kwargs = get_executor_kwargs(test_type,
                                                           test_environment.external_config,
                                                           test_environment.cache_manager,
+                                                          run_info,
                                                           **kwargs)
 
                     if executor_cls is None:
@@ -212,7 +220,7 @@ def main():
         elif kwargs["list_disabled"]:
             list_disabled(**kwargs)
         else:
-            return run_tests(**kwargs)
+            return not run_tests(**kwargs)
     except Exception:
         import pdb, traceback
         print traceback.format_exc()

tests/wpt/harness/wptrunner/wpttest.py

@@ -9,6 +9,9 @@ import os
 
 import mozinfo
 
+from wptmanifest.parser import atoms
+
+atom_reset = atoms["Reset"]
 
 class Result(object):
     def __init__(self, status, message, expected=None, extra=None):
@@ -58,8 +61,11 @@ class RunInfo(dict):
         self._update_mozinfo(metadata_root)
         self.update(mozinfo.info)
         self["product"] = product
-        if not "debug" in self:
+        if debug is not None:
             self["debug"] = debug
+        elif "debug" not in self:
+            # Default to release
+            self["debug"] = False
 
     def _update_mozinfo(self, metadata_root):
         """Add extra build information from a mozinfo.json file in a parent
@@ -83,27 +89,26 @@ class B2GRunInfo(RunInfo):
 class Test(object):
     result_cls = None
     subtest_result_cls = None
+    test_type = None
 
-    def __init__(self, url, expected_metadata, timeout=DEFAULT_TIMEOUT, path=None,
+    def __init__(self, url, inherit_metadata, test_metadata, timeout=DEFAULT_TIMEOUT, path=None,
                  protocol="http"):
         self.url = url
-        self._expected_metadata = expected_metadata
+        self._inherit_metadata = inherit_metadata
+        self._test_metadata = test_metadata
         self.timeout = timeout
         self.path = path
-        if expected_metadata:
+        self.environment = {"protocol": protocol, "prefs": self.prefs}
-            prefs = expected_metadata.prefs()
-        else:
-            prefs = []
-        self.environment = {"protocol": protocol, "prefs": prefs}
 
     def __eq__(self, other):
         return self.id == other.id
 
     @classmethod
-    def from_manifest(cls, manifest_item, expected_metadata):
+    def from_manifest(cls, manifest_item, inherit_metadata, test_metadata):
         timeout = LONG_TIMEOUT if manifest_item.timeout == "long" else DEFAULT_TIMEOUT
         return cls(manifest_item.url,
-                   expected_metadata,
+                   inherit_metadata,
+                   test_metadata,
                    timeout=timeout,
                    path=manifest_item.path,
                    protocol="https" if hasattr(manifest_item, "https") and manifest_item.https else "http")
@@ -117,22 +122,57 @@ class Test(object):
     def keys(self):
         return tuple()
 
-    def _get_metadata(self, subtest):
+    def _get_metadata(self, subtest=None):
-        if self._expected_metadata is None:
+        if self._test_metadata is not None and subtest is not None:
-            return None
+            return self._test_metadata.get_subtest(subtest)
-
-        if subtest is not None:
-            metadata = self._expected_metadata.get_subtest(subtest)
         else:
-            metadata = self._expected_metadata
+            return self._test_metadata
-        return metadata
+
+    def itermeta(self, subtest=None):
+        for metadata in self._inherit_metadata:
+            yield metadata
+
+        if self._test_metadata is not None:
+            yield self._get_metadata()
+            if subtest is not None:
+                subtest_meta = self._get_metadata(subtest)
+                if subtest_meta is not None:
+                    yield subtest_meta
+
 
     def disabled(self, subtest=None):
-        metadata = self._get_metadata(subtest)
+        for meta in self.itermeta(subtest):
-        if metadata is None:
+            disabled = meta.disabled
-            return False
+            if disabled is not None:
+                return disabled
+        return None
 
-        return metadata.disabled()
+    @property
+    def tags(self):
+        tags = set()
+        for meta in self.itermeta():
+            meta_tags = meta.tags
+            if atom_reset in meta_tags:
+                tags = meta_tags.copy()
+                tags.remove(atom_reset)
+            else:
+                tags |= meta_tags
+
+        tags.add("dir:%s" % self.id.lstrip("/").split("/")[0])
+
+        return tags
+
+    @property
+    def prefs(self):
+        prefs = {}
+        for meta in self.itermeta():
+            meta_prefs = meta.prefs
+            if atom_reset in prefs:
+                prefs = meta_prefs.copy()
+                del prefs[atom_reset]
+            else:
+                prefs.update(meta_prefs)
+        return prefs
 
     def expected(self, subtest=None):
         if subtest is None:
@@ -153,6 +193,7 @@ class Test(object):
 class TestharnessTest(Test):
     result_cls = TestharnessResult
     subtest_result_cls = TestharnessSubtestResult
+    test_type = "testharness"
 
     @property
     def id(self):
@@ -160,6 +201,8 @@ class TestharnessTest(Test):
 
 
 class ManualTest(Test):
+    test_type = "manual"
+
     @property
     def id(self):
         return self.url
@@ -167,9 +210,10 @@ class ManualTest(Test):
 
 class ReftestTest(Test):
     result_cls = ReftestResult
+    test_type = "reftest"
 
-    def __init__(self, url, expected, references, timeout=DEFAULT_TIMEOUT, path=None, protocol="http"):
+    def __init__(self, url, inherit_metadata, test_metadata, references, timeout=DEFAULT_TIMEOUT, path=None, protocol="http"):
-        Test.__init__(self, url, expected, timeout, path, protocol)
+        Test.__init__(self, url, inherit_metadata, test_metadata, timeout, path, protocol)
 
         for _, ref_type in references:
             if ref_type not in ("==", "!="):
@@ -180,7 +224,8 @@ class ReftestTest(Test):
     @classmethod
     def from_manifest(cls,
                       manifest_test,
-                      expected_metadata,
+                      inherit_metadata,
+                      test_metadata,
                       nodes=None,
                       references_seen=None):
 
@@ -194,7 +239,8 @@ class ReftestTest(Test):
         url = manifest_test.url
 
         node = cls(manifest_test.url,
-                   expected_metadata,
+                   inherit_metadata,
+                   test_metadata,
                    [],
                    timeout=timeout,
                    path=manifest_test.path,
@@ -219,11 +265,12 @@ class ReftestTest(Test):
             manifest_node = manifest_test.manifest.get_reference(ref_url)
             if manifest_node:
                 reference = ReftestTest.from_manifest(manifest_node,
+                                                      [],
                                                       None,
                                                       nodes,
                                                       references_seen)
             else:
-                reference = ReftestTest(ref_url, None, [])
+                reference = ReftestTest(ref_url, [], None, [])
 
             node.references.append((reference, ref_type))
 
@@ -243,7 +290,7 @@ manifest_test_cls = {"reftest": ReftestTest,
                      "manual": ManualTest}
 
 
-def from_manifest(manifest_test, expected_metadata):
+def from_manifest(manifest_test, inherit_metadata, test_metadata):
     test_cls = manifest_test_cls[manifest_test.item_type]
 
-    return test_cls.from_manifest(manifest_test, expected_metadata)
+    return test_cls.from_manifest(manifest_test, inherit_metadata, test_metadata)

tests/wpt/metadata/dom/nodes/Document-createElement-namespace.html.ini

@@ -1,3 +1,5 @@
 [Document-createElement-namespace.html]
   type: testharness
-  expected: TIMEOUT
+  expected:
+    if os == "mac": TIMEOUT
+    if os == "linux": CRASH

tests/wpt/metadata/dom/ranges/Range-cloneContents.html.ini

@@ -1,3 +1,3 @@
 [Range-cloneContents.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/dom/ranges/Range-deleteContents.html.ini

@@ -1,3 +1,3 @@
 [Range-deleteContents.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/dom/ranges/Range-extractContents.html.ini

@@ -1,3 +1,3 @@
 [Range-extractContents.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/dom/ranges/Range-insertNode.html.ini

@@ -1,3 +1,3 @@
 [Range-insertNode.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/dom/ranges/Range-surroundContents.html.ini

@@ -1,3 +1,3 @@
 [Range-surroundContents.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/dom/traversal/TreeWalker-acceptNode-filter.html.ini

@@ -1,3 +1,3 @@
 [TreeWalker-acceptNode-filter.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/Create-Secure-blocked-port.htm.ini

@@ -1,3 +1,3 @@
 [Create-Secure-blocked-port.htm]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/004.html.ini

@@ -1,3 +1,3 @@
 [004.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/005.html.ini

@@ -1,3 +1,3 @@
 [005.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/007.html.ini

@@ -1,3 +1,3 @@
 [007.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/008.html.ini

@@ -1,3 +1,3 @@
 [008.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/010.html.ini

@@ -1,3 +1,3 @@
 [010.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/011.html.ini

@@ -1,3 +1,3 @@
 [011.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/012.html.ini

@@ -1,3 +1,3 @@
 [012.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/017.html.ini

@@ -1,3 +1,3 @@
 [017.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/021.html.ini

@@ -1,3 +1,3 @@
 [021.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/constructor/022.html.ini

@@ -1,3 +1,3 @@
 [022.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/cookies/002.html.ini

@@ -1,3 +1,3 @@
 [002.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/cookies/004.html.ini

@@ -1,3 +1,3 @@
 [004.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/cookies/005.html.ini

@@ -1,3 +1,3 @@
 [005.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/cookies/007.html.ini

@@ -1,3 +1,3 @@
 [007.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces.html.ini

@@ -1,3 +1,3 @@
 [interfaces.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-defineProperty-getter.html.ini

@@ -1,3 +1,3 @@
 [bufferedAmount-defineProperty-getter.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-defineProperty-setter.html.ini

@@ -1,3 +1,3 @@
 [bufferedAmount-defineProperty-setter.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-initial.html.ini

@@ -1,3 +1,3 @@
 [bufferedAmount-initial.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/bufferedAmount/bufferedAmount-readonly.html.ini

@@ -1,3 +1,3 @@
 [bufferedAmount-readonly.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/close/close-basic.html.ini

@@ -1,3 +1,3 @@
 [close-basic.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/close/close-connecting.html.ini

@@ -1,3 +1,3 @@
 [close-connecting.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/close/close-multiple.html.ini

@@ -1,3 +1,3 @@
 [close-multiple.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/close/close-nested.html.ini

@@ -1,3 +1,3 @@
 [close-nested.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/close/close-replace.html.ini

@@ -1,3 +1,3 @@
 [close-replace.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/close/close-return.html.ini

@@ -1,3 +1,3 @@
 [close-return.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/constants/001.html.ini

@@ -1,3 +1,3 @@
 [001.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/constants/002.html.ini

@@ -1,3 +1,3 @@
 [002.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/constants/003.html.ini

@@ -1,3 +1,3 @@
 [003.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/constants/004.html.ini

@@ -1,3 +1,3 @@
 [004.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/001.html.ini

@@ -1,3 +1,3 @@
 [001.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/002.html.ini

@@ -1,3 +1,3 @@
 [002.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/003.html.ini

@@ -1,3 +1,3 @@
 [003.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/004.html.ini

@@ -1,3 +1,3 @@
 [004.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/006.html.ini

@@ -1,3 +1,3 @@
 [006.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/007.html.ini

@@ -1,3 +1,3 @@
 [007.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/008.html.ini

@@ -1,3 +1,3 @@
 [008.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/009.html.ini

@@ -1,3 +1,3 @@
 [009.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/010.html.ini

@@ -1,3 +1,3 @@
 [010.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/011.html.ini

@@ -1,3 +1,3 @@
 [011.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/012.html.ini

@@ -1,3 +1,3 @@
 [012.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/013.html.ini

@@ -1,3 +1,3 @@
 [013.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/014.html.ini

@@ -1,3 +1,3 @@
 [014.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/events/020.html.ini

@@ -1,3 +1,3 @@
 [020.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH

tests/wpt/metadata/websockets/interfaces/WebSocket/extensions/001.html.ini

@@ -1,3 +1,3 @@
 [001.html]
   type: testharness
-  expected: TIMEOUT
+  expected: CRASH