Mirrors/servo
1
0
Fork 0
mirror of https://github.com/servo/servo.git synced 2025-08-03 20:50:07 +01:00
Code Issues Projects Releases Packages Wiki Activity

Filter unknown flaky tests when filtering known intermittents

Browse source 
There are two kinds of flaky/intermittent tests in Servo. The
traditional kind is the test that fails on the CI, but has an associated
bug indicating that the test is an intermittent failure. Many of these
tests have completely unstable results, for instance those where an
unpredictable set of subtests fail. It's impossible to generate stable
results for these, so we have traditionally simply discard these
unexpected results.

Another kind of intermittent test is one that will produce an expected
result when rerun (ie will flake). Some of these are also labeled with
bugs, while some are not. In some cases, there is flakiness in some core
Servo functionality that can lead to *any* test flaking, such as a race
condition that can lead to an early screenshot for reftests. When these
kinds of tests do not have associated bugs, they cause the CI to fail.
In this case, it is impossible to label these tests as intermittent
because it can literally be any test.

This change, reruns failed tests in order to detect unlabeled tests in
the second category. Instead of blocking the CI when the second run
leads to expected results, the CI will now pass, but the flake will be
reported to the new flakiness dashboard. This prevents unrelated flakes
from slowing down the merge queue.
This commit is contained in:
Martin Robinson 2023-02-16 12:15:57 +01:00
parent 883dcbda45
commit 5e30ce8532
3 changed files with 79 additions and 60 deletions
Download patch file Download diff file Expand all files Collapse all files

38
etc/ci/report_aggregated_expected_results.py
View file

@ -97,28 +97,32 @@ class Item:
def get_results() -> Optional[Item]:
unexpected = []
known_intermittents = []
for filename in sys.argv[1:]:
with open(filename, encoding="utf-8") as file:
data = json.load(file)
unexpected += data["unexpected"]
known_intermittents += data["known_intermittents"]
unexpected += json.load(file)
unexpected.sort(key=lambda result: result["path"])
known_intermittents.sort(key=lambda result: result["path"])
def is_flaky(result):
return result["flaky"]
def is_stable_and_known(result):
return not is_flaky(result) and result["issues"]
def is_stable_and_unexpected(result):
return not is_flaky(result) and not result["issues"]
def add_children(children, results, filter_func, text):
filtered = [Item.from_result(result) for result in
filter(filter_func, results)]
if filtered:
children.append(Item(f"{text} ({len(filtered)})", "", filtered))
children = []
if unexpected:
children.append(
Item(f"Tests producing unexpected results ({len(unexpected)})", "",
[Item.from_result(result) for result in unexpected]),
)
if known_intermittents:
children.append(
Item("Unexpected results that are known to be intermittent "
f"({len(known_intermittents)})", "",
[Item.from_result(result) for result in known_intermittents])
)
add_children(children, unexpected, is_flaky, "Flaky unexpected result")
add_children(children, unexpected, is_stable_and_known,
"Stable unexpected results that are known to be intermittent")
add_children(children, unexpected, is_stable_and_unexpected,
"Stable unexpected results")
run_url = get_github_run_url()
if run_url: