Mirrors/servo
1
0
Fork 0
mirror of https://github.com/servo/servo.git synced 2025-10-02 09:39:14 +01:00
Code Issues Projects Releases Packages Wiki Activity

Remove test_mapping.json and python/servo/mutation (#39617)

Browse source 
As per #39585.

>It used to run on our old buildbot CI for a couple years. It was never
migrated, and we should clean it up.

_Originally posted by @jdm in
https://github.com/servo/servo/issues/39585#issuecomment-3351054660_

---------

Signed-off-by: Ashwin Naren <arihant2math@gmail.com>
This commit is contained in:
Ashwin Naren 2025-10-01 21:31:54 -07:00 • committed by GitHub
parent 65588cd5df
commit 1448fd5967
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 0 additions and 435 deletions
Download patch file Download diff file Expand all files Collapse all files

8
components/script/dom/test_mapping.json
View file

@ -1,8 +0,0 @@
{
"xmlhttprequest.rs": [
"XMLHttpRequest"
],
"range.rs": [
"dom/ranges"
]
}

13
etc/ci/mutation_test.sh
View file

@ -1,13 +0,0 @@
#!/usr/bin/env bash
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at https://mozilla.org/MPL/2.0/.
set -o errexit
set -o nounset
set -o pipefail
PS1="" source python/_virtualenv/bin/activate
# `PS1` must be defined before activating virtualenv
python python/servo/mutation/init.py components/script/dom

1
pyproject.toml
View file

@ -65,7 +65,6 @@ project-excludes = [
"**/*_tests.py",
"**/tests/**",
"python/mach/**/*.py",
"python/servo/mutation/**/*.py",
]
[tool.uv]

85
python/servo/mutation/README.md
View file

@ -1,85 +0,0 @@
## Implement Mutation Testing on Servo Parallel Browsing Project
The motivation for mutation testing is to test the breadth coverage of tests for source code. Faults (or mutations) are automatically seeded into the code, then tests are run. If tests fail then the mutation is killed, if the tests pass then the mutation lived. The quality of tests can be gauged from the percentage of mutations killed.
For more info refer [Wiki page](https://en.wikipedia.org/wiki/Mutation_testing).
Here Mutation testing is used to test the coverage of WPT for Servo's browser engine.
### Mutation Strategies
The mutation test consists of a Python script that mutates random lines in Servo's code base. The expectation from the WPT tests is to catch this bug caused by mutation and result in test failures.
There are few strategies to mutate source code in order to create bugs. The strategies are randomly picked for each file. Some of the strategies are:
* Change Conditional flow
* Delete if block
* Change Arithmetic operations
#### How To Add a New Strategy?
Write new class inheriting the Strategy class in mutator.py and include it in get_strategies method. Override mutate method or provide replace strategy regex if it works with mutate method of Strategy class.
### Test Run Strategy
The mutation test aims to run only tests which are concerned with the mutant. Therefore part of WPT test is related to the source code under mutation is invoked. For this it requires a test mapping in source folders.
#### test_mapping.json
The file test_mapping.json is used to map the source code to their corresponding WPT tests. The user must maintain a updated version of this file in the path where mutation testing needs to be performed. Additionally, the test_mapping.json will only consist of maps of source codes that are present in the current directory. Hence, each folder will have a unique test_mapping.json file. Any source code files that may be present in a path but are not mapped to a WPT in test_mapping.json will not be covered for mutation testing.
#### Sample test_mapping.json format
A sample of test_mapping.json is as shown below:
```
{
"xmlhttprequest.rs": [
"XMLHttpRequest"
],
"range.rs": [
"dom/ranges"
]
}
```
Please ensure that each folder that requires a mutant to be generated consists of test_mapping.json file so that the script can function as expected. Wildcards are not allowed in test_mapping.json.
If we want to run mutation test for a source path then there should be test_mapping.json in that path and all the subdirectories which has source files.
Eg: There should be test mapping in following folders if we run mutation test on 'components/script' path.
* components/script/test_mapping.json
* components/script/dom/test_mapping.json
* components/script/task_source/test_mapping.json
* components/script/dom/bindings/test_mapping.json
* ...
### Running Mutation test
The mutation tests can be run by running the below command from the servo directory on the command line interface:
`python python/servo/mutation/init.py <Mutation path>`
Eg. `python python/servo/mutation/init.py components/script/dom`
### Running Mutation Test from CI
The CI script for running mutation testing is present in /etc/ci folder. It can be called by executing the below command from the CLI:
`./etc/ci/mutation_test.sh`
### Execution Flow
1. The script is called from the command line, it searches for test_mapping.json in the path entered by user.
2. If found, it reads the json file and parses it, gets source file to tests mapping. For all source files in the mapping file, it does the following.
3. If the source file does not have any local changes then it mutates at a random line using a random strategy. It retries with other strategies if that strategy could not produce any mutation.
4. The code is built and the corresponding WPT tests are run for this mutant and the test results are logged.
5. Once it has completed executing mutation testing for the entered path, it repeats the above procedure for sub-paths present inside the entered path.
### Test Summary
At the end of the test run the test summary displayed which looks like this:
```
Test Summary:
Mutant Killed (Success) 25
Mutant Survived (Failure) 10
Mutation Skipped 1
Unexpected error in mutation 0
```
* Mutant Killed (Success): The mutant was successfully killed by WPT test suite.
* Mutant Survived (Failure): The mutation has survived the WPT Test Suite, tests in WPT could not catch this mutation.
* Mutation Skipped: Files is skipped for mutation test due to the local changes in that file.
* Unexpected error in mutation: Mutation test could not run due to unexpected failures. (example: if no && preset in the file to replace)

57
python/servo/mutation/init.py
View file

@ -1,57 +0,0 @@
# Copyright 2013 The Servo Project Developers. See the COPYRIGHT
# file at the top-level directory of this distribution.
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.
from os import listdir
from os.path import isfile, isdir, join
import json
import sys
import test
import logging
import random
test_summary = {test.Status.KILLED: 0, test.Status.SURVIVED: 0, test.Status.SKIPPED: 0, test.Status.UNEXPECTED: 0}
def get_folders_list(path: str) -> list[str]:
folder_list = []
for filename in listdir(path):
if isdir(join(path, filename)):
folder_name = join(path, filename)
folder_list.append(folder_name)
return folder_list
return folder_list
def mutation_test_for(mutation_path: str) -> None:
test_mapping_file = join(mutation_path, "test_mapping.json")
if isfile(test_mapping_file):
json_data = open(test_mapping_file).read()
test_mapping = json.loads(json_data)
# Run mutation test for all source files in mapping file in a random order.
source_files = list(test_mapping.keys())
random.shuffle(source_files)
for src_file in source_files:
status = test.mutation_test(join(mutation_path, src_file.encode("utf-8")), test_mapping[src_file])
test_summary[status] += 1
# Run mutation test in all folder in the path.
for folder in get_folders_list(mutation_path):
mutation_test_for(folder)
else:
logging.warning("This folder {0} has no test mapping file.".format(mutation_path))
mutation_test_for(sys.argv[1])
logging.basicConfig(format="%(asctime)s %(levelname)s: %(message)s", level=logging.DEBUG)
logging.info("Test Summary:")
logging.info("Mutant Killed (Success) \t\t{0}".format(test_summary[test.Status.KILLED]))
logging.info("Mutant Survived (Failure) \t{0}".format(test_summary[test.Status.SURVIVED]))
logging.info("Mutation Skipped \t\t\t{0}".format(test_summary[test.Status.SKIPPED]))
logging.info("Unexpected error in mutation \t{0}".format(test_summary[test.Status.UNEXPECTED]))
if test_summary[test.Status.SURVIVED]:
sys.exit(1)

199
python/servo/mutation/mutator.py
View file

@ -1,199 +0,0 @@
# Copyright 2013 The Servo Project Developers. See the COPYRIGHT
# file at the top-level directory of this distribution.
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.
import fileinput
import re
from re import Match
import random
from collections.abc import Iterator
def is_comment(line: str) -> Match[str] | None:
return re.search(r"\/\/.*", line)
def init_variables(if_blocks: list[int]) -> tuple[int, int, int, int, int]:
random_index = random.randint(0, len(if_blocks) - 1)
start_counter = 0
end_counter = 0
lines_to_delete = []
line_to_mutate = if_blocks[random_index]
return random_index, start_counter, end_counter, lines_to_delete, line_to_mutate
def deleteStatements(file_name: str, line_numbers: list[int]) -> None:
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() not in line_numbers:
print(line.rstrip())
class Strategy:
def __init__(self) -> None:
self._strategy_name = ""
self._replace_strategy = {}
def mutate(self, file_name: str) -> int:
line_numbers = []
for line in fileinput.input(file_name):
if not is_comment(line) and re.search(self._replace_strategy["regex"], line):
line_numbers.append(fileinput.lineno())
if len(line_numbers) == 0:
return -1
else:
mutation_line_number = line_numbers[random.randint(0, len(line_numbers) - 1)]
for line in fileinput.input(file_name, inplace=True):
if fileinput.lineno() == mutation_line_number:
line = re.sub(self._replace_strategy["regex"], self._replace_strategy["replaceString"], line)
print(line.rstrip())
return mutation_line_number
class AndOr(Strategy):
def __init__(self) -> None:
Strategy.__init__(self)
logical_and = r"(?<=\s)&&(?=\s)"
self._replace_strategy = {"regex": logical_and, "replaceString": "||"}
class IfTrue(Strategy):
def __init__(self) -> None:
Strategy.__init__(self)
if_condition = r"(?<=if\s)\s*(?!let\s)(.*)(?=\s\{)"
self._replace_strategy = {"regex": if_condition, "replaceString": "true"}
class IfFalse(Strategy):
def __init__(self) -> None:
Strategy.__init__(self)
if_condition = r"(?<=if\s)\s*(?!let\s)(.*)(?=\s\{)"
self._replace_strategy = {"regex": if_condition, "replaceString": "false"}
class ModifyComparision(Strategy):
def __init__(self) -> None:
Strategy.__init__(self)
less_than_equals = r"(?<=\s)(\<)\=(?=\s)"
greater_than_equals = r"(?<=\s)(\<)\=(?=\s)"
self._replace_strategy = {"regex": (less_than_equals + "|" + greater_than_equals), "replaceString": r"\1"}
class MinusToPlus(Strategy):
def __init__(self) -> None:
Strategy.__init__(self)
arithmetic_minus = r"(?<=\s)\-(?=\s.+)"
minus_in_shorthand = r"(?<=\s)\-(?=\=)"
self._replace_strategy = {"regex": (arithmetic_minus + "|" + minus_in_shorthand), "replaceString": "+"}
class PlusToMinus(Strategy):
def __init__(self) -> None:
Strategy.__init__(self)
arithmetic_plus = r"(?<=[^\"]\s)\+(?=\s[^A-Z\'?\":\{]+)"
plus_in_shorthand = r"(?<=\s)\+(?=\=)"
self._replace_strategy = {"regex": (arithmetic_plus + "|" + plus_in_shorthand), "replaceString": "-"}
class AtomicString(Strategy):
def __init__(self) -> None:
Strategy.__init__(self)
string_literal = r"(?<=\").+(?=\")"
self._replace_strategy = {"regex": string_literal, "replaceString": " "}
class DuplicateLine(Strategy):
def __init__(self) -> None:
Strategy.__init__(self)
self._strategy_name = "duplicate"
append_statement = r".+?append\(.+?\).*?;"
remove_statement = r".+?remove\(.*?\).*?;"
push_statement = r".+?push\(.+?\).*?;"
pop_statement = r".+?pop\(.+?\).*?;"
plus_equals_statement = r".+?\s\+\=\s.*"
minus_equals_statement = r".+?\s\-\=\s.*"
self._replace_strategy = {
"regex": (
append_statement
+ "|"
+ remove_statement
+ "|"
+ push_statement
+ "|"
+ pop_statement
+ "|"
+ plus_equals_statement
+ "|"
+ minus_equals_statement
),
"replaceString": r"\g<0>\n\g<0>",
}
class DeleteIfBlock(Strategy):
def __init__(self) -> None:
Strategy.__init__(self)
self.if_block = r"^\s+if\s(.+)\s\{"
self.else_block = r"\selse(.+)\{"
def mutate(self, file_name: str) -> int:
code_lines = []
if_blocks = []
for line in fileinput.input(file_name):
code_lines.append(line)
if re.search(self.if_block, line):
if_blocks.append(fileinput.lineno())
if len(if_blocks) == 0:
return -1
random_index, start_counter, end_counter, lines_to_delete, line_to_mutate = init_variables(if_blocks)
while line_to_mutate <= len(code_lines):
current_line = code_lines[line_to_mutate - 1]
next_line = code_lines[line_to_mutate]
if (
re.search(self.else_block, current_line) is not None
or re.search(self.else_block, next_line) is not None
):
if_blocks.pop(random_index)
if len(if_blocks) == 0:
return -1
else:
random_index, start_counter, end_counter, lines_to_delete, line_to_mutate = init_variables(
if_blocks
)
continue
lines_to_delete.append(line_to_mutate)
for ch in current_line:
if ch == "{":
start_counter += 1
elif ch == "}":
end_counter += 1
if start_counter and start_counter == end_counter:
deleteStatements(file_name, lines_to_delete)
return lines_to_delete[0]
line_to_mutate += 1
def get_strategies() -> Iterator[Strategy]:
return (
AndOr,
IfTrue,
IfFalse,
ModifyComparision,
PlusToMinus,
MinusToPlus,
AtomicString,
DuplicateLine,
DeleteIfBlock,
)
class Mutator:
def __init__(self, strategy: Strategy) -> None:
self._strategy = strategy
def mutate(self, file_name: str) -> int:
return self._strategy.mutate(file_name)

72
python/servo/mutation/test.py
View file

@ -1,72 +0,0 @@
# Copyright 2013 The Servo Project Developers. See the COPYRIGHT
# file at the top-level directory of this distribution.
#
# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
# option. This file may not be copied, modified, or distributed
# except according to those terms.
import subprocess
import os
import random
import logging
from mutator import Mutator, get_strategies
from enum import Enum
DEVNULL = open(os.devnull, "wb")
logging.basicConfig(format="%(asctime)s %(levelname)s: %(message)s", level=logging.DEBUG)
class Status(Enum):
KILLED = 0
SURVIVED = 1
SKIPPED = 2
UNEXPECTED = 3
def mutation_test(file_name, tests):
status = Status.UNEXPECTED
local_changes_present = subprocess.call("git diff --quiet {0}".format(file_name), shell=True)
if local_changes_present == 1:
status = Status.SKIPPED
logging.warning("{0} has local changes, please commit/remove changes before running the test".format(file_name))
else:
strategies = list(get_strategies())
while len(strategies):
strategy = random.choice(strategies)
strategies.remove(strategy)
mutator = Mutator(strategy())
mutated_line = mutator.mutate(file_name)
if mutated_line != -1:
logging.info("Mutated {0} at line {1}".format(file_name, mutated_line))
logging.info("compiling mutant {0}:{1}".format(file_name, mutated_line))
test_command = "python mach build --release"
if subprocess.call(test_command, shell=True, stdout=DEVNULL):
logging.error("Compilation Failed: Unexpected error")
logging.error("Failed: while running `{0}`".format(test_command))
subprocess.call("git --no-pager diff {0}".format(file_name), shell=True)
status = Status.UNEXPECTED
else:
for test in tests:
test_command = "python mach test-wpt {0} --release".format(test.encode("utf-8"))
logging.info("running `{0}` test for mutant {1}:{2}".format(test, file_name, mutated_line))
test_status = subprocess.call(test_command, shell=True, stdout=DEVNULL)
if test_status != 0:
logging.error("Failed: while running `{0}`".format(test_command))
logging.error("mutated file {0} diff".format(file_name))
subprocess.call("git --no-pager diff {0}".format(file_name), shell=True)
status = Status.SURVIVED
else:
logging.info("Success: Mutation killed by {0}".format(test.encode("utf-8")))
status = Status.KILLED
break
logging.info("reverting mutant {0}:{1}\n".format(file_name, mutated_line))
subprocess.call("git checkout {0}".format(file_name), shell=True)
break
elif not len(strategies):
# All strategies are tried
logging.info("\nCannot mutate {0}\n".format(file_name))
return status