Commit graph

1449 commits

Author SHA1 Message Date
Sandeep Hegde
1de6ab16e2 Mutation Testing
Introduced strategy design pattern

Added Strategies:
if true and if false
Replace String literals
Remove if blocks which do not have else.
Modify Comparision Statement - changing <= to < and changing >= to >
Duplicating statements
Added plus to minus and minus to plus mutaiton strategy

Classifying build failures for mutant as unexpected error - Skipping test run for mutant with compilation failure
Added logger messages instead of print
Randomized the mutation test order
Try new strategy on failure to mutate on a file
Updated Readme - Adding mutation strategy and mutation test execution flow
2017-12-06 15:17:28 -05:00
Simon Wörner
663d6bc0c5 Enable pip output in mach_bootstrap.py 2017-11-24 20:47:11 +01:00
Simon Wörner
f08b473c50 Redirect stdout and stderr to file to avoid deadlock 2017-11-24 20:47:00 +01:00
bors-servo
a812af51d6 Auto merge of #19325 - m-novikov:tidy-ignore-without-duplicates, r=jdm
Report an errror if a package has duplicates allowed but there are no duplicates

Resolves: #19306

<!-- Please describe your changes on the following line: -->

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix #19306 (github issue number if applicable).

<!-- Either: -->
- [X] There are tests for these changes

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19325)
<!-- Reviewable:end -->
2017-11-22 11:49:40 -06:00
Simon Sapin
b1ce298d4b Replace compiletest suite by doc-tests with compile_fail
compiletest-rs use internal rustc APIs and is broken in today’s Nightly.
rustdoc however is maintained with rustc and so much less fragile.
2017-11-22 16:25:17 +01:00
Simon Sapin
d96f0ff6a7 Use rustdoc --document-private-item instead of deprecated flags
… and use $RUSTDOCFLAGS instead of $RUSTDOC with a wrapper script
2017-11-22 15:32:40 +01:00
Maxim Novikov
e6d9fd8a4f Report an errror if a package has duplicates allowed but there are no duplicates
Resolves: #19306
2017-11-21 21:08:14 +01:00
Alan Jeffrey
be5eaa4447 Submit test-perf CSV files to S3. 2017-11-21 09:27:30 -06:00
Alan Jeffrey
ffd82a53df Renamed mach test-perf -submut to --submit. 2017-11-20 10:59:13 -06:00
bors-servo
50f11e3584 Auto merge of #18984 - dsandeephegde:master, r=jdm
Initial steps of Mutation testing

<!-- Please describe your changes on the following line: -->
- Added one strategy of mutation which is replacing occurrences && to ||.
- Added test mapping framework for running mutation tests corresponding to a mutant.
- Added one test_mapping.json to map source file in a folder to WPT test.
- Added README mentioning about Mutation testing.
- Added CI script to invoke mutation test.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes #18529 (github issue number if applicable).
- [x] These changes do not require tests because it is a python script to run mutation test and does not change any behavior.

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/18984)
<!-- Reviewable:end -->
2017-11-16 11:51:01 -06:00
Josh Matthews
c227604a2c Update web-platform-tests to revision 44702f2bc8ea98bc32b5b244f2fe63c6ce66d49d 2017-11-15 12:47:30 -05:00
Simon Sapin
3264453284 Stop using "alternate" rustc builds.
With https://github.com/rust-lang/rust/pull/45810,
normal Nightly now has LLVM assertions disabled.
2017-11-14 09:08:11 +01:00
Sandeep Hegde
a9493d0887 Corrected a typo 2017-11-13 21:12:53 -05:00
Sandeep Hegde
f817a9c371 Refactored mutate random line method 2017-11-13 20:47:06 -05:00
Sandeep Hegde
61794f8bbd Made test summary more descriptive and Updated Readme 2017-11-13 20:40:16 -05:00
Alan Jeffrey
109df3328a Add --base option to test-perf. 2017-11-13 18:58:06 -06:00
Anthony Ramine
0c2ccc8914 Fix a typo in tidy.py 2017-11-13 16:04:36 +01:00
Sandeep Hegde
cc6c2eea6e Added mutation test summary and made it exit with relevant exit code 2017-11-12 23:43:04 -05:00
tigercosmos
58b201d43e enhance android build debug info 2017-11-12 20:37:50 +08:00
Sandeep Hegde
b8199e11e0 Changed readme saying wildcard not allowed in test_mapping.josn 2017-11-10 20:48:47 -05:00
Sandeep Hegde
6e66c0d1ef Corrected typo in Readme 2017-11-10 20:22:12 -05:00
tigercosmos
2792369a6d remove unused script 2017-11-09 17:04:20 +08:00
Sandeep Hegde
f6f454505e fixed tidy error 2017-11-08 13:00:14 -05:00
Sandeep Hegde
84f694dc93 Added more information in Mutation Test Readme 2017-11-08 12:58:46 -05:00
Josh Matthews
86e3f0deef Use failure reporting API on CI. 2017-11-07 14:01:24 -05:00
Josh Matthews
c469cbbe72 Report instances of intermittent failures to a tracker. 2017-11-07 13:40:38 -05:00
Josh Matthews
c0f85579c5 Allow specifying alternative tracker endpoints. 2017-11-07 13:40:07 -05:00
Sandeep Hegde
d24d6ac8ef Skipping mutation test for file with local changes 2017-11-06 13:00:14 -05:00
Sandeep Hegde
ffbabf4a4c Added randomness to the mutation strategy
- now running only maximum one mutation per file.
- Beautified the output
2017-11-06 13:00:14 -05:00
Sandeep Hegde
5cc498dba2 Loging success message on mutation test success 2017-11-06 13:00:13 -05:00
panup21091993
5366264494 Added Readme.md file for mutation testing 2017-11-06 13:00:13 -05:00
Sandeep Hegde
2b8b98d3a6 Removed build and wpt-test output from mutation test log and refactored code. 2017-11-06 13:00:13 -05:00
Sandeep Hegde
6688b8a146 Changed method to mutate a line of code 2017-11-06 13:00:13 -05:00
Sandeep Hegde
58ab11cf9b Fixed few tidy errors 2017-11-06 13:00:13 -05:00
Sandeep Hegde
445ab9ae8d Reverting wrong comment. 2017-11-06 13:00:12 -05:00
Sandeep Hegde
ba2152900c Added Test Mapping framework and running through a path. 2017-11-06 13:00:12 -05:00
Sandeep Hegde
0d2ad9a5be Checking the wpt test failures/success. 2017-11-06 13:00:12 -05:00
Sandeep Hegde
abbdcf0afc XMLHTTPRequest Mutator - Initial step Mutation testing 2017-11-06 13:00:12 -05:00
bors-servo
4df647ed75 Auto merge of #19017 - tigercosmos:test, r=jdm
Print the full path for errors occurring in the servo crate

<!-- Please describe your changes on the following line: -->
`Cargo` will print the path where it runs.
Origin python script `cd` into the crate folder, so the root path is set in  the crate.
Now I use `--manifest-path PATH` to `cargo build`, so the root path is at `servo`.

Origin path in error message:
```
error: expected one of `!` or `::`, found `#`
  --> lib.rs:24:1
```

Now it would be:
```
error: expected one of `!` or `::`, found `use`
  --> ports/geckolib/glue.rs:11:1
```

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix #9895 (github issue number if applicable).

<!-- Either: -->
- [ ] There are tests for these changes OR
- [ ] These changes do not require tests because _____

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19017)
<!-- Reviewable:end -->
2017-11-03 09:05:17 -05:00
bors-servo
af1cd386f5 Auto merge of #19109 - tigercosmos:ndk, r=jdm
Check if the NDK version is 12

<!-- Please describe your changes on the following line: -->
Since we only support NDK 12(b), we should add some script to check for that.
User might download 15 or 16, and would get some errors.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix #19095 (github issue number if applicable).

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19109)
<!-- Reviewable:end -->
2017-11-03 08:14:27 -05:00
tigercosmos
3cdba605af Check if the NDK version is 12 2017-11-03 20:59:56 +08:00
tigercosmos
a0ae81610d Print the full path for errors occurring in the servo crate 2017-11-03 13:54:39 +08:00
bors-servo
92b49010b1 Auto merge of #19091 - MortimerGoro:webgl2_suite, r=jdm
Add WebGL conformance 2.0.0 tests

<!-- Please describe your changes on the following line: -->

This PR includes WebGL conformance 2.0.0 tests. For now I have disabled three folders in the new suite:
- `Conformance`: Tests WebGL 1.0 API like the current 1.0.3 suite we are using, but it includes a lot more tests and many expectations have to be updated. This is better to do in a separate PR because it will require to update github intermittent paths again and maybe detect new intermittents.
- `Conformance2\textures`: It causes a lot of timeouts because of using videos, svgs, and other complicated formats.
- `deqp`: It's a extra GPU testing suite included in the Khronos 2.0.0 suite. Disabled for now because it causes some timeouts.

So in a nutshell, it uses part of the 2.0.0 suite to test webgl 2.0 and keeps using 1.0.3 in order to test WebGL 1.0. It's good enough to enable TDD for the new WebGL 2.0 features

We can create follow-up issues for the next steps:
* deprecate 1.0.3 and use the new 'conformance' folder in 2.0.0 in order to test WebGL 1.0
* Enable `conformance2/textures` and  'deqp' tests once WebGL 2.0 implementation is more advanced or the timeouts are monitored in more detail.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [ ] These changes fix #__ (github issue number if applicable).

<!-- Either: -->
- [x] There are tests for these changes OR
- [ ] These changes do not require tests because _____

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19091)
<!-- Reviewable:end -->
2017-11-02 15:52:42 -05:00
bors-servo
86b9e7d7d6 Auto merge of #19070 - jdm:csstests, r=metajack
Enable CSS tests.

Do not merge this; I'm looking into what it takes to stop running the test-css jobs.

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19070)
<!-- Reviewable:end -->
2017-11-02 11:35:01 -05:00
Josh Matthews
28c20988f0 Remove all outdated CSS-related mach commands. 2017-11-02 09:22:30 -04:00
Josh Matthews
b8e44e14b2 Account for changes to Cargo.lock format. 2017-11-02 09:21:59 -04:00
Imanol Fernandez
90f657757f Add WebGL conformance 2.0.0 patches 2017-11-02 12:21:53 +01:00
Simon Sapin
93717356f1 Check full triples for "alt" compiler support, not just platforms.
Fixes https://github.com/servo/servo/issues/19075
2017-10-31 22:13:19 +01:00
Imanol Fernandez
4989f00916 Update WebGL conformance suite 1.0.3 from upstream 2017-10-31 16:00:12 +01:00
Imanol Fernandez
b5043de011 Create match command to update WebGL conformance suite from upstream 2017-10-31 16:00:12 +01:00