Commit graph

566 commits

Author SHA1 Message Date
Alan Jeffrey
2f9fe8ca22 Add a smoketest to the mac-rel-wpt1 builder. 2017-12-15 16:45:35 -06:00
E. Dunham
5ccca8c9b0 Add .taskcluster.yml
Template mostly from https://tools.taskcluster.net/quickstart/. Steps from the
linux-dev builder in etc/ci/buildbot_steps.yml are in taskcluster-test.sh.
2017-12-14 11:16:29 -08:00
bors-servo
b93579a8f0 Auto merge of #19507 - asajeffrey:test-perf-date-option, r=avadacatavra
Add a --date option to test-perf.

<!-- Please describe your changes on the following line: -->

To generate old test-perf results, we need a way to set the date when running `./mach test-perf`.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because they're test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19507)
<!-- Reviewable:end -->
2017-12-12 13:52:59 -06:00
Alan Jeffrey
a23b80e6e1 Capture loadEventEnd in test-perf. 2017-12-08 12:49:24 -06:00
Alan Jeffrey
8754695c82 Use urllib rather than httplib2 when downloading build perf data. 2017-12-08 10:26:17 -06:00
Alan Jeffrey
ee766b5e16 Add a --date option to test-perf. 2017-12-06 21:57:08 -06:00
bors-servo
b56d7b8070 Auto merge of #19450 - asajeffrey:buildbot-perf-upload-to-s3, r=jdm
Upload buildbot perf data to S3.

<!-- Please describe your changes on the following line: -->

Upload performance data to S3.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because this is test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19450)
<!-- Reviewable:end -->
2017-12-05 09:19:19 -06:00
bors-servo
58a52331d9 Auto merge of #19484 - asajeffrey:test-perf-different-port, r=jdm
Moved the test-perf http server to a different port.

<!-- Please describe your changes on the following line: -->

Run the test-perf http server on a different port than WPT uses.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because this is test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19484)
<!-- Reviewable:end -->
2017-12-05 04:43:18 -06:00
Alan Jeffrey
79b104bde7 Moved the test-perf http server to a different port. 2017-12-04 12:48:20 -06:00
Alan Jeffrey
d13ff4daf0 Move the trap instruction in /etc/ci/performance/test_all.sh. 2017-12-03 21:22:38 -06:00
Alan Jeffrey
524df9e18e Upload buildbot perf data to S3. 2017-12-01 11:47:07 -06:00
bors-servo
9da7663e29 Auto merge of #19344 - asajeffrey:buldbot-perf-download-json, r=jdm
Download the buildbot statistics and save them as CSV

<!-- Please describe your changes on the following line: -->

Download timing data from build.servo.org, and convert it to CSV for feeding to Google Data Studio.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because this is test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19344)
<!-- Reviewable:end -->
2017-12-01 06:52:26 -06:00
Alan Jeffrey
7151f24ca5 Download the buildbot statistics and save them as CSV. 2017-11-30 16:20:23 -06:00
Alan Jeffrey
4ad801930a Added explicit UTF-8 encoding to test-perf. 2017-11-30 12:45:26 -06:00
bors-servo
8b6b2e97c8 Auto merge of #19323 - asajeffrey:test-perf-run-nightly, r=edunham
Run test-perf on linux-nightly.

<!-- Please describe your changes on the following line: -->

Run test-perf on linux-nightly.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix #12988
- [X] These changes do not require tests because this is test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19323)
<!-- Reviewable:end -->
2017-11-27 16:12:14 -06:00
bors-servo
218edeb883 Auto merge of #19336 - servo:jdm-patch-6, r=emilio
Instruct how to deal with outdated manifests.

<!-- Reviewable:start -->
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19336)
<!-- Reviewable:end -->
2017-11-22 10:36:04 -06:00
Josh Matthews
b45e4fecff
Address tidy checks. 2017-11-22 10:48:51 -05:00
Simon Sapin
b1ce298d4b Replace compiletest suite by doc-tests with compile_fail
compiletest-rs use internal rustc APIs and is broken in today’s Nightly.
rustdoc however is maintained with rustc and so much less fragile.
2017-11-22 16:25:17 +01:00
Simon Sapin
d96f0ff6a7 Use rustdoc --document-private-item instead of deprecated flags
… and use $RUSTDOCFLAGS instead of $RUSTDOC with a wrapper script
2017-11-22 15:32:40 +01:00
Josh Matthews
b501c8fa03
Instruct how to deal with outdated manifests. 2017-11-22 08:16:54 -05:00
Josh Matthews
da46fe4812
Unbreak docs build. 2017-11-22 07:47:04 -05:00
Josh Matthews
5a36c481f0
Use same build environment during mutation testing. 2017-11-21 13:47:54 -05:00
Alan Jeffrey
1a59ea9733 Run test-perf on linux-nightly. 2017-11-21 09:45:34 -06:00
Alan Jeffrey
be5eaa4447 Submit test-perf CSV files to S3. 2017-11-21 09:27:30 -06:00
Alan Jeffrey
a57ba2a317 Include OS and architecture in test-perf data. 2017-11-20 14:12:21 -06:00
bors-servo
98c9e5f1ac Auto merge of #19245 - asajeffrey:test-perf-update-manifest, r=jdm
Update tp5n manifest for test-perf

<!-- Please describe your changes on the following line: -->

Updated test-perf tp5n manifest.

This PR provides a `test.manifest` file, that contains a subset of the tp5n tests that don't timeout (at least on my machine).

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because this is test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19245)
<!-- Reviewable:end -->
2017-11-20 11:09:38 -06:00
bors-servo
b5a205d92e Auto merge of #19257 - asajeffrey:test-perf-include-date-in-csv, r=jdm
Include a YYYYMMDD date field in the raw test-perf CSV to make Google Data Studio happy

<!-- Please describe your changes on the following line: -->

AFAICT Google Data Studio needs a YYYMMDD date string in the raw data to be able to provide time-indexed graphs. `TODATE` is not converting unix timestamps to YYYYMMDD.

https://www.en.advertisercommunity.com/t5/Data-Studio/Can-Data-Studio-Date-Dimension-use-epoch-in-seconds-from/td-p/839649

This PR is a workaround, and includes a date field in the raw CSV.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because this is test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19257)
<!-- Reviewable:end -->
2017-11-16 16:46:26 -06:00
Alan Jeffrey
ed36323e0b Include a YYYYMMDD date field in the raw test-perf CSV to make Google Data Studio happy. 2017-11-16 16:36:10 -06:00
bors-servo
50f11e3584 Auto merge of #18984 - dsandeephegde:master, r=jdm
Initial steps of Mutation testing

<!-- Please describe your changes on the following line: -->
- Added one strategy of mutation which is replacing occurrences && to ||.
- Added test mapping framework for running mutation tests corresponding to a mutant.
- Added one test_mapping.json to map source file in a folder to WPT test.
- Added README mentioning about Mutation testing.
- Added CI script to invoke mutation test.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes #18529 (github issue number if applicable).
- [x] These changes do not require tests because it is a python script to run mutation test and does not change any behavior.

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/18984)
<!-- Reviewable:end -->
2017-11-16 11:51:01 -06:00
Alan Jeffrey
b923559d55 Update tp5n manifest for test-perf. 2017-11-16 09:33:43 -06:00
bors-servo
ee0294380b Auto merge of #19233 - asajeffrey:test-perf-csv, r=jdm
Save test-perf data in csv

<!-- Please describe your changes on the following line: -->

Most data analytics and visualization tools (e.g. Google Data Studio and Amazon Quick Sight) seem to want .csv input, not .json. This PR makes .csv the default data format.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because it's test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19233)
<!-- Reviewable:end -->
2017-11-16 00:55:30 -06:00
bors-servo
e6b05fa204 Auto merge of #19206 - asajeffrey:test-perf-add-differ-summary, r=aneeshusa
Add total time to test_differ in etc/ci/performance

<!-- Please describe your changes on the following line: -->

Add the total time to test_differ in etc/ci/performance.

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because this is test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19206)
<!-- Reviewable:end -->
2017-11-15 18:34:12 -06:00
Alan Jeffrey
45d5d53e55 Added title to example test perf document. 2017-11-15 12:24:11 -06:00
Alan Jeffrey
a1dfce4d3c Made .csv the default file format for perf tests. 2017-11-15 12:24:10 -06:00
Alan Jeffrey
d817b224f5 Support saving test data to .csv as well as .json. 2017-11-15 12:24:09 -06:00
Alan Jeffrey
201f931fa1 Add total time to test_differ in etc/ci/performance. 2017-11-15 11:54:45 -06:00
Alan Jeffrey
b845d44aa2 Add 'Error loading page' case for validating correctness of test-perf data. 2017-11-15 11:42:21 -06:00
Anthony Ramine
23d7783e02 Make script depend on gecko-media
It's not used yet though.
2017-11-14 17:33:25 +01:00
Simon Sapin
3264453284 Stop using "alternate" rustc builds.
With https://github.com/rust-lang/rust/pull/45810,
normal Nightly now has LLVM assertions disabled.
2017-11-14 09:08:11 +01:00
bors-servo
5f33d35cde Auto merge of #19207 - asajeffrey:test-perf-add-base-url-option, r=edunham
Add --base option to test-perf.

<!-- Please describe your changes on the following line: -->

Add a `--base <URL>` option to test-perf, which is handy for two reasons: a) it reduces randomness in tests by allowing tests to be file URLs, and b) it doesn't require running the test suite as root (the tp5n manifest hardwires tests served from port 80 on localhost).

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because this is test infrastructure

<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/19207)
<!-- Reviewable:end -->
2017-11-13 19:51:32 -06:00
Alan Jeffrey
109df3328a Add --base option to test-perf. 2017-11-13 18:58:06 -06:00
Alan Jeffrey
45a63a8960 Update URL for tp5n archive. 2017-11-13 18:04:51 -06:00
Sandeep Hegde
2f900a0427 Added PS1 variable before activating virtualenv 2017-11-13 08:05:34 -05:00
Sandeep Hegde
ed47f89e79 Changed invocation of muatation test in CI to bash script to use virtualenv 2017-11-13 01:01:09 -05:00
Sandeep Hegde
cc6c2eea6e Added mutation test summary and made it exit with relevant exit code 2017-11-12 23:43:04 -05:00
Josh Matthews
86e3f0deef Use failure reporting API on CI. 2017-11-07 14:01:24 -05:00
Josh Matthews
c0f85579c5 Allow specifying alternative tracker endpoints. 2017-11-07 13:40:07 -05:00
Sandeep Hegde
58ab11cf9b Fixed few tidy errors 2017-11-06 13:00:13 -05:00
Sandeep Hegde
af605a6e09 Added Mutation Test to CI 2017-11-06 13:00:12 -05:00
Josh Matthews
00940911c5 Divide up CI jobs more evenly. 2017-11-02 09:22:01 -04:00