Auto merge of #14937 - shinglyu:stylo-perf, r=Manishearth

Read firefox path from environment variable for performance test

<!-- Please describe your changes on the following line: -->
This is for Stylo performance testing (at least before we got Talos running)

r?@Manishearth

---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [ ] These changes fix #__ (github issue number if applicable).

<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because independent from servo itself

<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14937)
<!-- Reviewable:end -->

etc/ci/performance/README.md

@@ -18,10 +18,11 @@ Servo Page Load Time Test
 ## CI for Gecko
 
 * Install Firefox Nightly in your PATH
-* Download [geckodriver](https://github.com/mozilla/geckodriver/releases) and add it to the `PATH`
+* Download [geckodriver](https://github.com/mozilla/geckodriver/releases) and add it to the `PATH` (e.g. for Linux `export PATH=$PATH:path/to/geckodriver`)
+* `export FIREFOX_BIN=/path/to/firefox`
 * `pip install selenium`
 * Run `python gecko_driver.py` to test
-* Run `test_all.sh --gecko --submit`
+* Run `test_all.sh --gecko --submit` (omit `--submit` if you don't want to submit to perfherder)
 
 # How it works
 
@@ -30,6 +31,25 @@ Servo Page Load Time Test
 * Each testcase is a subtest on Perfherder, and their summary time is the geometric mean of all the subtests.
 * Notice that the test is different from the Talos TP5 test we run for Gecko. So you can NOT conclude that Servo is "faster" or "slower" than Gecko from this test.
 
+# Comparing the performance before and after a patch
+
+* Run the test once before you apply a patch, and once after you apply it.
+* `python test_differ.py output/perf-<before time>.json output/perf-<after time>.json`
+* Green lines means loading time decreased, Blue lines means loading time increased.
+
+# Add your own test
+
+* Add you test case (html file) to the `page_load_test/` folder. For example we can create a `page_load_test/example/example.html`
+* Add a manifest (or modify existing ones) named `page_load_test/example.manifest`
+* Add the lines like this to the manifest:
+
+```
+http://localhost:8000/page_load_test/example/example.html
+# This is a comment
+# Pages got served on a local server at localhost:8000
+```
+* Modify the `MANIFEST=...` link in `test_all.sh` and point that to the new manifest file.
+
 # Unit tests
 
 You can run all unit tests (include 3rd-party libraries) with `python -m pytest`.

etc/ci/performance/gecko_driver.py

@@ -6,13 +6,22 @@
 
 from contextlib import contextmanager
 import json
+import os
 from selenium import webdriver
 from selenium.common.exceptions import TimeoutException
+import sys
 
 
 @contextmanager
 def create_gecko_session():
-    firefox_binary = "./firefox/firefox/firefox"
+    try:
+        firefox_binary = os.environ['FIREFOX_BIN']
+    except KeyError:
+        print("+=============================================================+")
+        print("| You must set the path to your firefox binary to FIREFOX_BIN |")
+        print("+=============================================================+")
+        sys.exit()
+
     driver = webdriver.Firefox(firefox_binary=firefox_binary)
     yield driver
     # driver.quit() gives an "'NoneType' object has no attribute 'path'" error.
@@ -90,6 +99,7 @@ def run_gecko_test(testcase, timeout):
 
     return [timings]
 
+
 if __name__ == '__main__':
     # Just for manual testing
     from pprint import pprint