The Servo repository is now (mostly) vendored in the Firefox Mercurial
repository. For size and duplication reasons, the tests/wpt directory is
not included in the vendored copy.
This causes problems when running `mach` from the Firefox repository
because `mach` references pip requirements files and module search
paths from WPT.
This commit adds code to detect when Servo's mach is running from a
Firefox source tree and to resolve WPT paths to the Firefox location
if appropriate. This enables `mach` to "just work" when running
from the servo/ directory in the Firefox repository.
The file looked for to identify the Firefox repository is identical
to what Firefox's `mach` script uses.
A potential issue with using Firefox's WPT files is that they may be
different from those in the Servo repository and this could lead to
differences in behavior - possibly even an error when loading/running
`mach`. However, the behavior before this commit was that Servo's
`mach` never worked in the Firefox repository (due to missing WPT
files). And post-commit it does. So this seems like a "perfect is the
enemy of good" scenario.
Allow running servo in software rendering mode from ./mach run
<!-- Please describe your changes on the following line: -->
This adds a flag to ./mach run to use software rendering on Linux (through the LIBGL_ALWAYS_SOFTWARE env var).
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix#14982.
<!-- Either: -->
- [X] These changes do not require tests because like said in the issue "There's no way to add automated tests for this".
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/15138)
<!-- Reviewable:end -->
Use Salt for mach bootstrap
<!-- Please describe your changes on the following line: -->
This is currently WIP, but allows Salt for `mach bootstrap`. Not looking for review yet, just posting for visibility. You can run `./mach bootstrap` and Salt will run, letting you know what changes it would make for bootstrapping (doesn't actually run yet though).
Currently, this reads from saltfs using gitfs, meaning it always tracks the master branch. (Note that this is blocked on https://github.com/servo/saltfs/pull/577 landing; in the meantime, I've included a small workaround in this PR to pull from my updated saltfs branch, which will need to be removed.) In the future, the relevant Salt code may be merged in tree as part of Docker support for TC, and the bootstrapper should be updated to read from in tree.
Also, our Windows machines have not been Salted, so the existing Windows bootstrappers are retained.
TODO:
- [x] Hook into existing bootstrapping code less hackily
- [x] Actually bootstrap instead of always using `test=True` mode (includes sudo)
- [x] Default to interactive mode (test first, then ask), with a force flag
- [x] Don't require running from the repository root directory
- [x] Make it easy to add support for other distros
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [ ] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [ ] These changes fix #__ (github issue number if applicable).
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they should be verified manually
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14974)
<!-- Reviewable:end -->
The Salt bootstrapper invokes Salt during `./mach bootstrap`
to install Servo's build dependencies.
It uses salt-call pinned to the same version of Salt as used in saltfs.
Currently, the implementation uses gitfs and reads directly from
the master branch of the saltfs repo;
in the future this should be changed when the relevant Salt states
are moved in-tree as part of Dockerization for TaskCluster.
We have not Salted our Windows machines, so the existing Windows
bootstrappers are retained. Currently this is only tested on
Ubuntu Trusty.
Salt uses various system python libraries,
including `python-apt` on Debian-based OSes to interact with apt.
`python-apt` does not seem to be installable via a requirements.txt
file, and the versions available on PyPI are far behind the versions
installed on actual Ubuntu machines.
Additionally, adding `python-apt` as an unconditional python dependency
would add bloat for users of other OSes, and lead to more churn
as additional OSes are supported.
However, as `python-apt` is already installed via apt on these machines,
we can allow Salt to instead use the module by using
`--system-site-packages` for the python virtualenv.
We also add the `-I` flag to `pip install` to ensure we have a local,
untouched copy of any other python packages used.
However, because this prints system-level Python packages in scope,
it slightly breaks isolation, so it is important to always pin
all dependencies in the requirements files.
- Default to interactive mode and remove the `--interactive` flag
- Use `--force` to skip interactivity
- Change MSVC dependency storage organization on disk: put each version
into its own folder and directly refer to the versioned folders,
providing immutability and making the installation list redundant
- Reuse `host_triple()` function to fix broken bootstrapper dispatching
- Simplify code:
- Remove or inline many unused and redudant functions and variables
- Prefer plain functions to classes
- Consolidate into fewer files, remove unnecessary bootstrapper/ dir
- Improve Python style
- Sort dependency list
Allow passing --nocapture argument to test-unit mach command
This adds the ` --nocapture` argument to the test-unit mach command.
Defaulting to false, when given it passes `-- --nocapture` to cargo test to show stdout during test-unit runs.
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#14595
<!-- Either: -->
- [ ] There are tests for these changes OR
- [ ] These changes do not require tests because _____
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14706)
<!-- Reviewable:end -->
Use MSVC toolchain if PLATFORM is defined
<!-- Please describe your changes on the following line: -->
`VSInstallDir` exists only when Visual Studio is installed.
`VS140COMNTOOLS` is defined by installing either Visual Studio or [Visual C++ Build Tools](http://landinghub.visualstudio.com/visual-cpp-build-tools)
It will allow to build MSVC based Servo without having to install whole Visual Studio (few GiB).
EDIT: Another (maybe cleaner) solution would be reverting 63ec96a57d
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [ ] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [ ] These changes fix #__ (github issue number if applicable).
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because `VS140COMNTOOLS` variable is already used in [mach.bat](https://github.com/servo/servo/blob/master/mach.bat)
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14346)
<!-- Reviewable:end -->
Tidy: Check Cargo.lock for packages with same version and different sources
<!-- Please describe your changes on the following line: -->
r? @Wafflespeanut
cc @SimonSapin
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix#14695
<!-- Either: -->
- [X] There are tests for these changes
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14715)
<!-- Reviewable:end -->
Raise warning when OSMesa path is not set
In Ubuntu 14.04, I still get crashes due to non-existence of OsMesa library (#13515). It turned out that sometimes I have paths like `osmesa-src-<hash>` without `out/lib/gallium`. Now, we raise a warning whenever we don't find a proper path.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
<!-- Either: -->
- [x] These changes do not require tests because it's related to mach.
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14601)
<!-- Reviewable:end -->
Define NATIVE_WIN32_PYTHON environment variable
r? @Wafflespeanut or @larsbergstrom
---
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/13650)
<!-- Reviewable:end -->
Various Windows installer fixes and improvements
r? @metajack or @larsbergstrom
---
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14531)
<!-- Reviewable:end -->
Majority of build_gecko.rs is just the straightforward conversion from
regen.py. There are two differences that:
1. Side in whitelist is changed to mozilla::Side
2. std::atomic__My_base is added to opaque types for Windows
Part of #13551 - Create packages under release/debug directories as appropriate
r? @aneeshusa
This change implements fixes for #13551 for Linux and MacOS targets.
`/python/servo/package_commands.py` was modified so that:
- On MacOS, it creates all intermediate packaging directories like `dmg`, `brew`, and `brew-tmp` under `target/(release|debug)`, rather than in `target` directly.
- On MacOS, all packaging artifacts (`.dmg`, brew `.tar.gz`) are packaged under `target/(release|debug)`, rather than in `target` directly.
- On Linux, the resulting `tar.gz` Servo package is placed under `target/(release|debug)`, rather than in `target`.
- Also did some extra cleanup around path parsing in the MacOS packaging code, to use `os.path` methods rather than straight `'/'` parsing with `split` and `join` where it was applicable.
`/etc/ci/upload_nightly.sh` was modified to:
- Look for artifacts in `target/release` for `mac`, `macbrew`, and `linux` platforms, rather than just `target/`.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#13551
- [x] These changes do not require tests because it is based on CI and packaging tools. They were manually tested for correctness.
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
…lease/debug as specified or detected. Modify macos packaging to create all packages under release/debug directory. Updated etc/ci/upload_nightly.sh to support uploading from either release/debug directory, depending on what was built
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/13845)
<!-- Reviewable:end -->
This commit updates the 'mach package' command to generate output
under the target/release and target/debug directories when run on
macOS and Linux.
etc/ci/upload_nightly.sh has also been updated to upload packages from
the release/debug directories.
Adding linting checks for buildbot_steps.yml
This pull request adds some tidy checks around YAML files, and specifically `buildbot_steps.yml`.
Tidy checks added:
* YAML files are checked for well-formedness/parse-ability
* Whether a YAML file has duplicate keys
* Whether a `buildbot_steps.yml` file contains only mappings to list-of-strings.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [x] These changes fix#13838 (github issue number if applicable).
<!-- Either: -->
- [X] There are tests for these changes OR
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
…ing checking for correct mappings and duplicate YAML keys. Added unit tests to test_tidy.py.
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14051)
<!-- Reviewable:end -->
