https://github.com/servo/saltfs/pull/687 added support
for specifying environment variables in `buildbot_steps.yml`.
Update the servo-tidy buildbot_steps.yml linter to reflect this.
Use the voluptuous Python library (BSD 3-clause license) for validation
in lieu of a much larger hand-written implementation.
Update the tidy self tests to take into account the new error messages.
Now that MinGW and MSYS have been removed from the Windows builders,
bash is not available to run the previous upload_nightlies.sh script.
Convert the script to Python 2 for cross-platform support.
Additionally, switch to the `boto3` library for easy uploading
without needing to install `s3cmd`,
and move the code into mach for easy `boto3` installation
as the new `./mach upload-nightly` command.
Also, hard-code the paths to the packages instead of using
globs to look for them, as the paths are static.
(The paths used to contain timestamps,
but we now insert timestamps when uploading to S3
to improve reproducibility.)
The Salt bootstrapper invokes Salt during `./mach bootstrap`
to install Servo's build dependencies.
It uses salt-call pinned to the same version of Salt as used in saltfs.
Currently, the implementation uses gitfs and reads directly from
the master branch of the saltfs repo;
in the future this should be changed when the relevant Salt states
are moved in-tree as part of Dockerization for TaskCluster.
We have not Salted our Windows machines, so the existing Windows
bootstrappers are retained. Currently this is only tested on
Ubuntu Trusty.
Salt uses various system python libraries,
including `python-apt` on Debian-based OSes to interact with apt.
`python-apt` does not seem to be installable via a requirements.txt
file, and the versions available on PyPI are far behind the versions
installed on actual Ubuntu machines.
Additionally, adding `python-apt` as an unconditional python dependency
would add bloat for users of other OSes, and lead to more churn
as additional OSes are supported.
However, as `python-apt` is already installed via apt on these machines,
we can allow Salt to instead use the module by using
`--system-site-packages` for the python virtualenv.
We also add the `-I` flag to `pip install` to ensure we have a local,
untouched copy of any other python packages used.
However, because this prints system-level Python packages in scope,
it slightly breaks isolation, so it is important to always pin
all dependencies in the requirements files.
This commit adds tidy checks for buildbot_steps.yml, as well as unit
tests. These checks include:
* Checking buildbot_steps.yml can be parsed by a YAML loader
* buildbot_steps.yml does not contain duplicate keys
* buildbot_steps.yml keys map to a list of strings
When I originally rewrote Servo's mach bootstrapping (using virtualenv
w/ requirements.txt in #7103), I didn't specify mach as a requirement
because a new version hadn't been published in a while. Now that 0.6
is out, I asked the mach maintainers to publish a new version on PyPI,
so now we can fetch it like the other Python dependencies.
Fixes https://github.com/servo/servo/issues/10728.
- also update requriements "pyflakes" from 0.8 to 0.8.1 due to following issue
```
Traceback (most recent call last):
File "/Users/Askeing/software/servo/python/_virtualenv/bin/servo-tidy", line 5, in <module>
from pkg_resources import load_entry_point
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3084, in <module>
@_call_aside
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3070, in _call_aside
f(*args, **kwargs)
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3097, in
_initialize_master_working_set
working_set = WorkingSet._build_master()
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 653, in _build_master
return cls._build_from_requirements(__requires__)
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in
_build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/Users/Askeing/software/servo/python/_virtualenv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 839, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'pyflakes==0.8.1' distribution was not found and is required by servo-tidy
```
https://github.com/servo/servo/issues/861#issuecomment-209547756
"I think the most important concern is that it's possible to modify tidy.py
and see how those changes affect ./mach test-tidy with the fewest possible
intermediate steps." - jdm
This takes publishing complexity away from the contributor when testing
changes and makes it an infra problem instead, where it's much easier to
automate & saner to manage pypi credentials
Prior to this commit:
* Our Python dependency story was a bit of a mess. We had complete
Python packages (wheels and directories) living in-tree, despite
not having any changes from upstream. This is particularly bad because
`setup.py` never gets run on these packages which could (sometimes
silently) unintended breakage.
* Python virtual environments (virtualenv) were only utilized for
testing web-platform tests
After this commit:
* A single virtualenv (`python/_virtualenv`) is activated upon *every*
call to mach
* A requirements file (`python/requirements.txt`) is added to describe
the dependencies needed by Python modules in `python/`. The child
commit immediately following this will remove all the dependencies
no longer needed in-tree (for the sake of keeping this commit
readable).
Relevant to https://github.com/servo/servo/issues/861
Fixes https://github.com/servo/servo/issues/6999
