This allows us to use `uv` for:
1. Installing a pinned Python version
2. Installing the dependency packages using `uv`'s pip compatible interface.
4. Bootstrapping `mach` without a Python installion on the host, using `uv
run`
This change also introduces a new 'composite' GitHub action to setup
python in the different CI workflows. There is no support for externally
managed python installations and virtual environments. These could be
added in the future.
Fixes#34095, #34547
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
This patch switches servo to use `uv` for both installing a pinned
Python version as well as installing the dependency packages using
`uv`'s pip compatible interface. It also introduces a new 'composite'
GitHub action to setup python in the different CI workflows.
There is no support for externally managed python installations and
virtual environments. These could be added in the future.
Fixes#34095
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
The default user name in Windows installations is of the form "FirstName
LastName", so it seems likely that there will be spaces in the user's
path. Based on my testing on Windows 11, the only Servo's bootstrap
script has trouble dealing with spaces in paths. This patch fixes that
by quoting such paths correctly. Our direct and indirect dependencies
seem to handle these without issue and Servo does build and run
correctly with this patch.
In this patch, the logic for gstreamer bootstrap now uses powershell
instead of directly invoking msiexec.exe via cmd.exe as I was unable to
get the installer to run correctly, even with quoting. Some extra hacks
were necessary to propagate the exit code correctly to mach.
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
I added this print statement, but after using `mach` for a while with
it, I think it is far too chatty. This change just removes the line.
Information is only printed when dependencies are installed.
We can now use the "new" pip resolver which should prevent the
installation of conflicting packages. Also, take this opportunity to
make bootstrap more resilient. Hash all dependencies to detect
situations where a newer marker file has been installed, but for an
older branch. This should ensure that dependencies are up to date even
when switching back and forth between older and new branches.
This also updates some dependencies to be the same as the ones used for
WPT tests, which is an issue caught be the resolver.
Fixes#10611.
It's often the case (especially with the taplo installation and on
Windows) that bootstrap is doing lots of stuff in the background for a
long amount of time. Without output it's hard to tell what exactly is
going on. This change adds more output to this process as well as
removing some Pythong 2.x era code.
1. Make the tidy output easier to follow
2. Integrate the WPT manifest cleanliness step into tidy
itself and don't run it if nothing has changed in the WPT
directory.
3. Fix an issue where Python test requirements were not installed,
which could cause issues with some modules not being found.
Fixes#30002.
* Replace virtualenv with Python's built-in venv.
* Apply Delan's suggestions and make a couple small fixes
- Fix a tidy warning about directories that don't exist
- Use shutil instead of the redundant get_exec_path
- Miscellaneous cleanups
* Fix typo in environment variable
* fix bug where pip still tries to the wrong site-packages
---------
Co-authored-by: Martin Robinson <mrobinson@igalia.com>
Co-authored-by: Delan Azabani <dazabani@igalia.com>
The version of `mozdebug` installable via pip is two years old and
Mozilla is slow to release new versions. It also uses `distutils` which
doesn't work on newer Python versions. Vendor it and stop using
`distutils` there.
* mach: use `importlib` module instead of `imp`
`imp` module has been deprecated since python 3.4
and has been removed in 3.12. The recommended alternative
is to use the `importlib` module that was introduced in
python 3.1
This is required to fix the CI failures in macos builds
since GitHub runner images for macos-13 now use python 3.12
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
* mach: use `importlib` module instead of `imp`
`imp` module has been deprecated since python 3.4
and has been removed in 3.12. The recommended alternative
is to use the `importlib` module that was introduced in
python 3.1
This is required to fix the CI failures in macos builds
since GitHub runner images for macos-13 now use python 3.12
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
---------
Signed-off-by: Mukilan Thiyagarajan <mukilan@igalia.com>
Servo is no longer completely vendored into Gecko. Instead parts of
Gecko are vendored into Servo. This change removes Python mach bootstrap
code that was written to accommodate the previous situation. It's no
longer necessary.
It seems that servo-tidy is only used by webrender in my GitHub
searches. WebRender could simply use `rustfmt` and the tidy on pypi
hasn't been updated since 2018. Converting tidy to a normal Python
package removes the maintenance burden of continually fixing the easy
install configuration.
Fixes#29094.
Fixes#29334.
On Windows with multiple Pythons installed, this was causing python2.7
to bootstrap a 3.7 virtualenv that it couldn't make use of.
PIP_NAMES wasn't used at all, and VIRTUALENV_NAMES ends up being unused
now.
The sys.platform check in need_pip_upgrade was doing effectively the same pip invocation regardless of what
platform we were evaluating to.
Additionally, removed some duplicate definitions of the python variable which was already in scope
higher up in the function.
Gecko is now using the wpt harness under the wpt tools directory
rather than a copy in an adjacent directory. Therefore the path to the
requirements files, and the required model paths, have changed.
The Servo repository is now (mostly) vendored in the Firefox Mercurial
repository. For size and duplication reasons, the tests/wpt directory is
not included in the vendored copy.
This causes problems when running `mach` from the Firefox repository
because `mach` references pip requirements files and module search
paths from WPT.
This commit adds code to detect when Servo's mach is running from a
Firefox source tree and to resolve WPT paths to the Firefox location
if appropriate. This enables `mach` to "just work" when running
from the servo/ directory in the Firefox repository.
The file looked for to identify the Firefox repository is identical
to what Firefox's `mach` script uses.
A potential issue with using Firefox's WPT files is that they may be
different from those in the Servo repository and this could lead to
differences in behavior - possibly even an error when loading/running
`mach`. However, the behavior before this commit was that Servo's
`mach` never worked in the Firefox repository (due to missing WPT
files). And post-commit it does. So this seems like a "perfect is the
enemy of good" scenario.
The Salt bootstrapper invokes Salt during `./mach bootstrap`
to install Servo's build dependencies.
It uses salt-call pinned to the same version of Salt as used in saltfs.
Currently, the implementation uses gitfs and reads directly from
the master branch of the saltfs repo;
in the future this should be changed when the relevant Salt states
are moved in-tree as part of Dockerization for TaskCluster.
We have not Salted our Windows machines, so the existing Windows
bootstrappers are retained. Currently this is only tested on
Ubuntu Trusty.
Salt uses various system python libraries,
including `python-apt` on Debian-based OSes to interact with apt.
`python-apt` does not seem to be installable via a requirements.txt
file, and the versions available on PyPI are far behind the versions
installed on actual Ubuntu machines.
Additionally, adding `python-apt` as an unconditional python dependency
would add bloat for users of other OSes, and lead to more churn
as additional OSes are supported.
However, as `python-apt` is already installed via apt on these machines,
we can allow Salt to instead use the module by using
`--system-site-packages` for the python virtualenv.
We also add the `-I` flag to `pip install` to ensure we have a local,
untouched copy of any other python packages used.
However, because this prints system-level Python packages in scope,
it slightly breaks isolation, so it is important to always pin
all dependencies in the requirements files.
In addition to minor changes for Windows, this forces Windows Python to
be used for all Windows builds (instead of using Windows Python only for
pc-windows-msvc builds).
