Allow building on paths with Unicode characters
This now works since the upgrade to Python 3, so we can remove
this code which prevents mach from running in these situations.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#10002.
- [x] These changes do not require tests because they just change build scripts.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Simplify the build instructions in README.md
- Add a chocolatey configuration that installs all dependencies and split
the instructions into a normal and manual installation section. In
addition fix a warning print statement for the Windows build that would
trigger a Python exception.
- Simplify the build instructions in the README and move all manual
setup details to: https://github.com/servo/servo/wiki/Building
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they do not change behavior.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
- Add a chocolatey configuration that installs all dependencies and split
the instructions into a normal and manual installation section. In
addition fix a warning print statement for the Windows build that would
trigger a Python exception.
- Simplify the build instructions in the README and move all manual
setup details to: https://github.com/servo/servo/wiki/Building
Default nightly builds to layout 2020
This PR switches the filenames used by nightly builds so that the default `servo-latest.{ext}` packages use the 2020 engine and the 'servo-latest-legacy-layout.{ext}` packages use the 2013 engine.
Since the platform workflows are reused by the main, quick-check, PR and WPT import flows and since the 2013/2020 flags are also controlled by try branches, I've kept the changes isolated to `nightly.yml` and `package_commands.py` and have avoided changing the input parameters. If we want to deault all the workflows to 2020, I can decline this PR and raise a new one.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they modify packaging logic.
1. The options specified in `rustdoc-with-private` are essentially the
default now so we can remove this script.
2. `./mach browse-doc` is redundant with `./mach doc --open` which uses
the underlying cargo functionality to browse the documentation.
Fixes#29888.
Fix windows nightly build failure
PR #29865 changed the target directory for windows to be different from the source folder. However, the 'upload-nightly' command assumes the packages are stored in the default cargo target directory (./target), so the nightly jobs for windows were failing to find the packages.
This PR fixes the 'upload-nightly' command to account for non-default target directories specified via CARGO_TARGET_DIR.
---
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they fix a bug in packaging logic.
PR #29865 changed the target directory for windows
to be different from the source folder. However,
the 'upload-nightly' command assumes the packages
are stored in the default cargo target directory
(./target), so the nightly jobs for windows were
failing to find the packages.
This PR fixes the 'upload-nightly' command to
account for non-default target directories
specified via CARGO_TARGET_DIR.
It seems that servo-tidy is only used by webrender in my GitHub
searches. WebRender could simply use `rustfmt` and the tidy on pypi
hasn't been updated since 2018. Converting tidy to a normal Python
package removes the maintenance burden of continually fixing the easy
install configuration.
Fixes#29094.
Fixes#29334.
Use layout 2020 by default
This PR switches `./mach build` to use layout 2020 by default. It doesn't switch the nightly builds served from download.servo.org to layout 2020. I can add that change to this PR if we are ready to make the switch in nightly builds as well.
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#29843
- [x] These changes do not require tests because modify mach build configuration.
Get clang-format from pip and upgrade to version 16
This allows relying on a specific version of clang-format and no longer use any version checks. In addition, we can use --dry-run -Werror in order to avoid having to run against every file individually.
Fix#29847.
Fix#29846.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#29847 and fix#29846.
- [x] These changes do not require tests because they update dev infrastructure.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
This allows relying on a specific version of clang-format and no longer
use any version checks. In addition, we can use --dry-run -Werror in
order to avoid having to run against every file individually.
Fix#29847.
Fix#29846.
Vendoring machanism for webgpu cts & update
- Add `update-webgpu` command to mach to vendor webgpu cts.
- Update webgpu cts (480edec387) & expectations (even though most are failing due to out of date impl)
- Ignore vendored path from tidy check
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#27508
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they are tests
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Remove more Python 2 compatibility code
- os.environ is always `str` in Python 3.
- The only string type is `str` so we can stop using `six.str_types`.
- `iteritems()` isn't necessary because dicts have the `items()` method.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they do not change behavior.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
- os.environ is always `str` in Python 3.
- The only string type is `str` so we can stop using `six.str_types`.
- `iteritems()` isn't necessary because dicts have the `items()` method.
Propagate status on `build_like_command_arguments`
As observed in #29805
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#29806
<!-- Either: -->
- [ ] There are tests for these changes OR
- [ ] These changes do not require tests because ___
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
Instead of always checking whether GStreamer is installed in mach's
`build_env`, only do this when actually building. Also, use the instance
variable to find features and look for the "media-gstreamer" feature
instead of looking for !"media-dummy."
Fixes#29797.
Integrate cross-compilation and media-stack handling into the
`build_like_command_arguments` decorator. This removes a lot of
repetition in the code and standardizes how targets are selected for all
similar commands.
Now cross compilation targets, feature flags, and helper variables are
stored in the CommandBase instance. This also avoids having to
continuously pass these arguments down to functions called by the
commands.
Implement `bootstrap-gstreamer` for all platforms
This change makes it so that the Platform classes can now handle installing GStreamer dependencies and properly setting up the environment including when cross-compiling. For Windows and Linux, prepackaged GStreamer is now installed into `target/dependencies/gstreamer` when not installed system-wide. In addition this change:
1. The Environment path append helper is moved to `util.py` and a new `prepend` version is added.
2. `set_run_env` and `build_dev` functions are combined and include more code from callers so environment setup is more often in the same place. Now code that used to call `set_run_env` calls `build_dev` and then `os.environ.update(...)`. We can further refine this with the `is_build` argument later.
4. Python typing information is added in many places.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#25335
- [x] These changes do not require tests because they are to the build scripts.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
This change makes it so that the Platform classes can now handle
installing GStreamer dependencies and properly setting up the
environment including when cross-compiling. For Windows and Linux
is now installed into `target/dependencies/gstreamer` when not installed
system-wide. In addition:
1. Creating and moving existing environment path append helpers to
`util.py`.
2. Combining the `set_run_env` and `build_dev` functions and moving
some outside code into them so that it can be shared. Now code that
used to call `set_run_env` calls `build_dev` and then
`os.environ.update(...)`.
3. Adding Python typing information in many places.
Signed-off-by: Martin Robinson <mrobinson@igalia.com>
We'd like to start offering servo nightly builds
with layout 2020 engine so that users can test
the new layout without building servo. These new
builds will be offered in addition to the current
2013 builds until we switch over to 2020 as the default.
Add a compositor-side scroll tree
This will allow the compositor to properly chain scrolling requests up
when a node has reached the extent of the scroll area. In addition, it
removes the use of the deprecated WebRender `scroll()` API. This fixes
scrolling on servo.org.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes:
- Fix#29402.
- Fix#27996.
- Fix#27624.
- Fix#24028.
- Fix#23918.
- Fix#21165.
- [x] There are tests for these changes
Fix slowness in macos copy_dependencies
The `copy_dependencies` logic is invoking `change_non_system_libraries_path` with absolute paths to gstreamer *plugin dylibs*.
`change_non_system_libraries_path` skips editing links in servo binary to relocatable dylibs, but the since the plugins have absolute paths, they are treated as 'non-relocable' and it will try to edit the *non-existent* link (since plugins are loaded dynamically) in servo bin . These unnecessary calls to change_link_name is the cause of the slowness identified in #29764
This PR fixes the issue by ensuring plugin dylibs are not passed to the first call to change_non_system_libraries_path that patches the links in the main servo binary.
With this fix, the time taken by copy_dependencies is reduced to ~3 seconds on my system.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#29764
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because they fix slowness in build step
The `copy_dependencies` logic is invoking
`change_non_system_libraries_path` with absolute paths
to gstreamer *plugin dylibs*.
`change_non_system_libraries_path` skips editing links
in servo binary to relocatable dylibs, but the since the
plugins have absolute paths, they are treated as
'non-relocable' and it will try to edit the *non-existent*
(since plugins are loaded dynamically) link in servo bin.
These unnecessary calls to change_link_name is cause
of the slowness identified in #29764
This PR fixes the issue by ensuring plugins are not
included in the call to change_non_system_libraries_path
Signed-off-by: Mukilan Thiyagarajan <me@mukilan.in>
This change removes:
- The custom `--dry-run` support for `cargo-update`. The real cargo
command now has `--dry-run` support and this code was broken for
Python 3.
- The Python 2 version of TemporaryDirectory. This is no longer needed
as we require Python 3.
Start organizing platform-specific Python code
This starts to split platform-specific Python code into its own module, which should help to tidy up our mach commands and make things more reusable. This is step one toward fixing #25335.
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `___` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes do not require tests because they just change mach scripts.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
This will allow the compositor to properly chain scrolling requests up
when a node has reached the extent of the scroll area. This fixes
scrolling on servo.org.
The official gstreamer .pkg distribution should contain
'relocatable' dylibs, but as discovered in #29732,
some dylibs have absolute links to liblzma.5.dylibs
Since /opt/homebrew is the default install location
on Apple Silicion, this will cause the packaging &
build steps to fail, even if 'xz' package is installed
via homebrew.
This is a temporary fix until upstream fixes the bug
and makes the package truly 'relocatable'.
Signed-off-by: Mukilan Thiyagarajan <me@mukilan.in>
This PR re-enables support for the gstreamer mediastack
in macOS by consuming the official binary '.pkg' files
from gstreamer.freedesktop.org
To maintain symmetry with other platforms, the '.pkg'
files are uploaded to servo-build-deps and fetched from
there using the new script 'etc/install_macos_gstreamer.sh'.
Unlike the Homebrew version, the official GStreamer is
distributed as a 'relocatable' framework i.e the dylibs all
have @rpath-relative install names and also link to other
dylibs using @rpath relative path. To address this difference
the 'servo' binary needs to be patched with 'install_name_tool'
to add an LC_RPATH command that sets the relative paths
that the dynamic linker should search when trying to satify
dependencies. In Servo's case, this will be a path relative to
the 'servo' binary itself i.e '@executable_path/lib/'
The additional 'lib' is due to a flaw in the gstreamer
packaging where the install names of some of the dylibs
have the prefix '@rpath/lib' and some of them just have '@rpath'.
This PR also fixes a couple of issues present in the
`mach build` process on MacOS:
1. `mach build` process was not copying transitive dependencies
of servo binary but only the first level dylibs
2. `mach build` process didn't patch the links to dylibs
in servo binary (and dependencies). This meant though
(some) dylibs were copied to local path, the binary
still loaded the dylibs from system GStreamer installation
i.e homebrew instead of the copieds dylibs
The build and runtime dependencies in etc/homebrew/Brewfile
and etc/homebrew/Brewfile-build have also been removed in This
PR.
Signed-off-by: Mukilan Thiyagarajan <me@mukilan.in>
The current version seem to be broken due to API change in one of the transitive dependency (urllib3) causing nightly builds to fail during package upload step with error:
```
ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_' (C:\a\servo\servo\python\_virtualenv3.7\Lib\site-packages\urllib3\util\ssl_.py)
```
