Convert nightly upload script to Python
Now that MinGW and MSYS have been removed from the Windows builders,
bash is not available to run the previous upload_nightlies.sh script.
Convert the script to Python 2 for cross-platform support.
Additionally, switch to the `boto3` library for easy uploading
without needing to install `s3cmd`,
and move the code into mach for easy `boto3` installation
as the new `./mach upload-nightly` command.
Also, hard-code the paths to the packages instead of using
globs to look for them, as the paths are static.
(The paths used to contain timestamps,
but we now insert timestamps when uploading to S3
to improve reproducibility.)
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [ ] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#16560 (github issue number if applicable).
<!-- Either: -->
- [ ] There are tests for these changes OR
- [x] These changes do not require tests because Buildbot will test them.
<!-- Also, please make sure that "Allow edits from maintainers" checkbox is checked, so that we can help you if you get stuck somewhere along the way.-->
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/16565)
<!-- Reviewable:end -->
Now that MinGW and MSYS have been removed from the Windows builders,
bash is not available to run the previous upload_nightlies.sh script.
Convert the script to Python 2 for cross-platform support.
Additionally, switch to the `boto3` library for easy uploading
without needing to install `s3cmd`,
and move the code into mach for easy `boto3` installation
as the new `./mach upload-nightly` command.
Also, hard-code the paths to the packages instead of using
globs to look for them, as the paths are static.
(The paths used to contain timestamps,
but we now insert timestamps when uploading to S3
to improve reproducibility.)
`mach bootstrap` will finish succesfully,
but then segfault inside glibc while exiting.
It seems to have to do with the MarkupSafe library used by Mako;
delaying the Mako import from the top level to inside functions where it
is used avoids the problem.
(It seems to interact with the call to pip to install the Salt
requirements in the virtualenv, as commenting that pip call out
separately will also fix the segfault.)
Also, use the Mako installed in the Python virtualenv while running
packaging commands (instead of the one from the bundled zip file),
and cleanup imports in the package_commands.py file.
Various Windows installer fixes and improvements
r? @metajack or @larsbergstrom
---
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/14531)
<!-- Reviewable:end -->
This commit updates the 'mach package' command to generate output
under the target/release and target/debug directories when run on
macOS and Linux.
etc/ci/upload_nightly.sh has also been updated to upload packages from
the release/debug directories.
New folder linux
<!-- Please describe your changes on the following line: -->
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [x] `./mach build -d` does not report any errors
- [x] `./mach test-tidy` does not report any errors
- [x] These changes fix#11983 (github issue number if applicable).
- [x] There are tests for these changes (./mach package)
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/12850)
<!-- Reviewable:end -->
The purpose of the code is to make a new temporary folder with the
necessary resources used to package Linux. Before this, the package
was built based on a modified target directory. By using a temporary
folder, it removes the need to rebuild Servo every time it gets
packaged for Linux.
Create .tar.gz package deterministically using Python's "tarfile"
<!-- Please describe your changes on the following line: -->
A development of #12108, creates a .tar.gz package using the `tarfile` and `gzip` modules, without external dependencies. Fixes#11981.
Also this fixes the issue when the existing `resources/` directory didn't allow to create a new package and failed with "File exists" error.
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix#11981 (github issue number if applicable).
<!-- Either: -->
- [ ] There are tests for these changes OR
- [X] These changes do not require tests because "more general approach to reproducibility testing is needed"
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/12244)
<!-- Reviewable:end -->
Change colons to dashes in package name
<!-- Please describe your changes on the following line: -->
Updating packaging so that packages use '-' instead of a ':'
r? @ConnorGBrewster
---
<!-- Thank you for contributing to Servo! Please replace each `[ ]` by `[X]` when the step is complete, and replace `__` with appropriate data: -->
- [X] `./mach build -d` does not report any errors
- [X] These changes fix#11984 (github issue number if applicable).
<!-- Either: -->
- [ ] There are tests for these changes OR
- [X] `./mach test-tidy` does not report any errors
- [X] These changes do not require tests because they are updating packaging
<!-- Pull requests that do not address these steps are welcome, but they will require additional verification as part of the review process. -->
<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="35" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/12088)
<!-- Reviewable:end -->
