tor-reproducer issueshttps://code.briarproject.org/briar/tor-reproducer/-/issues2023-07-13T11:04:05Zhttps://code.briarproject.org/briar/tor-reproducer/-/issues/15Linux binaries require glibc 2.292023-07-13T11:04:05ZakwizgranLinux binaries require glibc 2.29The Linux binaries won't run on systems with glibc 2.28 (eg Debian 10), perhaps because they were built on Debian 11. If we still want to support these older platforms (which is an open question) then we should ensure the binaries are co...The Linux binaries won't run on systems with glibc 2.28 (eg Debian 10), perhaps because they were built on Debian 11. If we still want to support these older platforms (which is an open question) then we should ensure the binaries are compatible with older glibc versions. If we don't then we can close this ticket.https://code.briarproject.org/briar/tor-reproducer/-/issues/14Optimize build time for macOS binaries2023-07-13T11:02:08ZSebastianOptimize build time for macOS binariesWith our current approach of building the tor binaries for macOS using the [tor-browser-build](https://gitlab.torproject.org/tpo/applications/tor-browser-build/) setup, the build takes rather long (~ 3.5 hours).
The cause of this is tha...With our current approach of building the tor binaries for macOS using the [tor-browser-build](https://gitlab.torproject.org/tpo/applications/tor-browser-build/) setup, the build takes rather long (~ 3.5 hours).
The cause of this is that the build does build lots of tools from source itself such as `cmake` and `clang` to name the building blocks that take up roughly 73% of the build time. Here's a full list of things it builds with the timestamps giving indication of how long it takes. The difference from the item to its predecessor is the time it takes to build, e.g. `cmake` takes 7 minutes in this instance (which is quicker than the actual CI hardware because the machine that was used here has better hardware):
```
drwxr-xr-x 2 z z 4,0K May 2 09:00 mmdebstrap
drwx------ 2 z z 4,0K May 2 09:01 mmdebstrap-image
drwxr-xr-x 2 z z 4,0K May 2 09:01 container-image
drwx------ 2 z z 4,0K May 2 09:08 cmake
drwxr-xr-x 2 z z 4,0K May 2 09:10 llvm-project
drwx------ 2 z z 4,0K May 2 09:12 ninja
drwx------ 2 z z 4,0K May 2 10:17 clang
drwx------ 2 z z 4,0K May 2 10:23 libtapi
drwx------ 2 z z 4,0K May 2 10:25 cctools
drwx------ 2 z z 4,0K May 2 10:31 macosx-toolchain
drwx------ 2 z z 4,0K May 2 10:34 openssl
drwx------ 2 z z 4,0K May 2 10:35 libevent
drwx------ 2 z z 4,0K May 2 10:38 tor
```
A solution could be to let the build just not build some of those and instead use Debian snapshot repos to get deterministic versions of cmake, clang etc. during build.
On an upstream ticket, it was suggested, that it's possible to disable the use of containers when building tor, which would result in the requirement of installing in the docker container all the dependencies needed for the build. (see https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40819#note_2899611)
Although this comment was given to solve a problem with docker in docker, I had the impression it also might help to reduce the build time. However it looks like `clang`, the most important building block (>66% of build time), is not among the tools that are then expected to be installed on the CI machine, it's still built from source.
I expect meddling with the tor build system to be a bit tricky even though it looks quite well structured. But I bet there is a way to make the build just use `clang` and `cmake` from the system instead of building it from source itself.
The same upstream ticket had a suggestion about reducing build times, too (see https://gitlab.torproject.org/tpo/applications/tor-browser-build/-/issues/40819#note_2899637):
> If you are able to keep the `out` directory accross builds, then it should not rebuild the dependencies.
I don't think though, that this is helpful for us. Well we could build those bits as part of the docker image, but as this runs as part of our pipeline as well, we won't gain much in total, only for reproducing locally maybe.https://code.briarproject.org/briar/tor-reproducer/-/issues/13Tor binaries are not being stripped properly2022-03-21T11:18:43ZakwizgranTor binaries are not being stripped properlyThe Tor binaries for 0.4.5.12 aren't being stripped properly. `file` reports the binaries as unstripped and running `strip` with no arguments significantly reduces the size of the binaries.
For the previous release (0.3.5.17), `file` re...The Tor binaries for 0.4.5.12 aren't being stripped properly. `file` reports the binaries as unstripped and running `strip` with no arguments significantly reduces the size of the binaries.
For the previous release (0.3.5.17), `file` reports the binaries as stripped and `strip` with no arguments has no effect on the size.https://code.briarproject.org/briar/tor-reproducer/-/issues/12Investigate whether we can use --enable-static-tor2022-03-10T14:24:04ZakwizgranInvestigate whether we can use --enable-static-torWhen configuring the Tor build, the `--enable-static-tor` flag sets two flags we're not currently using on all platforms: `--enable-static-zlib` and `-static`.
We had some issues with `--enable-static-zlib` on Android, and `-static` may...When configuring the Tor build, the `--enable-static-tor` flag sets two flags we're not currently using on all platforms: `--enable-static-zlib` and `-static`.
We had some issues with `--enable-static-zlib` on Android, and `-static` may cause portability issues with glibc and NSS on Linux. Find out more about these issues and decide whether we should use `--enable-static-tor` on each platform.https://code.briarproject.org/briar/tor-reproducer/-/issues/11Check git commit hashes when downloading source repos2022-03-10T14:06:45ZakwizgranCheck git commit hashes when downloading source reposTo hinder supply chain attacks, record the expected git commit hashes for the source repos in the version file alongside the tag names, and check the commit hashes when checking out the tags.To hinder supply chain attacks, record the expected git commit hashes for the source repos in the version file alongside the tag names, and check the commit hashes when checking out the tags.https://code.briarproject.org/briar/tor-reproducer/-/issues/10Share more code between platforms2022-03-10T14:07:20ZakwizgranShare more code between platformsRefactor the code to share more code between platforms, perhaps through inheritance. Depends on #9.Refactor the code to share more code between platforms, perhaps through inheritance. Depends on #9.https://code.briarproject.org/briar/tor-reproducer/-/issues/9Extract code from Android Makefile2022-03-10T14:07:20ZakwizgranExtract code from Android MakefileConvert some of the code in the Android Makefile to Python so we more easily compare the Android, Linux and Windows builds and share code between them.Convert some of the code in the Android Makefile to Python so we more easily compare the Android, Linux and Windows builds and share code between them.https://code.briarproject.org/briar/tor-reproducer/-/issues/8Do hardening of builds2022-02-15T10:04:12ZNicoDo hardening of buildsThe following discussion from !17 should be addressed:
- [ ] @akwizgran started a [discussion](https://code.briarproject.org/briar/tor-reproducer/-/merge_requests/17#note_59064): (+2 comments)
> I think you mentioned somewhere tha...The following discussion from !17 should be addressed:
- [ ] @akwizgran started a [discussion](https://code.briarproject.org/briar/tor-reproducer/-/merge_requests/17#note_59064): (+2 comments)
> I think you mentioned somewhere that you had to drop `-O3` (which is commented as "needed for FORTIFY_SOURCE" in the Linux build), but I couldn't find where you mentioned it. I can't remember where the FORTIFY_SOURCE comment came from - possibly the Guardian Project's original Makefile? Any idea whether it's important?https://code.briarproject.org/briar/tor-reproducer/-/issues/5Properly handle enviroment variables2022-04-03T10:52:06ZNicoProperly handle enviroment variablesThis script is growing bigger and bigger and while working on !17 I noticed that the environment variables from the Linux build are causing the Windows build to fail, because `os.environ.copy()` doesn't seem to do what it pretends to do....This script is growing bigger and bigger and while working on !17 I noticed that the environment variables from the Linux build are causing the Windows build to fail, because `os.environ.copy()` doesn't seem to do what it pretends to do. I.e., we still modify the original environment variables after the copy and thus I now manually pop them after each build (15971b47a2c7b7f01abb422b42dd0cc75f7a8045).
Since this is quite ugly, we should do something better in the future.
There are simple helper functions [like this one](https://gist.github.com/igniteflow/7267431#gistcomment-2551951) which would allow us to really only temporary set environment variables, but since this would require larger architectural changes, I didn't do it yet. Especially given that we might use official Tor binaries soon anyway.