Repository navigation
Android NDK r25b changes will break developers using r22b or older #103673
Description
Activity
Nominating for compiler discussion.
I would probably be somewhat in favor of fully rolling back the bump on soon to be beta 1.66 (we branch today), and landing a bump (edit: on master) only to r21/r22 so we retain compatibility for slightly longer.
We should in parallel issue a blog post with our strategy for this bump, and ideally a strategy for future bumps. It looks like Android's LTS releases are only supported until the next one is issued (once a year roughly) which means there's not really an overlap period in which we can stick with a supported copy until user's migrate. This situation definitely reminds me of our similar challenges on BSDs.
Reacted by Jon Gjengset and Evgeniy DushistovI see a couple issues here:
Difficulty in moving between ndk versions
I think we should strive to match the experience of the ndk itself here. In particular if there is no ndk release with both libgcc and libunwind, the Rust project shouldn't go out of its way to ease the upgrade experience beyond giving people enough warning. (If someone wants to contribute such a soft transition path and it's not a lot of maintenance overhead, that's fine.)
Still, matching the ndk experience requires being in line with the expected upgrade cadence of ndk releases. I think this is ultimately up to the Android target maintainers.
Tying rustup releases to ndk and target API levels
It doesn't seem ideal to have your version of Rust coupled to your target API level, especially on a frequently updating platform such as Android. (Unless you can mix code targeting different API levels in the same binary, in which case it might be fine?) I expect to see similar issues with Fuchsia, where the goal is to update the platform even more frequently.
Thinking out loud a bit, this makes me wonder if we should just point people toward
-Z build-stdand stabilize the functionality in some capacity. This gives people maximum flexibility to mix and match ndk versions, set target API levels, and manage upgrade paths on their own. On my machine it's not that much overhead (+~17s to initial build time), but I would like to be conscious of the impact to people who are building on slower laptops.On point 1: Totally agree that the Rust project isn't the right place to ease the android-ndk upgrade experience in general. Is there clear direction somewhere on what "enough warning" means? With the glibc version bump, the blog post was published almost two months in advance.
On point 2: In the general case, at least, rustup does fairly well with staying uncoupled from ndk versions. As I mentioned, a distribution of rust that uses ndk r22b will build fine against an application using as old as r10e, and possibly older. This weird situation is more a problem of the android-ndk than it is a problem of anything Rust has done. (As a side note, with target API levels, I understand the story to be a bit more complex, with the current min of
android-14changing toandroid-19with this version bump, which restricts the possible targets the application can declare, but users can declare a lower min or target if their ndk supports it, even if that causes issues for the stdlib).Rust is fairly unique here because, if you combine points 1 and 2, every project that uses Rust has a dependency by definition that will break them when it upgrades. C++ doesn't have this problem because their stdlib gets distributed as part of the ndk. Theoretically, a random C++ library might also have this problem if they link against
libgcc, but no one library is going to have the kind of reach across C++ that the Rust stdlib has across Rust. It's also not uncommon in my experience for Android C++ projects to download their dependencies from source anyway (fresco is a good example) and build them directly.Pointing people toward a stabilized version of
-Z build-stddefinitely seems like a preferred solution here. Initial builds are going to have other far greater bottlenecks, and as an added bonus, Android developers are often also binary-size-conscious whichbuild-stdwould help with. It's not clear to me whether you're suggesting that as a prerequisite for the ndk-bump or just as a useful tool in the future. Isbuild-stdfar enough along that some form of it could be stabilized in the right timeframe?I think there is enough meat here to fire up an
MCPFCP. Similar situation were handled that way. An MCP was suggested in the first place by @jyn514 in this comment.Anyway, this will be discussed tomorrow. Between the previous discussion on Zulip and this issue comments, I think there is already great context provided.
Reacted by Jon Gjengset and Alex Pinkus47 remaining items
It might break due to
workarounds you might have regarding
libunwind.But I can expand on that a bit more. I'm currently creating a pull request to the blog to help move things along.
PR for announcement created here.
Thanks for opening up the PR! I'll add my comments there.
I think the breakage @glandium is referring to is the same one I'm concerned about: anyone using a currently-supported NDK version (i.e. r22 or below) will be unable to build until they switch their NDK version. In other words, anyone not currently applying a workaround will be broken.
I'll share a suggestion on the PR for how we might message that without seeming too gloomy.Edit: @chriswailes was too fast for me and already addressed this on the PR -- thank you!the problem with making simple changes to a large open source project is that they're simple enough that everyone can comment on them, have an opinion and bike shed it to death. if this was some complex compiler issue, I'm sure it would have been fixed and released by now and there would have been at most 1-2 people commenting on the PR.
This was resolved in the end by accepting the breakage and notifying users via a blog post; see #105716 and rust-lang/blog.rust-lang.org#1055. So I think we can mark this as closed now.
- added 4 commits that reference this issue
on Nov 2, 2023
Discussion continued from #102332.
Background
Changes that were merged in #102332, slated for 1.66.0, will update the CI build scripts so that all Android targets1 use the
r25brelease2 of the Android NDK to build Rust binaries, moving fromr15c. Ther25brelease is the newest version of the NDK at time of writing (Oct 2022), and is a Long Term Support release (LTS)3.r15cpredates LTS designation, is over five years old, and is past the end of its support life. Rust's Android platform support document4, merged on 24 September 2022, indicates support for the most recent LTS release of the NDK, so the changes in #102332 are in line with that document.Currently, Rust developers who wish to use an
android-ndktoolchain newer thanr22bmust use build-time hacks to do so. This is because the Rust standard library useslibgccfor its unwinder implementation on Android, butlibgccis not included in new versions of the NDK. When usingr23bor newer in a Rust library, a developer will seeld: error: unable to find library -lgcc. #85806 added logic to select betweenlibgccorlibunwind, but this logic runs when building Rust itself, not when compiling the downstream application, so it is mainly useful with the nightly flag-Z build-std. Since older versions of the NDK do not havelibunwind, and newer versions do not havelibgcc, support forr23(and newer) is mutually exclusive from support forr22(and older), unless a developer applies a manual workaround5. This workaround, for buildingr25b, with a binary release that useslibgcc6, involves a fakelibgcc.acontainingINPUT(-lunwind). One common form of this workaround7 is used in theringcrate and referenced from setup scripts forflutter_rust_bridge; a slightly modified form can be found incargo-apk.Problem
When the updated CI script lands in a stable release, the Rust binary releases for Android targets will compute
has_unwind=true, and link againstlibunwindinstead oflibgcc, causing every project that builds using NDK versionr22bor older to fail with the messageerror: cannot find -lunwind. Because building withr23bor newer requires either nightly Rust or a build-script workaround, it is likely that most Android projects using Rust today are usingr22bor older and will therefore be broken by this change. One comment notes that Firefox will be broken8; a quick GitHub search also uncovers libraries likeopenssl-srcamong others that build using older NDK versions on CI.Depending on the version that a project must be upgraded from, going to
r23bcan be a non-trivial effort. The process for obtaining a toolchain changed inr199, so a developer must rewrite their CI compilation scripts. One commenter noted seeing compilation issues in a codebase containing C++ headers10, although they were able to resolve the issues. These are not reasons to never upgrade, but they indicate that a surprise NDK upgrade may be unwelcome work requiring more than just a version number bump.Options
Currently, an undetermined (but probably large) subset of the Android Rust ecosystem will find itself broken on December 15th, when Rust version 1.66.0 comes out. What should be done about it? Some ideas, in no particular order:
1. Roll back to version r15c by reverting the change
Reverting the change is the simplest option, and buys an indefinite amount of time to define a more graceful deprecation strategy. Doing so, however, prolongs the usage of a build tool that's well past end of life, and means that any features introduced in the last 5 years aren't available to Rust developers.
2. Downgrade to a newer toolchain that is older than
r23With a net-new change to use
r21e(newest compatible previous LTS version, released January 2021) orr22b(released March 2021)11, the binaries produced by CI could take advantage of a significantly newer toolchain and retain compatibility with older toolchains downstream. A Rust binary compiled usingr22bcan be linked into an application using a toolchain as old asr10e(and likely older, but I didn't have an older NDK at hand).However, both
r21eandr22bare still considered obsolete and unsupported. Upgrading to one of these versions means that breakage will still occur later, when we do decide to upgrade.3. Warn users about breakage over a longer time period
On the
r25bpull request, @jonhoo mentioned that a blog post would help alert users to the change. @jonhoo compared this to the change in minimum linux-gnu versions that happened in August12 and the accompanying announcement13. That change was pre-warned over several releases; in order to take a similar action here, ther25bPR would need to be reverted. While this impacts just the Android targets, rather than tier-1 linux targets as the glibc bump did, the impact on those developers is much more severe. By raising awareness of the issue and defining a timeline for the change, the blog post would help to reduce impact and limit unpleasant surprises.4. Make no change, but still warn users about the looming breakage
Even if no action is taken to delay this change, a blog post will be a prudent way to alert developers that the change is happening in six weeks. If it is safe to recommend the
INPUT(-lunwind)workaround for broad use, the blog post could recommend immediate migration tor23or newer before the change hits stable.Conclusion
As time marches on, developer tools evolve, and long-term support versions fade into unsupported obsolescence. It's natural and expected for Rust to drop support for older releases of the Android NDK, as with any other toolchain or environment that has passed end of life.
However, the current implementation will require a big-bang migration to
r23+for any project wishing to adopt1.66.0. Some developers may have adoptedr23, if they are attentive to Rust community workarounds, or brave enough to run their CI only on nightly. It's likely that many have not. In the interest of preserving existing project compatibility, it may be prudent to attempt a graceful migration tolibunwindwherever possible, which can be done after still upgrading to a version that's newer than the current one. By giving developers ample warning that allows them to adopt today's workaround, the Rust project can keep the ecosystem current without causing sudden surprises.Footnotes
Android targets:
arm-linux-androideabi,armv7-linux-androideabi,thumbv7neon-linux-androideabi-ndk,i686-linux-android-ndk,aarch64-linux-android,x86_64-linux-android↩Android NDK r25 revision history. ↩
NDK release process, detailing the implications of LTS releases. ↩
Platform support for
*-linux-androidand*-linux-androideabiand PR: Add a platform support document for Android #101780 ↩libunwindavailability in the Android NDK:Apparent first attestation of the
INPUT(-lunwind)workaround ↩echo "INPUT(-lunwind)"search results on GitHub ↩Comment from @glandium indicating Firefox breakage ↩
Standalone toolchain (deprecated) documentation ↩
Comment reporting compilation errors caused by the upgrade ↩
In order to use
r22b, the logic inlibrary/unwind/build.rswill also need to preferlibgccoverlibunwind, sincer22bincludeslibunwind.awithin the C++ STL inarm-linux-androideabionly, for reasons that are not clear. ↩https://github.com/rust-lang/rust/pull/95026 ↩
Increasing the glibc and Linux kernel requirements ↩