Conversation
The media module had no platform profiles beyond Windows, so a Linux or macOS build had no compiler or sysroot for its target and CMake simply built for the host. It now takes a toolchain file the way webrtc-jni does, and its profiles mirror webrtc-jni's by name and by the file each one selects, so both native libraries of a given classifier are built for the same target the same way. Keeping one set of toolchain files rather than a second copy is deliberate: they are where this project says which compiler and which sysroot a target uses, and two answers would drift. What configure is told about the target is worked out from what the toolchain file already set -- the architecture from CMAKE_SYSTEM_PROCESSOR, the target OS from the platform, the compiler from CMAKE_C_COMPILER, the target flags from CMAKE_C_FLAGS, the sysroot from CMAKE_SYSROOT -- so a target CMake can already build for needs no new case. Apple is the exception it has to make: there the architecture is a compiler flag rather than a different compiler. Cross-compiled arm and aarch64 start with --disable-asm. An assembler for a foreign architecture is a build prerequisite of its own, and nothing here is fast enough yet to need one; correctness on every platform first. Only windows-x86_64 is verified, because it is the only target this machine can build. It still builds from nothing and its configure line is unchanged apart from an explicit --arch=x86_64, and the module's 19 tests pass. Everything else waits on CI.
The media module is about to become part of the normal build, so every native build job has to be able to produce its FFmpeg, and none of them could. Windows gets MSYS2 with make and nasm, because FFmpeg's configure and makefiles are shell scripts and building it needs a POSIX shell even with MSVC. MSYS2_ROOT is exported so the media module's CMake finds it wherever the runner image put it. Linux and macOS get nasm, which is what FFmpeg assembles its x86 code with; macOS needs it even though the runner is Apple Silicon, because the Intel build is cross-compiled on it. The FFmpeg install directory is cached the same way the WebRTC checkout already is, keyed by FFmpeg version, platform and the hash of the dependency script and the module pom, so only the first build of a platform pays for it. The three native build jobs in each workflow now check out submodules, since the FFmpeg source is one. test-natives does not, because it runs against downloaded natives and never builds the module.
The module was opt-in behind a profile while it was being brought up. It now builds with everything else, which is also what lets CI prove it on the platforms this machine cannot build for. That collapses the awkward part of the examples module. The media example needed a source directory, a compile execution and an output directory of its own, purely because a JPMS "requires" cannot be made conditional and the module it names has to exist. With the module always there, the example moves in beside the others and the whole profile becomes one dependency and one "requires", exactly as its comment said it would. Anything building this project now needs the third-party/ffmpeg submodule checked out, and make and nasm to build it. The README and the guide say so where they used to describe the profile.
Every CI job failed at dependency resolution before a single mojo ran: Could not find artifact dev.onvoid.webrtc:webrtc-java-media:jar:windows-x86_64:0.19.0-SNAPSHOT The module declared a dependency on its own classifier artifact so that anything using it would get the natives along the way. That cannot work on a clean machine: the artifact is built and attached by this same module, so at the moment its dependencies are resolved nothing has produced it yet. It only passed locally because an earlier install had left the artifact in the local repository. webrtc-java can do this because webrtc-jni, a module earlier in the reactor, is what builds its natives, and because the artifact has been published besides. Neither is true here. So the dependency is gone, and anything using the module asks for the natives itself with a second dependency carrying the platform classifier. webrtc-examples does, and the guide says so. The module's own tests never needed it: Surefire is given the directory the CMake build collects the natives into. Verified by emptying dev/onvoid/webrtc/webrtc-java-media out of the local repository first, which reproduced the CI failure exactly, and then building clean: 168 tests in webrtc and 19 in webrtc-java-media, all five modules green.
Every job failed configuring the media module:
CMake Error at dependencies/ffmpeg/CMakeLists.txt:162 (set):
when parsing string
C:\Users\runneradmin/ffmpeg/windows-x86_64
Invalid character escape '\U'
Two things had to line up for this, which is why it never showed up
locally. Maven hands the install directory over as ${user.home} plus a
suffix, and on Windows that is a backslash path, which CMake reads as
the start of an escape sequence. And to_shell_path was a macro, so its
argument was substituted into the body as text rather than passed as a
value, leaving CMake to parse C:\Users\... as code.
It stayed hidden because that code only runs when FFmpeg has to be
built. This machine already had it installed, so every local build took
the short-circuit and never reached the macro. CI, starting from
nothing, reached it immediately.
The path is now normalised with file(TO_CMAKE_PATH) as it arrives, and
to_shell_path is a function, which passes its arguments as values.
Either change alone would have been enough; both are worth having,
since the first fixes this input and the second fixes the class of it.
Verified the way it should have been the first time: by deleting the
FFmpeg install directory so that the build path is the one that runs.
The failure reproduces exactly beforehand, and afterwards
"mvn clean package -DskipTests" builds FFmpeg from nothing and every
module succeeds, with the module's 19 tests green.
… with The first CI run that reached FFmpeg turned up three separate faults, all of them from reusing webrtc-jni's toolchain files. Those files are not neutral descriptions of a target: they encode how webrtc-jni links WebRTC's C++ ABI, and this module links no C++ ABI at all. On Linux they compile with -nostdinc++, because webrtc-jni supplies WebRTC's own libc++ through include paths of its own. This module inherited the flag and none of the paths, so its first #include <string> failed. It now uses toolchain files of its own that name the distribution's cross compilers and nothing else. linux-x86_64 needs no file at all, being built natively on the runner. Cross builds also tripped over stripping. make install strips what it installs with whichever strip is on the PATH, and the host's cannot read a binary for another architecture: "unable to recognise the format of the input file". FFmpeg is configured with --disable-stripping when cross-compiling, and the toolchain files name the cross binutils. On macOS the toolchain file set CMAKE_SYSTEM_NAME, which makes CMake report a cross build even when host and target agree, so configure was handed --enable-cross-compile for a native build along with --arch=aarch64. Apple's clang knows that architecture as arm64 and rejects the other outright, which configure reports only as being unable to create an executable. Neither macOS target is a cross build -- the Intel one runs under "arch -x86_64" -- so neither takes a file now, and the architecture is spelled the way Apple spells it. windows-aarch64 passed and keeps webrtc-jni's file, which suits it: there the C++ standard library comes from MSVC either way. Verified as far as this machine can: windows-x86_64 unaffected, with the full reactor green. The other targets wait on CI again.
windows-x86_64 built everything, including the media module, and then failed in webrtc-java-examples: Could not find artifact dev.onvoid.webrtc:webrtc-java-media:jar:windows-x86_64:0.19.0-SNAPSHOT The examples asked for the media module's natives as a dependency. CI runs "mvn package" and then "mvn -B jar:jar surefire:test", and the second of those invokes goals directly rather than running a lifecycle, so nothing reaches the package phase where that jar is attached and there is nothing to resolve against. The first command had succeeded moments earlier, which is why only the second failed. webrtc-java survives the same pattern only because its classifier artifact has been published and is already in the local repository. So the natives are off the dependency graph entirely. The example needs them at run time, and exec-maven-plugin is pointed at the directory the CMake build collects them into, which is how the media module's own tests have always found them. Verified by running both CI commands in order against a local repository with the media artifacts deleted: 168 tests in webrtc and 19 in webrtc-java-media, every module green, and the example still plays.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #291, which adds the media module itself. This is the part that makes it build everywhere rather than only on the machine it was written on.
What this does
Cross-compilation for all seven targets. The media module had no platform profiles beyond Windows, so a Linux or macOS build had no compiler or sysroot for its target and CMake simply built for the host. It now takes a toolchain file the way
webrtc-jnidoes, and its profiles mirrorwebrtc-jni's by name and by the file each one selects, so both native libraries of a given classifier are built for the same target the same way.Reusing those toolchain files rather than writing a second set is deliberate: they are where this project says which compiler and which sysroot a target uses, and two answers would drift apart.
What configure is told about the target is worked out from what the toolchain file already set — the architecture from
CMAKE_SYSTEM_PROCESSOR, the target OS from the platform, the compiler fromCMAKE_C_COMPILER, the target flags fromCMAKE_C_FLAGS, the sysroot fromCMAKE_SYSROOT— so a target CMake can already build for needs no new case here. Apple is the exception it has to make: there the architecture is a compiler flag rather than a different compiler.Cross-compiled arm and aarch64 start with
--disable-asm. An assembler for a foreign architecture is a build prerequisite of its own and nothing here is fast enough yet to need one; correctness on every platform first, speed where it is measured.CI. Windows gets MSYS2 with make and nasm, because FFmpeg's configure and makefiles are shell scripts and building it needs a POSIX shell even with MSVC;
MSYS2_ROOTis exported so the module's CMake finds it wherever the runner image put it. Linux and macOS get nasm, which macOS needs even though the runner is Apple Silicon, because the Intel build is cross-compiled on it.The FFmpeg install directory is cached the same way the WebRTC checkout already is, keyed by FFmpeg version, platform and the hash of the dependency script and the module pom, so only the first build of a platform pays for it. The three native build jobs in both workflows now check out submodules, since the FFmpeg source is one;
test-nativesdoes not, because it runs against downloaded natives and never builds the module.The module joins the normal build. That also collapses the awkward part of the examples module. The media example needed a source directory, a compile execution and an output directory of its own, purely because a JPMS
requirescannot be made conditional and the module it names has to exist. With the module always there, the example moves in beside the others and the whole profile becomes one dependency and onerequires.What is verified, and what is not
Verified on this machine:
mvn verifygreen: 168 tests inwebrtc, 19 inwebrtc-java-media, all five modules succeedingwindows-x86_64, its configure line unchanged apart from an explicit--arch=x86_64MediaFileExampleruns from a plainmvn exec:java, with no profile and no extra class pathNot verified: six of the seven platforms. Only
windows-x86_64can be built where this was written — no ARM64 cross tools, no Linux, no macOS. Everything for the other six is written from FFmpeg's documented cross-compile flags and this project's existing toolchain files, and CI is the only thing that can prove it. Expect the first run to want corrections.Worth deciding before merge
The last commit makes the media module part of the normal build, which means anything building this project now needs the
third-party/ffmpegsubmodule checked out, plus make and nasm to build it. That is a new prerequisite for every contributor, not just for those who want the media module.The original plan put that step after every platform builds in CI, not before. It is here so that CI builds the module at all and can prove the other six targets. If CI is not green, or if the prerequisite is not wanted yet,
4399bb2is the tip of the branch and drops cleanly on its own; the first two commits stand without it.