Can't summon soft_input with egui + winit + wgpu

[inferrna]

Hi. I'm trying to display soft keyboard in simple app by calling show_soft_input:
https://github.com/inferrna/hello_world_android_egui/blob/try_keyboard/src/lib.rs#L357
but nothing happens, even logcat contains no messages related to soft input, keyboard, etc.

Googling I found a lot complains about ANativeActivity_showSoftInput()
There is another way to do it — via JNI call. But it requires access to native_activity, which is currently hidden.

[rib]

Hi,

Good Android IME support is something that still needs work in Egui, Winit and android-activity unfortunately but is something I'd like to make some progress on.

You can see an initial attempt I made at enabling input method support in android-activity here: https://github.com/rib/android-activity/pull/24 but I hit some odd input issues at the time which I didn't have a chance to debug.

More recently I also looked at creating initial branches for Winit + Egui to experiment with this too:
https://github.com/rib/winit/tree/android-activity-ime-events
https://github.com/rib/egui/tree/android-winit-ime-support

I would probably recommend first looking at the GameActivity backend if you are looking to enable soft keyboard support since we should be able to utilize the GameText bindings in that backend to also support the InputMethod protocol.

Maybe you could take a look over some of the above branches to see if they are a helpful starting point.

At least in terms of using JNI to access the Activity you can use the ndk_context crate to get to the Activity, e.g. something like:

     let ctx = ndk_context::android_context();
     let jvm_ptr = ctx.vm();
     let jvm = unsafe { jni::JavaVM::from_raw(jvm_ptr.cast()).expect("Expected to find JVM via ndk_context crate") };
     let env = jvm.attach_current_thread_permanently().unwrap();

     let activity_ptr = ctx.context();
     let activity = unsafe { jni::objects::JObject::from_raw(activity_ptr as jni::sys::jobject) };

ndk_context provides a general (also usable for native Rust libraries that don't know whether they are part of a standalone Rust application) way to access a JavaVM on Android and an Activity/Context reference. android-activity initializes the state for ndk_context before your android_main() runs.

[tkkcc]

There is an implement in miniquad/miniquad, use a normal Activity.
not-fl3/macroquad#526
https://github.com/not-fl3/miniquad/search?q=showKeyboard

[rib]

@tkkcc if you just want to show the keyboard you should be able to use https://docs.rs/android-activity/0.4.0/android_activity/struct.AndroidApp.html#method.show_soft_input which should be equivalent to what macroquad does.

...but, unfortunately, that won't be very useful without also having input method support too.

macroquad has a custom subclass of the Activity class which effectively lets you call the InputMethodManager::showSoftInput (https://developer.android.com/reference/android/view/inputmethod/InputMethodManager#showSoftInput(android.view.View,%20int)) API via JNI

In android-activity the show_input_method API should do be equivalent to what macroquad does:

When android-activity is built for NativeActivity It calls an NDK API called ANativeActivity_showSoftInput which will end up calling an internal android_NativeActivity_setWindowFlags (https://github.com/aosp-mirror/platform_frameworks_base/blob/ffeb11af1c22a1b8e2af57aac338e1fb57261ec5/core/jni/android_app_NativeActivity.cpp#L202) and using some IPC to invoke this: https://github.com/aosp-mirror/platform_frameworks_base/blob/ffeb11af1c22a1b8e2af57aac338e1fb57261ec5/core/jni/android_app_NativeActivity.cpp#L249 and that will call a showIme method for NativeActivity.java - similar to the macroquad subclass of Activity: https://android.googlesource.com/platform/frameworks/base.git/+/master/core/java/android/app/NativeActivity.java#329

When android-activity is built for GameActivity the show_soft_input will call an internal GameActivity_showSoftInput that uses IPC like in NativeActivity to send a CMD_SHOW_SOFT_INPUT command that then calls GameTextInput_showIme which then calls a JNI method called setSoftKeyboardActive that also does the same thing as macroquad: https://android.googlesource.com/platform/frameworks/opt/gamesdk/+/refs/heads/master/game-text-input/src/main/java/com/google/androidgamesdk/gametextinput/InputConnection.java#113

I've at least smoke tested that show_soft_input has worked for me in the past when working with experimental branches to start enabling better IME support so I'm not currently sure why nothing was happening for @inferrna when calling show_soft_input via the linked example code.

[LinusDikomey]

I have a similar issue, I do use show_soft_input (in a native activity) but nothing happens. I just get the following message:
InputMethodManager: Ignoring showSoftInput() as view=android.app.NativeActivity$NativeContentView{a4db4 V.ED..... ......ID 0,0-1080,2084} is not served.

[tkkcc]

agdk-eframe demo is able to show and hide input method via button

code

use eframe::egui;
use eframe::{NativeOptions, Renderer};

#[cfg(target_os = "android")]
use winit::platform::android::activity::AndroidApp;

struct DemoApp {
    app: AndroidApp,
}

impl eframe::App for DemoApp {
    fn update(&mut self, ctx: &egui::Context, _frame: &mut eframe::Frame) {
        egui::CentralPanel::default().show(ctx, |ui| {
            if ui.button("ime show").clicked() {
                self.app.show_soft_input(false);
            }
            if ui.button("ime hide").clicked() {
                self.app.hide_soft_input(false);
            }
        });
    }
}

fn _main(mut options: NativeOptions, app: AndroidApp) {
    let demo_app = DemoApp { app };
    eframe::run_native("My egui App", options, Box::new(|_cc| Box::new(demo_app)));
}

#[cfg(target_os = "android")]
#[no_mangle]
fn android_main(app: AndroidApp) {
    use winit::platform::android::EventLoopBuilderExtAndroid;
    android_logger::init_once(android_logger::Config::default().with_min_level(log::Level::Info));
    let mut options = NativeOptions::default();
    let app2 = app.clone();
    options.event_loop_builder = Some(Box::new(move |builder| {
        builder.with_android_app(app);
    }));
    _main(options, app2);
}

i don't know if clone an AndroidApp will cause something wrong. i don't know how to access options.event_loop_builder from ui update fn.

[tkkcc]

tmp.mp4

now i can achieve some level (very buggy) of input experience. based on agdk-eframe demo, picking the last commit of

• https://github.com/rib/winit/tree/android-activity-ime-events
• https://github.com/rust-mobile/android-activity/tree/ime-support

and in game_activity/mod.rs, i put the check of textInputState before the check of android_app_swap_input_buffers. because the second is null, when the first is valid.

    pub fn input_events<F>(&self, mut callback: F)
    where
        F: FnMut(&InputEvent) -> InputStatus,
    {
        // eprintln!("555 {}", (*app_ptr).textInputState);
        unsafe {
            let app_ptr = self.native_app.as_ptr();
            if (*app_ptr).textInputState != 0 {
                let state = self.text_input_state();
                callback(&InputEvent::TextEvent(state));
                (*app_ptr).textInputState = 0;
            }
        }

        let buf = unsafe {
            let app_ptr = self.native_app.as_ptr();
            let input_buffer = ffi::android_app_swap_input_buffers(app_ptr);
            if input_buffer.is_null() {
                return;
            }
            InputBuffer::from_ptr(NonNull::new_unchecked(input_buffer))

issues:

1. the logic has some mistake. like the end of record, after one key, the long deleted text shows up.
2. existing string should be passed to gameactivity, may be using settextinputstate.
3. ime support should be transparent to lib user.
4. ime language is not system default. it can only be english.
5. hardward keyboard not work.

[lucasmerlin]

I'm also looking into keyboard support on android (and iOS). Based on @rib s work I have been able to get pretty decent support working on android 🥳

Screen_Recording_20230708_144153.mp4

(opening and closing the keyboard also works, even if it's not in the video)

Unfortunately for this to work, I had to add completely new events to winit, basically passing the TextInputState struct from android activity all the way through to egui, and then handling the logic there.
This is not ideal, but I'm not sure how much we could improve on this, with the way the GameTextInput works.

The most difficult part was finding the right places in egui to send the correct state updates when selecting a new text field or updating the selection.

If you'd like to try this you can use my relevant branches by adding this to your Cargo.toml (your project must be using game-activity for keyboard input to work):

[patch.crates-io]
winit = { git = "https://github.com/lucasmerlin/winit", branch = "v0.28.x_ime_support" }
egui = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
eframe = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
egui-wgpu = { git = "https://github.com/lucasmerlin/egui", branch = "mobile_ime_support"}
android-activity = { git = "https://github.com/lucasmerlin/android-activity", branch = "ime_support"}

These are the relevant branches:
https://github.com/lucasmerlin/winit/tree/mobile_ime_support
https://github.com/lucasmerlin/egui/tree/mobile_ime_support
https://github.com/lucasmerlin/android-activity/tree/ime_support (based on the work of @rib)

For this to feel native on android what's still missing would be egui showing the composition range with underlined text instead of the blue selection highlight. But I think that should be pretty straightforward.

Also missing of course is to update the insets / resize the window when the keyboard opens.

Next, I'll look into iOS input support, I already have opening the keyboard and typing working but I'll also try to get autocomplete and suggestions.

[fr-an-k]

show_soft_input() is still not working in NativeActivity with winit 0.29.10 (android-activity 0.5.0), nothing happens, no error on Android 8.1 or Android 11. I call it in android_main and after a touch event.

On Android 13 I get:
InputMethodManager: Ignoring showSoftInput() as view=android.app.NativeActivity$NativeContentView{454c3ca V.ED..... ......ID 0,0-1080,2023} is not served.

I suspect this has to do with using a GPU surface.
I will see if the window flags are correct (if I manage to access them).

https://github.com/rib/android-activity/pull/24 is a dead link.

GameActivity examples (rust-android-examples) immediately crash on my devices (Android 8.1, Android 11 and Android 13) with various errors, so I can't consider that for production.
On Android 13 the error is undefined symbol: ANativeActivity_onCreate; this may have to do with cargo apk/xbuild, they're quite limited.

I'll try to do it through JNI.

[MarijnS95]

https://github.com/rib/android-activity/pull/24 is a dead link.

Since this repository was migrated/transferred to the rust-mobile organization (whew you created the issue), then Rib forked the repo again for personal contributions, GitHub no longer redirects. Instead, replace rib with rust-mobile to find the right link: #24

[fr-an-k]

Ok, thanks.
It looks like a View is needed (created in onCreate) to request focus first, as done here:
https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/app/NativeActivity.java

Shouldn't ANativeActivity_onCreate simply call it's super.onCreate?

[MarijnS95]

Shouldn't ANativeActivity_onCreate simply call it's super.onCreate?

Why would that be required, given that NativeActivity exists to not have to write Java/JNI (to at least get a basic app going)?

The Java layer that powers NativeActivity does call super.onCreate:

https://cs.android.com/android/platform/superproject/main/+/main:frameworks/base/core/java/android/app/NativeActivity.java;l=180;drc=5d123b67756dffcfdebdb936ab2de2b29c799321

It's unclear what line your link is supposed to link to.

[fr-an-k]

Ok, I see.
I managed to get GameActivity to work with soft input (but not the latest winit/wgpu versions) on Android 8.1.

I guess you don't get soft input on a window surface; you'd need a view.

It might be possible to achieve this with NativeActivity if you create a SurfaceView in the same way that is done for GameActivity.

But I guess that's kind of deprecated and people will just have to use GameActivity for soft input on a GPU surface.

[ibaryshnikov]

It works with NativeActivity if you use getDecorView, here's an example which worked for me
https://github.com/mvvvv/StereoKit-rust/blob/master/src/tools/os_api.rs#L196
Or it may be a few lines of Java, something like

public static void JNI_showKeyboard() {
    InputMethodManager im = (InputMethodManager)Main.getContext().getSystemService(Context.INPUT_METHOD_SERVICE);
    im.showSoftInput(window.getDecorView(), InputMethodManager.SHOW_IMPLICIT);
}

Still, there's a number of issues which prevent it to be usable in current state:

• the window is not resized event with the android:windowSoftInputMode="adjustResize" property
• the keyboard doesn't react to changing locale or input methods withing one locale such as different ways to enter Japanese characters
• special characters are not handled properly, the ones which appear when you hold a key for awhile

[rib]

Although a few more general text input topics came up in the discussion here, I think the original issue here (show_soft_input() wasn't working with the native-activity backend, due to issues with ANativeActivity_showSoftInput) has been fixed by #178 (and tested with egui + winit)

It's perhaps worth noting here that the NativeActivity backend is not expected to ever get full InputConnection / IME support and so there's quite a significant limitation on what kind of text input can be supported, but with some keyboards you can at least get physical KeyEvents for basic ascii input.

For InputConnection / IME support it's recommended to use the game-activity backend if possible.