We build Aiden, a physical mobile AI agent that connects to a phone over USB and presents itself as a keyboard and mouse to reproduce real-world iPhone interactions. During real-device testing, we hit a bug that took controlled experiments to actually isolate, and it turned out to live inside iOS itself, not our stack.
The symptom
Ordinary letters worked fine. Modifier-key combinations intermittently didn't.
| Input | Expected result | Affected result |
|---|---|---|
x |
Inserts x
|
Inserts x
|
Shift+X |
Inserts X
|
May insert lowercase x
|
Command+A |
Selects all | No selection |
Command+V |
Pastes into the active field | No paste action |
Our write to the Linux USB Gadget device at /dev/hidg0 completed successfully every time. Nothing happened on the phone's screen.
Two iOS settings matter here: AssistiveTouch has to be enabled for an external mouse to work on iPhone at all, and Show Onscreen Keyboard (inside AssistiveTouch) keeps the software keyboard available while a physical keyboard is connected.
First finding: it's not the HID layer
System logs distinguish plain keyboard events from modifier-based key commands, and log the KeyboardFocus target each command resolves to.
Working case:
cmd-v -> <keyboardFocus; pid: 9085; token: MobileNotes>
Broken case, identical keypress:
cmd-v -> <keyboardFocus; pid: 3738; token: com.apple.springboard>
iOS recognized Command+V correctly both times. In the broken case, it routed the resulting command to SpringBoard (the Home Screen process, hosting a lot of system-level interactions) instead of the actual foreground app. The keystroke arrived fine, the routing afterward is where it breaks. We call this "iOS keyboard focus loss" internally, not an Apple term, just our shorthand: the keyboard stays connected, plain keys still land correctly, but modifier commands stop reaching the foreground app.
Ruling things out
Checked and did not fix it:
- Full Keyboard Access toggle (Settings > Accessibility > Keyboards & Typing), reproduced with it both on and off
- Boot Keyboard
protocol=1/subclass=1declaration - LED output behavior
- Apple keyboard descriptor and handshake sequence
- Keystroke timing
- Stale HID file descriptors after USB re-enumeration
Some of these fixed real, separate edge cases. None touched the core failure.
We also suspected toggling AssistiveTouch off/on after connecting broke the keyboard session, since that sequence reliably triggered the bug. A later round of tests removed AssistiveTouch toggling entirely and just added a mouse to an already-stable keyboard-only setup. Bug came back anyway. Demoted from root cause to one of several triggers.
The A/B test that actually isolated it
With AssistiveTouch left on throughout:
| Test setup | Device topology | Result |
|---|---|---|
| Aiden emulated composite device | Keyboard + pointer/mouse | Intermittently routed to SpringBoard when switching between apps |
| Physical gaming keyboard | Keyboard + firmware-declared virtual mouse | Command/Shift/Option commands routed to SpringBoard |
| Aiden keyboard-only | Keyboard-only, pointer: 0 | 32 Command+V attempts, 0 SpringBoard-only events |
| Same keyboard + physical USB mouse | Keyboard-only + separate mouse | 11 attempts, 4 SpringBoard-only events |
| Keychron K2 Max, Bluetooth mode | BLE keyboard + mouse collection | 32 attempts, 26 SpringBoard-only events |
The critical pair is rows 3 and 4: identical keyboard implementation, identical iOS settings, identical test procedure. Plugging in an ordinary USB mouse brought the failure back. Unplugging it, all 22 follow-up runs succeeded.
Conclusion: with AssistiveTouch enabled, if iOS detects pointer or mouse capability on the connected device at the same time as a keyboard, it can misroute modifier-based commands to SpringBoard instead of the foreground app.
This isn't specific to our HID implementation, it reproduces on a real gaming keyboard declaring a virtual mouse, and on a Bluetooth keyboard with a mouse collection in its report descriptor. That rules out an Aiden-specific compatibility explanation. The logs place the fault boundary inside iOS: the system receives the modifier key, generates the correct key command, then routes KeyboardFocus to SpringBoard anyway. Without iOS source access, we can't say why pointer/mouse capability affects that resolution, only that it reliably does.
We also tried substituting an HID touchscreen/digitizer for the mouse, keeping pointer: 0. Six Command+V attempts, all landed correctly. Further support for the pointer/mouse link, but any keyboard interface present still makes iOS hide the software keyboard, and a one-time Eject only restores it briefly, so this isn't viable for our actual product experience.
Our workaround
Aiden only needs modifier keys when executing a shortcut, and the trigger is tied specifically to pointer/mouse capability being present. So:
- Two separate USB HID profiles: normal operation keeps keyboard + pointer; a modifier-key action first re-enumerates to keyboard-only
- Different USB Product IDs and serial numbers between the two profiles, so iOS doesn't reuse stale state from the mouse-equipped profile
- Once the shortcut completes, pointer capability is restored
- Switching scope covers a whole agent action, not a single keypress, to avoid the software keyboard popping and retracting repeatedly mid-action
- The normal profile is restored on action completion, error, cancellation, or panic cleanup
To be direct: this is a workaround, not a fix. For the brief window a modifier key needs to take effect, we simply don't let iOS see pointer/mouse capability at all.
Has anyone else hit this?
It reproduces on hardware that has nothing to do with us, so if you're driving iOS with an external keyboard plus pointer and have seen a shortcut silently fail, this might be why.
Top comments (0)