This document describes how RootPresenter and Scenic process visually-related input, such as touch, mouse, and keyboard. We'll work roughly bottom up through the layers of abstraction, from device to gesture.
Other inputs, such as buttons, audio, and video, are out of scope for this document.
Drivers : gives us inputs as InputReport FIDL tables
RootPresenter : routes touch and mouse input from Drivers to Scenic
Scenic : routes touch and mouse input from RootPresenter to UI Clients
Keyboard Subsystem : routes and transforms text input to UI Clients
UI Client : consumes inputs from Scenic and Keyboard Subsystem to drive UI
Input Drivers provide access to input peripherals through the file system under /dev/class/input-report
. These are presented as structured FIDL tables that describe the device and the reports that will be sent.
Generally, the RootPresenter is the singleton process that has detailed and specific knowledge about the entire device, such as details about the display, peripherals, sensors, etc. It takes care of device management details, such as reading out InputReports reports from Drivers, and packages them into FIDL structs for consumption by Scenic or other entities.
It also instructs Scenic to create the top-level (or “root”) elements of the scene graph, and vends the Presenter API that UI clients use to attach their visual content to the scene graph.
The general transformation for an input event through RootPresenter is from the Driver, to InputReport
, to InputEvent
. The InputEvent
is sent to Scenic.
The InputReader
library is the code responsible for actually monitoring /dev/class/input-report
for new peripherals, and reacting to new reports from existing peripherals. It forwards new events for processing to other parts of RootPresenter. More information on InputReader
can be found here.
For each new peripheral (an input device), InputReader
assigns a new InputInterpreter
object that reads the InputReport descriptor report for a single input device, and performs bookkeeping by pushing a DeviceDescriptor
and its designated event forwarding channel, an InputDevice
, to the InputDeviceRegistry
FIDL protocol. (The InputDeviceRegistry
protocol also enables programmatic input injection from outside RootPresenter.) The InputDeviceRegistry
protocol is vended by RootPresenter, and in addition to bookkeeping (details below), informs each Presentation
about the new peripheral.
For each new event, InputInterpreter
reads a fuchsia.input.report:InputReport
, transforms it into a fuchsia.ui.input:InputReport
, and forwards it on InputDevice::DispatchReport
.
The implementation of DispatchReport
forwards the InputReport
to the registered InputDeviceImpl::Listener
, typically the RootPresenter itself. In turn, the InputReport
is forwarded to the active Presentation
.
For internal bookkeeping, each Presentation
keeps a mapping of InputDevice
ID to an associated DeviceState
. The DeviceState
is used to create a little persistent state for each peripheral, e.g., keeping track of a mouse device's DOWN/MOVE/UP state. In Presentation
, the InputReport
is routed to its relevant DeviceState
, where it is transformed into an appropriate InputEvent
, and is sent to the OnEventCallback
that was registered at the DeviceState
's constructor (when the peripheral was first added).
The InputEvent
is now handled by RootPresenter's OnEvent
callback. It looks for global hooks, displays a mouse cursor, adjusts for predetermined screen rotation, and finally enqueues the InputEvent
as an InputCmd
to Scenic.
Sensor HID reports are handled in an analogous fashion. Some differences are:
DeviceState
.InputReport
is typically enough for plumbing out to clients.In contrast to RootPresenter, Scenic has less knowledge about the device. Instead of knowing about peripherals, it receives InputEvent
FIDL structs from RootPresenter. Generally, it owns and manages the large-scale visual elements that each UI client creates (the scene graph), and handles mouse and touch input (hereafter: “pointer input”) dispatch to each UI client.
Root Presenter performs pointer event injections into Scenic over the [fuchsia.ui.pointerinjector
] protocol, which is a privileged capability.
Pointer events, such as touch, typically follow an ADD → DOWN → MOVE* → UP → REMOVE state sequence, encoded as PointerEventPhase
.
On ADD, we identify the set of potential clients by performing a hit test, and forward this event to these clients. To associate future touch events by the same finger to the same clients, we track the set of clients for that particular finger. Parallel dispatch is used to enable gesture disambiguation (TBD), where the touch events should eventually be owned by a single client.
On DOWN, we send a FocusEvent
to the single client that is “on top”. We also send a FocusEvent
with focused=false
to the previously focused client.
On MOVE and UP, we merely forward them to existing clients.
On REMOVE, we forward it to existing clients, and then remove the tracking association.
For an overview of pointer coordinate mapping, see Ray Casting and Hit Testing.
The Keyboard Subsystem, implemented in ime_service.cmx, receives KeyEvent
s, and routes them to UI clients.
The Keyboard Subsystem routes key events based on which client(s) have requested such events, and whether or not those clients have received focus.