Handling input changes

Touch-specific UI

Any time your app has touch-specific UI elements, e.g. an on-screen joystick in a game, you should consider what the user experience will be when touch is not being used. In some mobile games, a significant part of the screen is used up by the controls which are necessary for touch-based play, but are unnecessary if the user is using a gamepad or keyboard to play. Your app or game should provide logic to detect which input method is being actively used and adjust the UI accordingly. See the Implementation section below for some examples of how to do this.

On-screen prompts

Your app may be providing helpful on-screen prompts to your users. Be careful that these are not input-device dependent. For example:

• Swipe to…
• Tap anywhere to close
• Pinch-to-zoom
• Press ‘X’ to…
• Long-press to activate

Some apps may be able to adjust their wording to be input agnostic. This is preferable where it makes sense, but in many cases specificity is important and you may find yourself needing to show different messages depending on the input method being used, particularly in tutorial-type modes like on the apps first run.

Multiple-language considerations

If your app supports multiple languages, you will want to think through your string architecture. For example, if you support 3 input modes, and 5 languages, that could mean 15 different versions of every UI message. This is going to increase the amount of work required to add new features and amplify the likelihood of spelling errors.

One approach is to think of interface actions as a separate set of strings. For example, if you define the “close” action as its own string variable with input-specific variants like: “Tap anywhere to close”, “Press ‘Esc’ to close”, “Click anywhere to close”, “Press any button to close”, then all of your UI messages that need to tell the user how to close something can use this single “close” string variable. When the input method changes, simply change the value of this one variable.

Soft keyboard / IME input

Remember that if a user is using an app without a physical keyboard, text input can occur via an on-screen keyboard. Be sure to test that necessary UI elements are not occluded when an on-screen keyboard appears. See the Android IME visibility documentation⁠ for more information.

Prioritized state machine based on received events

One approach is to keep track of the current “active input state” - representing the input prompts currently shown on-screen to the user - by keeping track of actual input events being received by the app and transitioning between the states using prioritized logic.

Example: game with touch, keyboard/mouse, and controller support

Imagine you have developed a car racing game for touch-based mobile phones. Players can accelerate, decelerate, turn right, turn left, or use nitro for a speed boost.

The current touchscreen interface consists of an on-screen joystick on the bottom left of the screen for the speed and direction controls, and a button on the bottom right for nitro. The user can collect nitro canisters on the track and when the nitro bar at the bottom of the screen is full, a message appears above the button that says “Press for Nitro!”. If it’s the user’s first game or no input is received for a while, “tutorial” text appears above the joystick showing the user how to make the car move.

You would like to add keyboard and bluetooth game controller support. Where do you begin?

Input events

Here is some example code of how to detect input events from various kinds of input devices using the standard Android APIs. Use these events to drive your state machine, like above. You should adapt the general concept to the types of input events you expect and to your app or game.

// Drive the state machine based on the received input type
// onKeyDown drives the state machine, but does not trigger game actions
// Both keyboard and game controller events come through as key events
override fun onKeyDown(keyCode: Int, event: KeyEvent?): Boolean {
    if (event != null) {
        // Check input source
        val outputMessage = when (event.source) {
            SOURCE_KEYBOARD -> {
                MyStateMachine.KeyboardEventReceived()
                "Keyboard event"
            }
            SOURCE_GAMEPAD -> {
                MyStateMachine.ControllerEventReceived()
                "Game controller event"
            }
            else -> "Other key event: ${event.source}"
        }
        // Do something based on source type
        findViewById<TextView>(R.id.text_message).text = outputMessage
    }
    // Pass event up to system
    return super.onKeyDown(keyCode, event)
}

// Trigger game events based on key release
// Both keyboard and game controller events come through as key events
override fun onKeyUp(keyCode: Int, event: KeyEvent?): Boolean {
   when(keyCode) {
       KeyEvent.KEYCODE_N -> {
           MyStateMachine.keyboardEventReceived()
           engageNitro()
           return true // event handled here, return true
       }
   }
   // If event not handled, pass up to system
   return super.onKeyUp(keyCode, event)
}

// Touch and stylus events come through as touch events
override fun onTouchEvent(event: MotionEvent?): Boolean {
   if (event != null) {
       // Get tool type
       val pointerIndex = event.action and ACTION_POINTER_INDEX_MASK shr ACTION_POINTER_INDEX_SHIFT
       val toolType = event.getToolType(pointerIndex)

       // Check tool type
       val outputMessage = when (toolType) {
           TOOL_TYPE_FINGER -> {
               MyStateMachine.TouchEventReceived()
               "Touch event"
           }
           TOOL_TYPE_STYLUS -> {
                MyStateMachine.StylusEventReceived()
               "Stylus event"
           }
           else -> "Other touch event: ${toolType}"
       }

       // Do something based on tool type, return true if event handled
       findViewById<TextView>(R.id.text_message).text = outputMessage
   }
   // If event not handled, pass up to system
   return super.onGenericMotionEvent(event)
}

// Mouse and joystick events come through as generic events
override fun onGenericMotionEvent(event: MotionEvent?): Boolean {
   if (event != null) {
       // Check input source
       val outputMessage = when (event.source) {
           SOURCE_JOYSTICK -> {
                MyStateMachine.ControllerEventReceived()
               "Controller event"
           }
           SOURCE_MOUSE -> {
                MyStateMachine.MouseEventReceived()
               "Mouse event"
           }
           else -> "Other generic event: ${event.source}"
       }
       // Do something based on source type, return true if event handled
       findViewById<TextView>(R.id.text_message).text = outputMessage
   }
   // If event not handled, pass up to system
   return super.onGenericMotionEvent(event)
}