Support custom key binds (with keyboard and touchscreen support)

[FatalistError]

(updated)

Problem

It's extremely frustrating to create a control scheme that allows the player to do a number of different things when we are limited to only a few keys for controls. There's an express need for an additional expansion to the PC and desktop control scheme and possibly mobile.

currently, we're limited to about 3 different keys (if you're generous) for non-movement (or other hard-coded action) controls. This is frustrating as if you want to make a special menu, a action key (like a special ability), or a movement key (like crouch, prone), this is very limiting, and especially considering shift and zoom already having an action. While you can be creative, this basically still limits you to one key for any given action, and since most servers use aux for sprint... you basically have nothing.

Solutions

PC/DESKTOP

I propose two possible solutions (for PC)

Solution A: allow games to take matters into their own hands.
Allow mods to see the control binds of players, and see (almost) all pressed keys

this would be used to get key binds, this is so mods can plan around existing player key binds, and even create their own binding system
minetest.get_player_key_binds(player) = {forward = "w", left = "a" . . . etc}

this would be a list of active keys, if a key is active it should not be on the list. This list would exclude some keys, like windows button (and linux equivalent), the function keys, or anything else sensitive or not needed.
minetest.get_player_held_controls(player) = {w = true, a = true . . . etc}

these functions could alternatively be methods for the player object.

Best/adopted solution (for PC)

Solution B: allow game/world key binds
This would allow games to create their own custom key bind settings for each player, and allow the client to interface with it.

on joining, the player would receive a list of key binds, this would be directly interfaceable with the "change keys" button, this should be modifiable by the server in post. The client would store these key binds by server in a human readable file format. This is especially important as to allow players to copy across versions etc, these files also should never delete themselves.

mostly self explanitory:
the world keybinds would function as a sort of default
minetest.set_world_keybinds({keyname="key" . . . etc})
minetest.add_world_keybind(keyname, key)
minetest.on_player_change_keybinds(function(player, key, value))

(edit): a useful system to implement would also be the ability to send a request to get player keybinds...
this would allow the server to store the keybinds, and to replace current local keybinds for any reason (use case being if the client reinstalled, they wouldn't be screwed over.)

minetest.get_keybinds(playername)
minetest.send_keybinds(binds, playername) --returns true if accepted/unsuccessful, else false.

(during chat, or when a formspec is open, these would be ignored)

MOBILE

I propose one solution for mobile, this would be the addition of "buttons" to the HUD definition.
It's fairly simple actually:
the size of the button would be the same size as the image or text, just a box. The change would include this addition to the HUD definition:
button = function(player)
button_image = "image.png"
"button" would be called while the player presses down on it, when it's being pressed down upon, the image would change to button_image (if present). HOWEVER this system would require atleast 1 other PR(s)/issues to be put in place:
see relevant PRs:

#12264 user input mechanisms (for detecting touchpad) <---- most important, could be essential
#12367 window/screen size information (for better scaling)

If the inclusion of PR 12264 can not be made, HUD images with the button function should not be displayed on devices without capabilities to use them.

additional

Mobile (could) be delayed for obvious reasons
The engine does not currently have the framework to allow for the support of mobile in regards to controls... but this doesn't mean we should limit ourselves and not make improvements where we can.
We should improve PC control inputs and settings, and upgrade mobile options later, it's silly to limit ourselves to "X platform can't do Y so we should leave out this entirely basic feature for Z until we rework the entire engine".
Besides, games/mods that require additional controls are likely to hard to play on mobile, though I agree that's not our choice to make- it's the truth, and ignoring it is a massive oversight.
to reiterate
Of course this doesn't mean we shouldn`t add support for mobile, but why do we need to ruin the fun for PC players who want to make something special.

Add any other context or screenshots about the feature request here.

(once again) relevant PRs

#12264 user input mechanisms (for detecting touchpad) <---- most important, could be essential
#12367 window/screen size information (for better scaling)

[appgurueu]

On minetest.get_player_held_controls(player): It's apparently pretty hard to ensure that key names are consistent across platforms. I could still see this being useful if games / servers provided their own "configure your keybinds" dialogs though - when the user is forced to configure their keybinds, it's irrelevant which names are used as long as they are unique. Only "default" keybinds might fail; and for trivially named keys (e.g. ASCII letters) ensuring consistent key names shouldn't be too hard.

That said, this would have to be fleshed out: What if the player is typing something into chat - should it still send keybinds? What about formspecs (relevant: #8679)?

[FatalistError]

I think that for typing into chat, or any menu/formspec stuff it shouldn't, because there's currently no way to tell...

[FatalistError]

if there was a way to tell, then yes, I'd say that it'd be a good idea.

[Montandalar]

For mobile, perhaps gesture support could be helpful?

[FatalistError]

For mobile, perhaps gesture support could be helpful?

This is entirely unnecessary
Making a system for this would be complicated, and it serves no relevant or valid purpose.

[Montandalar]

A lazy post gets a lazy response. But you may have misunderstood what I mean by gestures. Gestures are the only way to interact with a phone that doesn't have external input devices like controllers or keyboards. Gestures include: tap, double tap, long tap, drag, pinch, pinch and rotate, swipe & multi-finger swipe, shake and others.

Double tap, and long tap are already bound to the placing and breaking actions, and dragging is bound to camera controls. A hot zone is used to emulate a kind of D-pad for movement. The mobile native way to enable extra actions is to use gestures. I would say at least let mod authors bind actions to multi finger swipe (left/right/up/down perhaps), pinch and shake.

Slightly off topic but here's an even more radical idea - maybe it's already possible? For vehicle controls on mobile, use the dpad for steering only and let the sneak/jump buttons be the brake and accelerator. Driving a mesecar or advtrains locomotive is pretty unintuitive with the dpad. If I try to turn too hard I brake, if I accidentally swipe a little bit left the train doors open...

My main point here is any decision we make about the engine can't simply ignore mobile. Yours does, and I don't think it holds merit for that reason. If I add the doodad key to PC controls so my mod can frobniscate my wim-wams, why am I going to have to explain to mobile users that "oh yeah, I know frobnostication would perfectly make sense as a shake gesture, for now it's just a button in that menu full of other buttons that do stuff"?

You'll have counterarguments, that's OK. But please don't disregard what I have to say out of hand again.

[FatalistError]

A lazy post gets a lazy response. But you may have misunderstood what I mean by gestures. Gestures are the only way to interact with a phone that doesn't have external input devices like controllers or keyboards. Gestures include: tap, double tap, long tap, drag, pinch, pinch and rotate, swipe & multi-finger swipe, shake and others.

Double tap, and long tap are already bound to the placing and breaking actions, and dragging is bound to camera controls. A hot zone is used to emulate a kind of D-pad for movement. The mobile native way to enable extra actions is to use gestures. I would say at least let mod authors bind actions to multi finger swipe (left/right/up/down perhaps), pinch and shake.

Slightly off topic but here's an even more radical idea - maybe it's already possible? For vehicle controls on mobile, use the dpad for steering only and let the sneak/jump buttons be the brake and accelerator. Driving a mesecar or advtrains locomotive is pretty unintuitive with the dpad. If I try to turn too hard I brake, if I accidentally swipe a little bit left the train doors open...

My main point here is any decision we make about the engine can't simply ignore mobile. Yours does, and I don't think it holds merit for that reason. If I add the doodad key to PC controls so my mod can frobniscate my wim-wams, why am I going to have to explain to mobile users that "oh yeah, I know frobnostication would perfectly make sense as a shake gesture, for now it's just a button in that menu full of other buttons that do stuff"?

You'll have counterarguments, that's OK. But please don't disregard what I have to say out of hand again.

It by no means ignores mobile, it implements the equivalent system, there's no possibility to have the solution work the same for mobile, because they're different platforms with different input methods. Additionally, you still have not explained the use cases for gestures outside of vehicles, but even that use doesn't make sense. Either way, this isn't the point.

The goal is to establish basic controls, I'm not saying it's necessarily a bad idea to add support for gestures, but it's something that should be at a later date, we all know that if there's a overly complicated suggestion or feature request, it'll take forever to get done or just get closed. It's more important to actually implement the BASICS. Also, a lot of the time, feature packed games like this that cannot practically use a menu for controls like this, aren't easy or practical for a mobile player to use anyway; this is why this issue is designed to primarily focus on the PC side of things, but still provides a viable equivalent solution to mobile.

If I add the doodad key to PC controls so my mod can frobniscate my wim-wams, why am I going to have to explain to mobile users that "oh yeah, I know frobnostication would perfectly make sense as a shake gesture, for now it's just a button in that menu full of other buttons that do stuff"?

nobody said anything about a menu, if it was about menus we already HAVE menus, it's called FORMSPEC. The HUD buttons would allow you to create a button anywhere on the screen that modders could use for controls. Whatever a "shake gesture" could do (which is a feature even most dedicated mobile games don't use) could be easily accomplished, and even possibly more practical as a button.

TL:DR:
it's not practical, logical, or frankly even viable to add gestures at this time.

[Montandalar]

Okay, let's say we can ignore gestures at the moment. What makes interactive tappable HUDs really a mobile-only feature? Plenty of desktop users want those too. Having formspecs be completely modal sucks, you can't see what's going on in the world behind the formspec.

Many games let you release the mouse from camera controls and interact with smaller UI elements. For instance in Garry's Mod, the spawn menu does not disable movement controls, but it redirects the mouse from camera controls to a cursor; the leaderboard can also be used with the mouse to mute players in voice chat. In Dark Souls, you can navigate the menus while continuing to move your character; some keys are bound to the menu, while the WASD movement keys continue to be useable, and the mouse can be used as a cursor and to click and scroll. In Mount & Blade Warband, you can open the tactics menu with Backspace, and this lets you click on various troop types, orders, and on the minimap to hold positions.

Let mods add non-modal formspecs. Mobile users will be able to tap elements on them at any time, and PC users can push a key to release the mouse to the UI instead of camera controls. Now you don't even need touchscreen detection - but i you combine this with touchscreen detection, you can add touchscreen-specific controls. Thus, in an indirect way, we have what you asked for: buttons that can be used for controls on touchscreens.

There, we now have three separable items:

1. Bindings for keyboard (and other input devices like controllers) - suggested by Adding custom keystrokes through Lua #2731 and in issues linked from there - such as Control Scheme in Android #8541.
   In fact the second suggestion in 8541 mentioned buttons just like you did - I still think non-modal HUD is the 'right implementation' of that basic idea, which brings me to..
2. Formspec/HUD hybrid that is non-modal - suggested in Issues and suggestions regarding HUD elements #10548
3. Touchscreen gesture bindings - not suggested yet from a quick search.

No priority on any of those, not my call. Any objection to those ideas? Would having 1 & 2 satisfy your feature request here? Then my suggestion is the separate 3rd item.

[FatalistError]

Okay, let's say we can ignore gestures at the moment. What makes interactive tappable HUDs really a mobile-only feature? Plenty of desktop users want those too. Having formspecs be completely modal sucks, you can't see what's going on in the world behind the formspec.

Many games let you release the mouse from camera controls and interact with smaller UI elements. For instance in Garry's Mod, the spawn menu does not disable movement controls, but it redirects the mouse from camera controls to a cursor; the leaderboard can also be used with the mouse to mute players in voice chat. In Dark Souls, you can navigate the menus while continuing to move your character; some keys are bound to the menu, while the WASD movement keys continue to be useable, and the mouse can be used as a cursor and to click and scroll. In Mount & Blade Warband, you can open the tactics menu with Backspace, and this lets you click on various troop types, orders, and on the minimap to hold positions.

Let mods add non-modal formspecs. Mobile users will be able to tap elements on them at any time, and PC users can push a key to release the mouse to the UI instead of camera controls. Now you don't even need touchscreen detection - but i you combine this with touchscreen detection, you can add touchscreen-specific controls. Thus, in an indirect way, we have what you asked for: buttons that can be used for controls on touchscreens.

There, we now have three separable items:

1. Bindings for keyboard (and other input devices like controllers) - suggested by Adding custom keystrokes through Lua #2731 and in issues linked from there - such as Control Scheme in Android #8541.
   In fact the second suggestion in 8541 mentioned buttons just like you did - I still think non-modal HUD is the 'right implementation' of that basic idea, which brings me to..
2. Formspec/HUD hybrid that is non-modal - suggested in Issues and suggestions regarding HUD elements #10548
3. Touchscreen gesture bindings - not suggested yet from a quick search.

No priority on any of those, not my call. Any objection to those ideas? Would having 1 & 2 satisfy your feature request here? Then my suggestion is the separate 3rd item.

Making it a formspec, doesn't make sense at all, and you wanna hear the issue with this one chief? say you want to have a menu button, let's say.... X if you want mobile users to be able to use that, you have to have a button on the screen, that's not well scaled for PC on their screen? so now you both have a button on the screen, and a button that PC users don't want on their screen for no reason. And what if you have a gun mod, and you want to put a fire button for mobile people, now there's this annoying and useless button on the screen.

Not to mention, I already proposed a solution for client detection not being implemented, it's very simple:

If the inclusion of PR 12264 can not be made, HUD images with the button function should not be displayed on devices without capabilities to use them.

it would be done entirely client side.

additionally, using the proposed methods you could create a button that opens a formspec, which is equivocal to what you suggested for a button "PC users can push a key to release the mouse to the UI instead of camera controls".

you're finding problems that don't exist, and arguing for the sake of arguing. I'm not going to entertain your shit anymore as thus far, everything you've said has been redundant. I'm not going to turn this whole comment section into a pissing competition.

[rubenwardy]

• Adding custom keystrokes through Lua #2731

I knew this issue was a duplicate, but couldn't find the original issue

[FatalistError]

I appreciate that man

[Montandalar]

You didn't explain why making a formspec made no sense. It's basic code reuse in my opinion. Of course the formspec needs to be scaled, and of course it should only be shown to mobile users. You need to learn at least some agreeableness - you currently have displayed 0 tolerance for any other ideas. You can be as right as you want and still rude. Thank you for making sure I won't continue to contribute to this issue thread, because I don't want to deal with someone who isn't going to change their mind at all. I hope whoever next comments in this thread has a better time than me. You may as well start writing the PR yourself because if anybody disagrees with you about how to implement this and you do the same to me as to them, it won't turn out just the way you want it, and then you'll be mad your ideas were ignored or something.

[rubenwardy]

I don't really have time to read all of this right now, but I'd appreciate if this issue could get a little less heated as it doesn't seem healthy

[herbert-west2]

I support this vital feature

[kneekoo]

Better input handing is very important for many use cases. And while not everything can be done the same way between platforms (PC vs mobile), one should not hold the other "hostage" towards offering a better experience. Ideally, each platform would be able to evolve separately, so that both platforms can evolve at the same time or at different paces without merge conflicts.

[appgurueu]

We have #14456 now, which should lay a good foundation for adding more customizable touchscreen controls. I'm thinking this could let us design an API roughly like this:

core.custom_control.register("mycontrol", {
    description = "Do thing",
    touch = {
        pos = {x = 0 ... 1, y = 0 ... 1}, -- default
        size = {x = 0 ... 1, y = 0 ... 1}, -- default
    },
    key = "a" ... "z" or "0" ... "9",
})

paired with something like a player:get_custom_control("mycontrol") or player:get_custom_controls().

Additionally, there should be a core.custom_control.enable / core.custom_control.disable to show / hide these controls: For example a "reload" control should only be usable when a weapon is equipped and otherwise not clutter the precious mobile screen space.

On mobile, this would just add the touch controls, and make them accessible to the touch control editor so they can be rearranged (unconditionally, even if they are presently disabled).

On desktop it would add the control to the key change dialog and automatically configure it, unless there is a collision with an existing keybind or no corresponding scancode can be found for the given keycode, in which case it would remain unconfigured.

In both cases, the configured keybind / control position should be remembered.

Not all keys have good, universal keycodes ("names"), but some basic ones like letters, digits, arrow keys should work pretty much universally (and our docs should make clear that modders can not expect anything fancier to work generally). This is important so that keys like "R" (for reload) can just work out of the box, without players having to configure them.
Worst case we can't find the scancode for a given keycode and the user has to configure it themselves.

[grorp]

There's an old API proposal by rubenwardy for this here: #11446 (comment)

[grorp]

For the touchscreen part:

In general, we need an API for games/mods to customize the touchscreen layout, not only to add custom buttons. Giving modders an API to place individual touchscreen buttons without giving them access to the predefined buttons means they'll have to rely on the default layout, which may change.

On the other hand, a small, scope-limited API like the one you proposed is still better than no API at all. We could add an API like this and deprecate it later in favor of a more comprehensive touchscreen controls API, or the more comprehensive API might never happen.

Now on your actual API proposal: You're missing texture and offset fields. It should also be possible to change the button's texture later (per-player)

See also
https://github.com/minetest/minetest/blob/ac7406c8a1013ea8bfc94c48dbc5dc97532c0cbd/src/gui/touchscreenlayout.h#L59-L65

[y5nw]

(Nitpicking:) For keyboard controls, there are IMO some considerations to be made (especially with #14940) regarding the implementation:

• An API for getting players' keyboard configurations should be able to get both the scancode and the keycode, as both have their usecases in terms of showing the control scheme to the user. (That said, this particular usecase should ideally be implemented with SSCSMs instead of server-sent UIs. Alternatively/Additionally, an escape sequence can be used as a placeholder for the key name.)

• Getting the (user facing) key name may require additional changes to the translation system depending on the implementation, as the key names are currently translated on the client-side using gettext instead of using the translation system provided by the engine. (A more general solution would be to move client UI strings to use the translation system provided by the engine, using its own textdomain, but this would be out of scope for a PR implementing custom keybindings and require updating a lot of translation strings.)

• There should be a way to specify alternative/secondary keybindings:

  • There will very likely be collisions when multiple mods define their own keybindings;
  • For symmetry (e.g. allow using Left or Right Control for an action); related: SDL: Use scancodes for keybindings #14964 (comment);
  • For different input devices (keyboard/joystick)

  An API for registering controls should be able to differentiate between the above cases.

For touchscreens controls, I recall reading somewhere (possibly from resolving merge conflicts) that touchscreen input internally looks for a corresponding keycode/scancode and (only if such a keybinding is present) sends the corresponding keypress event. I am not sure whether this remains the case, but this will need to be addressed as well if it is.