Gaming integration: how it works and what it never does

From mnema-talk-desktop v1.4.0, rendered on this site.

The gaming integration (sidepeek, overlay, global shortcuts, push-to-talk) is Beta and Windows-first. This page is the reference for the desktop client. The user-facing setup guide is in the main repository (desktop-app.md).

Anti-cheat safety statement

Online games run anti-cheat software, and a chat client must never look like a cheat. This is what the client does, and what it will never do.

It does:

  • Ask the operating system to tell it when the focused application changes (a documented, passive notification), and look up the image path of that one process.
  • Read read-only launcher manifests on disk (for example the list of installed Steam games) to learn which folders are game installs.
  • Draw its own windows (the sidepeek and the overlay) as ordinary topmost windows of its own process.

It never:

  • Enumerates processes or windows to find games. Detection starts from the focus-change notification and touches only the process that is in front.
  • Injects code or a DLL into any other process, hooks another process, reads or writes another process's memory, or opens a handle with more access than "query limited information" to ask for its image path.
  • Installs a driver or a kernel component.
  • Hooks a game's graphics API. There is no in-game rendering; the overlay is a separate window, which is why it cannot draw over exclusive fullscreen.
  • Sends synthetic input to a game or reads the game's keyboard or mouse state beyond the push-to-talk and shortcut keys the user configured.
  • Sends the image path, window title or process name of what you run to any server. A matched game name stays on the device.

Game detection

Detection is event driven. There is no timer that looks at the foreground window.

SystemSourceLooks up
WindowsSetWinEventHook(EVENT_SYSTEM_FOREGROUND, WINEVENT_OUTOFCONTEXT) on a dedicated thread with its own message loop (out of context: nothing is loaded into other processes)QueryFullProcessImageNameW on the foreground window's single PID, opened with PROCESS_QUERY_LIMITED_INFORMATION
macOS, Linuxnone: the gaming integration is Windows only. Detection is off and the settings say so.-

The foreground source sits behind the ForegroundSource trait (src/foreground.rs) with a fake for tests. The operating-system code is in src/foreground_windows.rs.

A window counts as a game only when it was announced by the source and is still the foreground window when the sidepeek or shortcuts act, so an announcement that has been overtaken by a newer focus change never shows a game.

Which game is it?

Three sources, checked in this order. The first match wins and decides the name.

  1. Games you added in the gaming settings: type an executable file name such as MyGame.exe, or press "Add the app I was just in" and confirm the file name you are shown. That name is the last other application the foreground event announced before you opened Mnema (never Mnema itself, Java, or a helper such as a crash handler). Only the bare file name is handed to the settings page, only when you press the button; the path stays in the host and nothing leaves the device. The list is kept locally (up to 32 names) and each entry can be removed again.

  2. Launcher libraries. The install folders of the games your launchers report. A program that lives under one of those folders is that game, and the game is named as the launcher names it. Read-only, from files and registry keys the launchers keep for themselves:

    LauncherReadNotes
    Steamsteamapps/libraryfolders.vdf and steamapps/appmanifest_*.acf of every librarySteam tools and runtimes (Steamworks Shared, SteamVR, Proton, Linux runtimes, Wallpaper Engine) are skipped
    Epic Games Store%ProgramData%\Epic\EpicGamesLauncher\Data\Manifests\*.itemadd-ons and unfinished installs are skipped
    GOG GalaxyHKLM\SOFTWARE\WOW6432Node\GOG.com\Games\* (gameName, path)
    EA appHKLM\SOFTWARE\WOW6432Node\EA Games\* and HKLM\SOFTWARE\EA Games\* (Install Dir)
    Battle.netnot readits install list is a binary database without a public format; Battle.net games are found by the bundled list or a name you add

    Every path read from a manifest or the registry is checked before the disk is touched: only absolute local drive paths are opened. UNC (\\host\share), device (\\?\, \\.\) and relative paths, .. segments and alternate data streams are skipped silently, so a crafted manifest cannot make Windows contact another machine. A manifest folder that is a drive root, a system folder or a broad folder such as Program Files itself is ignored, so one bad manifest cannot make every program a game. Manifests larger than 256 KiB are ignored.

  3. The bundled list (src/game-list.json): executable file names of widely played PC games. Matching is by file name only.

The launcher scan runs on its own thread: once at start, every 15 minutes, and whenever the gaming settings are saved. The foreground path only reads the last finished result. Java is never matched by name: javaw.exe counts as Minecraft only when its window title says so, and a Java program inside a launcher's game folder is that game. Crash handlers, uninstallers and redistributables inside a game folder are not the game.

The bundled list

src/game-list.json is maintained by hand. It holds about 600 well-known PC games from the last quarter century (shooters, battle royales, MOBAs, MMOs, survival, sports, racing, simulators, strategy and indie hits): a display name, one or more executable file names and, for some, a Steam app id kept for reference. It is deliberately not complete and has not been checked against installed copies. The launcher libraries (above) cover the rest: any game your Steam, Epic, GOG or EA app reports is found there without being in the list, and you can add a name yourself in the gaming settings. A missing or wrong name only means that game is found through its launcher or a name you add instead. A unit test keeps names, executables and ids unique and well formed. Corrections and verified names are welcome under the repository licence.

Sources

The list is our own work, written from public knowledge of the executable names games ship. An earlier catalog that came from such a list was deleted; the Git history still contains that old file (names and executable names only), and it is kept as is because rewriting public history would break signed release tags and forks. No file in the current tree carries data derived from it. The header of src/game-list.json states the provenance.

Push-to-talk and shortcuts

Keys are event driven. Nothing asks "is this key down?" in a loop.

SystemSourceNotes
WindowsRaw Input (RegisterRawInputDevices with RIDEV_INPUTSINK, keyboard and mouse) on its own thread with a hidden windowNo keyboard hook, no GetAsyncKeyState. Works while a game has the focus. A session lock or unlock, a console or remote disconnect, a logoff and a suspend or resume clear all keys, so a key cannot stay stuck down when its key-up happens where Raw Input cannot see it.
macOS, LinuxnoneThe gaming integration is Windows only.

What is read: for each change only the key (or the mouse side button 4 or 5) and whether it went down or up. Text is never looked at, no key history is kept and nothing about keystrokes is logged or sent. The host keeps the set of keys currently down and compares it with your four chords (overlay, mute, deafen, push-to-talk); only a change in whether a chord is down wakes the rest of the app. A press that begins and ends between two looks is still delivered once.

The UI thread never waits for the system or for a game. The input thread, the foreground-window thread and the library scan are threads of their own; the one shell question that can take seconds while a game starts (is something in exclusive fullscreen?) is asked on a worker thread, spaced two seconds apart, with at most one in flight; the toast follows on that thread. A window of another program is only ever asked with SendMessageTimeout and SMTO_ABORTIFHUNG, so a stuck game costs the timeout, not a hang. A watchdog thread asks the UI thread to answer a message every 100 ms; when it does not within 200 ms it writes what the UI thread was doing to logs/desktop.log in the data folder (mnema-data\data\logs beside the program for the portable copy, %LOCALAPPDATA%\com.mnema.desktop.shareddev\logs for the installed one). The Windows CI keeps a full-screen stand-in game in front while sending input and requires the host's windows to keep answering.

The source runs only while you are in a Talk with gaming on, and stops when you leave. On Windows the host asks the windowing library to release the process's own raw input registration first, because Windows allows one registration per process; if something else still holds it, the settings say so and push-to-talk works while Mnema is in front.

Push-to-talk and the other shortcuts act only while a detected game is in the foreground, as before. The web page accepts a ptt-press for 150 ms and drops the press 300 ms after the last one (a safeguard against a stuck microphone), so while the key is held the host repeats the press every 100 ms. That is a keepalive of a state known from the key event, not input polling. Leaving the game, leaving the Talk, a session lock or a stalled host releases push-to-talk.

Known limits

  • Raw Input does not see the secure desktop (UAC prompts, Ctrl+Alt+Del). A key released there is not reported; the session-lock reset and the page's 300 ms safeguard bound the damage.
  • Games that read the keyboard exclusively (some old DirectInput exclusive modes) do not change what Raw Input delivers, but we cannot test every title.
  • Not verified on real hardware or against any anti-cheat; see the safety statement above for what the client does and never does.

Games in exclusive fullscreen

The sidepeek and the overlay are ordinary topmost windows, so they cannot be drawn over a game that owns the display in exclusive fullscreen. Borderless windowed ("windowed fullscreen") works. The client does not try to get around this: no graphics hook, no injection.

While a detected game is in front and the hint has never been given, the host asks the shell one cheap question per beat, SHQueryUserNotificationState (Windows). QUNS_RUNNING_D3D_FULL_SCREEN means exclusive fullscreen; every other answer, including a borderless full-screen window, can be drawn over. After two looks in a row the host shows one toast (it waits in the notification centre while the game owns the display) that suggests borderless windowed mode and says that push-to-talk and shortcuts keep working. It is given once per installation (gaming-hints.json in the app's config folder) and never repeated. The same hint is in the gaming snapshot as exclusive_fullscreen_hint for the settings and the surfaces. Nothing about it leaves the device.

Something wrong on this page? Tell us.