Keyboards and mice are not universal. For millions of people living with motor disabilities, repetitive strain injuries, low vision, or conditions that make sustained physical interaction with a computer painful or impossible, the standard desktop experience can be challenging. Anthony is a voice-driven desktop orchestrator for Linux that lets you control your entire GNOME desktop by speaking using natural language. It uses AI for processing, but everything runs locally on your own machine so your data remains private.
Assistive technology for everyone
Assistive technology has come a long way. Screen readers (like Orca, on Linux) do critical work for blind and low-vision users. Sticky keys and on-screen keyboards help people with limited dexterity.
Pressing keys for keyboard shortcuts, navigating menus, clicking small targets, dragging windows are all actions that many people perform hundreds of times a day without ever thinking about it. For someone with cerebral palsy, advanced multiple sclerosis, ALS, a spinal cord injury, or even severe RSI from years of desk work, however, each one of those actions can range from difficult to impossible.
Voice control is an ideal solution. Nearly two decades after the industry's early attempts in 2008, the necessary technology has matured. We now possess the full stack: Reliable speech-to-text, embedding models for retrieval-augmented generation (RAG), and advanced large language models capable of reasoning, vision, and tool calling. Combined with neurally enhanced text-to-speech, a seamless voice experience is finally possible. But on Linux so far, the few Linux voice tools that exist tend to be limited dictation engines or proof-of-concept projects with a handful of hard-coded commands. None of them offer what users with disabilities actually need: Comprehensive, hands-free control over the full desktop experience.
Anthony was built to bridge that gap.
How Anthony works
Anthony listens continuously through your microphone using voice activity detection (VAD). There is no wake word to remember, no button to press. You simply speak, and the system acts. For users who can't reliably press a physical key to activate listening, this continuous mode is essential, because it transforms the computer from something you operate into something you talk to.
For users who prefer explicit control, a push-to-talk mode is also available. Flexibility matters, because disability is not monolithic. What works for one person may not work for another.
When you speak to Anthony, a local speech recognition model transcribes your words. A routing layer interprets your intent and then dispatches the appropriate action. Common commands are handled instantly (under 100 milliseconds) through pattern matching. More complex or open-ended requests flow through a local language model that determines the most likely intent, and then composes multi-step actions to accomplish it. Everything responds through audio, with voice feedback synthesized by a natural-sounding neural voice engine, so you never need to look at the screen to confirm that the command was understood.
Critically, none of the data of this interaction leaves your computer. Every model in the process runs locally, including speech recognition, language understanding, and text-to-speech. For users in medical or care settings, where privacy around health-related conversations is paramount, this is a requirement. For everyone else, it's exactly the kind of data privacy you'd expect from Linux.
What a hands-free desktop looks like
Imagine the interactions with a computer for someone controlling it entirely by voice.
"Open Firefox." The browser launches. "Go to my email." Anthony types the URL and navigates. No mouse needed.
"Read what's on the screen." Anthony takes a screenshot, passes it through a vision model, and describes the visible content aloud — text, layout, images, UI elements. For a low-vision user, this is a second pair of eyes. For a blind user working alongside a screen reader, it provides complementary spatial context that linear screen reading misses.
"Tile Firefox on the left. Tile the terminal on the right." Both windows snap into position. Window management — one of the most mouse-dependent tasks on any desktop — becomes a sentence.
"Turn on dark mode." Done. "Set brightness to thirty percent." Done. "Enable ‘do not disturb'." Done. System settings that normally require navigating through multiple menus and clicking small toggles become single voice commands.
"Type: Dear Dr. Chen, I am writing to follow up on our last appointment." Anthony types the text into whatever application is focused. For someone who can speak clearly but cannot type — a common profile for users with motor impairments in their hands and arms — this turns voice into a direct text input channel, not just a command interface.
"Close the text editor." Here, Anthony does something considerate. If the editor has unsaved work, it detects the save or discard dialog using the AT-SPI accessibility framework — the same framework that powers screen readers — and reads the available options aloud. "There is a dialog. The options are: Save, Discard, and Cancel." You say "Save," and Anthony activates the button. No unsaved work is lost because of an imprecise command. This kind of safety-aware interaction is not a convenience feature for a user who can't just undo a mistake with a keyboard shortcut, it's protection against real data loss.
Natural language instead of memorization
One of the most significant accessibility benefits of Anthony is that it doesn't require users to learn a command vocabulary. Traditional voice control systems ask you to memorize exact phrases such as "window maximize", "press control-shift-T", and "mouse click 400 200". These systems are tools for power users who happen to prefer voice. They are not accessible.
Anthony understands natural language. "Make the browser bigger", "full-screen Firefox", "maximize that window." All of these work. You describe what you want in your own words, and the system figures out the rest. This dramatically lowers the cognitive load of voice control, which matters especially for users with cognitive disabilities, acquired brain injuries, or anyone who simply shouldn't have to think about syntax when they're trying to get work done.
The system recognizes applications by common names, not window titles or process identifiers. "Close the text editor" works whether the window title says "Untitled Document - gedit" or "notes.txt - GNOME Text Editor". You think in terms of intent, and Anthony translates.
Conversational assistant
Beyond desktop control, Anthony can function as a general-purpose conversational assistant. Say "switch to chat mode" and you can ask questions, request explanations, or have multi-turn discussions — all by voice, all locally processed. Say "switch to command mode" and you are back to controlling your desktop.
For users who spend a lot of time at their computer and experience fatigue from physical interaction, having a single voice interface that handles both information needs and productivity tasks means fewer mode switches, fewer tools to manage, and less overall strain.
The trade-offs
Anthony requires capable hardware, including sufficient RAM, a Fedora-based GNOME installation, and ideally a discrete GPU. It is not yet a complete solution for every Linux user. It does not replace screen readers or other established assistive tools. It is best understood as a complementary layer: A voice-first control surface that removes the requirement of physical interaction for desktop computing.
It is also a work in progress. Accessibility is an ongoing commitment to listening to the people who need the tool most and adapting to how they actually use it.
Conclusion
Your computer should work for you, not the other way around.
The desktop computer remains one of the most powerful tools a person can use for communication, creativity, work, and independence. But that power is restricted by an interaction model built around two hands on a keyboard and one hand on a mouse. For a user whose body or situation does not fit that model, a computer's full potential stays locked behind a physical barrier.
Anthony aims to remove that barrier, not by uploading your private data to the cloud or forcing you to memorize commands, but by letting you speak naturally to your own computer, running entirely on your own hardware.
