What was missing
Until now, Golem gave Claude eyes and arms on another machine, but the arms did not reach the screen. Claude could run PowerShell, move files, manage services and take screenshots, and it still got stuck in front of a window that asked for three fields.
At the end of September, a technician at a software company was installing a product on a customer’s server with Golem. One of the installer’s steps was a screen with a few fields to fill in, and Claude said it could not write on it. It was right. It saw the screen through the screenshot, but it had no tool to touch it, and that is what Golem 1.5 brings.
Two different things called “agent”
These days the word “agent” almost always means an AI agent, and two pieces with that name show up in this article. It is worth telling them apart before going on.
- The AI agent is Claude, the model I talk to from the Claude app or from Claude Code. It understands what I ask, decides what to do and calls the tools it has at hand.
- The Golem agent is a program,
golem.exe, that runs on the other machine and has no intelligence of its own. It receives Claude’s calls, carries them out there (a command, a file copy, writing into a text box) and sends back the result. It is the piece that makes the tools in this article available to Claude.
Between the two sits the Golem client, on my computer, which connects Claude to the Golem agent on the other machine. From here on, “the Golem agent” always means the program, and I call Claude by its name.
Eight new tools
| Tool | What it does |
|---|---|
ui_windows | Lists the open windows: title, process, size |
ui_controls | Reads the controls of a window: buttons, text boxes, check boxes, lists |
ui_screenshot | The image of one window alone, even if another one covers it |
ui_set_text | Writes into a text box |
ui_click | Presses a button, ticks a check box or picks an item of a list |
ui_keys | Brings the window to the front and types: {ENTER}, {TAB}, ^s |
ui_focus | Brings a window to the front, restoring it if it is minimized |
ui_wait | Waits for a window to appear, or to close |
With this, the installer case goes like this: Claude starts the installer, waits for its window, reads its controls, presses “Next” until the page with the fields, fills them in and carries on. If the installer ends with a “Finish” window, it closes it.
If the installer asks for administrator rights, the Golem agent needs them too: with golem serve, start it as administrator; with Golem Assist, open it as administrator. Otherwise Golem detects it and answers that the window is out of its reach, without typing anything. Under the Guardian those windows are always out of reach.
By the name of the control, not by coordinates
The easy way would have been to move the mouse to coordinates taken from the screenshot and click. I did not want that. A click by coordinates lands wherever it lands: if the window moved, if another one came to the front or if the screen has another scale, it presses something else and nobody notices.
The tools talk to applications through UI Automation, the same interface Windows screen readers use. Each control is picked by its identifier, by its name or by its position in the window’s tree, and the text is written into the control without going through the keyboard. Only when an application exposes nothing do the keys or a mouse click come in, and the answer says which way was used.
Before typing or clicking, Golem checks that the window came to the front. If the person at the computer is working in another window and Windows does not let the focus change, nothing is sent and the answer says so. A key in the wrong window does more harm than none.
On a customer’s computer, with Golem Assist
Golem Assist is the window a customer opens when they need help: it gives them a twelve-digit number, they read it to me and I get in. It starts the Golem agent in the customer’s own session, with their desktop, so the hands work from the start. Now I can ask Claude to fill in the form the customer has in front of them, and the customer sees it being typed on their screen.
For that the customer needs version 1.5 of Golem Assist. The tools come from the Golem agent that runs on their computer, and that program is inside GolemAssist.exe.
On a server with nobody in front of it, with the Guardian
The Guardian runs as a Windows service, and services live in a session with no desktop. To get the hands to the screen, the Guardian starts a small windowless helper in the session of whoever is signed in to the server, and hands it the desktop tools and the screenshots. If nobody is signed in, the tools say so instead of failing in a strange way. The helper closes with the Golem session.
The helper runs with the normal rights of that session’s user. A window opened as administrator takes no keys from a process that is not elevated, and Golem detects it and says so.
Passwords do not stay in the log
Golem writes down every call in golem-audit.log, next to the executable, so the owner of the machine knows what was done on it. When Claude types a password, it does it with secret=true, and then neither the answer nor the log carry the text. The password does go through the conversation with Claude, which needs to know it to type it; if you do not want that, type it yourself.
With an ERP in Visual FoxPro
What UI Automation sees inside a Visual FoxPro application depends on the version and on the screen. In the certification of this version, the login screen of an ERP made in VFP Advanced 10.2 showed 73 controls with their names and values, and the clicks landed in place. With other versions and other screens much less may be visible.
Even so, to work with an ERP in VFP the way I recommend is FoxAgent as a child server of Golem. It drives the screens from inside the application, with its forms and its data, and it does not depend on what the application exposes to the outside.
In look-only mode
With --read-only, the look-only mode, since 1.5.1 the three that only read the screen are published (ui_windows, ui_controls and ui_screenshot), just like screenshot. The five that write, click, focus or wait stay out.
Tested
Before publishing it I ran the usual tests and the new ones: the Golem agent writing and clicking in a test window and in the Windows 11 Notepad, the Guardian installed with its helper, an elevated window that refuses to take keys and no helper left alive after removing the Guardian. The current version, 1.5.3, is signed, and the hashes of every file are in the installation guide.