altengine

Published · altengine team

How to Automate a Windows Desktop App (and Let an AI Agent Write the Script)

Somewhere in most companies there is an application with no API, no export worth using, and no documentation — and a person who opens it every morning to retype yesterday's numbers. The work is mechanical. Automating it is not, because the only interface is a window. This post walks the whole path: enrol the machine, drive the window from a script, survive the dialogs, and then hand the genuinely hard part — working out what to click — to an AI agent that can look at the running application.

The problem is never the typing. It's the selectors.

Writing a script that clicks a button is easy. Knowing which button, in an application built in 2004 by someone who has left, is the part that burns a week. There is no reference to read. The usual process is a person with a UI spy tool, alt-tabbing between the app and the editor, guessing at a control name and running the script again.

That is the part worth changing, and it is the reason this feature and the MCP server are better together than apart. We'll build the script by hand first, because you need to know what the agent is writing.

Step 1: enrol a machine

An automation instance is a fleet. Inside it: agents (enrolled machines, tagged with labels), scripts (deployed, versioned bundles), runs (one execution, with a log, artifacts, a cost and an outcome), and schedules (cron, declared centrally, fired by the machine's own clock).

Mint an enrollment token in the console and download the installer it offers — the token is in the filename and the installer reads it back out. By hand, it's the supervisor's install:

altengine-supervisor.exe --install ^
  --instance fleet ^
  --token <enrollment-token>

The machine trades that token once for its own credential. One trap worth knowing up front: altengine-worker.exe enroll only saves a credential. A machine enrolled that way never connects, and shows in the console as having never connected — which looks like a network problem and isn't.

Labels are stamped on at enrollment — site-dallas, has-quickbooks — and they're how you target work without naming one computer.

Step 2: find a window, drive a control

A script finds windows through ui, and everything that waits waits for a fact — a window exists, an element became visible, the pixels stopped moving:

const win  = ui.findWindow("Invoice Ledger");                  // waits for it
const grid = win.find("automationId", "resultsGrid", 120000);  // and for this, for 2 minutes

win.focus();
win.sendKeys("^{f}");                  // Ctrl+F
win.typeText("2026-10-01");            // characters, not key events
grid.find("name", "Export").click();

find takes by as one of name, role, automationId or class. It waits — the fleet's action timeout, 30 seconds, unless you pass milliseconds.

Three distinctions that cause most early bugs:

  • find waits; findAll doesn't. The asymmetry is deliberate — find is looking for one thing that must be there, while findAll is also how a script asks is there anything?, and making that take thirty seconds to answer "no" would be worse. But on a grid that fills in after a query returns, findAll gives you whatever existed at that instant: nothing, or half a table, with no error anywhere. Pass a timeout when the screen is still loading.
  • typeText carries characters; sendKeys sends keys. An application watching for key presses will not see typeText at all.
  • A modifier applies to a brace group. "^{s}" is Ctrl+S and "^s" is refused. That's what keeps "+1" and "100%" ordinary text instead of keystrokes.

And never write a sleep. A sleep encodes a guess about a machine you are not sitting at: too short on the morning the ERP is slow, too long every other day. Note that wrapping a find in ui.waitFor does not extend it — find throws at the action timeout and waitFor re-throws immediately. Put the timeout on the find.

Step 3: the dialogs that lie to you

The best example of why desktop automation has a reputation: on Windows 11 the file name field in a Save dialog is a ComboBox with an Edit inside it, and the dialog reads neither. It tracks the name through real key events. So setControlText, setValue and typeText all change what's on screen, all read back correctly — and the file is then written under the prefilled name.

const dialog = ui.findWindow((w) => w.className === "#32770" && /save as/i.test(w.title), 15000);
dialog.sendKeys("%{n}");        // the "File name:" accelerator
dialog.sendKeys("^{a}");        // replace what is prefilled
dialog.sendKeys(fullPath);      // real keystrokes, not typeText
dialog.clickButton("Save");     // Enter would take autocomplete's suggestion
dialog.waitUntilClosed(10000);
if (!fs.exists(fullPath)) throw new Error(`saved somewhere else: ${fullPath}`);

The dialog closing tells you a save happened, never where. Check the path.

For the interruptions that arrive whenever they like — a session expiry, a confirm-overwrite, a "keep both files" — register a guard that runs before every poll of every wait:

const g = ui.guard(() => {
  const expired = ui.window("Session expired");
  if (expired) { expired.clickButton("OK"); signIn(); return true; }
  return false;                       // nothing of mine was on screen
});
// …later, when that phase is over: g.stop();

Returning true also resets the enclosing wait's deadline, which is the detail that makes it usable: a script that asked for fifteen seconds meant fifteen seconds of waiting for its dialog, not fifteen minus the time spent clearing three modals out of the way.

Step 4: when there is no control tree at all

Plenty of software in this category draws its own grid, toolbar and text, and publishes nothing to read. That's what vision is for:

const hit = vision.find("Run Report", { region: win.rect() });
if (hit) mouse.click(hit.center);
vision.waitForText("Export complete", { timeout: 60000 });
vision.waitForStill({ region: win.rect() });        // the pixels stopped moving

Try win.controls() and el.click() first — vision is slower, needs the window visible, and depends on what is actually on screen. Every result carries a rect and a center in screen coordinates, even when you passed a region.

Need to shell out instead? sys.exec returns {ok, code, stdout, stderr, timedOut, truncated}:

const r = sys.exec("powershell", ["-Command", "Get-Content C:\\erp\\status.txt"]);
if (!r.ok) log.warn("exit " + r.code + ": " + r.stderr);

Step 5: now let an agent write the selectors

Everything above is the part you can only write if you can see the application. So let the agent see it.

Connect one to the MCP server — an API key with MCP / AI agent access switched on (it's off by default), or a signed-in connection for a hosted client. Alongside the automation_* tools for listing agents and starting runs, the agent gets a set that reaches a live desktop:

  • desktop_list_windows — what's actually open
  • desktop_inspect_tree — dump the accessibility tree
  • desktop_capture_screenshot — photograph a window
  • desktop_test_selector — try a selector and be told in one round trip that it was wrong
  • desktop_eval_js — evaluate statements against real state
  • desktop_save_script — save the result as a version

That loop is the whole point. Nobody can write selectors for a twenty-year-old line-of-business application from documentation, because there isn't any — so the agent looks at the running application instead, guesses, is corrected in one round trip, and iterates until the script works. The week with the UI spy tool becomes a conversation.

If somebody already does the task by hand, start there instead: record them doing it once. The agent reads the recording with automation_get_capture — the accessibility tree at every click, with the pointer inside the control that was clicked, and a keyframe of the screen at any moment — and writes the first draft from that, with no live access to the machine. Then it verifies the selectors against the live application as above. The automate-desktop-app prompt walks an agent through both routes in order.

What the agent deliberately cannot do

This is the one place on the platform where a tool reaches a machine somebody is sitting in front of, so it takes two separate permissions and both are required:

  • The credential must carry the live desktop inspection scope. It is not implied by any other scope — a key that manages instances does not silently also look at screens.
  • A developer window must be open on that specific machine. An organization owner opens one from the console, or an owner's credential with full access opens one with automation_open_dev_window. It lasts up to an hour, expires on its own, belongs to the person who opened it, and is re-checked on every single call. While it is open, whoever is sitting there sees a small notice at the top of the screen saying an AI assistant has live access.

Which gives the property worth having: a leaked key gets a list of machine names. It does not get a photograph of somebody's desk. Refusals say which of the two is missing, so an agent that hits one asks you to open the window rather than retrying. desktop_eval_js additionally needs confirm: true — it types and clicks in whatever application is open, in front of whoever is there.

More broadly, the MCP surface covers building, not governance. There is no tool for billing, members, deleting an organization or creating API keys, and no scope that produces one — at any access level, including full. A credential that can mint credentials escapes its own limits, which would make every other restriction decorative.

Step 6: hand the work over for good

Once the script exists, the machine does the job on its own schedule. A few things shape how that behaves:

  • Lanes decide which process runs the script, and therefore what it can touch. Desktop is the signed-in user's session — the only lane with windows, input, clipboard and screenshots, and the default. Headless runs in the service session under a restricted token, with every privilege stripped. Maintenance is for the setup work.
  • Scripts are exclusive by default. Anything touching windows, keystrokes or the foreground shares one screen, so those runs serialize per machine. A script that only makes HTTP calls declares --parallel.
  • Runs can execute as a local account. One run creates it, later runs execute as it, headless, under its own identity — the password is held by the agent and never returned, and the agent deletes every account it created when it is uninstalled.
// once — create the account (maintenance lane, machine installed with --managed-account)
job.result(user.ensure('buildbot', { admin: false }))

// then — run anything as it; needs neither of those
altengine automation run --instance fleet --run-as buildbot nightly

Test before any of that touches the control plane. The worker runs a script locally with nearly the same engine:

altengine-worker.exe run nightly.js --param date=2026-10-01 --trace trace.json

No run record, and artifact.put*, env.set, job.waitForData and vm.run throw. The trace is written even when the run fails, and it's the same trace a dispatched run uploads — so what you read locally is what support reads.

One note if you're planning to develop against the emulator: it serves every cloud MCP tool under the same names, but not automation_*, and desktop_* cannot be emulated at all. Those reach a real Windows desktop through a real agent, and there is nothing local to stand in for one.

What it costs to run

Three things: the time each enrolled machine spends connected, a small fee per run started, and the time runs actually spent executing. Connected agents are $5.00 per 1 million agent-seconds — about $13 a month for an always-on machine — and runs are $1.00 per 1,000 started.

A machine idling all day between two nightly jobs pays for being connected and almost nothing else, which is the honest shape: your PC costs us nothing, and our end of its connection does. Setting a machine up isn't billed at all — a run pays nothing for the time it spends installing VirtualBox, fetching Windows installation media or building a guest. Each instance carries a per-run cost ceiling, and the machine itself stops a run when its budget is spent, so a script with no time limit still cannot spend more than you allowed.

On the agent side, MCP tool calls are $0.02 per 1,000, and connecting, listing tools and reading resources are free. An agent session of a few hundred calls costs well under a cent. Requests are rate-limited to 600 per minute per organization — an agent doing real work makes a handful of calls a second, while a runaway loop makes thousands.

The first $3 of usage each month is free for every organization, with no card to start. Full rates on the pricing page.

Where to start

Pick the task someone does every morning. Enrol the machine it runs on, open a developer window, and let an agent list the windows and dump the tree while the application is actually open — before you write a line of script. The selectors are the week you were going to lose, and they're the part you no longer have to guess at.

The full API is in the Automation reference, with the script surface in the script API and the hard-won details in tips & recipes.

Start free in the console

← Back to the blog