Automation API
Automation runs your scripts on your own Windows machines. An agent installed on a PC in your office holds an outbound connection to altengine and executes JavaScript you deploy — driving a desktop application, or an HTTP endpoint that only exists on your network. No inbound port, no VPN, no firewall change.
Closed beta. Automation is enabled per organization. If the Automation tab isn't in your console, ask us to switch it on.
The pieces
| Piece | What it is |
|---|---|
| Instance | A fleet: its agents, scripts, schedules, runs and settings. |
| Agent | One enrolled machine, tagged with labels. |
| Script | A deployed, versioned bundle. Parameters make one script many jobs. |
| Run | One execution, with a log, artifacts, a cost and an outcome. |
| Schedule | Cron (one or several lines), time zone and parameters. Declared here, fired by the machine's clock. |
Enrol a machine
Mint an enrollment token in the console, then download the installer it offers — the token is in the filename and the installer reads it back out. To enrol by hand:
altengine-worker.exe enroll ^
--control-plane https://api.altengine.net ^
--instance fleet ^
--token <enrollment-token>The machine trades the token once for its own credential. Tokens last days by default, are listed in the console, and can be revoked in a click.
Labels are stamped on at enrollment (site-dallas, has-quickbooks) and are how you target work without naming one computer.
Machines a machine enrols
An enrolled machine can enrol machines of its own — the VirtualBox guests it hosts — under a name it picks. Off until you tick Allow machines to enrol their own guests on the instance.
altengine-worker.exe enroll --child-key lab-1 --name "Lab 1" --labels site-dallas
{"agent_id":"…","agent_secret":"…","instance":"fleet","child_key":"lab-1","created":true}The credential is printed and never saved on the host — it belongs to the machine the host is about to start. Inside that machine:
altengine-worker.exe enroll --instance fleet --agent-id <id> --agent-secret <secret>The same key is always the same machine. A host re-enrols a guest on every boot and after every re-image; through single-use tokens each of those would be a different machine, and yesterday's runs, artifacts and schedules would be stranded on a row nothing will ever connect to again. Re-enrolling returns the same id with a fresh secret, drops the old socket, and reports created: false. Omitting --name or --labels leaves what the machine already has rather than clearing it.
| Bound | What |
|---|---|
| Key | Lower-cased. Letters, digits, dot, dash and underscore, starting with a letter or digit, up to 64 characters. |
| Scope | The key is the host's own. lab-1 on two machines is two guests, and no host can name, adopt or overwrite another's. |
| Cap | Guests one machine may own at once: 4 by default, up to 64. Deleting a guest frees its place. |
| Depth | A guest cannot enrol guests. |
| Credential | The host's own. No organization API key goes on the machine. |
A guest is an ordinary machine: its own runs, its own artifacts straight to storage, targetable by its own labels. The Agents tab nests it under its host. Driving guests from a script is vm.*.
Windows installation media
A host can also build a guest that does not exist, by running Windows setup unattended. Point the instance at the media it should use. Bring your own — we do not host Windows.
| Setting | What it takes |
|---|---|
| Windows installation media | A path on that machine, a UNC share, or an http(s) URL. The first two are used where they lie; a URL is downloaded once into a cache on each machine and reused by every later guest there. |
| SHA-256 | Required when the location is a URL, and the pair is refused without it. What this resolves to is booted as an operating system, so it is verified before it is used — and re-verified before each use, rather than trusted for its name. |
| Edition in that media | Which edition inside the ISO, counting from 1. 0 lets the machine choose, which it can only do when the choice is obvious; otherwise it refuses and lists what it found. |
A script names media of its own for one guest with vm.ensure — which is what a host owning both a Windows 11 desktop and a Server guest needs, since neither ISO is "the fleet's". Building a guest is not billed, and does not count against the run's cost ceiling: see pricing.
What the agent installed
The agent records what it put on the machine, and whether the thing was already there. An uninstall reverses only its own entries, so removing the agent from a PC that already ran VirtualBox leaves that VirtualBox alone.
- A hypervisor that was already installed when the agent arrived is never removed, whatever happens afterwards.
- One the agent installed stays anyway if the machine still has virtual machines the agent did not create — removing the hypervisor would take them with it.
- If the machine's VM list cannot be read at all, nothing is removed.
The same reversal runs when a machine is revoked from the console — it uninstalls itself, and takes its own files with it.
Write a script
An ES module with job, env, log, sys, fs, csv, crypto, time, http, db, net, task, ui, mouse, clip, vision, browser, vm and artifact in scope — the script API reference has every function, and tips & recipes has the patterns. Everything is synchronous — no promises, no await — except handing back a file, which returns before the upload finishes.
const day = job.params.date;
sys.launch("C:\\ERP\\erp.exe");
const win = ui.findWindow("Invoice Ledger"); // waits
win.focus();
win.find("name", "Date").setValue(day);
win.find("name", "Run report").click();
const grid = ui.waitFor(() => win.find("automationId", "resultsGrid"),
{ timeout: 120000, describe: "the invoice grid" });
for (const row of readRows(grid)) job.emit(row);
job.next({ date: nextDay(day) }); // tomorrow is its own runDon't write sleeps. ui.findWindow and win.find wait, and every window lookup waits for the window to stop changing before handing it back. Use ui.waitFor with a describe, so the failure names what never appeared.
Three that are easy to get wrong — see tips & recipes:
win.typeText()enters text;win.sendKeys()presses keys ({ENTER},^{s},{SHIFT+}…{-SHIFT}).- Match a dialog by what it is — class
#32770and its buttons — not its title.close()is a request a dialog can decline; press its button withclickButton. win.find()returns the first match. Checkwin.findAll(by, query).lengthwhile you write the selector.
Reach your network
const r = http.fetch("http://erp.internal/api/invoices?day=" + day, {
headers: { authorization: "Bearer " + env.ERP_TOKEN },
});
if (r.status === 404) return; // nothing that day
for (const row of r.json()) job.emit(row);No outbound allowlist — the script runs on your machine, on your network. An HTTP error is data: check r.status and r.ok, nothing throws. Responses over 32 MB are truncated and say so.
A file is http.download rather than a fetch: it streams to disk, so there is no size cap, and it lands atomically or not at all. http.downloadAll does several at once.
Deploy and run
export ALTENGINE_URL=https://api.altengine.net
export ALTENGINE_KEY=ak_...
altengine automation deploy --instance fleet --name nightly ./nightly.js
altengine automation run --instance fleet --param date=2026-08-29 --wait nightlyThe CLI bundles the script — the agent resolves no imports. Test it on the machine first, with no control plane involved:
altengine-worker.exe run nightly.js --param date=2026-08-29 --trace trace.jsonExclusive and parallel
Scripts are exclusive by default: anything touching windows, keystrokes, the clipboard or the foreground shares one screen, so those runs are serialized per machine. A script that only makes HTTP calls declares --parallel and runs alongside others up to the instance's limit.
Lanes
Which process on the machine runs a script, and therefore what it can touch. A different axis from exclusive and parallel: that one says whether two runs may share a desktop, this one says whether a desktop is involved at all.
| Lane | What it is |
|---|---|
| Desktop | The signed-in user's session. The only lane with windows, input, the clipboard and screenshots, so it needs somebody signed in. The default. |
| Headless | The service session, as a restricted account. fs, csv, http, db, net and the browser — signed in or not, so a headless script behaves the same at 3pm as at 3am. |
| Maintenance | The service session as LocalSystem: installers, service control, vm.install. Off unless both the instance and the machine allow it. |
Set it on the Scripts tab, or as lane on the deploy request. Omitting it leaves the deployed setting alone, so a redeploy cannot silently relocate a headless script onto a session where its windows do not exist.
Endpoints
| Endpoint | What it does |
|---|---|
POST /v1/automation/{instance}/scripts | Upload a new script version. |
GET /v1/automation/{instance}/scripts | Scripts and which version is live. |
GET /v1/automation/{instance}/agents | Machines, and which are connected. |
POST /v1/automation/{instance}/runs | Start a run. 202 if no machine is connected yet. |
GET /v1/automation/{instance}/runs/{id} | A run, with short-lived download links. |
GET /v1/automation/{instance}/runs/{id}/logs | Live while it runs, complete afterwards. |
POST /v1/automation/{instance}/runs/{id}/cancel | Stop a queued or running run. |
GET/POST /v1/automation/{instance}/schedules | Read and declare schedules. |
POST /v1/automation/{instance}/data/{key} | Send a value to whichever job is waiting for that key. 202 when no machine confirmed it. |
GET/PUT /v1/automation/{instance}/env | Credentials for the fleet. A read returns names, never values. |
POST/GET /v1/automation/{instance}/enroll-tokens | Provision machines from your own tooling. Needs a full grant. |
Takes an organization API key; end-user identity tokens are rejected, as with containers. To let a page start a run, have it call a function that decides.
Results are files
job.emit() lands as _emitted.ndjson. Everything else:
artifact.put(name, bytes, contentType?) | Bytes you built in memory. |
artifact.putFile(name, path, {contentType, remove}?) | A file already on disk. Never read into memory, so there is no size limit. |
artifact.putFolder(path, {name, contentType, remove}?) | Zipped in the background. Defaults to <folder>.zip. |
Files upload straight to storage from the machine, and download links are minted per request and expire on their own. Something is told once the run and its files have both landed — see when a run finishes.
A run's files are objects in a blob instance, created alongside the fleet and named after it. Every artifact carries a blobkey, so a function, a container job or anything else holding a blob grant reads a run's output through the blob API instead of a credential minted for the purpose. Deleting that instance is refused while a fleet still points at it. Point a fleet at a different store — one in another region, or one two fleets share — from its settings.
These return before the upload finishes, so a script keeps working while a large export goes up. Two consequences:
- You cannot catch a storage failure at the call site. If an artifact cannot be uploaded after retries, the run fails — a run reported
donealways has its output. - A file handed over by path must still be there when the upload reaches it. Do not delete it yourself; pass
remove: true.
job.outDir
The export you want is usually written by the application, not by your script — the ERP's Save As dialog. Point it at job.outDir:
win.setControlText(nameBox, job.outDir + "\\invoices.zip");
win.clickButton("Save");
ui.waitFor(() => !dialog.exists(), { describe: "the export to finish" });
artifact.putFile("invoices.zip", job.outDir + "\\invoices.zip");Anything under job.outDir is uploaded and then deleted, and the whole directory goes when the run ends. Files elsewhere are left alone.
The directory is bounded: when more output is waiting to upload than the machine's budget allows, the next artifact.put waits. While output is still going up the run stays running with a phase of uploading.
Logs are written on the machine and pulled on demand — a live tail while a run goes, and the complete log uploaded as an artifact when it finishes, so a decommissioned machine still answers.
When a run finishes
A finished run — files already uploaded — calls a function of yours, or POSTs to a URL. Name it on the run, or on the instance for every run that does not.
POST /v1/automation/{instance}/runs
{ "script": "invoices", "on_complete": "reports/run-finished" } // a function
{ "script": "invoices", "webhook_url": "https://erp.example.com/hook" } // a URL| Target | On the run | On the instance | How it arrives |
|---|---|---|---|
| A function | on_complete | Call on completion | {functions-instance}/{function}. Invoked in place — nothing crosses the internet, and it is billed as an ordinary invocation. |
| A URL | webhook_url | Webhook | An https POST, HMAC-signed with the instance secret. |
Both carry the same body — type: "run.completed", the run, and its artifacts. The webhook's artifacts carry short-lived download URLs; the function's do not, because it has env.blob and every artifact's blobkey.
The function runs with its own configured grants. No credential is minted for it, so naming it here grants the function nothing it did not already have — it grants the fleet the ability to start it.
- One run, one target. Sending both is refused, and naming either one on the run replaces both instance settings — so "call my function for this run" cannot also fire the fleet's standing webhook. Every link of a chain inherits it.
- Five attempts, either form. The run carries
on_completeorwebhookwithstatus,attemptsandat. - A function that throws lands in the functions console with its stack. A URL that fails is
error:500and nothing else.
Files
The export is written by the application, so the next thing a script does is wait for it and read it. The full pattern is in tips & recipes.
const path = job.outDir + "\\invoices.csv";
win.clickButton("Export");
fs.waitFor(path, { timeout: 120000 }); // waits for it to stop GROWING
const out = csv.write(job.outDir + "\\open.csv");
for (const row of csv.read(path, { encoding: "cp1252" })) {
if (row.Status === "OPEN") out.write(row);
}
out.close();
artifact.putFile("open.csv", job.outDir + "\\open.csv");fs.waitFor waits for the file to stop growing, not to appear.
Stream; don't read. csv.read and fs.lines hand you one record at a time off the file handle, so a filter over a 2 GB export holds one row. fs.readText is for small files and refuses anything over 64 MB.
fs.waitFor(path, {timeout, stableFor}?) | Wait for a file to finish being written. |
csv.read(path, opts?) | Iterate rows as objects keyed by the header. |
csv.write(path, opts?) | .write(row), .writeAll(rows), .close(). |
csv.parse(text) / csv.format(rows) | For a body already in memory — an http.fetch response. |
fs.lines(path) / fs.writer(path) | Line at a time, for anything that is not CSV. |
fs.readText / read / write / writeText | Whole file. Capped at 64 MB. |
fs.exists / stat / list / mkdir / move / copy / remove | Removing a directory needs {recursive: true}. |
fs.tempDir() | Scratch space. Unlike job.outDir, nothing here is uploaded or cleaned up. |
CSV options: delimiter (",", "\t", "tab", "pipe", "semicolon"), encoding ("utf8", "cp1252", "latin1"), header, columns, skip, strict.
Old Windows software does not emit UTF-8. Reading a Windows-1252 export as UTF-8 does not fail — it corrupts accented names silently. Pass { encoding: "cp1252" }.
Parsing is lenient by default: stray quotes, rows shorter than the header, repeated column names. Duplicate headers become Amount, Amount_2; a missing field reads "". Pass { strict: true } if a malformed row should fail the run.
There is no path allowlist. The script runs as the signed-in user; its permissions are the boundary.
Reading the screen
For software that draws its own grid, toolbar and text, and so publishes no control tree. Not a rare case in this category.
const hit = vision.find("Run Report", { region: win.rect() });
if (hit) mouse.click(hit.center);
vision.waitForText("Export complete", { timeout: 60000 });
vision.waitForStill({ region: win.rect() }); // the pixels stopped movingTry win.controls() and el.click() first — vision is slower, needs the window visible, and depends on what is on screen.
Every result carries a rect and a center in screen coordinates, even when you passed a region. find matches a phrase across consecutive words on one line, case-insensitively unless you pass { exact: true }, and returns null when the text is not there; waitForText throws and says what was readable.
For a control with no text, match a picture. Templates match by score, not by exact pixels:
vision.findImage({ path: "C:\\templates\\save.png" }, { threshold: 0.9 });
vision.findImage({ base64: "iVBOR…" }); // embedded in the script
vision.bestImageMatch({ path: "…" }).score; // the closest, when nothing matchedOCR uses the engine built into Windows and needs a language pack for the signed-in user (Settings > Time & language). To check a machine, run the agent's tests\ocr.js.
Run a program
const r = sys.exec("powershell", ["-Command", "Get-Content C:\\erp\\status.txt"]);
if (!r.ok) log.warn("exit " + r.code + ": " + r.stderr);Returns {ok, code, stdout, stderr, timedOut, truncated}. Options: timeout, cwd, input, env, encoding.
Arguments are an array, never one command string. For a shell, name it: sys.exec("cmd", ["/c", …]).
A non-zero exit is data, like an HTTP status — msiexec returns 3010 for "installed, needs a reboot". Only a program that could not be started throws. A timeout is a result too, so whatever it printed before being killed survives, and it kills the program and everything it started.
sys.launch(exe, …args) starts a program without waiting — use it for the application you are about to drive, then ui.findWindow. sys.platform is "windows", "linux" or "darwin".
Schedules
A schedule is cron, a time zone, and parameters. Cron may be several expressions, one per line — a timetable like "weekdays at 07:00 and 13:00, Saturdays at 09:00" is three. It runs when any line matches, and two lines landing on the same minute still produce one run.
It is pushed to the machine, which fires it from its own clock — so a nightly export still runs when your internet is down, and the run's record arrives when the connection returns.
Target one machine or a set of labels. A labelled schedule reaches every machine carrying those labels and produces exactly one run.
When a machine was off at the appointed time, choose skip, or run the most recent missed occurrence, once. Never all of them.
One machine on a different timetable
One branch runs the export at 03:00 because its accounting system is busy at 02:00; one till sends a different account number; one PC being decommissioned should stop. Open that machine from the Agents tab, go to Schedules, and Customise — cron, time zone, parameters, and whether it runs here at all. A field left empty follows the fleet, and keeps following as the fleet's own setting changes.
Which field you change decides whether that machine still shares the fleet's run.
| Change | The run |
|---|---|
| Cron, time zone, parameters | That machine takes an occurrence of its own. On a labelled schedule its 02:00 run is a second run, not a replacement for the shared one — different parameters make it a different job, and it is billed as one. |
| Run on this machine | Shared. Every machine carrying the label still competes for the one run, minus the ones switched off. |
That switch is a ceiling: it stops a schedule on one machine and can never start one. A schedule switched off for the fleet is off everywhere.
A machine may hold exceptions to 25 schedules, and one exception's parameters are capped at 4 KB. Follow the fleet clears an exception.
Running a schedule on guests
A schedule can run on the targeted machines' guests instead of on the machines themselves — Run on those machines' guest VMs instead of on them, with an optional list of guest labels to narrow which ones (target_children and child_labels over the API).
That is one run per guest, unlike host labels above, where every matching machine competes for a single run. The host keeps the clock and switches each guest on: a powered-off VM is not in an outage, so it would miss every occurrence and then fire once at 9am for the 2am job. The occurrence is claimed before the guest is up, so the run waits queued exactly as one waits for an office PC that is asleep.
Guest labels are all-of, like a host's, and a label nothing carries reaches nothing rather than everything — otherwise a typo puts the dealer export on every guest the host owns.
Long work is a chain
A backfill through two years should be a run per day, not one run for the year. job.next({...}) queues the next link with different parameters; each retries on its own and survives a reboot. Only successful runs chain.
Credentials
Set credentials on the instance and read them as env.NAME, rather than putting them in the source:
curl -X PUT https://api.altengine.net/v1/automation/fleet/env \
-H "Authorization: Bearer $ALTENGINE_API_KEY" \
-d '{"env": {"PORTAL_USER": "dana", "PORTAL_PASSWORD": "…"}}'Values are write-only: a read returns names. They travel with the dispatch, so a run that starts while the office internet is down can still sign in.
env.set(name, value) writes one back from inside a script — see forced password changes.
Waiting for a code
A portal sends a one-time code to somebody's mailbox. The script can be told:
win.clickButton("Send code");
const code = job.waitForData("otp.dealer-42", { timeout: 300000 });Whatever already receives that message posts it in — most naturally a function, which has a public URL and reaches the waiting job through env.automation.deliver. An organization API key works too, and job.dataUrl(key) hands out a URL scoped to one run.
A value that arrives before the script asks is held, not dropped.
What a delivery answers
The machine acknowledges the value, so the answer is what it confirmed — not what was written to its socket. Three outcomes:
| Status | Outcome | What to do |
|---|---|---|
200 | The machine filed it in the waiting job's mailbox. | Nothing. |
202 | Undetermined — written to an open socket, not confirmed within five seconds. | Do not send a replacement. The job may already have it, and a second one-time code invalidates the first. Let the job's own timeout decide. |
409 | Nothing was written anywhere: no connected machine is running a job that matches. | Retry, or look at the run — a code that arrives after the script gave up is this. |
Instance-wide, the counts are per run — a delivery can land on one machine and go unconfirmed on another:
POST /v1/automation/fleet/data/otp.dealer-42
{ "key": "otp.dealer-42", "delivered": 1, "runs": ["run_a"], "unreachable": [], "undetermined": ["run_b"] }The status is 200 while delivered is above zero, 202 when only undetermined has anything in it. To one run — job.dataUrl(key)'s URL, or /runs/{id}/data/{key} — it is delivered: true, or delivered: false with undetermined: true and a sentence saying so.
Bounds
A run has no time limit by default — a backfill is finished when it runs out of history. What bounds it instead:
| Bound | Default | Catches |
|---|---|---|
| Cost ceiling per run | $1.00 | The real bound. Converted to authorised seconds; the machine stops itself when they are spent. |
| Stall detector | 15 min | A run that has stopped doing anything — stuck on a dialog nobody will dismiss. |
| Livelock check | on | A run doing the same thing forever — clicking OK on a dialog that keeps coming back. It is never idle, so nothing else catches it. |
| Action timeout | 30 s | One step. "The invoice grid never appeared" is a failure you can act on. |
| Wall clock | none | Set timeout_ms when there is a real deadline. |
When a run fails
Everything needed to understand the failure is captured at the moment it happens and uploaded with the run, so it is debuggable with the machine switched off:
| File | Contents |
|---|---|
_log.ndjson | Everything the script logged. Uploaded by every run. |
_trace.json | Each step, in order, with timings and the one that failed. |
_context.json | The machine, agent build, script version, parameters, and every open window. |
_screen.png | The desktop at the moment of failure. |
_screen.zip | The two minutes before it. Off by default — see below. |
_screen.json | That recording's index: when each frame was taken, and how the capture went. |
Nothing typed, pasted or entered as a value appears in the log or trace — they record how much was typed and into which control. Parameters are masked by name. The screenshot is taken by default; the recording is not.
Watching the two minutes before it failed
Turn on screen recording — per instance, or for a single machine from the Agents tab; prefer the single machine — and a failed run leaves one behind. Play it back from that machine's page in the Agents tab. It needs no live access and no desktop: it is the run's own uploaded record, readable with the PC switched off.
The screen is held in a ring in memory and thrown away when a run ends well, so a successful run writes nothing anywhere. Nothing but the screen is recorded — there is no mouse or keyboard hook of any kind.
| Bound | Value |
|---|---|
| How far back | 2 minutes. |
| Frame rate | 2 a second, adaptive. The capture loop keeps itself under a fifth of one core, so a very large desktop records fewer frames rather than slowing the person sitting at it. _screen.json reports what one frame cost, so a thin recording has an explanation. |
| Size | 6 MB and 1,500 frames. Past either, the oldest seconds are dropped and the index says how many. |
| Which runs | Only desktop-lane ones. A headless or maintenance run has no desktop to photograph, and a run whose screen never changed produces none either. _context.json says which it was. |
It is not an MP4 — it is JPEG frames in a zip, which any zip tool opens. Nothing needs encoding software on the machine.
A recording is an ordinary artifact: it is an object in the fleet's blob instance and is billed on that instance's storage line, like every other file a run keeps.
Checking on a machine
A machine is in one of three states, and the third is not a shade of the second.
| State | What it means |
|---|---|
| Online | It reported in within the last few minutes. Work sent to it starts. |
| Offline | It worked and stopped. Office PCs sleep, so this is not an error on its own — work sent to it stays queued until it reconnects. last_seen is when the machine was last heard from, not when it was noticed missing. |
| Never connected | never_connected on the API. The credential was minted and nothing ever came up on it, so its hostname, OS and agent version are blank. This is not a machine that has stopped working; revoking removes the row. |
online means the machine answered, not that a socket is open: an agent reports in on a timer, and one that goes quiet is dropped from the roster inside the window. A machine that loses power is offline in four to seven minutes rather than whenever its connection eventually collapses.
| Field | What it says |
|---|---|
liveness | "reporting", or "unverified" for a machine whose agent is too old to report — there a held socket is the whole of the evidence, and a PC that was switched off reads as online until its credential expires. Present only while online; the console says Online, unchecked. Updating the agent makes it a checked answer. |
heard_within_seconds | The window "reporting" is a guarantee over. Read it rather than hard-coding it. |
A machine reports its own hostname, OS and agent version when it connects. Whoever enrolled it may say what it is called and give it labels, and nothing else about it is taken on their word.
Three views in the console answer three different questions.
| Question | Where |
|---|---|
| Why did this run fail? | Runs → Details. Parameters, artifacts and the log. While a run is going that log is a live tail from the machine; once it ends it is the complete log the run uploaded, so a finished run is readable even from a PC that has since been switched off. |
| Why is this machine unhealthy? | Agents → Agent log. Why it reconnects, why an upload failed, why it refused a run. A machine connected but doing nothing looks, from the run list, like a quiet night. |
| What is on its screen right now? | Agents → View screen, once you have opened live access on that machine. Lists its windows and photographs one. The image is not stored. |
Live access is what the last of those needs, and it is also what an AI assistant needs. It is off by default, cannot be turned on by an API key, lasts up to an hour and closes by itself. Revoking a machine closes it immediately.
Building a script against the live application
Open live access on a machine, then View screen. Four panels do the whole loop:
| Panel | What it answers |
|---|---|
| Screen | What is on it right now. Click, double-click, right-click and scroll go to the machine; tick a box and your keystrokes do too. Pick one window or the whole desktop. |
| Elements | The accessibility tree — the names, roles and automation ids a selector can match on. “Try as selector” carries one straight into the next panel. |
| Selector | What your selector actually matched, and how many things. That count is the point: on Windows' own Save As dialog, role=edit matches 46 elements and 44 of them are cells in the file list. |
| Console | Statements against real state, in one session, so a variable set in one is there for the next. Anything a script can do. Save the ones that worked as a script version when you are done. |
A saved version is not live until you activate it on the Scripts tab. An assistant connected over MCP has the same four capabilities, gated by the same window.
The same loop on the machine itself
A developer sitting at the machine — at the desk, over RDP, in a lab VM — needs none of that: no enrolment, no control plane, no live access window.
altengine-worker.exe studio --port 9099 --file .\nightly.jsServes an editor and the inspector at http://127.0.0.1:9099: the windows, a window's accessibility tree, a screenshot, a selector tested against the live application, statements evaluated in a session that remembers the last one, and the buffer run with its log streaming line by line as it goes. It is the same code the console and MCP reach, so a selector that resolves here resolves in a deployed run.
The port is the capability. The studio binds 127.0.0.1, there is no flag to change that, and there is no sign-in. Anything on that machine that can open the port can drive the desktop and run code as whoever started the process. Run it on a machine you are working at; it is not a way to give somebody else access to one.
One run at a time — a run that has not declared itself parallel owns the screen, so a second is refused rather than queued. A run started here stops after 10 minutes unless it names its own limit, and never runs past an hour.
Saving writes a file on disk. Publishing a version is altengine automation deploy or the console — that route needs an organization API key, which does not belong on an enrolled machine.
The editor is Monaco when it can find one: point --monaco at a directory holding Monaco's vs/ folder. Otherwise the page uses a plain editor and says so on its badge. It never fetches from a CDN, so a site with no outbound access gets the plain editor rather than a page that waits forever.
Recording somebody doing the task
Start a capture from Agents → View screen → Capture, then have somebody at that PC do the task by hand. The recording keeps the screen and the accessibility tree: a video shows a click at (412, 388), and the tree beside it says which control was there — a name, a role and an automation id a selector can be written against.
Watch it back in the same panel. The scrubber moves the picture, and the sample list beside it says what was on screen at that instant.
It needs live access open on the machine, and there is a second permission at the machine itself:
The person at the machine is told, and can stop it. The agent will not start a capture unless it can put a notice on the screen it is about to record. The notice stays on top, does not take focus from what they are working in, and stops the recording when clicked. A machine with no interactive desktop, or one sitting at a lock screen, is refused with that reason rather than recorded.
Closing the live access window stops any capture running on that machine.
What a capture leaves behind
| File | Contents |
|---|---|
_capture.json | The manifest: what the capture was allowed, what it counted, why it stopped, and the segment list with a per-frame index. |
_capture-NNNNNN.zip | One segment of the recording — JPEG frames in a plain zip, the same format as a failure video. |
_capture-ui.ndjson | The UI samples, one JSON object per line: when, which window, where the pointer was, and the tree. |
The recording is cut into segments — at 6 MB, 600 frames or 60 seconds, whichever comes first — and each one is uploaded while the person is still working. So a long capture is fetched a piece at a time, and one that was interrupted keeps everything up to the last completed segment. The manifest is written last: a capture with segments and no manifest is one that did not finish.
The tree is sampled on interaction
Not on a clock, and not per frame. The agent walks the tree when the foreground window changes, and when the screen changes and then goes still — the moment after a click has finished redrawing, which is the state the person just produced. There is no keyboard or mouse hook of any kind.
A sample is 6 levels deep and 1,200 nodes, and two are never closer together than 750 ms. A window that stops answering is abandoned after 10 seconds and that sample carries the reason instead of a tree. Untick Record the UI tree as well as the screen to capture only the video.
Bounds
These are the machine's ceilings. The console can ask for less, never more.
| Bound | Value |
|---|---|
| Length | 15 minutes by default; up to 60. |
| Size | 192 MB by default; up to 512 MB. |
| Frames | 20,000, and 180 segments. |
| Frame rate | 2 a second by default, up to 4, adaptive. The capture loop keeps itself under a fifth of one core, so a very large desktop records fewer frames rather than slowing the person sitting at it. The manifest reports what one frame cost, so a thin recording has an explanation. |
| UI samples | 400, and 6 MB of them. |
Passing a bound stops the capture and names which one. Nothing is dropped from the beginning — a recording that ends before the person did says so on its face, in truncated and above the player. Sampling has its own two bounds, and hitting one leaves the rest of the capture with video but no tree; that is reported too.
How a capture ended
stopped | What happened |
|---|---|
person | Whoever was at the machine clicked the notice. |
console | Stopped from the console. |
closed | Live access on that machine ended. |
bounds | It reached a limit; truncated names which. |
refused | The machine would not start it — locked, no desktop, or already recording. The row is kept, so "I pressed the button and nothing happened" has an answer. |
lost | The machine stopped reporting mid-capture. Whatever it uploaded is kept. |
A capture is not a run: no script executed, and it produces no run row and no log. It adds no billing line of its own — its files are objects in the fleet's blob instance, billed on that instance's storage line like every other artifact, and the machine's connection is already billed. A person working by hand is not run time.
A machine has to be signed in
Anything that drives windows — clicks, keystrokes, the accessibility tree — needs somebody signed in to that PC. A locked machine has no desktop to type into, and a run that needs one says exactly that rather than blaming the application.
Work that only uses http, fs, csv or the browser does not care. If a job has to survive the office going home, write it that way — sys.desktop() tells a script which half of itself it can run.
Building a script with an AI agent
Connected over MCP, an assistant can list a machine's windows, dump the accessibility tree, photograph a window, test a selector, and evaluate statements against real state until the script works — then save it as a version. That is how you write selectors for an application with no documentation.
It needs both: the live desktop inspection scope on the credential, and a developer window that a person opens on that machine from the console. The window lasts up to an hour and expires on its own.
Cost
Time an agent spends connected, a fee per run, and time runs spent executing. A machine idling between two nightly jobs pays for the connection and little else. Files a run keeps are billed by blob, on the storage line of the instance holding them. See pricing.
Updates
The agent updates itself from a release manifest signed with an offline key, verified against a pinned publisher certificate. The connection carries a version number and never a download link. Windows only.