← Back to API docs

Datastore API

The Datastore API is a managed JSON document store with a bounded, App Engine Datastore–style query engine — multi-field filters, sorting, aggregates, and atomic transactions. Each namespace is an isolated, strongly consistent store, so reads always see the latest committed writes. All paths below are relative to a datastore instance you provision in the console.

Authentication and base URL

Send requests to https://api.altengine.net with an organization API key as a bearer token. Reads require a read grant, writes a write grant, and deletes a full grant — see Authentication.

Authorization: Bearer ae_yourkeyid.your-api-key-secret

Every path is scoped to an instance, namespace, and collection: /v1/datastore/{instance}/ns/{namespace}/col/{collection}/…. Data is organized as instance → namespace → collection → document. A namespace is an isolated store (up to 10 GB) that you create on demand — ideal for one-database-per-tenant designs; a collection is a named group of documents within it. Namespaces and collections are created automatically on first write. Use _default for the default namespace (it can't be an empty path segment).

A namespace is placed once, when it is first used: near that first request, or in the region set on the instance (console → Datastore → your instance → Settings → Region). Changing the region places only namespaces created afterwards; existing namespaces stay where they are.

Datastore requests are rate-limited per instance to a sustained 100 requests per second (6,000 per minute) by default, with a matching burst; you can set a lower limit per instance in the console (Datastore → your instance → Settings). Separately, a namespace holds at most 8 writes (puts, deletes, transactions, index builds) in flight at once; a write past that is refused. Both return 429 with a RATE_LIMITED code and Retry-After: 1. A refused write was not applied, so wait and send the same request again — even an auto-id put.

Endpoints

Documents

A document is any JSON object, addressed by a key that is unique within its collection. Objects may nest freely; there is no schema to declare up front.

POST /v1/datastore/app/ns/acme/col/users/documents
{
  "documents": [
    { "key": "u1", "data": { "name": "Ada", "age": 36, "addr": { "city": "SF" } } }
  ]
}
# → { "keys": ["u1"] }

Putting a document with an existing key replaces it. Send up to 500 documents per request.

Every document read back — by get, query, or join — also carries created and updated, write times in milliseconds since the Unix epoch, next to key and data. A put that replaces a document keeps its created. These are the values __created__ and __updated__ sort on.

Keys: strings or numbers

Keys are strings — you can use any UTF-8 string (1–512 bytes, no NUL byte): a slug, an email, a UUID, whatever you like. A JSON number is also accepted and is stored as its decimal string, so 5 and "5" address the same document.

{ "documents": [
  { "key": "[email protected]", "data": { "plan": "pro" } },
  { "key": 1042,              "data": { "sku": "A-1042" } }   // stored as "1042"
] }

Auto-generated keys

Omit key and the instance generates one for you; the strategy is set per instance in the console (Datastore → your instance → Settings). The generated keys come back in the keys array, in input order.

Batch get and delete

# Get documents by key in one request (found ones only; up to 500 keys)
POST /v1/datastore/app/ns/acme/col/users/documents/get
{ "keys": ["u1", "u2", "u3"] }
# → { "documents": [ { "key": "u1", "data": { … }, "created": 1760000000000, "updated": 1760000000000 }, … ] }

# Batch delete by key (requires a full grant)
POST /v1/datastore/app/ns/acme/col/users/documents/delete
{ "keys": ["u1", "u2"] }
# → { "deleted": 2 }

Caching pattern: run a query with "keys_only": true to fetch just the matching keys (a small, cheap payload), keep a client-side cache of documents you've already loaded, then batch-get only the keys you're missing.

Query

Post a structured query to a collection. where filters, order sorts, limit bounds the page, and cursor pages forward. Fields are dot-paths into the document (nested paths like addr.city work); the reserved selectors __key__, __created__, and __updated__ sort by key and by write time.

POST /v1/datastore/app/ns/acme/col/orders/query
{
  "where": [
    { "field": "status", "op": "=",  "value": "open" },
    { "field": "total",  "op": ">",  "value": 100 }
  ],
  "order": [{ "field": "total", "dir": "asc" }],
  "limit": 25
}
# → { "documents": [ { "key": "…", "data": { … } } ], "cursor": "…"|null }

A sorted query takes one range filter, and sorts by it first. A query that sorts may use at most one range filter (<, <=, >, >=, !=), and that field must be the first sort key — which is why the example above orders by total rather than by __updated__. No index can serve two range filters with a sort, or a range on one field ordered by another, so such a query is refused with 400 INDEX_REQUIRED and no index will help. Sort by the range field first, or make the filter an equality. A query with no order may filter ranges on several fields.

To page, pass the returned cursor back on the next request; when it is null there are no more results. Supported operators: =, !=, <, <=, >, >=, and in (a non-empty array). Add "keys_only": true to return just the matching keys ({ "keys": [ … ], "cursor": … }) instead of full documents. A query page is capped at 500 documents (default 25). Repeated identical queries are served from cache, which a write to the same collection immediately invalidates, so you never read stale data.

Joins

Enrich each result with a referenced document from another collection via join. Each join reads the value at local_field and looks up that key in the foreign collection, attaching the result under as. The lookup is by the foreign collection's key (its primary key), so a join is always index-served — never a full scan — and bounded by the page size. Up to 5 joins per query.

POST /v1/datastore/app/ns/acme/col/orders/query
{
  "where": [{ "field": "status", "op": "=", "value": "open" }],
  "order": [{ "field": "total", "dir": "desc" }],
  "join": [
    { "as": "customer", "collection": "users", "local_field": "user_id" }
  ]
}
# → { "documents": [
#      { "key": "o1", "data": { "user_id": "u1", … },
#        "joins": { "customer": { "key": "u1", "data": { "name": "Ada" } } } }
#    ], "cursor": … }

When the referenced document doesn't exist (or the local field is absent), joins.<as> is null. Joins are only for where/order queries, not keys_only ones.

A join reads the foreign collection by key without applying that collection's access rules, so joins require an organization API key — a query carrying a join from an end-user id_token is refused. Client-side apps read one collection at a time through their own row rules.

Indexes

Queries are index-served: a query that a declared index cannot satisfy returns 400 with an INDEX_REQUIRED code naming the field to index. A plain listing (no filter, no sort) needs no index.

Declare an index on one or more fields. Single-field indexes are enough to filter on several fields at once — the engine seeks one and filters the rest. A composite index (multiple fields) additionally serves a query that filters on the leading field(s) and sorts by the next, and backs grouped aggregates. Add "unique": true to enforce uniqueness of the field value within the collection.

# Declare indexes
POST /v1/datastore/app/ns/acme/col/orders/indexes
{ "fields": ["status"] }                 # single-field
# and, for filter-by-status + sort-by-total:
{ "fields": ["status", "total"] }        # composite

# List / drop
GET    /v1/datastore/app/ns/acme/col/orders/indexes
DELETE /v1/datastore/app/ns/acme/col/orders/indexes/{id}

A field may carry a direction suffix, :asc (the default) or :desc. An index serves a sort whose directions all match it or all oppose it, so a uniform sort needs no suffix; a mixed sort needs one. __created__ or __updated__ may be the last field of a composite, which serves a filter sorted by write time.

# order: priority asc, due desc
{ "fields": ["priority", "due:desc"] }

# where status = "open", order: __updated__ desc
{ "fields": ["status", "__updated__"] }

You can also manage indexes visually in the console (Datastore → your instance → Indexes). Declaring an index on a populated collection builds it once over the existing documents.

Auto-indexing (on by default). If a query or aggregate needs an index that doesn't exist yet, the engine creates the one it suggests and runs your request — so a query shape an index can serve does not fail while you are building. It does not rescue every query. These still return INDEX_REQUIRED, because there is no index to create: a sort combined with two range filters, or with a range on a field other than the first sort key (see Query above); a sort on __key__, __created__, or __updated__ followed by another field; and a shape that would need an index of more than 8 fields. The one-time build is billed as writes (one row per existing document), and the response includes "auto_indexed": { "fields": [ … ], "indexes": [ [ … ] ], "rows_written": N }: indexes lists every index built for the request (a read rule with OR branches can need one per branch), fields is the first of them, and rows_written is their total. It applies to end-user tokens too, so a client query can trigger a build — a collection is capped at 64 indexes, and at the cap a query that needs a new one returns 400 FAILED_PRECONDITION. Turning the setting off is how you stop client-driven builds entirely. Turn it off per instance (console → Datastore → your instance → Settings → uncheck Automatically create missing indexes) to instead get a 400 INDEX_REQUIRED naming the field to add — the right choice in production.

Aggregates & metrics

Compute grouped metrics without reading every document. metrics lists the aggregates (count, sum, avg, min, max), group lists the dimension fields, and where filters first. Grouped and filtered fields must be index-served, same as queries.

POST /v1/datastore/app/ns/acme/col/orders/aggregate
{
  "group": ["region"],
  "metrics": [
    { "fn": "count",              "as": "orders" },
    { "fn": "sum", "field": "total", "as": "revenue" }
  ],
  "order": [{ "field": "revenue", "dir": "desc" }]
}
# → { "groups": [ { "group": { "region": "west" }, "metrics": { "orders": 2, "revenue": 240 } } ] }

as names a metric in metrics; it must match [A-Za-z_][A-Za-z0-9_]*. A metric without a valid as is named by its position — m0, m1, … — with no error. order sorts groups by a group field or a metric's name; any other field returns 400. limit caps the number of groups returned.

Cost & indexing tradeoffs

Datastore bills on three axes — reads, writes, and stored data — and indexing moves all three; see the pricing page for exact rates.

So indexing is a two-sided lever, and the goal is the middle:

• Under-indexing makes queries scan (or fail with INDEX_REQUIRED): slow, and read-expensive because you pay for every row examined.
• Over-indexing makes every write more expensive and grows storage. An index on a field you never filter, sort, or group by is pure cost with no benefit.

Rule of thumb: index what you query — and nothing more. Declare an index for each field (or field combination) you actually filter, sort, or group on. Prefer a composite index when you filter and sort together — one composite is cheaper to maintain than several single-field indexes. Drop indexes you've stopped querying.

Auto-indexing (see Indexes above) can create indexes you didn't plan, each one adding write and storage cost.

Transactions

Run several operations atomically within one namespace — they all commit or none do. Transactions are strongly consistent. Operations run in order:

POST /v1/datastore/app/ns/acme/transaction
{
  "operations": [
    { "op": "check",  "collection": "accounts", "key": "a1", "exists": true },
    { "op": "mutate", "collection": "accounts", "key": "a1", "increment": { "balance": -30 } },
    { "op": "put",    "collection": "ledger",   "data": { "delta": -30 } }
  ]
}
# → { "keys": [null, "a1", "…"] }

Atomic increment is race-free — concurrent increments to the same field never lose an update.

Live updates

An instance can publish a change event to a channel instance in the same organization after each committed put to a listed collection. Turn it on in the console (Datastore → your instance → Settings → Live updates): name the channel instance, then list the collections, each with an optional partition field.

# what a subscriber to posts.thread-42 receives
{ "channel": "posts.thread-42",
  "data": { "op": "put", "ns": "acme", "collection": "posts", "keys": ["p17", "p18"] },
  "ts": 1760000000000 }

An event carries keys, never document data: fetch the documents with a batch get, which applies the reader's own access rules. One put publishes one event per distinct channel. ns is "" for the default namespace. Only POST …/documents publishes — deletes and transactions do not. Publishing happens after the write commits and does not affect its response; events are billed to the channel instance as messages.

Backups & restore

Every namespace has continuous point-in-time recovery for the last 30 days at no extra charge. From the console (Datastore → your instance → Backups) you can copy a namespace's current snapshot and restore the namespace to a past timestamp or a saved snapshot.

Limits

Errors

Errors use standard HTTP status codes and the same JSON shape as the rest of the API — see Errors.