A JSON Document Store That Doesn't Fight Your Schema
altengine's Datastore API is a JSON document store you reach over REST: POST any JSON object with a key, then filter, sort, join, and aggregate it. There are no property types to declare, no embedded-entity workaround for nesting, and no eventual consistency to design around — a namespace always reads back its latest committed write. What it does keep is the one bound worth keeping: every query has to be index-served.
Put whatever shape you have
A document is any JSON object, addressed by a key that is unique within its collection. Objects nest freely. There is nothing to declare first — namespaces and collections are created on the first write.
POST /v1/datastore/app/ns/acme/col/users/documents
Authorization: Bearer ae_yourkeyid.your-api-key-secret
{
"documents": [
{ "key": "u1", "data": { "name": "Ada", "age": 36, "addr": { "city": "SF" } } }
]
}
# → { "keys": ["u1"] }Paths are instance → namespace → collection → document. A namespace is an isolated store of up to 10 GB that you create on demand, which makes one-database-per-tenant the easy path; use _default if you don't want tenancy. Up to 500 documents go in per request, and putting a key that already exists replaces it.
Keys are strings — any UTF-8 string from 1 to 512 bytes, so a slug, an email, or a UUID all work. A JSON number is accepted too and stored as its decimal string, so 5 and "5" address the same document. Omit the key entirely and the instance generates one; the strategy is per instance, and the choices are uuid (the default), scattered, serial, or manual to switch auto-id off so a keyless put is a 400.
What the old constraints actually bought
If you shipped on the classic App Engine Datastore, you remember the shape of the friction. Every property carried a declared type. Nesting meant reaching for embedded entities rather than just writing an object. Joins did not exist, so you denormalized or made a second round trip. Composite indexes were declared up front in a file, and a query that outran them failed in production. And outside an entity group, queries were eventually consistent.
That is a fair list, and it is worth being fair about the other side too: those constraints are why the thing was fast at scale and why the bill did not surprise anyone. A query that can only be served by an index cannot quietly become a full-collection scan when your data grows. That property is worth keeping. Most of the rest was the price of a much older storage engine, not a law of nature — so altengine keeps the document model and the bounded query engine, and loosens everything else.
Queries: filters, sort, cursors
A query is a JSON body posted to a collection. Fields are dot-paths into the document, so addr.city works the same as a top-level field, and the reserved selectors __key__, __created__, and __updated__ sort by key and by write time.
POST /v1/datastore/app/ns/acme/col/orders/query
{
"where": [
{ "field": "status", "op": "=", "value": "open" },
{ "field": "total", "op": ">", "value": 100 }
],
"order": [{ "field": "__updated__", "dir": "desc" }],
"limit": 25
}
# → { "documents": [ { "key": "…", "data": { … } } ], "cursor": "…"|null }The operators are =, !=, <, <=, >, >=, and in over a non-empty array. Pass the returned cursor back to page forward; null means you are done. A page is capped at 500 documents and defaults to 25. Add "keys_only": true and you get just the matching keys back — a light payload that pairs well with a batch get for whatever your client hasn't cached yet. Repeated identical queries are served from cache, and a write to the same collection invalidates it immediately, so the cache never hands you a stale row.
Joins, by key, still index-served
A join attaches a referenced document from another collection to each result. It reads the value at local_field and looks that up as the key of the foreign collection — which is exactly why it stays cheap: a primary-key lookup, bounded by your page size, never a scan.
POST /v1/datastore/app/ns/acme/col/orders/query
{
"where": [{ "field": "status", "op": "=", "value": "open" }],
"order": [{ "field": "total", "dir": "desc" }],
"join": [
{ "as": "customer", "collection": "users", "local_field": "user_id" }
]
}
# → documents carry "joins": { "customer": { "key": "u1", "data": { "name": "Ada" } } }Be clear about the limits: up to 5 joins per query, a missing reference gives you joins.<as> as null, and joins don't combine with keys_only. A join also reads the foreign collection without applying that collection's row-level access rules, so it requires an organization API key — a query carrying a join from an end-user identity token is refused. A browser-side app reads one collection at a time through its own rules instead.
Grouped aggregates
Counts and sums are a first-class request rather than something you assemble by paging the whole collection.
POST /v1/datastore/app/ns/acme/col/orders/aggregate
{
"group": ["region"],
"metrics": [
{ "fn": "count", "as": "orders" },
{ "fn": "sum", "field": "total", "as": "revenue" }
],
"order": [{ "field": "revenue", "dir": "desc" }]
}
# → { "groups": [ { "group": { "region": "west" },
# "metrics": { "orders": 2, "revenue": 240 } } ] }The functions are count, sum, avg, min, and max. Grouped and filtered fields have to be index-served, same as a query.
The index rule, and the switch on it
This is the part that stayed strict, so it deserves a straight explanation. A query a declared index cannot satisfy returns 400 with an INDEX_REQUIRED code naming the field to index. A plain listing with no filter and no sort needs no index at all. Single-field indexes get you further than you might expect — the engine seeks one and filters the rest — so you need a composite index mainly when you filter on leading fields and sort by the next one, or when you back a grouped aggregate. "unique": true on an index enforces uniqueness of that field within the collection.
What has changed is that missing an index no longer blocks you mid-build. Auto-indexing is on by default: a query that needs an index it doesn't have gets one created and then runs. It is never silent — the one-time build is billed as writes, one row per existing document, and the response carries "auto_indexed": { "fields": [ … ] }.
In production, turn it off. It's a per-instance setting, and with it off you get the 400 INDEX_REQUIRED instead. The reason is cost, not purity: every index adds an index row per document write and takes storage, so you want to decide which indexes exist rather than let a stray query decide for you.
Consistency and transactions
Each namespace is strongly consistent — reads always see the latest committed writes, with no entity groups to design around. Transactions run several operations atomically within one namespace, up to 500 of them: put and delete, a mutate for field-level set / increment / remove with optional upsert, and a check precondition that aborts the whole transaction with 409 if it fails.
POST /v1/datastore/app/ns/acme/transaction
{
"operations": [
{ "op": "check", "collection": "accounts", "key": "a1", "exists": true },
{ "op": "mutate", "collection": "accounts", "key": "a1",
"increment": { "balance": -30 } },
{ "op": "put", "collection": "ledger", "data": { "delta": -30 } }
]
}The atomic increment is race-free: concurrent increments to the same field never lose an update. Every namespace also carries continuous point-in-time recovery for the last 30 days at no extra charge, so undoing a bad bulk edit is a console operation rather than a restore project.
What it costs, and what it doesn't
Three axes: writes at $2.00 per million rows, reads at $0.50 per million where one read covers up to 100 rows examined, and stored data at $0.80 per GB-month. There is no idle cluster — a namespace you aren't querying costs only its stored data.
Metering reads by rows examined is what makes the index rule pay off: a get or a small index-served page is a single read, while an unselective query pays for every row it touches. The fast query and the cheap query are the same query. Writes move the other way — a document in a collection with three indexes is four writes, not one — which is why "index what you query, and nothing more" is the whole tuning story. The first $3.00 of usage each month is free for every organization and shared across services, with no card to start; see the pricing page for the rest.
Where it isn't the right tool
There is no raw SQL and no ad-hoc analytical scan — that is the deal you make for a bounded query engine. Joins are one hop by foreign key, not arbitrary relational joins. A namespace tops out at 10 GB (you make more namespaces), puts, deletes, transaction operations and query pages each cap at 500, and an instance is rate-limited to a sustained 100 requests per second by default. And Datastore does not do full-text ranking or facets — that's what the Search API is for. The full surface, including index management and the cost tradeoff table, is in the Datastore API reference.
← Back to the blog