GO-7383 AnyBlock JSON: a readable, validatable object format - #3241
Open
requilence wants to merge 345 commits into
Open
GO-7383 AnyBlock JSON: a readable, validatable object format#3241requilence wants to merge 345 commits into
requilence wants to merge 345 commits into
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Coverage provided by https://github.com/seriousben/go-patch-cover-action |
`type_properties[].object_types` is a type key slot: it shares the envelope's term ledger, its census and its `type_keys` legend. Nothing asserted it. The envelope truncates to the positions §2 models — one type, plus a template's target — so the census's whole stated purpose, a document naming several types, lives entirely in the slot no assertion watched. Removing the ledger back-off proves the hole. Seed 0 of the hostileVocab variant then exports a type whose `owner` property targets ["task","task","squatter"] for the stored ["69bbfc78877a91b1d12d1a7c","task","squatter"] — the bundled Task target lost and duplicated onto the custom type, with the legend binding `task` to the custom key — and the seed stayed green. The reader-side capture is a PropertyResolver answering false, so the snapshot is identical to the resolver-less read: applyTypeProperties resolves `object_types` into the definition it hands the resolver, and the recommended lists it writes carry property ids, not targets, so there is nowhere else to see this slot from outside.
`writableTypeSlug` refuses a vocabulary answer when the SLUG cannot be
written or when the stored KEY cannot be — the second because the stored key
would become a legend VALUE, which the schema bounds at 128 characters. The
corpus reached only the first half: pool 6 supplies a 140-character stored
type key, but hostileVocab had no answer for it, so `slug == key`
short-circuited before the guard ran. Deleting `|| !isWritablePropertyKey(key)`
left every package test green.
hostileVocab now spells that key `diary` — a perfectly good spelling for a
stored key that is not one. With the guard, export writes the 140-character
key verbatim into `type`, which is a value slot and unbounded (§3); without
it, Marshal emits `type_keys: {"diary": "yyy…(140)"}` and its own Validate
answers /type_keys/diary: legend stored key … is 140 characters; the bound
is 128 (§3) — I1.
The key is named (overLongTypeKey) rather than repeated, so the pool, the
type-property targets and the vocabulary cannot drift apart.
The same hole one namespace over. `writableSlug` refuses a vocabulary answer
when the stored KEY cannot be written, because that key becomes a legend
VALUE and the schema bounds a legend value like a member name. Deleting
`|| !isWritablePropertyKey(key)` left the whole package green.
The reason is worth recording: every /properties emit site filters unwritable
keys BEFORE anything slugs them, and so does the census, so the guard is
reachable only from a BLOCK slot — a dataview sort or filter naming a stored
relation key verbatim — and the corpus named none. The corpus dataview now
carries a filter on the stored key "a\nb" (appended after the picks, so the
seeds above it are unchanged) and hostileVocab spells it `ab`.
Revert-check: deleting the guard half makes seed 1 emit
`property_keys: {"ab": "a\nb"}`, and Marshal's own Validate answers
/property_keys/ab: legend stored key "a\nb" carries a control character (§3).
The seam refuses an unwritable resolved property key in `type_properties[].key`
— landed deliberately, with an I2 argument in its own commit message — but
nothing on the validation side carried the rule. The slot is a JSON string
VALUE, not a member name, so `propertyNames` never sees it and `minLength: 1`
was the only bound it ever had:
{"version":1,"kind":"object_type","key":"k",
"type_properties":[{"key":"kkk…(140)"}]}
validated clean and then failed to import with
/type_properties/0/key: resolved property key … is 140 characters; the bound
is 128 (§3). Same for a key carrying a newline. Validate promising "this
document imports" for a document that does not is exactly what I2 forbids,
and it held on every non-widening axis (default, bundled, type-moves-template).
Three sides now agree, which is what the invariant asks for:
- the schema bounds the spelling (maxLength 128 + the control-character
pattern), because an external validator runs the schema and nothing else
(§12);
- propertyNameIssues restates it so the verdict says what is wrong with the
string rather than printing a regex, and suppresses the schema's duplicate
at the same pointer — the mechanism the legends already use;
- export drops such an entry with a warning instead of emitting one. An
unwritable stored key has no spelling but itself, no legend can rescue it,
and emitting it hands back an archive the seam refuses — I1. The corpus
now serves one (hp3, on the hidden section) so the drop is load-bearing.
Revert-checks: removing the export drop makes I1 fail at seed 0 of every
variant with /type_properties/2/key: maxLength: got 140, want 128; removing
both validation halves makes I2 fail on the two new documents in the default,
bundled and type-moves-template axes.
typeterm.go's twin, on the other namespace. The envelope `key` is the raw
STORED key and is never translated (SPEC §2), but every property SLOT is: the
keys of `properties` and `type_properties[].key` each carry a term that
resolves through the §3 chain — the document's own `property_keys` legend,
then the bundled derived table, then verbatim. The scans built and compared
tables raw; the converter reads them by the resolved key.
ScanFormats keyed the format table by the spelling, and the converter looks it
up by the stored key: import.go hands Options.ResolveFormat the output of
importer.propertyKey, and Options.ResolveProperties a PropertyDefinition whose
Key is likewise resolved. So a bundle carrying a `property_keys` legend missed
the table entirely — the value passed through as raw JSON (a date stayed a
string, a select minted no option) and NO Relation object was minted for the
property at all. Converting a two-document bundle whose legend backs
`priority` produced `"6a32d485…": "High"` in the details, four relation
options where two were declared — two orphaned under the spelling by
newBatch's pre-mint, two more under the real key — and no relation for either.
CheckPropertyFormats compared raw against raw, so it agreed with itself and
disagreed with the converter, both ways. Fail-open: the document's own legend
says `priority` here is a space-minted relation, but the raw probe hit
bundle.GetRelationFormat("priority"), got the BUNDLED relation's format, and
reported clean. Fail-closed, and worse in practice: `properties` spells
bundled keys as their api slugs (§3) and `bundle` is keyed by stored keys — it
has never heard of `due_date` — so the canonical spelling was reported as
having no declared format, which anyblockconvert turns into a hard error
unless -lenient.
CheckSharedSelects grouped by spelling, so two documents naming one stored key
two ways did not merge though their option pools do, and two documents whose
legends bind one spelling to different keys merged though they do not.
The authored-targetObjectType probe in CheckTemplateTargets is the same defect
on a slot the type-side fix left alone: it read doc.Properties under the stored
key while the codec decides by the resolved one. `target_object_type` reached
the detail and the check missed it, rejecting a template the converter wires;
a legend moving the `targetObjectType` spelling means the detail is NOT
written, and the check took it as authored and skipped the document whole.
resolvePropertyTerm (propertyterm.go) runs the chain the way a package-only
reader does — which is what anyblockconvert and anyblockvalidate are, since
neither passes an Options.Keys vocabulary: the legend, then anyblockjson's
exported BundledKeyVocabulary, whose pass-through on an unknown term is chain
step 4. Unlike the type namespace there is no `template` reservation to carry.
The legend lookup is the only line restated here; there is no exported entry
point in anyblockjson that composes the two, so
TestLintResolvesPropertyTermsLikeTheCodec pins the composition against the key
Unmarshal actually stores the value under, and drift fails there.
A lint that asks the questions in a different order than the code it lints answers a different question. batch.objectTypeIds asks typeIds FIRST and only falls through to the bundled url; CheckTargetTypes asked the bundle first and short-circuited on a hit. So a bundle that defines an `object_type` document with a bundled key and no `id` passed the lint — `page` is in the bundle table, done — while the converter took the local arm, found the empty id, and appended an EMPTY STRING to relationFormatObjectTypes. That target names nothing, is invisible in every UI, and re-exports as a shorter list than it went in as. Registering the id-less type at all is deliberate (TypeIds says so): it is what lets the converter tell "defined here, but unaddressable" from "bundled" — but only the lint was reading the distinction backwards. Reordered to defined-with-id / defined-without-id / bundled / neither, which is objectTypeIds' order exactly. The id-less finding now also says when the key is one the bundle table knows, because otherwise being told `page` is "defined here but carries no id" reads as nonsense. TestBatch_IdlessLocalTypeYieldsAnEmptyTargetId pins the converter half in cmd/anyblockconvert, so the two halves cannot drift out of order again without a test naming the other one failing.
…recondition)
KeyVocabulary's doc comment asked for one thing — whatever `…Slug` emits,
`…Key` must invert — and §11's round-trip guarantee rests on a second one it
never stated: no answer, in either direction, may bind a spelling the bundled
table binds to a DIFFERENT key.
A vocabulary can satisfy the stated contract completely and still lose a type.
The legend is why: a document owes an entry only for a spelling the reader's
own chain cannot invert, and "the reader's own chain" means the bundled table,
which ships with every reader. So the bundled `task` key, spelled `task`, is
written with no entry at all — and a reader whose vocabulary answers
TypeKey("task") == "69bbfc…" resolves it through that answer instead:
source=[ot-template ot-task] -> back=[ot-template ot-69bbfc…]
A template for the bundled Task type comes back as a template for an
unrelated custom type, silently. The property namespace has the same shape.
Not a live defect: storeresolver refuses both halves — keyMaps.roundTrips will
not SPELL a key with a slug the bundled table binds elsewhere, and the
bundledKey check in keyMaps.key will not BIND one (12 re-pointed objects in a
36 808-object sweep is what the second half cost when it was missing). A
hand-written Options.Keys can, which is what the rule is for.
Three things hold it now:
- the precondition is on the interface, with the mechanism and what breaks;
- TestKeyVocabulary_ShadowingSlugBreaksInversion pins the failure with the
conforming twin beside it, and asserts through a predicate
(typeSlugShadowsBundled) so "conforming" is checkable rather than eyeballed;
- I1 grows a reader-vocabulary axis. §11.1 states the guarantee for export
and import "wired with equivalent resolvers" and nothing exercised it —
every reader in the sweep was package-only, so no seed was ever read back
through the vocabulary that wrote it. roundTripVocab is hostileVocab's
conforming twin (hostileVocab is a writer vocabulary: its Key direction is
pure bundled, so it is not an inverse pair at all).
That axis also asserts the whole snapshot equals the package-only round trip's:
with equivalent resolvers a vocabulary is a spelling choice and may not change
what the document MEANS. It is the only assertion watching the property
namespace, where the type-slot assertions cannot see.
Revert-check: pointing roundTripVocab's customTypeKey at `task` — one
character of the fixture, making it shadow — fails the new axis at seeds
0, 2, 4, 5 … on all three assertions (envelope types, type-property targets,
and the snapshot comparison), and passes again at `tsk7`.
Also: I2's `type-space` axis mapped the custom type key to `task2`, a slug no
document in the corpus spells, so the collision it exists for was absent by
construction. It maps to `task` now.
Export was not a fixpoint. Exporting an object, importing it and exporting it
again produced a different document — the same object, two spellings — and §9's
"provided ids are preserved so re-exports diff cleanly" is worth nothing if
the terms move instead.
The census was why. seedTypeTermLedger reserved every stored type key the
SNAPSHOT named, while the document spells only what §2 models — one type, plus
a template's target — and only the type properties export actually writes.
Every key in that gap was reserved on behalf of a term no reader ever sees,
and the reservation backed a real slug off. Two shapes, both re-measured on
this branch before touching anything:
ObjectTypes ["ot-custom1","ot-cust"], vocabulary spelling custom1 as cust
gen1: "type": "custom1" (backed off — `cust` is in the census)
gen2: "type": "cust" + type_keys (the truncated entry is gone)
a recommended list holding a keyless definition that targets `cust`
gen1: object_types ["custom1"] (backed off)
gen2: object_types ["cust"] + type_keys
Nothing was protected by the wider reservation: a key the document never
names cannot be taken as another key's spelling by a reader who never sees
it. modelledTypeKeys now runs the same reduction for the census and for the
emit — the emit passing a flag so the keyless-entry warnings are reported
once — and the census asks writableTypePropertyKey, the same question
buildTypeProperties asks.
I1 grows the snapshot-side half of §11.2 to catch this: Export(S) ==
Export(Import(Export(S))), compared with ids omitted because import mints an
id wherever the snapshot had none, which §11.2 already exempts.
Three pools the corpus lacked make the first shape reachable from ordinary
data: a NON-template with a second type (every multi-type pool was a template
or a keyless pair, so the truncating branch had no shape at all), a
non-template with three, and a list whose every entry is keyless.
Revert-checks: restoring the census's old envelope walk fails I1 at hostileVocab
seeds 17, 91, 217, 262 ("exporting the snapshot that came back must reproduce
the document") plus the new fixpoint test's first subtest; restoring the old
type-property walk fails its second subtest.
The property census has the same over-reservation by construction — which
blocks survive is decided during buildBlocks, so its walk cannot know — and
its comment now names the fixpoint as the cost. No corpus shape reaches it.
Documentation for the three behaviour changes on this branch, plus two
corrections in one family.
§11 gains the precondition its guarantees are stated for. "Equivalent
resolvers" was never defined, and it is stronger than KeyVocabulary said: a
vocabulary may not bind a spelling the bundled table binds to a different
key. A vocabulary that does can still be a strict inverse pair — and a
template for the bundled `task` type comes back as a template for an
unrelated custom type, because a spelling the bundled table inverts is
written with no legend entry at all.
§11 also gains the snapshot-anchored guarantee, Export(S) =
Export(Import(Export(S))), and §3 the census rule that makes it true:
reserving keys the document does not spell backs a real slug off, so the same
object exported before and after a round trip produced two documents.
§3 gains the property namespace's answer to a question the type namespace
answered the other way: a property key slot carries the writable-key rule
wherever it sits, `type_properties[].key` included, because /properties is
that namespace's home surface and cannot express a key that is not a member
name — while the type namespace's home surface is `type`, a value, which is
why its primary slots are unbounded.
§3's argument for having no type deny rule stops resting on "export strips no
object types", which is false — export truncates to the positions §2 models,
and v0.14 added a bullet saying so four bullets down. What export strips is
POSITIONAL, never a particular key, which is what the derivation actually
needs.
Two documentation corrections, both teaching a spelling the format has not
used for two versions:
- §1's Naming section still described the pre-vocabulary rule — property
keys "written exactly as stored: `iconEmoji`, `dueDate`" — and argued
that a key ↔ key mapping was impossible because Validate takes no
resolver. The answer to that was to put the mapping in the document,
which is what property_keys/type_keys are.
- §3's "what is not a key slot" paragraph named the format's OWN fields in
their pre-snake_case spellings — `kind: "objectType"`,
`defaultTemplateId`, a callout's `iconEmoji`/`iconImage` — none of which
the schema has used since v0.8, and all of which contradict §1's rule
that every identifier the format defines is snake_case. The paragraph's
point (these are not key slots) is intact; only the examples were stale.
And §2a's type-document example wrote `"iconEmoji": "✅"` in `properties` two
lines above prose calling the same key `icon_emoji` (reported by a sibling
agent). The example is the first thing a reader copies.
buildTypeProperties dropped a definition with no key silently and one whose
key is merely unwritable with a warning — the same drop, for the same reason,
reported two ways. The empty key is the one a vocabulary bug actually
produces, so it is the one most worth reporting.
§3 already promises this in the type namespace ("every drop is reported
through OnWarning") and the sentence added for the property half says the
same.
A table cell that qualifies for the string shorthand is rendered without going through blockToJSON, which is where the emit-once mark lives (§11). The branch set that mark and never consulted it, so all it did was order the two arrivals of a block with two parents: reached through the cell first, the other parent was silenced correctly; reached through the other parent first, the cell wrote the block A SECOND TIME — once nested under its parent (under a disambiguated label, since the derived cell id is reserved) and once as the cell's text. One stored block came back from import as two. The branch now reads the mark, and a cell whose block was already emitted renders empty — the first arrival wins, exactly as blockToJSON decides it everywhere else. The existing regression test only covered the safe order, which is how a mark that was written but never read looked fine for so long; it now runs both orders and follows the document back through Unmarshal, so a second emission is counted as a block rather than as a phrase. The hostile corpus grows the same shape — a cell that is also the child of a plain block standing before the table — and I1 asserts the block is written at most once. 155 of the 300 seeds catch the old behaviour.
A table owns `<rowId>-<colId>` for every row×column pair, written cell or
not (§6.1), and §6.1 says all three surfaces claim the same set. Import
claimed only the half the document spelled: claimAuthoredIds walks the
authored row and column ids and crosses them, so a table whose row or
column id is GENERATED left every cell id that row or column implies
unreserved. Two things then landed on it, and both were reachable with a
generator the caller supplies (the convert wiring derives ids from file
paths):
- a block the document authored on that id — the derived cell was built
on top of it, and the snapshot came back with two blocks sharing an id;
- an id generated later — the trailing paragraph took the cell's id, with
the same result.
So the whole grid is claimed once every row id is resolved, before any cell
is built (building one generates ids of its own), and a generated row or
column id is now rejected when the cell ids it implies are taken — the
generated side is the one that yields, since a derived id has no spelling
of its own.
The test calibrates itself against the generator rather than hard-coding
its answers: it imports the table alone to learn what the row and column
will be called, and only then builds the collision. Hard-coding would pass
vacuously the day the number of genId calls changes.
A snapshot's block list is not its block tree: unlinked subtrees survive in it, and a table among them is not in the document at all. Export reserved its grid anyway — seedIdLabels walked every block in the list — so the ids of blocks the document DOES contain turned on a block nobody can see: a paragraph legitimately named `r9-c9` was written as `r9-c9_2` because an orphaned table happened to have a row `r9` and a column `c9`. Not cosmetic, then: §9 promises an id that is already legal is preserved so re-exports diff cleanly, and this renamed one on the authority of a block that never reaches the output. The reservations now run over the ChildrenIds closure of the export's entry point. The walk over-approximates deliberately — it follows edges blockToJSON declines to descend — because under-reserving is the dangerous direction: a grid that IS emitted and not reserved is a document Marshal's own Validate rejects (I1). That makes the entry point load-bearing, which the fragment export did not have: indexBlocks infers a root as "the first block nobody references", and a subtree slice carrying its own parent (or any spare entry) moves that somewhere else. MarshalBlockSubtree now says its root is subtree[0], the block the emit actually starts from; the test for it fails as an I1 violation without that line.
jsonPath joined the schema library's RAW instance-location tokens, so a
member name carrying `/` or `~` produced a pointer addressing somewhere
else. The restated key-slot checks build theirs escaped (RFC 6901), and
the ledger that suppresses the schema's second opinion is keyed by
pointer — so the two spellings never met and `{"property_keys": {"a/b":
""}}` came back three times for one empty legend value: once at
`/property_keys/a~1b` and twice at `/property_keys/a/b`, a location the
document has no member at. Against §12's one fault, one issue.
transformDateFilter returns a filter whose format is not date before it computes any range, so a preset on a text or select property is stored UI state that decides nothing and its day count is never read. The counting-preset rule checked the condition half of that gate and not the format half, so a leaf the app runs exactly as written — `status greater number_of_days_ago` — was refused for a count nothing would have read. The gate reads the format import will attach (impDvFormat): the dataview's own properties list first, the bundled table on the resolved stored key second, so a hand-written dataview with no properties list still has its `due_date` filter judged as the date filter it becomes. On the fragment surface the same resolution now runs the term through the reader's vocabulary first, the way importer.filterFromJSON does — the documented `due_date` slug resolved no format at all before, and the format is what says whether a preset means anything. The comment over countingPresets predated the condition scoping and still claimed a missing count silently means "today", full stop; and getDateRange's exactDate default reads the same field too, as a timestamp rather than a count.
A preset under a condition transformDateFilter does not substitute into decides nothing: the view matches on the condition alone, so a filter written as "verified this week" quietly means "verified, ever". The reader said nothing about it. It warns now, through the OnWarning channel, and the document goes through. Refusing it was never an option: export writes the pairing because stored filters carry it — the corpus produces it on its own, and making this an error fails I1 on every variant — so an error here would turn one stored filter into an object that cannot be exported. The message names the six conditions a preset is applied on, and the leaf it is about. The format half of the same gate stays silent on purpose: a filter's format usually comes from outside the document, so "not a date" there is as often "not known here", and a warning that fires on a correct filter is what makes every warning cheaper to ignore (§12).
The walk that claims every property-name site has an addressable message followed map values only, and half this schema's subschemas hang off array-valued keywords — the block dispatch under `allOf`, the table cell under `anyOf`, the filter node under `oneOf`. A site inside any of them was unchecked while the test reported a clean sweep. Descending finds nothing new today: all four sites are plain map values, and the assertion is unchanged. That makes the descent unfalsifiable against the schema itself, so it gets a fixture of its own — the same array shapes the schema uses, with a site hidden in each, which the map-only walk misses all three of.
Three documents the invariant corpus could not express: a counting preset whose property the block declares as text (inert by format, where the two above it are inert by condition or applied by both), one on a property nothing resolves, and a member name carrying a JSON-pointer metacharacter — accepted as a property, refused as a legend value, both addressed escaped.
§6.2: a preset applies on a date property under six conditions, both halves stated; the counting-preset operand rule is scoped to where the preset applies; a preset under a condition that does not apply warns, and the format half deliberately does not. §12: an issue path is a JSON pointer and a segment from the document is escaped as one, which is what the one-fault-one-issue suppression is keyed by.
FormatInfo.ObjectTypes was written by ScanFormats and read by nothing. It came from 843f95e, which gave an objects/files property its target types so a minted relation carries relationFormatObjectTypes — the client only offers the current-user filter value when the targets include the built-in Participant type. That feature is live, but its carrier moved: objectTypeIds takes anyblockjson.PropertyDefinition, whose ObjectTypes the codec resolves through the §3 chain. What was left behind held the RAW, unresolved spellings, beside a struct whose other keys are now all resolved — the same shape as the five translation bugs just fixed, waiting for its first reader.
recordPropertyKey and recordTypeKey decided whether a term owes a legend
entry by asking the BUNDLED table alone. That is the wrong table for the
reader most likely to read the document back: the writer's own space, whose
vocabulary import consults first.
A vocabulary that conforms to every rule KeyVocabulary states can still bind
a spelling the document writes verbatim, and a space grows exactly that
shape by DELETING something. A UI-deleted type vacates the slug namespace
(storeresolver's corpse policy) while every object it typed keeps the stored
key, and the freed spelling becomes another live type's api key:
export writes TypeSlug("initiative") = "initiative" (no legend entry)
import binds TypeKey("initiative") = "69bbfc78…"
=> an object typed ot-initiative comes back as ot-69bbfc78…
silently. The property namespace produces the loud half of the same fault:
two spellings then address one property, and Unmarshal refuses a document
Marshal has just written — an I1 break.
Export now asks both tables and records the identity entry when either
would answer something other than the key being written. The entry is
authoritative for every reader, which is what a legend is for. Shipped
vocabularies are unaffected in the common case (no golden moved): the
bundled table needs no entry it did not already owe, and storeresolver's
roundTrips/keyMaps.key already refuse the collisions they can see.
KeyVocabulary also gains the third precondition storeresolver has always
implemented and the interface never stated: a live stored key outranks the
vocabulary's own slug binding. Without it a document naming a relation by
its stored key lands on whichever other relation minted that string as its
api key.
TestKeyVocabulary_ShadowingSlugBreaksInversion is restated rather than
weakened: the writer-side arm is now covered by the legend, so the
precondition is pinned where it still bites — a READER whose vocabulary
shadows the bundled table for a document no writer could have warned it
about.
SS3 says every dropped object type is reported through OnWarning. modelledTypeKeys reported the KEYLESS drop and said nothing about the positional one, so Marshal(Page, ["ot-page", "ot-task"]) emitted a document carrying one type, returned no warnings, and the user's second type was gone with nothing anywhere to say so — not in the document, which has no slot for it, and not to the caller, who is the one still holding it. The truncation itself is the format's shape (SS2 models one type, plus the target type on a template), so the warning says that, and names the entry at the position it stood in among the snapshot's object types — before the keyless entries close ranks, so a caller can match it against its own list.
`text` names two stored formats, and SS3 makes it resolve per key: a key
already known to be shorttext — every bundled `name`, `icon_emoji`,
`cover_id`, and whatever the wiring's ResolveFormat recognizes — keeps that
format, and only an unknown key becomes longtext. That is what keeps a
bundled short-text property from being rewritten on every round trip.
applyTypeProperties ran the rule (declaredFormat); BuildRecommendedLists,
the PATCH-types channel for the same array, read the name literally:
{"key": "name", "format": "text"} document door -> shorttext
PATCH door -> longtext
So one array meant two different things depending on which endpoint wrote
it, and the property a PATCH mints is created with a format the document
path would not have given it. The rule now lives in one function over
Options, which is all either door has.
The rule checked that `value` was PRESENT. `"value": null` — and a string, a boolean, a list — passed both Validate and Unmarshal, and the count still read as 0, i.e. today: getDateRange reads the operand with domain.Value.Int64, which answers 0 for every kind that is not a number. That is the exact trap the error message describes, admitted by the rule meant to close it. No I2 break either, which is what kept it invisible: both surfaces accepted. The rule now reads the operand — a whole number in [0, 36500], the bound the compact grammar already puts on daysAgo(n) (SS6.2.1), so two forms of one filter language admit the same filters. A number no float64 can hold stays checkNumbers' issue, at the same pointer, because one fault is one issue (SS12). Export gets the matching half, or the tightened rule would make an object unexportable: a stored operand that is not a day count has no written form, so it is written as the count the query engine reads out of it (0 for a non-number, the truncation for a fraction, the bound for anything past it) and reported through OnWarning. The I1 corpus carries such a filter now.
A path addresses the fault. Neither of these did. buildTypeProperties reported a dropped SS2a entry at /type_properties/<len(out)> — the index the next SURVIVING entry takes. For [bad, good] it warned about /type_properties/0 while entry 0 of the document is the healthy one, so a caller that followed the pointer read the wrong property. A dropped entry has no index in the document; the array is its address, and the message names the key, which is the same shape a dropped property key is reported with. importer.typeKey's template-spelling guard hardcoded /type wherever it fired, including from /template_for and from /type_properties/N/object_types/M — a field the pointer does not name at all in a type document. The slot is a parameter now.
SS3: the legend's emission rule now names both tables a writer can ask — the bundled one and the vocabulary in force — and says what a delete does to a space's slug namespace, which is where a conforming vocabulary starts binding a spelling that objects still carry as a stored key. SS11.1: KeyVocabulary's third precondition, written down. It was always implemented by storeresolver and never stated: a live stored key outranks the vocabulary's own slug binding. SS3 (object types): both kinds of drop are reported, the keyless one and the keyed one the positional truncation leaves nowhere to go. SS6.2: a counting preset needs a day count, not merely a `value` — with the [0, 36500] bound the compact grammar already applies, and the export half that keeps the tightened rule from making an object unexportable.
A type keeps its recommended properties in four detail lists by role. type_properties (SS2a) collapses them into one labelled array and import rebuilds all four, writing an empty list for a role nothing occupies. Most types have no file-role property, so the store has no recommendedFileRelations key at all and the round trip adds an empty one. That is a real difference and it is normalization: an absent list and an empty list say the same thing, and the empty list is the only way the format can express clearing a role, because type_properties cannot name a section that exists with no members. Unrecorded, it buried the sweep. A 34 339-object account over 148 spaces reported 1 351 objects differing; 1 344 of them differed by nothing else. The comparator now records the absent-to-empty step, and only that step: a role list arriving with members, one that lost its members, and an empty list on any other key are all still reported. Whether the object state should carry all four lists consistently is a question about the state rather than the format, filed as GO-7451.
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
The human's resolution of the native sweep's state_drift_timestamps finding. The evidence that admitted the drop: two exports of the same 7 spaces, 1,164 documents compared field-by-field — the only drifting kind is participant (22 of 22) and the only drifting field created_date; on the full 155-space run, 2,322 drifts against 2,492 participants, every other kind byte-stable. The mechanism: a participant is derived from the ACL and has no creation change, so the store stamps createdDate with time.Now() on every cold build (detailsinject) — a load timestamp wearing the name of a fact. Implemented as a FORMAT rule, not an exporter rule — the typeProvenanceKeys pattern, scoped to the other machine-derived kind — so the codec and the exporter agree and a hand-authored bundle behaves the same: export omits the key on participants whatever it holds, import drops it there (stale, not wrong), and snapshotdiff consults the same predicate in the SAME change (the standing rule for every drop; late teaching once cost 1,344 false failures, and this one would have cost one per corpus participant). creator and last_modified_by stay on participants by the human's decision (100% _anytype_profile placeholder — upstream's bug to fix, not the format's to paper over); the design doc records my §15 #12 dissent and the near-zero cost of reversing either choice.
The native exporter has had no way to be ASKED for: a client calls ObjectListExport with a model.Export.Format, and the enum stopped at GRAPH_JSON = 5. This adds AnyBlockJSON = 6 and regenerates. Additive by construction — every existing value keeps its number, so a client that has never heard of the format is unaffected, and one that sends 6 against an older middleware gets the unknown-format path rather than a silently different export. Generation was validated before the edit rather than after: `make setup-protoc-go` builds the fork the repo pins (go.mod replaces gogo/protobuf with anyproto/protobuf), and a dry `make protos-go` against an unmodified tree produced ZERO changed files — so the local toolchain reproduces the committed output exactly and nothing in this diff is generator drift. The generated diff is accordingly the enum and nothing else: 521 lines of gzipped FileDescriptorProto (11242 → 11250 bytes), 14 enum-map lines re-aligned because "AnyBlockJSON" is longer than "GRAPH_JSON", and the five lines of the constant itself. Routing is deliberately NOT in this commit: nothing yet maps the new value onto the exporter, so requesting it today behaves exactly as an unknown format did. That keeps the regenerated protobuf reviewable on its own.
Two seams the RPC wiring needs, and one bug it makes matter. ExportCollected writes a bundle from a doc set the caller already has; Export keeps collecting for itself and calls it. The export service runs the same ClosureDerived collection for every format before it knows which writer the request wants, so without this door an RPC-driven bundle would query the whole space twice. CollectRequest is exported so the two spellings of that collection cannot drift. EmitRunner makes the emit pool injectable. The package's own fixed-width pool stays the default (tests, cmd tooling); the export service passes a process.Queue-backed runner instead, which is what will keep this format's progress reporting and ProcessCancel identical to the five legacy formats. Because a task may then run on any goroutine the runner owns, the per-worker resolver set becomes a small recycling pool — minting one per task would re-read the space's relation snapshot for every document, which is exactly what the per-instance caches exist to prevent. The bug: emit ignored ctx.Done() entirely. The producer fed every id and every task ran to completion, so a cancelled export of a large account kept cold-loading objects — the expensive half — long after the user said stop. Now the internal pool stops feeding on cancellation, every task checks ctx.Err() before it loads anything (the queue-backed runner already holds all N tasks, so only the per-task check can stop that one), and an abandoned run returns without writing index.json or properties.json: a bundle whose index states documents the emit never wrote is worse than no bundle. Tasks in flight are allowed to land, each holding a loaded object and possibly a half-written file. ExportDocument renders one object standalone — no bundle files — for the single-object in-memory export, and does not close the object out of the cache: close-after-write pays for itself across thousands of documents, not for the one the user is probably looking at.
The enum landed alone last commit so the regenerated protobuf was reviewable alone; this routes it. closureForFormat had to learn the format, and the predicate it asked was called isAnyblockExport and meant "protobuf or pb.json" — a name that predates this format by years and now reads as its exact opposite, five lines from the routing that decides what an AnyBlock JSON export collects. Extending it would have left that collision in place, so it is gone: one switch listing the three self-contained formats. A table test pins all seven formats' closures, because a wrong answer here is invisible until a bundle silently ships without its types. exportByFormat hands the format to the native exporter with the doc set already collected and a queue-backed runner, so emit runs as the same process.Queue tasks every other format runs as: the progress numbers a client watches keep counting, and ProcessCancel still stops the export — answered in the legacy path's own cancel shape (0 succeeded, no error, the half-written output removed). The queue is created at the format's emit width, which on mobile is 2 rather than 4, because for this format the queue's width IS the resident content set. The Export_Protobuf profile file stays exclusive to protobuf: that branch already tests the format exactly, so the new one never reaches it. ExportSingleInMemory answers with ONE document, no bundle files — the design's position (Q7), resting on the format's rule 7: a document carries its own property names and formats, so index.json and properties.json have nothing to add about a single object. The in-memory file-object refusal is now shared by both paths rather than duplicated. The picker: anyblock.Exporter types its Picker as CachedObjectGetter on purpose — close-after-write is that exporter's memory model, so a picker that cannot close must be a compile error, not a failed type assertion at runtime. The alternative to widening export.picker was resolving a second component holding the same pointer, which is state that can drift. In production the wider type resolves to the same component the narrow one did: core/block.Service is the only ObjectGetter the app registers, and it already answers CachedObjectGetter for the indexer. The two fixtures follow: export_test.go swaps in the strictly richer mock (no assertion changes — the legacy formats never call TryRemoveFromCache, and the mock would fail the test if they did), the anyblock fixture registers its closing picker as the app's component rather than the bare getter mock, and core/publish — the other package that builds this service in a test — gets the same wrapper, closing nothing, since publishing never takes the native path. Two consequences worth naming. dirWriter gains RemoveFile, which nothing in the legacy formats calls: it is the native exporter's un-write hook for a blob whose stream fails half way, reached through an optional interface assertion, and without it the RPC path — the one real users take — would keep the truncated file the anyblock package went to the trouble of removing. And anyblock's own test moves to an external test package: it builds the real export service for its collection seam, and package export now routes back into anyblock, so an in-package test would close that import cycle. Nothing there needed unexported access.
Q6 shipped as its recommendation (a) — a new enum value, pbjson untouched. The answer records the three things the wiring settled that the question never asked: the doc set travels from the export service instead of being collected twice, emit runs as the export queue's own tasks (so progress and cancellation are the ones clients already know), and the single-object in-memory export is one document with no bundle files. §1.5 gains the two runners and, more importantly, the cancellation the first implementation simply did not have.
A backup is usually an archive, and the zip writer is the one export writer whose paths are not the filesystem's — the entry names come straight from the plan's slash-separated bundle paths. Same fixture, Zip: true, every document plus the two bundle files accounted for.
Five documents go: the three core/api/APIV2*.md drafts and docs/AgentApiV2Research.md (a separate workstream, to be re-added on its own branch), and PREFREEZE_REVIEW.md, whose own header warns it was written against SPEC v0.6 and spells identifiers in the camelCase draft the format left behind. The deletion surfaced why citing a document's internal numbering from code is a bad trade. APIV2_ADDRESSING.md was retired in a8ded19, and five Go comments went on citing it — including `§7.5a-1`, a section of a file that has not existed for weeks. Ten more cited APIV2.md, which this commit removes. Thirteen dead pointers in total, none of which any reader could have followed. Every one of those comments already stated its rule in words, so the fix is subtraction: the citation goes, the sentence stays. `FoldApiKey is the forgiving-layer fold (APIV2_ADDRESSING.md §7.5a-3): lowercase with `_` and `-` stripped` loses nothing by becoming `FoldApiKey is the forgiving-layer fold: lowercase with `_` and `-` stripped`. Error and warning MESSAGES lose their section references too — 18 of them. The reasoning is the same but sharper: a comment's reader has the repo open, an error's reader does not. SPEC.md ships in this repo, not with the client, so a refusal reaching an authoring agent or a client dialog cited a document its reader could not obtain. Checked one by one, every message already named its rule, so none is poorer for it — two that carried the reference inline rather than trailing were rephrased ("format %d has no §3 name" → "format %d has no name in this format"). Test assertions followed: three asserted on the citation rather than the rule and now assert the words that carry the meaning ("in property_settings", "written in full"); opaque labels like "§11 I1" now read "I1: Marshal never emits what its own Validate rejects". Untouched, deliberately: the ~100 test labels and the code comments that state their rule and THEN cite it. Those follow the rule this commit applies — say the thing, then point — and the pointer is useful to a reader who has the repo. OVERVIEW/PRINCIPLES/PRINCIPLES_SHORT/ANOMALIES stay for now, pending a per-document verdict on what each still holds that SPEC.md does not.
apiObjectKey was minted with `strcase.ToSnake` alone at every site that
writes it, and snake-casing leaves in place every character it does not
understand. Measured over a 38,123-object account, 27 of 1,530 stored api
keys sit outside the grammar the api advertises for a key: the name
`Lists [in work]` minted `lists_[in_work]`, `Manual export & import`
minted `manual_export_&_import`, and `➡️ Medium` minted `[?]_medium` —
the emoji arriving as unidecode's literal `[?]`. All but four are on
options, whose key is derived from the name and nothing else.
The format already had the correct rule and applies it when it derives a
key from a name: derive, then constrain to `^[a-zA-Z0-9_]+$` and a length
bound. The application now uses the same rule. bundle grows the pair as
MintApiSlug (a supplied key) and MintApiSlugFromName (a display name),
beside the halves they compose; nothing about the derive or sanitize
halves changes, so the format side is untouched.
Sites moved onto it:
- objectcreator.injectApiObjectKey — the mint for every type, property
and option created outside the api. It also now stores NOTHING when
the mint comes back empty (a name of only emoji): an object with no
derivable slug is addressed by its internal key, which is a real
address, while an empty apiObjectKey is not.
- createRelationOption's name fallback, which stored the raw
transliteration — spaces and brackets and all.
- The six api sites that take a caller-supplied key: property, type and
tag, create and update. They also had no length bound at all, and
`sanitizedString` on that path was only TrimSpace despite its name.
For a caller-SUPPLIED key the choice was to sanitize or to refuse.
Sanitize, with one exception, because:
- conversion is the advertised contract — the key field on every create
and update request documents that a key "should always be snake_case,
otherwise it will be converted to snake_case", and refusing would
break input that works today (`Due Date` → `due_date`);
- the caller is told what was minted: create and update answer with the
object, whose key field carries the stored key, which is the whole
thing refusal would have bought;
- and the status quo is worse than either, since it accepts the key and
then stores a spelling the same endpoint promised it would not.
The exception is a key nothing survives from (`➡️`, `!!!`, a key in a
script with no letters in the grammar). That is not a conversion — there
is no spelling to convert to — and storing nothing would silently drop a
key the caller explicitly asked for, so those get ErrBadInput naming the
grammar. A supplied key is not transliterated on the way, unlike a
display name: a name has nobody to ask, while answering the key `Задача`
with `zadacha` would name the object something its author never wrote.
Existing stored keys are NOT migrated; this changes what is minted from
here on.
No existing test expectation needed changing — there were no tests for
injectApiObjectKey, and every api test used a key already in the grammar.
The new tests use the corpus's own hostile inputs as fixtures.
audioGenre is renamed "Genre" -> "Audio genre" (the genre relation keeps "Genre"). This one is a genuine prerequisite for spelling bundled keys by their display names: both keys travel in exported documents, and two live wire spellings reading "Genre" would collide the moment the name becomes the address. The space TYPE is renamed "Space" -> "Space settings" (it maps to the Workspace smartblock, i.e. the space's settings object), while spaceView keeps "Space". This pair is hygiene rather than a blocker: measured across the 28,560-document corpus, space/space_view/space_settings/workspace appear as type-key spellings in 0 documents each, so neither name is wire-reachable today. Renaming now keeps the bundled name table injective before a CI rule starts enforcing that. Both renames are user-visible labels.
One uniform rule, decided in NAME_ADDRESSING.md (committed here) and now built: a key's wire spelling is the entity's display name, NFC-normalized, otherwise verbatim — bundled and space-minted alike, in every slot. `createdDate` spells "Creation date", `"type": "page"` becomes `"type": "Page"`, and a property named `#`, `C++` or `Дата выполнения` is a key exactly as written. No derived identifier remains anywhere in the format; `apiObjectKey` is never read (the API surface's key convention is an explicitly separate decision and untouched). What raw naming deletes, it deletes rather than reimplements: the normalization ladder with its empty-normalization fallback and leading-`_` escapes (label.go collapses to "NFC(name), else nothing"; the surviving normalizer serves only the informative `#name` reference suffix and moved to refs.go), the v0.38 wire-alias table (alias.go — the display names carry the rename, the relation TYPE is named "Property"; the sixteen alias spellings are hard-cut, pre-freeze, per §9 Q5), and storeresolver's apiObjectKey read. Collisions are per DOCUMENT, not per space. The census now runs a collision plan (planKeyTerms): a name two censused keys share, or one equal to a censused stored key, degrades EVERY claimant deterministically — stored key when readable, else `<name> (<tail6>)`, else stored key — so a plain spelling is never one of two same-named claimants and the suffix is stable across exports. storeresolver grants shared names to every holder (the space-wide git rule is gone), refuses only a name that is another live entity's stored key, and exposes candidates through the new ScopedKeyVocabulary capability: the importer resolves a shared name within the declared type (1 ambiguous type in 1,753 measured) and raises a loud error asking for the legend when the type is not enough — never a guess, never a phantom while two live properties bear the name. Continuity needs no compatibility table: legacy derived slugs resolve through the extended fold (fold(ToSnake(key)) == fold(key) by construction; the fold also drops spaces and default-ignorable code points), custom legacy spellings through their exhaustive legends. A DENIED key's fold class deliberately answers nothing, keeping `format` and `include_time` legal custom names (the phantom-member warning stays a warning). Three new warnings watch the seams: key-spelling hygiene at Validate (edge whitespace, invisible code points — warn, never trim), and at import the glued-annotation and stale-name-phantom diagnoses, once per term (§9 Q2 decided as every-verbatim-term, deduplicated). The bundled name table is CI-guarded (bundledname_test.go): wire-reachable names unique, invertible, writable, clean; the nine "Underlying file id" transients are the tolerated stripped remainder, and the space/spaceView fold-class overlap is pinned as the one accepted exception (both measured at 0 documents). I1 and I2 are attacked in rawnames_test.go with the names raw addressing newly admits, in-document collisions, and the loud-error paths; golden files regenerate under the new spelling; SPEC.md rewrites §3's authority text and re-derives the manifest/dictionary keying wording (canonical spellings, stated correctly for the first time), with a loud v0.48 changelog entry.
The corpus sweep caught a fixpoint break the unit corpus could not: export writes the attribution lines (creator, lastModifiedBy) and import DROPS them, so a generation-2 census no longer holds them — and in a space with a custom property named "Created by" (a real production space has one, a multi_select beside bundled creator), the custom claimant was suffixed in generation 1 and un-suffixed in generation 2. Every such document reported not_byte_stable. planKeyTerms gains a yielding set: an attribution claimant never contests a spelling. Alone on its name it takes it as before; contested at all it takes its own stored key (always readable — the attribution keys are bundled camelCase) and the normal claimants keep the verdict they will re-derive without it. The plain-name trust property survives: a plain spelling still never denotes one of two claimants that both live in the document's next generation. cmd/anyblockroundtrip moves with it, twice: stripAttribution asks the bundled vocabulary for the attribution spellings instead of restating "creator"/"last_modified_by" (the restated strings silently stopped stripping at the re-spell and every attribution-bearing object reported unstable at once), also matching the stored keys the yield now writes; and it collapses a legend the strip emptied, because in a twin space the attribution spelling owes a legend entry that generation 2 legitimately lacks.
The only invariant break found under sustained attack: a stored dataview filter carrying _filter_template_2_ on a property the same block declares as text EXPORTED with zero warnings and was then REFUSED by this package's own Validate — the rule existed only on the import side, with no export guard, so Marshal emitted what Validate rejects (I1) and one stored filter made the whole object unexportable. Resolved the way the format already resolved the identical tension for the neighbouring date-preset rule: the mismatch is a WARNING. The filter is real stored data the app keeps as UI state; it matches nothing until the property is object- or file-valued, and the warning says so. The fragment surface (UnmarshalFilters) moves with it — the same rule at two severities would let one filter validate on one door and refuse on the other. Pinned with the exact shape the break was found on: the stored pair round-trips, Validate accepts Marshal's output, and the warning names the token and the property.
schema/authoring/object.schema.json's $defs/documentId still banned the pre-v0.45 six reserved bare words, so ValidateAuthoring accepted the ids chat, bin, allObjects and recentOpen — and chat/bin are the two most common listing widgets, so an authored bundle would collide there first. The index authoring schema beside it already held all ten; now both do, and a test pins the full list plus one ordinary id staying legal.
…tems A batch of independent SPEC.md corrections, each verified against the code or the corpus rather than re-derived: - §2c's install-path claims were falsified by our own c86601f: CreateObjectsForExperience applies a bundle's name and icon on a new-space install (setWorkspaceSettings with isBundle=true, gated on isNewSpace), and its Markdown/AI branch calls createWidgets. The "discarded on this path" and "never calls getWidgets or createWidgets" passages described the pre-fix bug as if it were the design. cmd/anyblockconvert/profile.go copied the stale claim and is corrected with it. - The deleted-icon drop is now documented: §9's "references to a tombstone are untouched" gains its one deliberate exception (an icon is optional where a link target is not — ObjectDeletionResolver / DroppedDeletedIconRef, 134 corpus bookmarks measured), and §11's normalization set lists the drop beside the missing-reference rule with the exported predicate the comparator reads. - §13 no longer contradicts itself about the exporter: core/block/export/anyblock SHIPS, wired into the export service's format switch; the core/converter/anyblockjson shim it once promised as follow-up was superseded and never existed. Import remains the follow-up. - headerRelationsLayout occurs on 51 snapshots, not the 0 the §3 justification claimed; the rule (stays a raw number) survives on the corrected evidence, which is now stated with the miscount acknowledged. - Three normative "lowerCamel" survivals now say snake_case, which is what the enums have spelled since §1's rule landed. - §15 stops listing as open what the body settled: #6 (icon block, mooted by the v0.39 lift), #7 (type_properties naming, settled by v0.32 as type_settings/property_definitions), #8 (property documents — §2d and §2f are that section). - Every dangling section reference is repaired: the §11.1/§11.2/§11.3 and §7.5a/§7.1–§7.3 pointers land on the sections that exist today, and the citations of the deleted APIV2.md / APIV2_REVIEW_SMALLMODEL.md (removed in 8fba18c) now describe the retired design record without pointing at files that are not there.
The study is a decision record, and the build made four calls it did not: the author-directed Space rename (space -> "Space settings", with the fold class consequence pinned in CI), the attribution-yield rule the corpus sweep forced on the per-document ladder, and the answers to the two open questions the study left the author (alias spellings: hard cut; warning scope: every verbatim term, deduplicated).
The collision plan modelled ONE written-then-dropped population — the attribution keys, which export writes and import drops, and which yield rather than contest. buildProperties drops on four more predicates after the census has already counted the key: a type document's install provenance, a participant's load timestamp, an admitted system-stamped key whose value is empty, and a name-over-number key holding a string its vocabulary cannot name. A key dropped by any of those four is written NOWHERE, yet it still claimed a spelling and still reserved its own stored key as one. So it degraded its rival in generation 1, and generation 2 — which no longer holds the key at all — spelled that rival plainly. Export stopped being a fixpoint. Reproduced on all four; the shortest is `isHidden: false` beside a custom property named "Hidden", which writes "Hidden (b90aa1)" once and "Hidden" the next time. The four predicates now live in one place, droppedPropertyKey, which the emit and the census both ask — the census silently, so the one arm that warns still fires exactly where the value is lost. A dropped key that a BLOCK also names stays censused: the block really does spell it, and the block survives the drop. Not a yielding claimant, which is the weaker remedy: a yielding key still occupies a member of the document it is written into, so it still takes an uncontested spelling. A dropped key occupies nothing.
…y key
JSON Schema matches a member name literally; the format resolves a property
key case- and separator-insensitively. A rule written as a literal over a
key the codec spells many ways is not a narrower rule, it is a rule with
holes in it, and both authoring rules that lived in the schema had one.
The type-document rule REFUSED the canonical spelling. `properties:
{required: ["name"]}` accepted `{"name": "Habit"}` and rejected `{"Name":
"Habit"}` — the spelling every exporter writes and every example shows.
Swapping the literal would have moved the hole onto the lowercase form,
which the codec accepts just as happily. The rule now runs in a semantic
pass on the resolved key: any spelling that resolves to the name property
satisfies it, and a type document with no name at all still fails.
The derived-key list lost its teeth for nine keys. The schema bans the
pre-raw-name spellings, so `creator`, `createdDate`, `lastModifiedBy`,
`lastModifiedDate`, `addedDate`, `revision`, `internalFlags`,
`featuredRelations` and `isArchived` were still refused — while "Created
by", "Creation date" and the rest sailed through. Those nine are exactly
the keys the full format DROPS rather than refuses, so nothing downstream
caught them: an author's value disappeared without a word where it used to
be refused at authoring time. The remaining twenty-six list entries were
never at risk; Validate refuses them first.
So the nine live in Go now, as stored keys, enforced on the resolved key
through the same chain Validate's own admission loop uses. The schema keeps
its literal list — it is what an agent actually reads — with both spellings
carried, and a test pins the two together in both directions so neither can
rot again. The `layout_align` narrowing had the same defect and gets the
same treatment.
The semantic pass runs before the schema pass so the better message wins:
"'not' failed" does not tell an author which key they wrote or why the app
owns it.
authoredKey — the identity a property-definition entry states when it gives only a `name` — still ran the api-slug derivation, which is the one derived identifier left in a format that says it has none. The schema had already been rewritten to say "a name alone is enough: the name IS the spelling"; the code disagreed. It did not merely rename. It transliterated, and then truncated: "Cooking Time" arrived as `cooking_time` (no longer the name any resolver holds), "Тоггл" as `toggl`, "作業内容" as `zuo_ye_nei_rong`, "C++" as `c`, and "☕" and "#" as the empty string, which the seam then refused as an unwritable key. All nine names are legal spellings now, and each is its own address. NFC and otherwise verbatim, like every other key slot. No length bound is imposed at the derivation any more: the bound belongs to the spelling, and both callers already refuse a resolved key that cannot be written — at the entry's own JSON pointer, which is a better report than a silently truncated term. The old bound was the object-ref length, 255, on a term the schema bounds at 128; there is one bound now, asked in one place. The two legend descriptions in the published schema still called a spelling a slug. They say spelling.
FoldKeyTerm ran NFC first and dropped `_`, `-` and whitespace second, which made it non-idempotent on its own output. Dropping a separator puts two runes next to each other that were not neighbours before, and a composable pair only composes when it is adjacent: "A_" followed by a combining acute folded to a decomposed `á`, while the precomposed "Á" folded to U+00E1. Two spellings a reader calls one word, in two fold classes. Normalizing again after the map puts every result in one form.
Three defects with one shape: the importer reads the LENGTH of a candidate list as "how many live entities answer to this spelling", and three places could put one entity in a list twice. Two candidates is a hard refusal, so a bookkeeping repeat made Unmarshal refuse a document Marshal had just written. `grant` appended to keysByLabel without the first-wins guard its neighbours all carry — keyById, idByKey, propertyIdsByKey here, relKeyToId one file over, and GetRelationByKey answering a duplicated relationKey with records[0]. A stored key is read off the `relationKey` DETAIL rather than off the row identity, so a legacy row and its derived twin are two rows and one entity; the same shape reaches the type namespace through uniqueKey. Latent but reachable. TypePropertyKeys had it independently and does NOT need duplicate rows: it concatenates the type's four recommended lists, nothing declares them disjoint, and a property listed as featured AND ordinary made the type unable to single out its own property — the exact opposite of what the type-scoped resolution exists for. That one is live. So: dedup where the maps are built, dedup again where the answer is assembled, and count DISTINCT keys at both importer sites, because the refusal is the expensive end. The interface doc now says the list is a set, sorted; it never did. The ambiguity refusal also reported at /property_internal_keys — which reads as a pointer and is not one, and names a member that is ABSENT in exactly the documents that trigger the refusal, since the legend is what would have resolved it. Each caller passes the slot that spelled the term. The type namespace already did this correctly and is unchanged. The test double the whole TestRawNames_ suite runs against was weaker than the shipped vocabulary — no dedup, and a fall-through to the bundled table where storeresolver refuses — so the suite was blind to all of the above. It now mirrors storeresolver method for method, including grant's degradation ladder. That ladder's rung for "a name that is another live entity's stored key" turned out to be unreachable through the old double despite the file claiming it as coverage; it has a test now. Also: the list form of the importer's property-key inversion had no callers (link blocks go through propertyKeysAt), and a property-definition entry's ambiguity now reports at the definition's own pointer rather than at /properties.
Deleting the v0.38 alias table put "Relation" back on the wire. That table
respelled sixteen `relation*` keys as `property_*`; five of the sixteen are
carried by a display name that already said "property", and the other
eleven were still NAMED "Relation …" in the bundle. Measured on native
output over the 77-space corpus: "Relation key" on 4,009 documents,
"Relation option color" on 2,716, "Relation value is readonly" on 311,
"Recommended relations" on 165, "Featured Relations" on 117, "Header
relations layout" on 92 — plus the casing split, one concept spelled
"Featured Relations" beside "Recommended relations".
So the eleven are renamed, in sentence case: relationKey, relationOptionColor,
relationReadonlyValue, relationFormatObjectTypes, featuredRelations,
headerRelationsLayout and the four recommended-list keys, plus the
`relationOption` TYPE. All ten relations are hidden, so no user-facing label
moves. The bundled name table stays injective under the fold; the only
residual collision is the deliberate pre-existing "Underlying file id" nine.
A side effect worth having: the deleted alias spellings resolve again, with
no compatibility table. `property_option_color` folds onto "Property option
color" because the fold strips case and separators, so both the old alias
and the derived-slug shape of every new name land in the right class.
**A rename that cannot propagate is worse than no rename**, and two already
could not. The reviser early-returns unless the bundled revision exceeds the
local one, and NOTHING here carried a revision bump: the `space` type's
"Space" -> "Space settings" was written three commits ago and stayed at
revision 3, and `audioGenre`'s "Genre" -> "Audio genre" could not propagate
at any revision, because a non-system relation was unreachable to the
reviser entirely. That one was measured emitting `{"property": "Audio
genre", "internal_key": "audioGenre", "name": "Genre"}` in 76 of 77 spaces —
addressed by one string, named by another, and shown to users under the old
name.
Every renamed system relation and both types get a revision, which is
sufficient: system objects are derived into every space at creation and the
system path applies Name unconditionally. The two NON-system relations
needed a path, and adding them to systemRelations.json was declined — that
list means "business logic depends on this", and membership makes an object
undeletable and derives it into every space, which is a large change to buy
a name fix. Instead the reviser now answers for any bundled relation, with
a filter set of exactly {Revision, Name} on the non-system arm, and applies
the name only when the local one is still a previous bundled name. A user
who renamed their own installed relation keeps their name; the revision is
stamped either way, so nothing is revisited.
The guard needs the previous name, which the bundle does not store, so the
reviser keeps an explicit table of bundled renames. docs/Flow.md documents
appending to it as the third step of renaming a non-system relation, and
its reviser link was pointing at a path that has not existed for some time.
Four non-system bundled relations now fall in scope of the revision check;
two are the targets and two (`author`, `assignee`, at revision 1 from an
old edit) take a one-time revision stamp and no name change.
The writer maps every invalid byte to U+FFFD; the collision plan compares raw Go strings. So two display names differing ONLY in their invalid bytes looked distinct to the plan, took no suffix, and then rendered as one member name. A JSON object holds a member once: one value silently replaced the other, and Validate passed, because by the time it read the document the collision had already happened. Reproduced — both values gone, document valid. Folding it into the collision plan instead was weighed and does not work. The plan would have to compare the RENDERED forms, which are equal, so there is no spelling it could hand either claimant. The honest answer is that a name whose bytes cannot be written is not a spelling, which is what the writable-key rule already says about an empty key and a control character; both claimants now fall back to their stored key, which is always its own address, and both values survive. The rule lives with the other writable-key rules rather than in the schema, because the schema cannot state it: a parsed JSON document always holds valid UTF-8, the decoder having already replaced anything else, so only export can break it. The retired normalization grammar dropped U+FFFD as a matter of course; the exposure arrived with raw names. Zero occurrences in the corpus — this is hardening. Also: headerRelationsLayout was recorded as having ZERO occurrences in the argument for leaving five bare-integer enums unnamed, in four places. It is on 51 documents and holds two distinct values (44 ones, 7 zeros) — which is what typesettings.go already said about it one file over. The verdict does not change, but its ground does: leaving it bare now rests on volume alone, 51 against imageKind's 4,079, and it is the weakest of the five on that account. Said so where the claim is made.
Four corrections, one dead pointer pair, and one rule the refactor changed
without anybody writing it down.
**"Every claimant degrades" needed its carve-outs.** A claimant is a key the
document actually WRITES, and two populations are not that. A key the
`properties` emit drops is written nowhere, so it claims no spelling and
reserves no stored key — the census fix. The attribution keys are the
opposite: export writes them and import drops them, so they yield rather
than contest. Both exist for one reason, now stated: a claimant that will
not be there next time must not decide anybody else's spelling.
**"Anywhere else is a validation error" is a warning.** A filter template on
a non-object property was a refusal on the import side with no export guard,
so Marshal wrote what Validate rejected and one stored filter made a whole
object unexportable. Both doors warn.
**The compact filter grammar's bare-key rule needed restating.** It used to
be a stronger claim than it is: when a spelling was a normalized slug, the
slug was minted THROUGH the grammar's identifier predicate, so everything
the format could write this grammar could parse. A spelling is a raw display
name now and names carry spaces. Nothing became unreachable — resolution
folds away case and separators, so bare `due_date` addresses "Due date" and
`Дата_выполнения` addresses "Дата выполнения" — but a name no identifier
folds onto ("C++", "50% done", a name colliding with a keyword) has no
compact form, and the parser already says so and names the structured
filters array. Recorded in the spec and in the predicate's own comment
rather than left as a claim that had quietly stopped being true.
**Examples and prose still spelled retired slugs.** The well-known-property
table now shows spellings beside stored keys; the attribution example, the
dictionary's `installed` list and the loose-value example show what export
writes; and the prose that names stored keys names them (`lastModifiedBy`,
`isFavorite`, `createdDate`), not a snake_case form that is neither key nor
spelling. Changelog entries keep the spelling of their own release.
Two citations survived the file they cited: `docs/AgentApiV2Research.md` and
`PREFREEZE_REVIEW.md` were deleted deliberately, and the sentences that
pointed at them now carry the substance instead.
Also records the two authoring-subset rules that moved out of the schema
onto the resolved key, and why they could not stay literals.
`testdata/authoring/habit_tracker/` is what the spec calls the bundle an authoring agent should be shown first, and it was still written in the retired slug spelling: `last_done` where the property is named "Last done", `name`/`description`/`is_favorite` where the format writes "Name", "Description" and "Favorited", `"type": "page"` where the bundled type is spelled "Page". Its own dictionary schema had already been rewritten to say `property` is "the property's spelling — its display name, written exactly: \"Streak\", \"Last done\"", so the worked example contradicted the schema it declares. The dictionary entries now state a `name` and nothing else, which is the shortest true statement of the rule: the name IS the spelling, and every other file in the bundle references the property by that name. The published authoring schema said "keyed by property key: `name` (the title)" and offered `"page"`, `"task"`, `"collection"` as type examples; both now show the spelling a reader will actually meet, and say that resolution folds case and separators so the stored key reaches the same place. The index schema described the resolution chain as "the shipped ladder", a mechanism that no longer exists, and called a widget's shown properties keys when they are spellings. The bundle-coherence test skipped the built-in type by comparing the term to the literal `page`. It asks the ladder now — a term any bundled type answers to is a built-in and needs no declaration — so the test does not have to be edited every time an example changes its spelling. Also repoints a dead §3.6 citation in the published object schema at the section that carries the one-shape rule today.
Making the bundled renames reachable gave the reviser twelve pending objects in every space, and in a SUBSCRIBED space this account may only read, all twelve failed the same way. Measured on one such space in a 155-space account: twelve "insufficient permissions" errors on every space load, joined into one multi-line ERROR, and — worse — thirteen documents whose two consecutive exports differed. The thirteen are the real cost. An unwritable change is applied to the LOADED document before the push is refused, so between the refusal and the next eviction the space reports a value that will never persist: the type that had just been renamed read "Space settings" on one export and "Space" on the next, and seven property documents appeared in one export and not the other. A reader of a shared space was seeing the owner's objects under names the owner had not given them. One refusal answers for the whole space — the verdict belongs to the ACL, not to the object — so the pass stops at the first, and reports a skip rather than a failure. A reader genuinely cannot revise the system objects of a space they do not own; the owner's client revises them and the change syncs. Saying so once per load is the whole of what is true. This is not new behaviour, only newly reachable: any revision bump has always done this in a read-only space, and there was simply nothing pending before. What remains is one attempted write per read-only space per load, because nothing short of attempting it tells us the answer — the source's own ReadOnly() is hardcoded false on tree sources and the restriction layer takes no ACL input, so there is no permission to read up front.
Six documents, one class of defect: each stated something the code stopped doing and nobody re-read them. **"`relation` appears nowhere in the format" was false in four of them.** Renaming the eleven bundled names makes most of it true, and the rest is worth stating precisely instead of absolutely: the format's own vocabulary — member names, kinds, block types, every bundled display name — no longer says the word, but a document records a STORED key verbatim wherever fidelity demands an identity rather than a name (the envelope `internal_key`, the legend values), and user data is user data. A property someone named "Relation" is their word, carried. **OVERVIEW.md had not been touched by the raw-name refactor at all.** Its flagship example still showed the retired spelling, its decision 3 said properties are "addressed by key", and its file table pointed at a document that was deleted. Refreshed end to end against what the exporter produces. **The key bound is counted in characters, not bytes** — three sites. **NAME_ADDRESSING.md had drifted from itself.** Its §5.2(ii) ruled the property-vocabulary renames unnecessary on a lift that covers three keys, not eight, naming a `max_values` member that does not exist; §5.5 said an unresolvable term is an error where the importer stores it verbatim with a warning; §9 recorded the Space rename in the wrong direction; a §5.x cross-reference was off by one; "§9 Q5" was Q1; and `alias.go` was described as rewritten when it was deleted. The addendum grows from five implementation calls to nine — the dropped-key carve-out in the collision ladder, the last derived identifier, what a bundled rename needs to reach existing accounts, and what happens when the reviser meets a space this account may only read. Also, found while verifying: three more dead file pointers, a claim in ANOMALIES that no golden carries a legend (all four do now), a PRINCIPLES example resting on an object legend that was deleted, and a SPEC version in a header eleven drafts old.
… prose Three cleanup passes over the Go, all of them consequences of the raw-name refactor having outrun its own commentary. **59 message strings lose their section citations, and none remain.** A prior commit stripped 18 of about 64 on the principle that the person reading a refusal has no copy of the spec, so a section number costs them the words that would have helped. Every removal states the rule instead: the `id`/`type` refusal now says those are the envelope's own members, the deny-rule warning says import refuses the internal keys export strips, and the batch checker's dictionary-wins message says the dictionary is the file that defines the property. Test assertion labels keep their citations — their reader does have the repo — and the two assertions that matched message TEXT moved with the messages. **Twelve citations pointed at sections that no longer exist.** §4.4/§4.5 became §7a, §7.5a-5 became the resolution chain's own step, §8.38 and §8.41 became the rules they named, and the four §7.5a-1 citations on the api-slug surface lost theirs entirely: that material describes the API's own key convention now, so no spec section is the right target. The chain-step numbering was also off by one in several comments — verbatim is step 5, the fold is step 4 — and in three places in the spec itself. **The prose still described the retired ladder as live**, most heavily in `cmd/internal/anyblockbatch`, whose code was correct and whose commentary was written entirely in a vocabulary the format no longer speaks. `TestDocumentSpellsSlugs` is `TestDocumentSpellsNames`. What survives the sweep is what is still true: `PropertySlug`/`TypeSlug` keep their names on purpose, the api-slug surface is real, a bundle-local id is a slug in a different sense, and the collision LADDER is a live mechanism with three rungs — only the resolution chain stopped being one.
… itself
Pre-freeze pass over the format, its specification and its authoring surface.
**Format version 2.** Every pre-freeze draft carried version 1 while the
grammar moved under it, so a draft and a frozen document were
indistinguishable. The integer moves once at the freeze and 1 is refused
outright at the version gate, naming re-export as the repair — not migration,
which no single pre-freeze grammar could define, but a clean refusal in place
of a silent misread. The hand-rolled `{"type":"template"}` check dies with it:
the version gate now catches every pre-freeze shape, where that check caught
one by hand. Its `template_for` guard was never a rule, only de-duplication of
two contradictory repair messages, so it goes too.
**Four defects, each with tests that can fail:**
- the fragment surface ran no schema validation at all, so `UnmarshalFilters`
and `UnmarshalSorts` silently dropped unknown members that a whole document
refuses — the one place the validate-before-decode discipline did not hold,
on the surface most exposed to careless callers
- NFC normalization was missing on the read path, so two byte-distinct
spellings of one name imported as two visually identical properties
- property-key slots outside `/properties` were unbounded: a megabyte key and
raw control bytes round-tripped through all seven
- the widget `limit` bound was inlined in two schemas and had already drifted;
the cap turns out to be deliberate — the fallback design requires the
asymmetry — so it keeps its value and gains one home
**The spec says what the format does, not what the spec used to say.** The
version changelog is gone (1,571 lines), §15 is a decisions ledger rather than
a list of open questions, and no pre-release version number survives anywhere.
What remained was narration of the document's own revision history; the rules
that narration carried stayed, only their dates went.
**Gaps closed rather than trimmed.** §10 states the unknown-input policy the
format had always followed but never named: closed slots refuse, open slots
degrade, and a new field's author says which it joins. §3 gains the seventh
named-enum key it claimed six of. §13 documents `ScopedKeyVocabulary`,
`ObjectDeletionResolver` and the seven exported drop predicates it had left
out — the API contract external implementers build against.
Also: `NAME_ADDRESSING.md` deleted, its decision now living in §3; the
vocabulary lineage citations dropped; `anyblockroundtrip` derives its volatile
date members from the bundled table instead of hardcoding retired slugs, which
had silently stopped matching; and two handoffs recording what the pre-freeze
audits found and what the authoring-mistake catalogue should test next.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| for _, k := range nonCanonical { | ||
| claims[nfcTerm(k)]++ | ||
| } | ||
| out := make(map[string]V, len(m)+len(nonCanonical)) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces AnyBlock JSON — a readable, strictly-validatable JSON representation of an Anytype object — plus the tooling to convert, validate and verify it. It is intended to replace
.pb.json(jsonpb ofSnapshotWithType) as the export/import format, and it is the document shape API v2 will serve and accept (that work is stacked on top of this branch).The format specification's public home will be anyproto/any-block.
pkg/lib/anyblockjson/SPEC.mdis the normative version this implementation is built against; it should move there rather than being maintained in two places.Nothing here is wired into an existing import/export path — the package, the schema and the CLIs are additive, and no shipped behaviour changes.
The decisions worth reviewing
Each is argued at length in
SPEC.md; the short version, because the reasoning is what deserves scrutiny rather than the field names:indent, not a nested tree. A nested tree needs a recursive schema, and a recursive schema cannot be used with constrained decoding or provider strict mode — which is the mechanism that rescues small models. Truncation also degrades to a valid prefix rather than unparseable output. The cost is honest: an off-by-oneindentmis-parents a block where a misplaced bracket would have been a parse error, so import enforces strict monotonicity with a documented CommonMark-style lenient clamp.text, not UTF-16 mark ranges. Models cannot maintain offset bookkeeping; markdown keeps formatting where the formatting is.false,0,"",[]. This one was decided by data: the first production sweep flagged 14,032 "issues" that were all default scalars dropped by canonicalization, and the ruling was that a user setting a property to empty is a fact.relation→property,smartBlockType→kind, REST-API format names. "Relation" appears nowhere in the format.Import(Export(S)) ≡ N(S), withExport∘Importidempotent and byte-stable, whereNis a documented normalization.Verified against real data
cmd/anyblockroundtripruns the contract against a real account — export every object of every space to pb, convert pb → AnyBlock → pb, and check §11. The last full sweep covered 35,369 objects at 99.86%, with every remaining failure traced and fixed until only accepted anomalies were left. It caught two silent data-loss bugs no unit test had.Every real-data oddity found along the way is written up in
pkg/lib/anyblockjson/ANOMALIES.mdrather than smoothed over.What's included
pkg/lib/anyblockjson— export, import, inline mark codec, tables, dataview, type documents, validation, the hand-authored JSON Schema (2020-12).cmd/anyblockconvert,anyblockvalidate— convert and check documents in batch.cmd/anyblockroundtrip— the production sweep above.cmd/anyblockrecover— rebuild source documents from snapshots.cmd/anyblockinstall— bundle → live space.pkg/lib/anyblockjson/schema/{object,index}.schema.json.Regenerating
go test ./pkg/lib/anyblockjson/... -updatetypebefore validating a block) so errors read asat /blocks/7/columns: a table requires columnsrather than "does not match any of 23 schemas".ANYTYPE_MNEMONIC=… go run ./cmd/anyblockroundtrip -root <copy-of-data-dir>. Run it against a copy of the data dir — the repo is single-process locked.Note for reviewers
Three planning documents for the stacked API v2 work landed on this branch by sequence rather than by topic:
core/api/APIV2.md,core/api/APIV2_REVIEW.md,core/api/APIV2_REVIEW_SMALLMODEL.md. They describe the consumer, not the format. Left in place rather than rewriting history; they are superseded by the versions on the API v2 branch.