-
Notifications
You must be signed in to change notification settings - Fork 23
feat: produce and transport resolved symbol tables from Python #332
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -269,6 +269,45 @@ impl PySerializedSymbolTable { | |
| } | ||
| } | ||
|
|
||
| /// Build a ``SerializedSymbolTable`` from its JSON transport | ||
| /// text, as produced by ``to_json_str``. | ||
| /// | ||
| /// This is the inverse of ``to_json_str`` and exists for callers | ||
| /// that carry the transport form across a process or service | ||
| /// boundary — a scheduler persisting the table produced by | ||
| /// ``create_job`` and later handing it to a worker, for instance. | ||
| /// Prefer ``from_symtab`` when the source is an in-memory | ||
| /// ``SymbolTable``. | ||
| /// | ||
| /// Raises ``ValueError`` if the text is not valid JSON. Note that | ||
| /// the *contents* are validated lazily: a well-formed JSON | ||
| /// document whose entries are not valid symbol table entries is | ||
| /// accepted here and rejected by ``to_symtab``. | ||
| #[classmethod] | ||
| fn from_json_str(_cls: &Bound<'_, pyo3::types::PyType>, json: &str) -> PyResult<Self> { | ||
| let inner = openjd_expr::SerializedSymbolTable::from_json_str(json).map_err(|e| { | ||
| pyo3::exceptions::PyValueError::new_err(format!( | ||
| "Failed to parse SerializedSymbolTable JSON: {e}" | ||
| )) | ||
| })?; | ||
| Ok(Self { inner }) | ||
| } | ||
|
|
||
| /// Serialize to the JSON transport text: an array of | ||
| /// ``{"name", "type", "value"}`` objects in canonical | ||
| /// (lexicographic) path order. | ||
| /// | ||
| /// Use this to move a table across a process or service boundary; | ||
| /// pair it with ``from_json_str`` to reconstruct. The result is | ||
| /// stable for a given table, so it is safe to store or compare. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The two guarantees documented here — an array of name/type/value objects "in canonical (lexicographic) path order", and "stable for a given table, so it is safe to store or compare" — only hold for instances built via
Since the docstring actively invites callers to store and compare the text, consider scoping the claim (e.g. canonical order and byte-stability hold for tables built via |
||
| fn to_json_str(&self) -> PyResult<String> { | ||
| serde_json::to_string(&self.inner).map_err(|e| { | ||
| pyo3::exceptions::PyValueError::new_err(format!( | ||
| "Failed to serialize SerializedSymbolTable: {e}" | ||
| )) | ||
| }) | ||
| } | ||
|
|
||
| /// Deserialize this serialized symbol table into a full | ||
| /// ``SymbolTable`` suitable for inspection or modification. | ||
| /// ``path_format`` controls how PATH-typed values are | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,13 +1,15 @@ | ||
| # Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
|
|
||
| import json | ||
| import pickle | ||
| from pathlib import Path | ||
| from typing import cast | ||
|
|
||
| import pytest | ||
|
|
||
| import sys | ||
|
|
||
| from openjd.expr import ExprValue, SymbolTable, TypeCode | ||
| from openjd.expr import ExprValue, SerializedSymbolTable, SymbolTable, TypeCode | ||
| from openjd.expr import PathFormat | ||
|
|
||
| HOST_PATH_FORMAT = PathFormat.WINDOWS if sys.platform == "win32" else PathFormat.POSIX | ||
|
|
@@ -254,3 +256,125 @@ def test_contains_namespace(self) -> None: | |
| assert "Param" in st | ||
| assert "Param.X" in st | ||
| assert "Other" not in st | ||
|
|
||
|
|
||
| class TestSerializedSymbolTableJson: | ||
| """``to_json_str`` / ``from_json_str`` are the supported way to move a | ||
| serialized symbol table across a process or service boundary.""" | ||
|
|
||
| def test_round_trip_preserves_values(self) -> None: | ||
| # GIVEN | ||
| symtab = SymbolTable( | ||
| { | ||
| "Job.Name": "my-job", | ||
| "Step.Name": "render", | ||
| "Param.Count": 42, | ||
| "Param.Scale": 1.5, | ||
| "Param.Debug": True, | ||
| } | ||
| ) | ||
| serialized = SerializedSymbolTable.from_symtab(symtab) | ||
|
|
||
| # WHEN | ||
| json_text = serialized.to_json_str() | ||
| restored = SerializedSymbolTable.from_json_str(json_text).to_symtab() | ||
|
|
||
| # THEN | ||
| assert restored["Job.Name"] == ExprValue("my-job") | ||
| assert restored["Step.Name"] == ExprValue("render") | ||
| assert restored["Param.Count"] == ExprValue(42) | ||
| assert restored["Param.Scale"] == ExprValue(1.5) | ||
| assert restored["Param.Debug"] == ExprValue(True) | ||
|
|
||
| def test_transport_shape(self) -> None: | ||
| """The transport form is an array of {name, type, value} objects in | ||
| canonical path order, with scalars carried as strings.""" | ||
| # GIVEN | ||
| symtab = SymbolTable({"Job.Name": "my-job", "Param.Count": 42}) | ||
|
|
||
| # WHEN | ||
| entries = json.loads(SerializedSymbolTable.from_symtab(symtab).to_json_str()) | ||
|
|
||
| # THEN | ||
| assert entries == [ | ||
| {"name": "Job.Name", "type": "string", "value": "my-job"}, | ||
| {"name": "Param.Count", "type": "int", "value": "42"}, | ||
| ] | ||
|
|
||
| def test_to_json_str_is_stable(self) -> None: | ||
| # GIVEN | ||
| symtab = SymbolTable({"Param.B": 2, "Param.A": 1}) | ||
|
|
||
| # WHEN | ||
| first = SerializedSymbolTable.from_symtab(symtab).to_json_str() | ||
| second = SerializedSymbolTable.from_symtab(symtab).to_json_str() | ||
|
|
||
| # THEN | ||
| assert first == second | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This asserts determinism (same input, same output twice in one process) but not the canonical-ordering property the docstring actually promises. Both To pin down "canonical (lexicographic) path order" — and catch nondeterminism from the underlying first = SerializedSymbolTable.from_symtab(SymbolTable({"Param.B": 2, "Param.A": 1})).to_json_str()
second = SerializedSymbolTable.from_symtab(SymbolTable({"Param.A": 1, "Param.B": 2})).to_json_str()
assert first == second(Python dict literals preserve insertion order, so these two do reach the Rust side in different orders.) |
||
|
|
||
| def test_empty_table_round_trips(self) -> None: | ||
| # WHEN | ||
| json_text = SerializedSymbolTable.from_symtab(SymbolTable()).to_json_str() | ||
|
|
||
| # THEN | ||
| assert json_text == "[]" | ||
| assert SerializedSymbolTable.from_json_str(json_text).to_symtab().symbols == set() | ||
|
|
||
| def test_from_json_str_accepts_hand_built_transport(self) -> None: | ||
| """A caller that builds the transport form itself, rather than going | ||
| through ``from_symtab``, gets the same result.""" | ||
| # GIVEN | ||
| hand_built = '[{"name": "Job.Name", "type": "string", "value": "hand-built"}]' | ||
|
|
||
| # WHEN | ||
| symtab = SerializedSymbolTable.from_json_str(hand_built).to_symtab() | ||
|
|
||
| # THEN | ||
| assert symtab["Job.Name"] == ExprValue("hand-built") | ||
|
|
||
| def test_from_json_str_rejects_malformed_json(self) -> None: | ||
| # WHEN / THEN | ||
| with pytest.raises(ValueError, match="Failed to parse SerializedSymbolTable JSON"): | ||
| SerializedSymbolTable.from_json_str("not json at all") | ||
|
|
||
| def test_from_json_str_defers_content_validation_to_to_symtab(self) -> None: | ||
| """Well-formed JSON that is not a valid table is accepted by | ||
| ``from_json_str`` and rejected by ``to_symtab``.""" | ||
| # GIVEN | ||
| well_formed_but_wrong = '{"not": "an array"}' | ||
|
|
||
| # WHEN | ||
| serialized = SerializedSymbolTable.from_json_str(well_formed_but_wrong) | ||
|
|
||
| # THEN | ||
| with pytest.raises(ValueError, match="expected JSON array"): | ||
| serialized.to_symtab() | ||
|
|
||
| def test_json_round_trip_matches_pickle_round_trip(self) -> None: | ||
| """The JSON form carries the same content as the pickle form, which | ||
| round-trips through the same transport text.""" | ||
| # GIVEN | ||
| symtab = SymbolTable({"Job.Name": "my-job", "Param.Count": 42}) | ||
| serialized = SerializedSymbolTable.from_symtab(symtab) | ||
|
|
||
| # WHEN | ||
| via_json = SerializedSymbolTable.from_json_str(serialized.to_json_str()).to_symtab() | ||
| via_pickle = pickle.loads(pickle.dumps(serialized)).to_symtab() | ||
|
|
||
| # THEN | ||
| assert via_json.symbols == via_pickle.symbols | ||
| for name in via_json.symbols: | ||
| assert via_json[name] == via_pickle[name] | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This test is circular and cannot fail. If the intent is to guard against pickle and JSON diverging in the future, comparing the two text forms directly would be a tighter check: assert pickle.loads(pickle.dumps(serialized)).to_json_str() == serialized.to_json_str()Otherwise dropping it leaves nothing untested — |
||
|
|
||
| def test_path_values_round_trip_with_host_format(self) -> None: | ||
| # GIVEN | ||
| symtab = SymbolTable({"RawParam.Scene": "/proj/scene.blend"}) | ||
| json_text = SerializedSymbolTable.from_symtab(symtab).to_json_str() | ||
|
|
||
| # WHEN | ||
| restored = SerializedSymbolTable.from_json_str(json_text).to_symtab( | ||
| path_format=HOST_PATH_FORMAT | ||
| ) | ||
|
|
||
| # THEN | ||
| assert "RawParam.Scene" in restored | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This test does not exercise PATH-typed values, so it cannot catch a regression in path serialization.
ExprValue(str(input_file), type="path", path_format=HOST_PATH_FORMAT)Separately, the only assertion is membership ( |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Design note on the lazy-validation choice: for the use case the docstring names — a scheduler persisting the transport text and later handing it to a worker — deferring content validation to
to_symtabpushes the error to the worst possible place. The scheduler happily accepts and stores a bad payload, and the failure surfaces later on the worker, far from the input that caused it.Validating structure at ingest (i.e. parsing into the real entry type rather than a loose JSON document) would make
from_json_stra real trust boundary and give the error at the point where the caller still has the offending text in hand. If the laziness is deliberate — e.g.openjd_expr::SerializedSymbolTable::from_json_strintentionally keeps the raw document so unknown future entry kinds pass through untouched — it would help to say so in the docstring, since as written it reads as an accident rather than forward-compatibility.