Implement default values for references

README-extensions.md

@@ -0,0 +1,161 @@
+# reclass-rs Extensions
+
+Initially, reclass-rs only implemented features present in [kapicorp-reclass], including extensions introduced by kapicorp-reclass.
+This document covers extensions that are currently unique to reclass-rs.
+
+## Non-compatible `compose_node_name` option
+
+Reclass-rs supports the `compose_node_name` option.
+By default, in contrast to kapicorp-reclass, reclass-rs preserves literal dots in composed node names.
+For example, given the following inventory, the node's internal path is parsed as `['path', 'to', 'the.node']`:
+
+```
+.
+├── classes
+│   ├── cls1.yml
+│   └── cls2.yml
+└── nodes
+    └── path
+        └── to
+            └── the.node.yml
+```
+
+However, optionally, reclass-rs can be configured to handle `compose_node_name` the same way that kapicorp-reclass does, by naively splitting node names on each dot.
+To enable this compatibility mode, set `compat_flags: ['ComposeNodeNameLiteralDots']` in your inventory's `reclass-config.yml`.
+In compatibility mode, the node's internal path for the previous inventory is `['path', 'to', 'the', 'node']`.
+
+## Verbose warnings
+
+Reclass-rs supports boolean config option `verbose_warnings`.
+When verbose warnings are enabled, reclass-rs produces the following informational messages on standard error:
+
+* warnings when dropping unrendered values which could potentially contain missing references.
+* informational messages when replacing a missing reference with a default value.
+
+## Default values for references
+
+> [!IMPORTANT]
+> This feature is currently experimental and may change at any time.
+
+Reclass-rs supports specifying default (fallback) values for references.
+The format for specifying a default value is
+
+```
+${some:reference:path::the_default_value}
+```
+
+We chose `::` as the separator between the reference path and the default value because `::` currently can't appear in references in valid kapicorp-reclass inventories (due to [kapicorp/kapitan#1171](https://github.com/kapicorp/kapitan/issues/1171)).
+
+Reclass-rs first resolves nested references in the reference path and default value and then splits the reference contents into the path and default value.
+References with default values can be used everywhere that references are supported, including class includes.
+
+> [!NOTE]
+> Currently, default values are only applied once all nested references have been successfully resolved.
+> A missing nested reference without its own default value will result in an error even if the top-level reference specifies a default value.
+
+For further processing, the default value is parsed as YAML.
+Incomplete YAML flow values always raise an error (also for existing references that specify an incomplete YAML default value).
+
+> [!NOTE]
+> While it's generally possible to specify arbitrarily complex YAML default values inline, we recommend specifying complex default values through a nested reference.
+> When providing complex default values inline, it may be necessary to carefully escape characters of the YAML value in order to ensure the reference parser keeps the YAML value intact.
+
+### Example
+
+```yaml
+# nodes/test.yml
+classes:
+  - class1
+  - class2
+
+parameters:
+  _base_directory: /tmp
+```
+
+```yaml
+# classes/class1.yml
+parameters:
+  helm_values:
+    a: a
+    b: b
+```
+
+```yaml
+# classes/class2.yml
+parameters:
+  _compile:
+    jsonnet:
+      - input_paths:
+          - ${_base_directory}/foo.jsonnet
+        type: jsonnet
+        output_path: foo
+    helm:
+      - input_paths:
+          - ${_base_directory}/charts/foo
+        type: helm
+        helm_values: ${helm_values}
+        output_path: foo
+    kustomize:
+      - input_paths:
+          - ${_base_directory}/kustomization.jsonnet
+        type: jsonnet
+        output_path: ${_base_directory}/kustomizations/foo
+      - input_paths:
+          - ${_base_directory}/kustomizations/foo/
+        type: kustomize
+        output_path: foo
+
+  # Fall back to jsonnet rendering if method isn't configured
+  compile: ${_compile:${method::jsonnet}}
+```
+
+The rendered parameters for node `test` shown above:
+
+```yaml
+parameters:
+  _reclass_:
+    environment: base
+    name:
+      full: test
+      parts:
+        - test
+      path: test
+      short: test
+  _base_directory: /tmp
+  _compile:
+    helm:
+      - helm_values:
+          a: a
+          b: b
+        input_paths:
+          - /tmp/charts/foo
+        output_path: foo
+        type: helm
+    jsonnet:
+      - input_paths:
+          - /tmp/foo.jsonnet
+        output_path: foo
+        type: jsonnet
+    kustomize:
+      - input_paths:
+          - /tmp/kustomization.jsonnet
+        output_path: /tmp/kustomizations/foo
+        type: jsonnet
+      - input_paths:
+          - /tmp/kustomizations/foo/
+        output_path: foo
+        type: kustomize
+  compile:
+    - input_paths:
+        - /tmp/foo.jsonnet
+      output_path: foo
+      type: jsonnet
+  helm_values:
+    a: a
+    b: b
+```
+
+> [!TIP]
+> This example can also be found as node `n9` and classes `component.defaults` and `component.component` in this repository's `tests/inventory-reference-default-values`.
+
+[kapicorp-reclass]: https://github.com/kapicorp/reclass

README.md

@@ -30,6 +30,14 @@ The implementation currently supports the following features of Kapicorp Reclass
   * reclass-rs uses the `fancy-regex` crate for regex patterns in `class_mappings`.
     The `fancy-regex` crate should support most regex patterns supported by Python.
 
+Additionally the implementation introduces the following new features:
+
+* Option `verbose_warnings` which generates additional informational messages on standard error
+* Default values for references
+
+> [!TIP]
+> Documentation for Reclass extensions introduced by reclass-rs can be found at [README-extensions.md](./README-extensions.md).
+
 The following Kapicorp Reclass features aren't supported:
 
 * Inventory Queries

src/refs/mod.rs

@@ -109,6 +109,13 @@ impl ResolveState {
         )
     }
 
+    fn render_default_value_error(&self, refpath: &str, e: &anyhow::Error) -> anyhow::Error {
+        let current_key = self.current_key();
+        anyhow!(
+            "Error parsing default value for reference '{refpath}' in parameter '{current_key}': {e}"
+        )
+    }
+
     pub(crate) fn render_flattening_error(&self, msg: &str) -> anyhow::Error {
         let current_key = self.current_key();
         anyhow!("In {current_key}: {msg}")
@@ -207,7 +214,10 @@ impl Token {
                 }
                 // Construct flattened ref path by resolving any potential nested references in the
                 // Ref's Vec<Token>.
-                let path = interpolate_token_slice(parts, params, state, opts)?;
+                let (path, default) = extract_path_and_default(
+                    interpolate_token_slice(parts, params, state, opts)?,
+                    "::",
+                );
 
                 if state.seen_paths.contains(&path) {
                     // we've already seen this reference, so we know there's a loop, and can abort
@@ -223,9 +233,18 @@ impl Token {
                 let k0 = refpath_iter.next().unwrap();
                 // v is the value which we update to point to the next value as we recursively
                 // descend into the params Mapping
-                let mut v = params
+                let v = params
                     .get(&k0.into())
-                    .ok_or_else(|| state.render_missing_key_error(&path, k0))?;
+                    .ok_or_else(|| state.render_missing_key_error(&path, k0));
+                if let Some(verr) = v.as_ref().err()
+                    && let Some(dv) = default
+                {
+                    if opts.verbose_warnings {
+                        eprintln!("[INFO] Returning default value due to error: {verr}");
+                    }
+                    return dv.map_err(|e| state.render_default_value_error(&path, &e));
+                }
+                let mut v = v?;
 
                 // newv is used to hold temporary Values generated by interpolating v
                 let mut newv;
@@ -249,9 +268,20 @@ impl Token {
                         // trivial case: v is a Mapping, we can just lookup the next value based
                         // on `key`.
                         Value::Mapping(_) => {
-                            v = newv
+                            let nv = newv
                                 .get(&key.into())
-                                .ok_or_else(|| state.render_missing_key_error(&path, key))?;
+                                .ok_or_else(|| state.render_missing_key_error(&path, key));
+                            if let Some(verr) = nv.as_ref().err()
+                                && let Some(dv) = default
+                            {
+                                if opts.verbose_warnings {
+                                    eprintln!(
+                                        "[INFO] Returning default value due to error: {verr}"
+                                    );
+                                }
+                                return dv.map_err(|e| state.render_default_value_error(&path, &e));
+                            }
+                            v = nv?;
                         }
                         // Sequence lookups aren't supported by Python Reclass. We may implement
                         // them in the future.
@@ -266,6 +296,9 @@ impl Token {
                             "We should have rendered Value::String and Value::ValueList into some other variant"
                         ),
                         // A lookup into any other Value variant is an error
+                        // NOTE(sg): We do not fall back to the default value (if any) here, since
+                        // references that try to recurse into a scalar value should always be
+                        // surfaced.
                         _ => {
                             return Err(state.render_lookup_error(
                                 &path,
@@ -384,6 +417,32 @@ fn interpolate_string_or_valuelist(
     }
 }
 
+fn extract_path_and_default(
+    path_with_default: String,
+    sep: &str,
+) -> (String, Option<Result<Value>>) {
+    if let Some((path, default)) = path_with_default.split_once(sep) {
+        let default_value: Result<Value> = serde_yaml::from_str(default)
+            .map(|v: serde_yaml::Value| {
+                let v = Value::from(v);
+                match v {
+                    // Convert parsed string default value into Value::Literal to avoid incorrect
+                    // additional reference resolution. We know that we've already resolved the
+                    // top-level nested references at the time we call `extract_path_and_default`.
+                    // NOTE(sg): We don't want to recursively replace `Value::String` with
+                    // `Value::Literal` so that default values which are references that expand to
+                    // complex values still get resolved correctly.
+                    Value::String(s) => Value::Literal(s),
+                    _ => v,
+                }
+            })
+            .map_err(|e| anyhow!(format!("{e}")));
+        (path.to_string(), Some(default_value))
+    } else {
+        (path_with_default, None)
+    }
+}
+
 #[derive(Debug)]
 /// Wraps errors generated when trying to parse a string which may contain Reclass references
 pub struct ParseError<'a> {