Plugin system and "renderers"

[dlespiau]

Wouldn't it be nice if we could bring into jk objects from external sources? For instance we could render a helm chart, bring the result into jk as an array of objects, modify objects as we see fit and write the final result to files.

This PR introduces plugins and the first type of plugins: Renderer. A Renderer takes a js object as input and returns a js value. This PR includes a test Echo Renderer plugin that simply outputs the input it receives.

Plugins are identified by a json document with some metadata about the plugin, including where to find plugin binaries for the current OS/architecture.

echo.json

{
  "name": "echo",
  "version": "0.1.0",
  "binaries": {
    "linux-amd64": "jk-plugin-echo"
  }
}

simple usage example

import { render } from '@jkcfg/std/render';

render('echo.json', { message: "success" }).then(r => std.log(r.message));

TODO:

• Retrofit a design doc
• Support downloading the plugin description and the plugin binaries from http URLs decided not to do this.
• Validate the system by implementing a helm plugin in @jkcfg/kubernetes (helm plugin kubernetes#55)
• (maybe) Port the jk <-> go communication to the new internal RPC method?
• (maybe) Port the plugins to gRPC instead of go RPC so we support plugins written in any language and not just go?
• Explore a bit more a --debug-plugin option that would make plugin output to stderr
• Turn the echo plugin into a test plugin
• Add a test returning an error from the test plugin
• Add options to the RPC (to be future proof, eg. pass the directory hash where the binary is)
• Remove the need for the plugin.json file
• Add a way to define a function as a Renderer

Updates: #133

[squaremo]

• Retrofit a design doc

Looking forward to this :-)

• Support downloading the plugin description and the plugin binaries from http URLs

I would advise leaving this for later, in order to

1. not bite off too much in one go
2. defer decisions that will have repercussions beyond plugins

To expand on 2., I will admit here to be very nervous about anything that looks like packaging and distribution, and anything that does automatic downloads. The first bout of nerves is that it will effectively determine how packaging works for everything to do with jk, but without including all the other stuff in the consideration. The second is because automatically downloading stuff introduces all sorts of concerns about trust and repeatability.

[squaremo]

Wooo RFC \o/

[squaremo]

What does kind actually affect? I don't see it mentioned after here.

[squaremo]

"... to be to encode"

[squaremo]

I reckon it's worth mentioning | here -- i.e., no plugins, just pipe the output of another command into jk (or write files and use them as input to jk, etc.). One of the problems I remember from our discussions of this is that you can't have more than one source of input via |; another (slightly harder to pin down) is that it's better to have control from within JavaScript code, so you can e.g., encapsulate the fact of the external tooling.

[squaremo]

Another formulation of this would be to expect a relative path to the binary. This would make sense if the binaries are shipped with the code wrapping them, for example.

[squaremo]

I'm guessing the motivation here is that occasionally it's a pain to have it not build because you've introduced a new dependency. You can just issue go mod tidy when that's the case.