I think this separation makes sense:
- Conduit designed to run a pipeline end to end, including i/o, given a config
- If you want more fine-grained control, that's fine, just work with the underlying Hamilton API
A how-to guide for this would be useful. One concrete use-case could be parameter optimisation where a pipeline needs to be run many times, varying one or more config inputs, and the outputs used to compute an objective function then discarded instead of writing to disk.
I think this separation makes sense:
A how-to guide for this would be useful. One concrete use-case could be parameter optimisation where a pipeline needs to be run many times, varying one or more config inputs, and the outputs used to compute an objective function then discarded instead of writing to disk.