Thinking about dynamic linking

[nperez]

Hi there,

This isn't a bug report or anything, but I wanted to get your perspective on a change I'd like to make.

I've been studying WAForth with my own host using wasmer in Go where I'd ideally store the compiled modules given to the host via $shell_load and be able to load them later (via Forth word such as LOAD-WORD and a new imported function $shell_load_by_name).

I got a proof of concept working by exporting the $create function and also exposing the dictionary index. I manually patch the modules to align with the current state of the dictionary before instantiating (basically driving WAForth from the outside). For simple words that only rely on the core words, this works great, but as soon as I reference other self-written words, ordering on loading starts to matter due to how calls are compiled.

So I am considering evolving the compiler to switch to referencing imported global vals for dictionary indexes beyond the core words when compiling words. Then, prior to instantiation, I can interrogate the module for what imports it is expecting and also interrogate the current dictionary state and supply the right indexes. And if there are words referenced but not in the dictionary, the host can attempt to load them.

I recognize that this will probably induce a big performance hit by replacing the constants with global.gets, but I am not super concerned for my project.

Do you think this approach could work? Are there any caveats I should consider? Is there another approach I should consider? Would my changes be of interest to upstream?

[remko]

I haven't given it enough thought to say what the possible problems can be. I'm not sure if you intend to change more than the function table, because the code might assume a certain layout of the dictionary etc., which you may not be able to tell.

If it's just the loading part that you're trying to make dynamic, have you looked at trying to do everything at link time (maybe using static code analysis) to avoid a runtime overhead? For example, waforthc also tracks shell_load, but then just concatenates everything into one big module. One thing I wanted to do is make waforthc also walk through the code, and replace all call_indirect instructions with call instructions (since everything is in one module after concatenating), so you get rid of the indirect call overhead. Maybe you can collect all the call_indirects when doing a shell_load to understand the dependencies, and keep track of the dependencies so you can load them later when dynamically linking. You could also rewrite the call_indirect instructions at dynamic load time, although as I said, there may be other parts of the code that rely on the order of words in the dictionary etc.