implement and use a proper nanopass framework

[zerbina]

While the initial goal was to use a nanopass-based architecture for the
compiler, this is not really the case, at present. The characteristic of
nanopasses are that:

1. they focus on a small transformation or analysis task
2. they contain no traversal boilerplate
3. the input and output language are well-defined

The current pass architecture only insufficiently matches the above.
While some tricks are used to keep the traversal logic small, it's
still present.

The goal with this PR is to implement a nanopass framework and write a
compiler for the source language with it.

Passes and languages will become much easier to write. Compared to the
current state, some notable improvements are that:

• it is ensured at compile time (instead of at run-time) that
  constructed AST adheres to the IL's grammar
• AST can embed arbitrary non-AST data
• traversal logic in passes is auto-generated in most cases
• languages are defined and available within the host language (i.e.,
  NimSkull)

This makes the passtool obsolete, though it will have to be kept around
for now, as - in order to keep the scope of this PR smaller - skully
is going to continue using the previous pass pipeline.

---

To-Do

• add simple source location tracking support to the nanopass framework
• add record type support to the nanopass framework
• address all important remaining inline TODO comments
• reject language definitions containing forms that violate internal expectations
• finish the new nanopass-using compiler

[zerbina]

Language definition and extension works, but the pass DSL is still missing. The passes module contains some pass examples using a preliminary syntax, but it's possible that the syntax won't work and has to be replaced with something else.

[saem]

Is the intention, that after nanopass is implemented, to look at what it would mean to go with an extension/inheritance vs composition based API?

[zerbina]

Yep, I want to have something that's certain to work first, which is also the reason metadata is not part of the PR.

[saem]

I like the internal procs, I'm finding them quite a bit more readable than having them at the module level.

[saem]

First cut at documentation, the next thing I'm going to work at is describe, hopefully via examples/graphically, how various functions decompose grammar structures, especially parseForm

If you want me to make any major changes in terms of docs, let me know, right now this is mostly forcing me to understand the code as I work through it and try to explain it.

[saem]

Suggested change

	## defining intermediate languages (their syntax and grammar) and passes.
	## defining intermediate languages (their syntax and grammar) and passes.
	##
	## Nanopass references:
	## Short paper (primer): https://legacy.cs.indiana.edu/~dyb/pubs/nano-jfp.pdf
	## Dissertation (main src): https://andykeep.com/pubs/dissertation.pdf
	##
	## Summary:
	## The goal of a nanopass compiler is to build a compiler consisting of small
	## fine grained passes that are easier to develop, understand, and debug. Each
	## pass defines a grammar and rules for moving it to the grammar of the pass
	## that comes before it. This recursive process results in reducing a high
	## level language into the lowest level that can be passed to codegen.
	##
	## Assuming one responsibility per pass approach, a pass could do one of the
	## following:
	## - simplify: reduce a high level construct to lower level constructs
	## - analyse: make some determinations about various parts of code
	## - verify: ensure that code maintains certain properties
	## (this is not an exhaustive list, it’s meant to be illustrative)
	##
	## Each pass defines a language `defineLanguage(name, [base], body)`, giving it
	## a `name`, and optional `base` language to extend, followed by the
	## definition.
	##
	## If no base language is provided, definitions take the form:
	## - `mv(type)`, e.g.: `n(int)`, where a terminal form (`mv`) is created for
	## its true type (i.e.: `int|string|float|etc`)
	## - `nonterminal(nt) ::= NamedForm(nt, ...repeatingMv)`, where a
	## `nonterminal` form is created with a metavariable named `nt`, and is
	## defined by the right-hand nonterminal form `NamedForm`, with pattern
	## defined by prior metavariables
	## - `foo(f) ::= mv | Named(...f)` where `|` is used to provide alternation
	##
	## If a base language is provided, forms can be prefixed with +/- to add or
	## remove them from the new language:
	## - `foo(f) ::= -mv` removes `mv` from the definition of `foo(f)` above, but
	## retains `Named(…f)`, N.B.: `|` can be used to join multiple additions
	## and/or removals

An initial stab at describing nanopass for this module, I favoured brevity over completeness.

[zerbina]

Nice, looks good, and good call on favoring brevity. There's still the link to the paper for those curious/interested in more details.

I think it'd be a good idea to also link to the dissertation Scheme's nanopass framework is based on, given that the latter was the main inspiration for the implementation (so far).

[saem]

Updated with the dissertation link, I should be able to sort out the rest of it once I have the routines below documented a bit more.

[saem]

I've updated again, I personally think I'm missing some nuance, not necessarily in the description, but in my mental model about non-terminal form definitions: nonterm(nt) ::= Form(...nt), here the lhs nonterm is a nonterminal form, with nonterm being it's name, and ranged over by nt. Where Form(...nt) is a nonterminal named form, but also a patten?

I need to sort out the confusion above, but I'll leave that for tomorrow/day after once I've reread the code some more.

[zerbina]

(You know parts of the following explanation already, but I'm still including them for completeness' sake)

A non-terminal definition has the following syntax <name> '(' <metavar>+ ')' '::=' <prod>, where prod is recursively defined as:

prod ::= <metavar>              # metavariable production
      |  <name> '(' <elem>* ')' # form production
      |  <prod> '|' <prod>      # alternative

In words, a production is either an alternative (of productions), a meta-variable, or a form. A form describes the shape of a syntax tree (number of children and their types). Elements of a form must be either meta-variables or ...<metavar> (zero or more occurrences of an AST fragment matching the meta-var). Your understanding is correct that the forms here are effectively patterns.

Therefore, a non-terminal definition describes multiple judgements. As an example, consider:

expr(e) ::= x | If(e1, e2, e3)

This says multiple things:

1. there exists a non-terminal expr and it's ranged over by e
2. there exists an AST of the form (If e e e) (s-expr representation), meaning a tree with tag If and three children, where all three children are exprs
3. the ASTs ranged over by meta-variable x (which could either range over a non-terminal, or a terminal) are considered an expr
4. ASTs with the form (If e e e) are considered an expr

Or, more formally:

$$ \begin{align*} &expr\ \text{nonterminal}\\ &e\ \in\ expr\\ &x\ \in\ ...\\ &x\ :\ expr\\ &(\mathbf{If}\ e\ e\ e)\ :\ expr\\ \end{align*} $$

The number suffix for form elements is there for overloading purposes, but this feature isn't entirely thought through yet and is likely to change once passes are implemented.

---

What I've found helpful is too look at non-terminals from a type perspective (the meta-language does it this way, for example). There, a non-terminal can be seen as a sum type, while the tags are type constructors and Tag(...) is a type constructor definition.

I hope this helps clear up your confusion.

[zerbina]

I've pushed some cleaned-up progress on the nanopass framework, plus an updated version of the ILs/passes I'm currently using for testing.

It's important to note that the current ILs and passes only exist for testing and demonstration -- they are incomplete, incorrect, and there are some glaring problems with ordering. With my local nanopass framework, I'm able to compile and run passes.nim, where "run" here means that the code doesn't crash and produces an AST that has the correct shape.

What's mostly final is the general shape of the passes' body (i.e., there being zero or more processor procedures), the rest (syntax, naming, etc.) is still in-flux to varying degree.

---

One major outstanding question is how to best represent type information. Currently, types are coupled with languages via their construction forms being part of the language definitions. Type information is part of the forms via "type tags", that is, type constructions being the first element in most forms (after type checking). For example:
(If (BoolTy) (...) (...) (...)), or (TupleCons (TupleTy (IntTy) (FloatTy)) 1 1.0).

This has some major downsides:

• in-memory size of ASTs. Type construction sub-trees are potentially duplicated many times. Not only does this mean increased space taken up by the total AST, it also means more processing work for passes (more AST to traverse, copy, etc.).
• types cannot be referenced from the outside. Symbols may want to refer to types too, but cannot, at the moment. There are some ways to make this work (even without extra nanopass framework support), but I don't particularly like them.
• recursive types aren't supported directly. To be able to represent recursive types, some form of type symbols (i.e., types bound to identifiers) is needed, which adds complexity to passes that need to inspect types (but there might be ways around this).

There are multiple possible approaches/solutions here.

For one, the same strategy as the one the legacy ILs use can be used. That is, keeping a list of types on the module and referring to them via an index (effectively a symbolic reference). This requires manually handling book-keeping and lookup, however, which is cumbersome and prone to mistakes and I'd rather not go down that route.

Another approach could be to move type constructions to their own set of languages. The "main" languages would then refer to types via name terminals (this is a strategy I'm experimenting with as part of some NimSkull work, for what it's worth), or by embedding the types directly as terminals. Type lowering would need to happen in separate passes, which is troublesome when embedding the types directly as terminals, as the relevant "main" language passes and their associated type lowering pass would need to happen atomically, and I'm not sure how that could be made to work in practice.

Yet another approach could be to have the nanopass framework transparently "deduplicate" AST fragments, using node references internally (when deduplicating). While this would only address the size concern, having this feature might be a good idea in general.

[zerbina]

The most recent push contains the current implementation progress. Except for the terminal support, the basics are now there. Beyond properly implementing terminal, what's also still missing to reach the originally intended feature-set is node meta-data support (source location, node origin, etc.).

Originally, the idea was for language definitions to only define syntax, staying close to Scheme's nanopass framework. I've iterated the problems/troubles regarding types, and after discussing the topic with @saem, we've agreed that having some basic type support in the framework is much better than pushing everything type-related onto the framework user.

I'm also going to add symbols as a concept and feature to the nanopass framework. Previously, the idea was to push symbol handling onto the framework user (e.g., via using terminals) but this has some issues, a major one being that symbols referring to types (which are represented as syntax) are hard if not impossible to do that way. I've not decided on how exactly this integration will look like, however.

[zerbina]

To make the code easier to navigate and manage, I've split the nanopass module into multiple ones, with the nanopass module now acting as a "hub" module.

[zerbina]

I consider the nanopass framework finished now (ping @saem, in case you want to review it already). Although I wasn't able to reach all my goals (e.g., dedicated support for types, flexible and customizable node meta-data), I still believe it to be good enough to facilitate development of the compiler for a good while.

The implementation sure isn't great, though some parts are worse than others, with parsing and processing language definitions (nplangdef.nim) being the worst offender (it's incredibly complex and convoluted). Still, the code should be modular enough to be able to replace parts of the framework's implementation without having to rewrite the whole thing.

Pattern matching via match is alright now, even if less powerful than I wanted it to be (due to limitations of NimSkull). Guard expressions (i.e., arbitrary boolean conditions) are not implemented, but the implementation of match should be extensible enough to cleanly add guard expressions at a later point.

---

What remains is finishing the compiler rewrite, a big chunk of which I have already completed locally. I'm not going to spent much effort on creating a well thought-out and efficient compiler, and instead will focus mainly on getting this PR finished as quickly as possible, so that I can go back to working on the language.

With the previous, pseudo nanopass architecture, ILs and passes had to be very carefully designed, as once implemented, they were very hard to reorder or change significantly, effectively ossifying immediately, but this is not a problem with the nanopass framework anymore (at the very least, it's significantly less of a problem), hence the compiler having a shoddy implementation not being something to worry about right now (as long as it's robust).

I'm also going to drop the attempt at coming up with an interim concrete syntax, as it's just an unnecessary distraction from the rest of the PR.

[saem]

I did a quick scan and this is really nice, I've been away a bit too long and I didn't quite grasp the scope of the rework (sorry!). This is really cool.

(I made a soft suggestion, which I spotted as I was skimming through)

[saem]

Suggested change

	## Version of ``find`` that allows providing an inline predicate,
	## evaluated for every checked item.
	## Version of ``find`` that allows providing an inline predicate,
	## evaluated for every checked item, produces the last matched
	## value.

It's like find where it searches forward, but it also gives you the last item, like a reverse find.

[zerbina]

Oh, you're right, I forgot to break out of the loop there; will fix.