From fdaed05c6b728471a566e94201dbcca935998192 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 08:04:35 +0200 Subject: [PATCH 01/43] init cabal-exactprint proposal --- proposals/cabal-exactprint.md | 68 +++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 proposals/cabal-exactprint.md diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md new file mode 100644 index 0000000..85bd71d --- /dev/null +++ b/proposals/cabal-exactprint.md @@ -0,0 +1,68 @@ +# Cabal Exactprint + + +## Summary + +blah blah + +## Motivation + +blah blah + +## Proposed Change + +We propose to leverage the existing `Field ann` data type, as well as the `Parsec` and `Pretty` instances to implement +Cabal Exactprint. + +Here are two tasks that complement each other and can be worked on somewhat independently: +- Implement exact printing `[Field ann]`. That is, `exactRenderFields . readField = id` should hold. + +- Implement a modification/addition/removal algebra. A notable wish in the original meta thread + is about being able to programmatically modify cabal files. + With this mechanism, we expose a typed way to modify cabal files that only changes the part that has been touched. + Unmodified parts of the file stayes the same thanks to exaprint. + +## Alternatives Considered + + + +TriviaTree + +Barbie / TTG + +TypedFields + + +## Backwards Compatibility / Migration + + + +Some types need to be extended in TTG style (c.f. Depedency) + +## Interested parties + + + +Users of cabal, cabal-add, etc. + +## Implementation Notes + + + + +## Open Questions + + + +Desired API. + +## References + + + +Link to +- cabal-add +- cabal-fmt +- cabal-gild + +- my four attempts From f025565eb71d17406a8a7cd952500df6b03c60e6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 08:26:01 +0200 Subject: [PATCH 02/43] add more detailed explanation --- proposals/cabal-exactprint.md | 45 ++++++++++++++++++++++++++++------- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 85bd71d..a2e18bd 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -14,13 +14,42 @@ blah blah We propose to leverage the existing `Field ann` data type, as well as the `Parsec` and `Pretty` instances to implement Cabal Exactprint. -Here are two tasks that complement each other and can be worked on somewhat independently: -- Implement exact printing `[Field ann]`. That is, `exactRenderFields . readField = id` should hold. - -- Implement a modification/addition/removal algebra. A notable wish in the original meta thread - is about being able to programmatically modify cabal files. - With this mechanism, we expose a typed way to modify cabal files that only changes the part that has been touched. - Unmodified parts of the file stayes the same thanks to exaprint. +Before starting, we modify the cabal lexer and field parser's definition to retain comments. +Currently Cabal doesn't store any of the comments. This is already implemented. + +Firstly, we implement exact printing `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold. +This method is chosen for its flexibility. As long as we respect the invariants of `[Field ann]` during modification, +unchanged parts in the output should stay the same, and changed parts should be local. + +Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. +A notable wish in the original meta thread is about being able to programmatically modify cabal files. +With this mechanism, we expose a typed way to modify cabal files that only changes the part that has been touched. +Unmodified parts of the file stayes the same thanks to exaprint. +We will use the `Parsec` and `Pretty` to implement the typed modification framework. +Each field in a Cabal file is represented as a field name in association with a some field lines. +Upon modification, we proceed with the following steps: +- Should the field lines be non empty, join them into a single field line `fl`. +- Run the `Parsec` instance of a desired type `τ` on the joined field lines `fl`, obtain the data `p` these field lines represent. +- Run the user's transformation function `t` on `p`, obtaining `p'`. +- Run the `Pretty` instance of `τ` to obtain a new textual representation `fl'`. +- - Should the field be multiple (e.g. `build-depends` or `license-files`), + For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by the parser. + This solves the problem of in-field trivia, such as comma placement and redundant parenthesis in `build-depends`.\ + - Otherwise, we replace the entire string. +- Run modifications similar to this until no more is needed. +- Traverse all fields that has been modified to correct lines that have been mooved. + - If a field `f` is pushed below due to addition before `f`, we increment the line number accordingly. + - If a field `f` is pulled up due to removal before `f`, we can either do nothing (leaving empty lines before `f`) or decrement the line number accordingly. + Modification is be a hybrid of addition and removal. + +These two tasks can be implemented somewhat independently. + + + + ## Alternatives Considered @@ -37,7 +66,7 @@ TypedFields -Some types need to be extended in TTG style (c.f. Depedency) + ## Interested parties From 95200055e13815c36c60c891bca4ab5744a5b707 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 11:07:29 +0200 Subject: [PATCH 03/43] describe API --- proposals/cabal-exactprint.md | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index a2e18bd..92342d0 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -11,7 +11,7 @@ blah blah ## Proposed Change -We propose to leverage the existing `Field ann` data type, as well as the `Parsec` and `Pretty` instances to implement +We propose to leverage the existing `Field ann` data type, as well as the `Parsec` and `Pretty` classes and their instances to implement Cabal Exactprint. Before starting, we modify the cabal lexer and field parser's definition to retain comments. @@ -49,7 +49,32 @@ Do we use TTG for nested types ? I don't think so, but if we want to be perfectionist it can be cosidered. --> - +We want to allow the user to describe a single modification (here-below named `Edit`) by specifying a focus and a transformation. +Here we add a new dependency `myNewDep` as an example. +This can be described as "within the section library with no arguments [^1], within the field `build-depends`, add (append) a `myNewDep." + +In pseudo Haskell of the API we intend to build: +```haskell +addDependency :: Edit +addDependency = + ModifySection + -- focus on a section + (hasSectionName "library" <> hasSectionArgument []) + -- don't transform the section name nor arguments + id + -- transform nested fields or sections + $ AddField + -- focus on a field, creat it should it not exist + (hasFieldName "build-depends") + -- inject a new dependency into the list of dependencies + (addFieldLinesListLike @Dependency myNewDep) + +addFieldLinesListLike :: forall t. (Parsec t, Pretty t) => t -> ([FieldLine Position] -> [FieldLine Position]) +``` + +Interpreting all the foci of a `Edit` tree describes a set of paths down the tree of fields. +At the leaf (in the above example, `AddField`) we help user build a function that modifies `[FieldLine Position]` +by providing `addFieldLinesListLike`. ## Alternatives Considered From e85c3625e6544236082bc62281fe66ef1d3e8a28 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 11:31:06 +0200 Subject: [PATCH 04/43] reread, add details --- proposals/cabal-exactprint.md | 60 +++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 27 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 92342d0..24a6305 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -14,65 +14,67 @@ blah blah We propose to leverage the existing `Field ann` data type, as well as the `Parsec` and `Pretty` classes and their instances to implement Cabal Exactprint. -Before starting, we modify the cabal lexer and field parser's definition to retain comments. -Currently Cabal doesn't store any of the comments. This is already implemented. +As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. +Currently Cabal doesn't store any of the comments. This is already implemented in [https://github.com/haskell/cabal/pull/11252][#11252]. Firstly, we implement exact printing `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold. This method is chosen for its flexibility. As long as we respect the invariants of `[Field ann]` during modification, unchanged parts in the output should stay the same, and changed parts should be local. Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. -A notable wish in the original meta thread is about being able to programmatically modify cabal files. +A notable feature request in [https://github.com/haskell/cabal/issues/7544](Exact-printer Mega-issue #7544) is about being able to programmatically modify cabal files. With this mechanism, we expose a typed way to modify cabal files that only changes the part that has been touched. Unmodified parts of the file stayes the same thanks to exaprint. + We will use the `Parsec` and `Pretty` to implement the typed modification framework. -Each field in a Cabal file is represented as a field name in association with a some field lines. +Each field in a Cabal file is represented by a field name in association with some field lines. Upon modification, we proceed with the following steps: -- Should the field lines be non empty, join them into a single field line `fl`. -- Run the `Parsec` instance of a desired type `τ` on the joined field lines `fl`, obtain the data `p` these field lines represent. -- Run the user's transformation function `t` on `p`, obtaining `p'`. -- Run the `Pretty` instance of `τ` to obtain a new textual representation `fl'`. +- Should the field lines be non empty, join them into a single field line `fl` with indentation and newlines. +- Run the `Parsec` instance of a desired type `τ` on the joined field lines `fl`, obtain data `p` which these field lines represent. +- Apply user's transformation function `t` on `p`, obtaining `p'`. +- Run the `Pretty` instance of `τ` on `p'` to obtain a new textual representation `fl'`. - - Should the field be multiple (e.g. `build-depends` or `license-files`), For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by the parser. This solves the problem of in-field trivia, such as comma placement and redundant parenthesis in `build-depends`.\ - Otherwise, we replace the entire string. - Run modifications similar to this until no more is needed. -- Traverse all fields that has been modified to correct lines that have been mooved. - - If a field `f` is pushed below due to addition before `f`, we increment the line number accordingly. - - If a field `f` is pulled up due to removal before `f`, we can either do nothing (leaving empty lines before `f`) or decrement the line number accordingly. - Modification is be a hybrid of addition and removal. +- Traverse all fields that has been modified to correct lines that have been moved. + - If a field `f` is pushed below due to addition before `f`, we increment the line numbers of `f` and its following siblings accordingly. + - If a field `f` is pulled up due to removal before `f`, we can either do nothing (leaving empty lines before `f`) or decrement the line numbers of `f` and its following siblings accordingly. + - Modification is be a hybrid of addition and removal. -These two tasks can be implemented somewhat independently. +Exactprint and the modification framework can be implemented independently. - +To validate an exactprint implementation, we test the property `exactRenderFields . readFields = id` against Hackage; +to validate a modification framework implementation, we add golden tests for different cases to ensure that important invariants are preserved, +namely that `Position` of fields are not overlapping. -We want to allow the user to describe a single modification (here-below named `Edit`) by specifying a focus and a transformation. +We want to let user describe a single modification that we call `Edit` by specifying a focus and a transformation. Here we add a new dependency `myNewDep` as an example. -This can be described as "within the section library with no arguments [^1], within the field `build-depends`, add (append) a `myNewDep." +This modification can be expressed in plain English as "within the section library with no arguments [^1], within the field `build-depends`, add (append) a `myNewDep." In pseudo Haskell of the API we intend to build: ```haskell -addDependency :: Edit -addDependency = +appendDependency :: Edit +appendDependency = ModifySection - -- focus on a section + -- Focus on a section. (hasSectionName "library" <> hasSectionArgument []) - -- don't transform the section name nor arguments + -- Don't transform the section name nor arguments. + -- This mechanism can be useful to implement transformation on if conditions. id - -- transform nested fields or sections + -- Transform nested fields or sections. $ AddField - -- focus on a field, creat it should it not exist + -- Focus on a field, create it should it not exist. (hasFieldName "build-depends") - -- inject a new dependency into the list of dependencies + -- Inject a new dependency into the list of dependencies. (addFieldLinesListLike @Dependency myNewDep) +-- A helper function that adds a given thing into a list of 'FieldLine Position'. addFieldLinesListLike :: forall t. (Parsec t, Pretty t) => t -> ([FieldLine Position] -> [FieldLine Position]) ``` -Interpreting all the foci of a `Edit` tree describes a set of paths down the tree of fields. +Interpreting all the foci of a `Edit` tree describes a set of matching paths down the tree of fields. At the leaf (in the above example, `AddField`) we help user build a function that modifies `[FieldLine Position]` by providing `addFieldLinesListLike`. @@ -120,3 +122,7 @@ Link to - cabal-gild - my four attempts + +[^1]: + In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, + and `library` is a section that can take a library name as a section argument. From b8846b55895544c409a50e6925f7be7321b832e0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 11:33:28 +0200 Subject: [PATCH 05/43] copy description of different approaches from blog --- proposals/cabal-exactprint.md | 222 +++++++++++++++++++++++++++++++++- 1 file changed, 218 insertions(+), 4 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 24a6305..ebc46af 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -80,13 +80,227 @@ by providing `addFieldLinesListLike`. ## Alternatives Considered - +Here's an exhaustive list of the changes we tried in chronological order. -TriviaTree +- "Defer merging of common stanza" [#11277](https://github.com/haskell/cabal/pull/11277) -Barbie / TTG + During the parsing of a section, Cabal walks through all the import fields and merges them with the remaining section fields that aren't imports. + This is why the `cabal format` command prints out a Cabal file where all common stanza are inlined. -TypedFields + We store unmerged section data within `GenericPackageDescription` while exposing + accessors that merge the imports upon access. + + This will be necessary if we choose to implement cabal-exactprint using field grammar and is not a + exactprint implementation. + +- "Trivia-tree" [#11425 (proof of concept)](https://github.com/haskell/cabal/pull/11425) implements a untyped tree `TriviaTree` using existential type. + With it, we can imtate the shape of a recursive type `τ` freely and construct the same shape but with annotation as nodes. + Constructing and destructing `τ` guides us to store/read annotations accordingly. + This has the benefit of not duplicating all types we want to annotate, but due to its existential type nature, + it is very easy to get things wrong and parsers/printers lose their roundtrip guarantee during composition. + +- "Barbie/Trees-that-grow" [#11690 (proof of concept)](https://github.com/haskell/cabal/pull/11690) tries to do the same thing as Trivia-tree in a typed way. + + We draw inspiration from ghc-exactprint and its trees that grow model, annotating data structurally on each extension point. + Reaching the end of the design space of this approach (with just enough fields implemented to make two cabal files Hackage roundtrip 100%), + inherent problems of using `GenericPackageDescription` to solve this problem started to catch my eyes. + This is the first successful approach where syntactic roundtrip property of `Pretty`/`Parsec` are preserved with composition. + The details of the inherent problems will be mentioned later in details. + +From then, I started experimenting using `[Field Position]` as the CST to implement cabal-exactprint. + +- "typed-fields" [leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690) was the first attempt in this family. + + Rereading the original [cabal-exactprint meta thread](https://github.com/haskell/cabal/issues/7544), I realized that a big part of the + demand was to modify `[Field Position]` in a typed way which doesn't necessarily need GPD. + To allow typed modification in the fields, we extended the `Field` data type to have more constructor (the goal was one per known cabal field). + + This attempt proved that modifying or printing `[Field Position]` (or something isomorphic to it) is a lot easier. + Its structure is not lost and better reflects what was written in the Cabal file. + Also we would avoid threading everything through field grammar, which proved to be unwieldy. + + However, this resulted in the `FieldLine` bearing a too specific type for the field grammar and casting will be necessary, + rendering the specific type information of each field useless for later parsing. + It can only be a exactprint implementation and won't benefit other parts of Cabal-syntax. + +- "transform-fields" [leana8959/cabal/transform-fields](https://github.com/leana8959/cabal/tree/transform-fields) is the most recent attempt. + + Bypassing field grammar, I want to achieve in-place modification leveraging the existing `Parsec` and `Pretty` instance. + I want to allow easy typed modification to `[Field Position]` while maintaining its invariants, + which leaves us with printing `[Field Position]` faithfully. + + + + + +### "Trivia-Tree" + +`TriviaTree` is an open recursive type implemented using existential type. ++ To construct a `TriviaTree` node, the constructor can be applied on data `p` + along with its associated trivia, and optionally the trivia of `p`'s children. + ++ To deconstruct a `TriviaTree` node, a function is provided to try to retrieve the associated + trivia given some data `p`. We use the `Eq` instance to lookup in the trivia tree. This is + internally implemented with `Data.Map`. + + Should the trivia exist, it will be returned, along with the `TriviaTree` that are children to `p` + for further lookup using the data within `p`. + +As long as the construction and the deconstruction matches up, the trivia can be successfully +recovered. + +Inspired by _[Biparsers: Exact Printing for Data Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910), +trivia tree is passed around along the data. Each parser is extended to return a pair `(p, t)` +where `p` is the data parsed and `t` is the associated `TriviaTree`. The printer is extended to +receive `(p, t)` to print the data `p` with its associated `TriviaTree` `t`. + +This model was appealing because it would allow us to maintain the same amount of fields in each +constructor of a type that should support cabal-exactprint, supporting backwards compatibility. + +Trivia tree is poor in structural composition. +Let's describe the exactprint invariant as for a given `inp` and a type `τ`, `(print @τ . parse @τ) inp == inp`. +This reads as "parsing `inp` as `τ` and then printing it results to the same string". +Some subnodes satisfying the exactprint invariant +doesn't guarantee that building from these nodes will lead to a bigger node that also satisfy the +exactprint invariant. +This is because a structural "off-by-one" (e.g. applying one unnecessary constructor or forgetting to unwrap one level of `TriviaTree`) +can lead to not finding any trivia. We can't tell if some trivia is missing because the +TriviaTree was constructed incorrectly, deconstructed incorrectly, or if that node doesn't have trivia to begin with. +During implementation there was a lot of "dumping the AST to see where I messed up" +in the construction or elimination of `TriviaTree`. The engeerning cost was too high. + +TriviaTree is also plagued with the `Newtype` idiom that Cabal uses liberally. +"Ala" parser methods in field grammar are often written in a `f :: Type -> Type` context, and f is then +instantiated to `Identity` or some other wrapper type (e.g. `SpecVersion` for `CabalSpecVersion`) to +redirect the `Parsec`/`Pretty` instance used. +This doesn't work well with the trivia tree model at all, which is designed to associate a data with +its surrounding trivia. We use the data to lookup in the trivia tree. +With newtypes, it is unsure whether we save/look up trivia with the _`pack`ed_ data or the +_`unpack`ed_ data, and an incoherence will not be caught by the type checker but manifest as no trivia. +To make matters worse, field grammar uses `Newtype` as type level parser/printer combinators. `List sep b a` +is a good example. Applying this `Newtype` changes the parsec to parse zero or more `a`, using +the `Newtype` `b`. Field grammar interacts with newtypes over the `Newtype b a` class, not knowing +the multiplicity of the newtype used makes the lookup idea of trivia tree even more convoluted to +implement. + +In hope for more correctness while constrained to only make backwards-compatible changes, Jappie proposed draw inspiration from Barbie. + +### "Barbie/Trees-that-grow" + +[Barbie](https://hackage.haskell.org/package/barbies-2.1.1.0) is a pattern that parameterize a data declaration with a higher kinded +type parameter (commonly `f :: Type -> Type`). +By leveraging this type parameter, we can share the spine of the data type but have each leaf in a different context. + +This is somewhat what we wanted: annotating each leaf. +As for the backwards-compatibility, we wanted to use the `TypeFamilies` extension to conditionally +return the same type, keeping the unannotated type completely identical to existing implementation. +Also, not using the trivia tree frees us from looking up, and `Newtype`s no longer bother us despite +the types becoming really nasty. + +Here's an example of using this method to encode the `targetBuildDepends :: MonoidalFieldAla Dependency` +field of `BuildInfo`. `Dependency` is a normal Cabal type. + +```haskell +-- | Toggle whether a GPD component has annotation or not. +data ParsingPhase + = {-| Concrete syntax tree -} Conc + | {-| Abstract syntax tree -} Abst + +-- Return identical type conditionally. +type family IfConc (m :: ParsingPhase) (f :: Type -> Type) (a :: Type) where + IfConc Abst _ a = a + IfConc Conc f a = f a + +type MonoidalFieldAla (m :: ParsingPhase) (a :: Type) = + IfConc m List + ( IfConc m ( (,,) [Comment Position] BS.ByteString ) + ( IfConc m ((,) Positions) a + ) + ) +``` + +After refining the idea, it ended up being quite similar to the famous [trees that grow](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) idiom used in GHC to implement ghc-exactprint. + +A bearable albeit major inconvenience is constraints. +In the existing codebase there are some adhoc transformations done for backwards-compatibility. +For example, license-file and license-files are both parsed to a list of licenses, concatenated and +then inserted into `GenericPackageDescription`. +To make this still work with trees that grow annotation, it requires adding a +constraint saying that the annotated licenses still form a Monoid, making the +already long constraint tuple even longer. See [before](https://github.com/haskell/cabal/blob/b498d6a911509e6dade136cfbeaad30ad9382b78/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L585-L605) and [after](https://github.com/leana8959/cabal/blob/a91c3fe5d5f0f01c350cc938a8d0c8460d452031/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L552-L590). +The `ConstraintKind` extension is used to deduplicate that constraint from each field grammar, for +the sanity of everybody involved. + +A notable problem is we lose the shape of the original `[Field Position]`. +Components of `GenericPackageDescription` don't know the section they belong +to, and each data don't know which `FieldLine` they were originally parsed +from. +This was previously not known because the limitation of trivia tree didn't allow us to go this +far. + +- Regarding losing the shape of the sections: + + In a Cabal file, it is possible to have trivia for each section as well. + The library stanza "library" is normalized to lower case in Cabal, but + to achieve 100% roundtrip, we need to be able to save the original string + (which I call _cased name_). + Worse, cabal doesn't parse a simple component but a component wrapped in a conditional tree `CondTree`. + `Library` is represented in a suboptimal way where non-conditional fields such as library name is nested within `CondTree`. + The stop-gap solution would be to insert a `Maybe ByteString` that is the + original cased name into the parsed library at the top level of the `CondTree`. + +- Regarding losing the shape of a field: + + `monoidalField` fields parses things that are monoids and merges them during parsing. + This means that the following two cabal fields have exactly the same meaning. + + ```cabal + build-depends: foo, bar + ``` + + ```cabal + build-depends: foo + build-depends: bar + ``` + + In order to remedy this, we store the position of the field name position (where "build-depends" + occurs) as a trivia, in association with each element of the dependency list. + During pretty printing, it suffices to group items of the dependency list by the field name position + to which they belong to reconstruct the grouping. The positioning issue is also solved this way, two + birds one stone. + + A variant such problem is the adhoc concatenation of different fields for + backwards-compatibility. + `license-file` and and `license-files` are both valid fields. In fact, the both accept zero or more + than one license file, and the final parse is the concatenation of the two fields. + This requires marking from which field the data comes originated, so at printing time + we can recover what was originally written. + +These problems illustrate that while it is possible to implement cabal-exactprint with `GenericPackageDescription`, +it is not a good fit because it would require copying the information on all non terminals of +`[Field Position]` to all the leaves (i.e. sections to `GenericPackageDescription` components and +monoidal fields that will be merged to parsed `FieldLine`s). + +This lead to the development of a new family of attempts based on transforming the `Field Position` +directly, without going to and from `GenericPackageDescription`. + +### Typed-Fields + +`Field ann` represents either a field with its string content or a section that has many fields. +We want to use this type as CST while allowing typed modification that people were hoping for in the original thread. + +The idea is to create a `TField ann` (typed field) data type that has a constructor for each of the fields that cabal supports. +Each of the field will bear the value of the parsed type that its corresponding field has. + +This was an interesting idea, but it would be a type that is only used for cabal-exactprint. +Field grammar finds a field by its string name. While in `TField` we retain the original field name, +the value we get out of it will be typed. In other words, the type of the value we get from a TField +depends on which field name it is. This lead to opening the box of pandora called dependent sum and +dependent map, which was soon closed due to the complexity it bears. It wouldn't be possible to use +template haskell in Cabal-syntax to generate `GEq` and `GOrdering` instances due to it being a GHC boot +library. Writing these instances by hand will also be tedious due to the size of the +pattern match being quadratic to the size of the constructor, which is the amount of cabal fields we +support. ## Backwards Compatibility / Migration From 237e11fb07c3be8d52eae74ed5531a060c716d77 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 11:33:49 +0200 Subject: [PATCH 06/43] add todo; refine open questions --- proposals/cabal-exactprint.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index ebc46af..ffbf984 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -322,13 +322,14 @@ Users of cabal, cabal-add, etc. ## Open Questions - - -Desired API. +We are still investigating if describing it is possible or beneficial to describe the modification API in terms of lens. ## References - + Link to - cabal-add From 5acd18d8edc29dc0147b838d4a313129749df06a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 11:53:09 +0200 Subject: [PATCH 07/43] fixes, add implementation notes --- proposals/cabal-exactprint.md | 38 +++++++++++++++++------------------ 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index ffbf984..609e829 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -80,7 +80,7 @@ by providing `addFieldLinesListLike`. ## Alternatives Considered -Here's an exhaustive list of the changes we tried in chronological order. +Below is an exhaustive list of the changes we tried in chronological order since september 2025 and what I learned from these attempts. - "Defer merging of common stanza" [#11277](https://github.com/haskell/cabal/pull/11277) @@ -90,7 +90,7 @@ Here's an exhaustive list of the changes we tried in chronological order. We store unmerged section data within `GenericPackageDescription` while exposing accessors that merge the imports upon access. - This will be necessary if we choose to implement cabal-exactprint using field grammar and is not a + This will be necessary if we choose to implement cabal-exactprint using field grammar and is not a exactprint implementation. - "Trivia-tree" [#11425 (proof of concept)](https://github.com/haskell/cabal/pull/11425) implements a untyped tree `TriviaTree` using existential type. @@ -102,14 +102,14 @@ Here's an exhaustive list of the changes we tried in chronological order. - "Barbie/Trees-that-grow" [#11690 (proof of concept)](https://github.com/haskell/cabal/pull/11690) tries to do the same thing as Trivia-tree in a typed way. We draw inspiration from ghc-exactprint and its trees that grow model, annotating data structurally on each extension point. - Reaching the end of the design space of this approach (with just enough fields implemented to make two cabal files Hackage roundtrip 100%), - inherent problems of using `GenericPackageDescription` to solve this problem started to catch my eyes. + Reaching the end of the design space of this approach with just enough fields implemented to make two cabal files Hackage roundtrip 100%, + inherent problems of using `GenericPackageDescription` as CST to implement exactprint started to catch my eyes. This is the first successful approach where syntactic roundtrip property of `Pretty`/`Parsec` are preserved with composition. The details of the inherent problems will be mentioned later in details. From then, I started experimenting using `[Field Position]` as the CST to implement cabal-exactprint. -- "typed-fields" [leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690) was the first attempt in this family. +- "typed-fields" [leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690). Rereading the original [cabal-exactprint meta thread](https://github.com/haskell/cabal/issues/7544), I realized that a big part of the demand was to modify `[Field Position]` in a typed way which doesn't necessarily need GPD. @@ -123,16 +123,6 @@ From then, I started experimenting using `[Field Position]` as the CST to implem rendering the specific type information of each field useless for later parsing. It can only be a exactprint implementation and won't benefit other parts of Cabal-syntax. -- "transform-fields" [leana8959/cabal/transform-fields](https://github.com/leana8959/cabal/tree/transform-fields) is the most recent attempt. - - Bypassing field grammar, I want to achieve in-place modification leveraging the existing `Parsec` and `Pretty` instance. - I want to allow easy typed modification to `[Field Position]` while maintaining its invariants, - which leaves us with printing `[Field Position]` faithfully. - - - - - ### "Trivia-Tree" `TriviaTree` is an open recursive type implemented using existential type. @@ -305,20 +295,30 @@ support. ## Backwards Compatibility / Migration - - - +Because we don't touch the field grammar infrastructure at all, we don't forsee any backwards-compatibility issues. ## Interested parties -Users of cabal, cabal-add, etc. +As outlined in [https://github.com/haskell/cabal/issues/7544](Exact-printer Mega-issue #7544), +this would benefit the functionality of Cabal itself many ways, namely the following: + +- New command `cabal add` that adds a dependency automatically by editing the cabal file. +- `cabal gen-bounds` can modify the bounds of a cabal file. +- `cabal format` can format a cabal file in a canonical way while preserving comments. + +- `cabal init` can leverage the "addition" part of the modification framework and generate cabal files easily. ## Implementation Notes +[Jappie](https://jappie.me)'s previous proposal has been accepted and funded by the Haskell Foundation. +Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to implement and iterate the previous proposal. +Due to the design evoving drastically over time, this is the most up-to-date proposal describing our ideas after refinding them after a year. + +I will conditinue to work on this personally. ## Open Questions From b6e187788e558c4f85aa3ae437e19482817360b3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 11:57:09 +0200 Subject: [PATCH 08/43] fix link syntax --- proposals/cabal-exactprint.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 609e829..d1dc29e 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -15,14 +15,14 @@ We propose to leverage the existing `Field ann` data type, as well as the `Parse Cabal Exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. -Currently Cabal doesn't store any of the comments. This is already implemented in [https://github.com/haskell/cabal/pull/11252][#11252]. +Currently Cabal doesn't store any of the comments. This is already implemented in [#11252](https://github.com/haskell/cabal/pull/11252). Firstly, we implement exact printing `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold. This method is chosen for its flexibility. As long as we respect the invariants of `[Field ann]` during modification, unchanged parts in the output should stay the same, and changed parts should be local. Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. -A notable feature request in [https://github.com/haskell/cabal/issues/7544](Exact-printer Mega-issue #7544) is about being able to programmatically modify cabal files. +A notable feature request in [Exact-printer Mega-issue #7544](https://github.com/haskell/cabal/issues/7544) is about being able to programmatically modify cabal files. With this mechanism, we expose a typed way to modify cabal files that only changes the part that has been touched. Unmodified parts of the file stayes the same thanks to exaprint. @@ -301,7 +301,7 @@ Because we don't touch the field grammar infrastructure at all, we don't forsee -As outlined in [https://github.com/haskell/cabal/issues/7544](Exact-printer Mega-issue #7544), +As outlined in [Exact-printer Mega-issue #7544](https://github.com/haskell/cabal/issues/7544), this would benefit the functionality of Cabal itself many ways, namely the following: - New command `cabal add` that adds a dependency automatically by editing the cabal file. From e5aadf56d9a463a7a8522c026a37110151709f85 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 11:57:39 +0200 Subject: [PATCH 09/43] clarify that comment-parser is not yet merged --- proposals/cabal-exactprint.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index d1dc29e..d0f9289 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -15,7 +15,7 @@ We propose to leverage the existing `Field ann` data type, as well as the `Parse Cabal Exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. -Currently Cabal doesn't store any of the comments. This is already implemented in [#11252](https://github.com/haskell/cabal/pull/11252). +Currently Cabal doesn't store any of the comments. This is already implemented in [#11252](https://github.com/haskell/cabal/pull/11252) which is yet to be merged. Firstly, we implement exact printing `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold. This method is chosen for its flexibility. As long as we respect the invariants of `[Field ann]` during modification, From 14d000f4447b106466673af327a257502645ea41 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 12:03:20 +0200 Subject: [PATCH 10/43] fixes and rephrasing --- proposals/cabal-exactprint.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index d0f9289..02e0b6c 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -24,7 +24,7 @@ unchanged parts in the output should stay the same, and changed parts should be Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in [Exact-printer Mega-issue #7544](https://github.com/haskell/cabal/issues/7544) is about being able to programmatically modify cabal files. With this mechanism, we expose a typed way to modify cabal files that only changes the part that has been touched. -Unmodified parts of the file stayes the same thanks to exaprint. +Unmodified parts of the file stayes the same thanks to exactprint. We will use the `Parsec` and `Pretty` to implement the typed modification framework. Each field in a Cabal file is represented by a field name in association with some field lines. @@ -43,7 +43,7 @@ Upon modification, we proceed with the following steps: - If a field `f` is pulled up due to removal before `f`, we can either do nothing (leaving empty lines before `f`) or decrement the line numbers of `f` and its following siblings accordingly. - Modification is be a hybrid of addition and removal. -Exactprint and the modification framework can be implemented independently. +Exactprint and the modification framework can be implemented and tested independently. To validate an exactprint implementation, we test the property `exactRenderFields . readFields = id` against Hackage; to validate a modification framework implementation, we add golden tests for different cases to ensure that important invariants are preserved, @@ -52,8 +52,7 @@ namely that `Position` of fields are not overlapping. We want to let user describe a single modification that we call `Edit` by specifying a focus and a transformation. Here we add a new dependency `myNewDep` as an example. This modification can be expressed in plain English as "within the section library with no arguments [^1], within the field `build-depends`, add (append) a `myNewDep." - -In pseudo Haskell of the API we intend to build: +In pseudo Haskell of the API we intend to build the aforementioned example modification can be described as: ```haskell appendDependency :: Edit appendDependency = @@ -78,6 +77,9 @@ Interpreting all the foci of a `Edit` tree describes a set of matching paths dow At the leaf (in the above example, `AddField`) we help user build a function that modifies `[FieldLine Position]` by providing `addFieldLinesListLike`. +We strive to make the API flexible and will expose ways to modify `[Field Position]` directly, and validate/fixup the coordinates after changes. +However we don't try to guarantee that this will always be correct. + ## Alternatives Considered Below is an exhaustive list of the changes we tried in chronological order since september 2025 and what I learned from these attempts. @@ -338,6 +340,5 @@ Link to - my four attempts -[^1]: - In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, +[^1]: In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, and `library` is a section that can take a library name as a section argument. From 7e3d61a80531b2d9bf608cafe186add28c97c8e3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 12:23:11 +0200 Subject: [PATCH 11/43] clarify that I'm funded --- proposals/cabal-exactprint.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 02e0b6c..17879c3 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -320,7 +320,7 @@ this would benefit the functionality of Cabal itself many ways, namely the follo Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to implement and iterate the previous proposal. Due to the design evoving drastically over time, this is the most up-to-date proposal describing our ideas after refinding them after a year. -I will conditinue to work on this personally. +I will conditinue to work on this myself under the funding of Jappie and Haskell Foundation. ## Open Questions From 025319a3c8c8d09dafaea93074db3bd59402e869 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 22 Jul 2026 12:23:35 +0200 Subject: [PATCH 12/43] remove mentioning defer merging common stanzas --- proposals/cabal-exactprint.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 17879c3..365eee4 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -84,17 +84,6 @@ However we don't try to guarantee that this will always be correct. Below is an exhaustive list of the changes we tried in chronological order since september 2025 and what I learned from these attempts. -- "Defer merging of common stanza" [#11277](https://github.com/haskell/cabal/pull/11277) - - During the parsing of a section, Cabal walks through all the import fields and merges them with the remaining section fields that aren't imports. - This is why the `cabal format` command prints out a Cabal file where all common stanza are inlined. - - We store unmerged section data within `GenericPackageDescription` while exposing - accessors that merge the imports upon access. - - This will be necessary if we choose to implement cabal-exactprint using field grammar and is not a - exactprint implementation. - - "Trivia-tree" [#11425 (proof of concept)](https://github.com/haskell/cabal/pull/11425) implements a untyped tree `TriviaTree` using existential type. With it, we can imtate the shape of a recursive type `τ` freely and construct the same shape but with annotation as nodes. Constructing and destructing `τ` guides us to store/read annotations accordingly. From f480a8401077fb50474cef71a720c8bfb9a480f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 23 Jul 2026 09:37:19 +0200 Subject: [PATCH 13/43] proofread, fixes --- proposals/cabal-exactprint.md | 58 +++++++++++++++++++---------------- 1 file changed, 31 insertions(+), 27 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 365eee4..a3ab3a7 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -17,31 +17,35 @@ Cabal Exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. Currently Cabal doesn't store any of the comments. This is already implemented in [#11252](https://github.com/haskell/cabal/pull/11252) which is yet to be merged. -Firstly, we implement exact printing `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold. -This method is chosen for its flexibility. As long as we respect the invariants of `[Field ann]` during modification, -unchanged parts in the output should stay the same, and changed parts should be local. +Firstly, we implement exact printing from `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold, serving `[Field ann]` as the concrete syntax tree (CST). +We chose it as the CST for its flexibility. As long as we respect its invariants during modification, +unchanged parts in the output should stay the same, and changed parts should translate to local transformation in the output string. +In our current prototype, we are already able to roundtrip 119662 out of 194557 cabal files of hackage (~60%) with an implementation that is concise and simple. +To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't figure out whether a whitespace was a tab or a space. +These will require changes to the lexer which we have previously done in [#11252](https://github.com/haskell/cabal/pull/11252). Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in [Exact-printer Mega-issue #7544](https://github.com/haskell/cabal/issues/7544) is about being able to programmatically modify cabal files. -With this mechanism, we expose a typed way to modify cabal files that only changes the part that has been touched. -Unmodified parts of the file stayes the same thanks to exactprint. +With this mechanism, we expose a typed way to modify cabal files. For example, translating an endomorphism over `Version` to an endomorphism over `[FieldLines ann]`, +which allows the user to modify the `cabal-version` field while having all the position validation already dealt with behind the scenes. -We will use the `Parsec` and `Pretty` to implement the typed modification framework. +We will use the `Parsec` and `Pretty` classes to implement the typed modification framework. Each field in a Cabal file is represented by a field name in association with some field lines. Upon modification, we proceed with the following steps: + - Should the field lines be non empty, join them into a single field line `fl` with indentation and newlines. - Run the `Parsec` instance of a desired type `τ` on the joined field lines `fl`, obtain data `p` which these field lines represent. - Apply user's transformation function `t` on `p`, obtaining `p'`. - Run the `Pretty` instance of `τ` on `p'` to obtain a new textual representation `fl'`. - - Should the field be multiple (e.g. `build-depends` or `license-files`), For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by the parser. - This solves the problem of in-field trivia, such as comma placement and redundant parenthesis in `build-depends`.\ + This solves the problem of in-field trivia, such as comma placement and redundant parenthesis in `build-depends`. - Otherwise, we replace the entire string. -- Run modifications similar to this until no more is needed. - Traverse all fields that has been modified to correct lines that have been moved. - If a field `f` is pushed below due to addition before `f`, we increment the line numbers of `f` and its following siblings accordingly. - If a field `f` is pulled up due to removal before `f`, we can either do nothing (leaving empty lines before `f`) or decrement the line numbers of `f` and its following siblings accordingly. - Modification is be a hybrid of addition and removal. +- Run modifications similar to this until no more is needed. Exactprint and the modification framework can be implemented and tested independently. @@ -53,6 +57,7 @@ We want to let user describe a single modification that we call `Edit` by specif Here we add a new dependency `myNewDep` as an example. This modification can be expressed in plain English as "within the section library with no arguments [^1], within the field `build-depends`, add (append) a `myNewDep." In pseudo Haskell of the API we intend to build the aforementioned example modification can be described as: + ```haskell appendDependency :: Edit appendDependency = @@ -73,18 +78,18 @@ appendDependency = addFieldLinesListLike :: forall t. (Parsec t, Pretty t) => t -> ([FieldLine Position] -> [FieldLine Position]) ``` -Interpreting all the foci of a `Edit` tree describes a set of matching paths down the tree of fields. +The set of all the foci of a `Edit` tree describes a set of matching paths down the tree of fields. At the leaf (in the above example, `AddField`) we help user build a function that modifies `[FieldLine Position]` by providing `addFieldLinesListLike`. -We strive to make the API flexible and will expose ways to modify `[Field Position]` directly, and validate/fixup the coordinates after changes. -However we don't try to guarantee that this will always be correct. +We strive to make the API flexible and will expose ways to modify `[Field Position]` directly. However we don't try to guarantee that this will always be correct. ## Alternatives Considered Below is an exhaustive list of the changes we tried in chronological order since september 2025 and what I learned from these attempts. - "Trivia-tree" [#11425 (proof of concept)](https://github.com/haskell/cabal/pull/11425) implements a untyped tree `TriviaTree` using existential type. + With it, we can imtate the shape of a recursive type `τ` freely and construct the same shape but with annotation as nodes. Constructing and destructing `τ` guides us to store/read annotations accordingly. This has the benefit of not duplicating all types we want to annotate, but due to its existential type nature, @@ -107,7 +112,7 @@ From then, I started experimenting using `[Field Position]` as the CST to implem To allow typed modification in the fields, we extended the `Field` data type to have more constructor (the goal was one per known cabal field). This attempt proved that modifying or printing `[Field Position]` (or something isomorphic to it) is a lot easier. - Its structure is not lost and better reflects what was written in the Cabal file. + The shape of a cabal file is not lost and better reflects what was originally written. Also we would avoid threading everything through field grammar, which proved to be unwieldy. However, this resulted in the `FieldLine` bearing a too specific type for the field grammar and casting will be necessary, @@ -130,7 +135,7 @@ From then, I started experimenting using `[Field Position]` as the CST to implem As long as the construction and the deconstruction matches up, the trivia can be successfully recovered. -Inspired by _[Biparsers: Exact Printing for Data Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910), +Inspired by _[Biparsers: Exact Printing for Data Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910)_, trivia tree is passed around along the data. Each parser is extended to return a pair `(p, t)` where `p` is the data parsed and `t` is the associated `TriviaTree`. The printer is extended to receive `(p, t)` to print the data `p` with its associated `TriviaTree` `t`. @@ -151,13 +156,13 @@ During implementation there was a lot of "dumping the AST to see where I messed in the construction or elimination of `TriviaTree`. The engeerning cost was too high. TriviaTree is also plagued with the `Newtype` idiom that Cabal uses liberally. -"Ala" parser methods in field grammar are often written in a `f :: Type -> Type` context, and f is then +"Ala" parser methods in field grammar are often written in a `f :: Type -> Type` context, and `f` is then instantiated to `Identity` or some other wrapper type (e.g. `SpecVersion` for `CabalSpecVersion`) to redirect the `Parsec`/`Pretty` instance used. This doesn't work well with the trivia tree model at all, which is designed to associate a data with its surrounding trivia. We use the data to lookup in the trivia tree. With newtypes, it is unsure whether we save/look up trivia with the _`pack`ed_ data or the -_`unpack`ed_ data, and an incoherence will not be caught by the type checker but manifest as no trivia. +_`unpack`ed_ data, and any incoherence will not be caught by the type checker but manifest as no trivia during lookup. To make matters worse, field grammar uses `Newtype` as type level parser/printer combinators. `List sep b a` is a good example. Applying this `Newtype` changes the parsec to parse zero or more `a`, using the `Newtype` `b`. Field grammar interacts with newtypes over the `Newtype b a` class, not knowing @@ -168,14 +173,14 @@ In hope for more correctness while constrained to only make backwards-compatible ### "Barbie/Trees-that-grow" -[Barbie](https://hackage.haskell.org/package/barbies-2.1.1.0) is a pattern that parameterize a data declaration with a higher kinded +[Barbie](https://hackage.haskell.org/package/barbies-2.1.1.0) is a pattern that parameterizes a data declaration with a higher kinded type parameter (commonly `f :: Type -> Type`). By leveraging this type parameter, we can share the spine of the data type but have each leaf in a different context. This is somewhat what we wanted: annotating each leaf. -As for the backwards-compatibility, we wanted to use the `TypeFamilies` extension to conditionally +As for backwards-compatibility, we wanted to use the `TypeFamilies` extension to conditionally return the same type, keeping the unannotated type completely identical to existing implementation. -Also, not using the trivia tree frees us from looking up, and `Newtype`s no longer bother us despite +Also, not using trivia trees frees us from looking up, so `Newtype`s no longer bother us despite the types becoming really nasty. Here's an example of using this method to encode the `targetBuildDepends :: MonoidalFieldAla Dependency` @@ -187,7 +192,7 @@ data ParsingPhase = {-| Concrete syntax tree -} Conc | {-| Abstract syntax tree -} Abst --- Return identical type conditionally. +-- Return identical type if not annotated. type family IfConc (m :: ParsingPhase) (f :: Type -> Type) (a :: Type) where IfConc Abst _ a = a IfConc Conc f a = f a @@ -209,12 +214,10 @@ then inserted into `GenericPackageDescription`. To make this still work with trees that grow annotation, it requires adding a constraint saying that the annotated licenses still form a Monoid, making the already long constraint tuple even longer. See [before](https://github.com/haskell/cabal/blob/b498d6a911509e6dade136cfbeaad30ad9382b78/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L585-L605) and [after](https://github.com/leana8959/cabal/blob/a91c3fe5d5f0f01c350cc938a8d0c8460d452031/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L552-L590). -The `ConstraintKind` extension is used to deduplicate that constraint from each field grammar, for -the sanity of everybody involved. A notable problem is we lose the shape of the original `[Field Position]`. Components of `GenericPackageDescription` don't know the section they belong -to, and each data don't know which `FieldLine` they were originally parsed +to, and each data don't know which `FieldLine` of which `Field` they were originally parsed from. This was previously not known because the limitation of trivia tree didn't allow us to go this far. @@ -228,7 +231,7 @@ far. Worse, cabal doesn't parse a simple component but a component wrapped in a conditional tree `CondTree`. `Library` is represented in a suboptimal way where non-conditional fields such as library name is nested within `CondTree`. The stop-gap solution would be to insert a `Maybe ByteString` that is the - original cased name into the parsed library at the top level of the `CondTree`. + original cased name into the parsed library only at the top level of the `CondTree`. - Regarding losing the shape of a field: @@ -252,18 +255,19 @@ far. A variant such problem is the adhoc concatenation of different fields for backwards-compatibility. - `license-file` and and `license-files` are both valid fields. In fact, the both accept zero or more + `license-file` and and `license-files` are both valid fields. In fact, they both accept zero or more than one license file, and the final parse is the concatenation of the two fields. This requires marking from which field the data comes originated, so at printing time we can recover what was originally written. -These problems illustrate that while it is possible to implement cabal-exactprint with `GenericPackageDescription`, +These problems illustrate that while it is possible to implement cabal-exactprint using `GenericPackageDescription` as CST, it is not a good fit because it would require copying the information on all non terminals of `[Field Position]` to all the leaves (i.e. sections to `GenericPackageDescription` components and monoidal fields that will be merged to parsed `FieldLine`s). This lead to the development of a new family of attempts based on transforming the `Field Position` directly, without going to and from `GenericPackageDescription`. +The latest attempt of this family is the approach described in this proposal. ### Typed-Fields @@ -305,11 +309,11 @@ this would benefit the functionality of Cabal itself many ways, namely the follo -[Jappie](https://jappie.me)'s previous proposal has been accepted and funded by the Haskell Foundation. +[Jappie](https://jappie.me)'s [previous proposal](https://github.com/haskellfoundation/tech-proposals/pull/65) has been accepted and funded by the Haskell Foundation. Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to implement and iterate the previous proposal. Due to the design evoving drastically over time, this is the most up-to-date proposal describing our ideas after refinding them after a year. -I will conditinue to work on this myself under the funding of Jappie and Haskell Foundation. +I will continue to work on this myself under the funding of Jappie and Haskell Foundation. ## Open Questions From 327f5e2c4dd0181e2f087bbaad59c6cb6a96055c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 23 Jul 2026 10:05:27 +0200 Subject: [PATCH 14/43] document attempts --- proposals/cabal-exactprint.md | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index a3ab3a7..ea7411d 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -305,6 +305,19 @@ this would benefit the functionality of Cabal itself many ways, namely the follo - `cabal init` can leverage the "addition" part of the modification framework and generate cabal files easily. +It would also benefit existing programs that depend on Cabal: + +- [cabal-add](https://github.com/Bodigrim/cabal-add) + + It has three strategies to add dependencies that are tried in sequence. + All of the strategies use parsed fields to guide stringy manipulation directly within the source file. + In comparison `cabal-exactprint` will allow users to manipulate `[Field ann]` instead. Or, even better, we should be able to implement cabal-add in cabal directly. + +- [cabal-fmt](https://github.com/phadej/cabal-fmt) + + It parses the cabal file twice: once with `readFields` from cabal, and again with its own parser to find all the comments. + This can be simplified the new `readFieldsWithComments` in [#11252](https://github.com/haskell/cabal/pull/11252). + ## Implementation Notes @@ -321,17 +334,8 @@ We are still investigating if describing it is possible or beneficial to describ ## References - - -Link to -- cabal-add -- cabal-fmt -- cabal-gild +- [Biparsers: Exact Printing for Data Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910) -- my four attempts [^1]: In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, and `library` is a section that can take a library name as a section argument. From 7d48e0ae750bb0a548f3d34b9226d85f14820e43 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 23 Jul 2026 10:08:07 +0200 Subject: [PATCH 15/43] convert to typst --- proposals/cabal-exactprint.typ | 448 +++++++++++++++++++++++++++++++++ 1 file changed, 448 insertions(+) create mode 100644 proposals/cabal-exactprint.typ diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ new file mode 100644 index 0000000..f9d0634 --- /dev/null +++ b/proposals/cabal-exactprint.typ @@ -0,0 +1,448 @@ += Cabal Exactprint + + +== Summary + +blah blah + +== Motivation + +blah blah + +== Proposed Change + +We propose to leverage the existing `Field ann` data type, as well as +the `Parsec` and `Pretty` classes and their instances to implement Cabal +Exactprint. + +As a preliminary task, we modify the cabal lexer and field parser's +definition to retain comments. Currently Cabal doesn't store any of the +comments. This is already implemented in +#link("https://github.com/haskell/cabal/pull/11252")[\#11252] which is +yet to be merged. + +Firstly, we implement exact printing from `[Field ann]`. That is, +`exactRenderFields . readFields = id` should hold, serving `[Field ann]` +as the concrete syntax tree (CST). We chose it as the CST for its +flexibility. As long as we respect its invariants during modification, +unchanged parts in the output should stay the same, and changed parts +should translate to local transformation in the output string. In our +current prototype, we are already able to roundtrip 119662 out of 194557 +cabal files of hackage (\~60%) with an implementation that is concise +and simple. To increase the percentage of successful roundtrip, we need +to detect CRLF/LF and exactprint accordingly; furthermore, we can't +figure out whether a whitespace was a tab or a space. These will require +changes to the lexer which we have previously done in +#link("https://github.com/haskell/cabal/pull/11252")[\#11252];. + +Secondly, we implement a modification/addition/removal framework to +facilitate building modification functions. A notable feature request in +#link("https://github.com/haskell/cabal/issues/7544")[Exact-printer Mega-issue \#7544] +is about being able to programmatically modify cabal files. With this +mechanism, we expose a typed way to modify cabal files. For example, +translating an endomorphism over `Version` to an endomorphism over +`[FieldLines ann]`, which allows the user to modify the `cabal-version` +field while having all the position validation already dealt with behind +the scenes. + +We will use the `Parsec` and `Pretty` classes to implement the typed +modification framework. Each field in a Cabal file is represented by a +field name in association with some field lines. Upon modification, we +proceed with the following steps: + +- Should the field lines be non empty, join them into a single field + line `fl` with indentation and newlines. +- Run the `Parsec` instance of a desired type `τ` on the joined field + lines `fl`, obtain data `p` which these field lines represent. +- Apply user's transformation function `t` on `p`, obtaining `p'`. +- Run the `Pretty` instance of `τ` on `p'` to obtain a new textual + representation `fl'`. +- - Should the field be multiple (e.g.~`build-depends` or + `license-files`), For each item `it`, we swap out the old textual + represent with the new one, using the location of `it` provided by + the parser. This solves the problem of in-field trivia, such as + comma placement and redundant parenthesis in `build-depends`. + - Otherwise, we replace the entire string. +- Traverse all fields that has been modified to correct lines that have + been moved. + - If a field `f` is pushed below due to addition before `f`, we + increment the line numbers of `f` and its following siblings + accordingly. + - If a field `f` is pulled up due to removal before `f`, we can either + do nothing (leaving empty lines before `f`) or decrement the line + numbers of `f` and its following siblings accordingly. + - Modification is be a hybrid of addition and removal. +- Run modifications similar to this until no more is needed. + +Exactprint and the modification framework can be implemented and tested +independently. + +To validate an exactprint implementation, we test the property +`exactRenderFields . readFields = id` against Hackage; to validate a +modification framework implementation, we add golden tests for different +cases to ensure that important invariants are preserved, namely that +`Position` of fields are not overlapping. + +We want to let user describe a single modification that we call `Edit` +by specifying a focus and a transformation. Here we add a new dependency +`myNewDep` as an example. This modification can be expressed in plain +English as "within the section library with no arguments #footnote[In +cabal, sections can have arguments. If-else conditions are actually +sections where the condition is the single argument, and `library` is a +section that can take a library name as a section argument.];, within +the field `build-depends`, add (append) a \`myNewDep." In pseudo Haskell +of the API we intend to build the aforementioned example modification +can be described as: + +```haskell +appendDependency :: Edit +appendDependency = + ModifySection + -- Focus on a section. + (hasSectionName "library" <> hasSectionArgument []) + -- Don't transform the section name nor arguments. + -- This mechanism can be useful to implement transformation on if conditions. + id + -- Transform nested fields or sections. + $ AddField + -- Focus on a field, create it should it not exist. + (hasFieldName "build-depends") + -- Inject a new dependency into the list of dependencies. + (addFieldLinesListLike @Dependency myNewDep) + +-- A helper function that adds a given thing into a list of 'FieldLine Position'. +addFieldLinesListLike :: forall t. (Parsec t, Pretty t) => t -> ([FieldLine Position] -> [FieldLine Position]) +``` + +The set of all the foci of a `Edit` tree describes a set of matching +paths down the tree of fields. At the leaf (in the above example, +`AddField`) we help user build a function that modifies +`[FieldLine Position]` by providing `addFieldLinesListLike`. + +We strive to make the API flexible and will expose ways to modify +`[Field Position]` directly. However we don't try to guarantee that this +will always be correct. + +== Alternatives Considered + +Below is an exhaustive list of the changes we tried in chronological +order since september 2025 and what I learned from these attempts. + +- "Trivia-tree" + #link("https://github.com/haskell/cabal/pull/11425")[\#11425 (proof of concept)] + implements a untyped tree `TriviaTree` using existential type. + + With it, we can imtate the shape of a recursive type `τ` freely and + construct the same shape but with annotation as nodes. Constructing + and destructing `τ` guides us to store/read annotations accordingly. + This has the benefit of not duplicating all types we want to annotate, + but due to its existential type nature, it is very easy to get things + wrong and parsers/printers lose their roundtrip guarantee during + composition. + +- "Barbie/Trees-that-grow" + #link("https://github.com/haskell/cabal/pull/11690")[\#11690 (proof of concept)] + tries to do the same thing as Trivia-tree in a typed way. + + We draw inspiration from ghc-exactprint and its trees that grow model, + annotating data structurally on each extension point. Reaching the end + of the design space of this approach with just enough fields + implemented to make two cabal files Hackage roundtrip 100%, inherent + problems of using `GenericPackageDescription` as CST to implement + exactprint started to catch my eyes. This is the first successful + approach where syntactic roundtrip property of `Pretty`/`Parsec` are + preserved with composition. The details of the inherent problems will + be mentioned later in details. + +From then, I started experimenting using `[Field Position]` as the CST +to implement cabal-exactprint. + +- "typed-fields" + #link("https://github.com/haskell/cabal/pull/11690")[leana8959/cabal/typed-fields];. + + Rereading the original + #link("https://github.com/haskell/cabal/issues/7544")[cabal-exactprint meta thread];, + I realized that a big part of the demand was to modify + `[Field Position]` in a typed way which doesn't necessarily need GPD. + To allow typed modification in the fields, we extended the `Field` + data type to have more constructor (the goal was one per known cabal + field). + + This attempt proved that modifying or printing `[Field Position]` (or + something isomorphic to it) is a lot easier. The shape of a cabal file + is not lost and better reflects what was originally written. Also we + would avoid threading everything through field grammar, which proved + to be unwieldy. + + However, this resulted in the `FieldLine` bearing a too specific type + for the field grammar and casting will be necessary, rendering the + specific type information of each field useless for later parsing. It + can only be a exactprint implementation and won't benefit other parts + of Cabal-syntax. + +=== "Trivia-Tree" + +`TriviaTree` is an open recursive type implemented using existential +type. + To construct a `TriviaTree` node, the constructor can be applied +on data `p` along with its associated trivia, and optionally the trivia +of `p`'s children. + +- To deconstruct a `TriviaTree` node, a function is provided to try to + retrieve the associated trivia given some data `p`. We use the `Eq` + instance to lookup in the trivia tree. This is internally implemented + with `Data.Map`. + + Should the trivia exist, it will be returned, along with the + `TriviaTree` that are children to `p` for further lookup using the + data within `p`. + +As long as the construction and the deconstruction matches up, the +trivia can be successfully recovered. + +Inspired by +#emph[#link("https://dl.acm.org/doi/full/10.1145/3704910")[Biparsers: Exact Printing for Data Synchronisation];];, +trivia tree is passed around along the data. Each parser is extended to +return a pair `(p, t)` where `p` is the data parsed and `t` is the +associated `TriviaTree`. The printer is extended to receive `(p, t)` to +print the data `p` with its associated `TriviaTree` `t`. + +This model was appealing because it would allow us to maintain the same +amount of fields in each constructor of a type that should support +cabal-exactprint, supporting backwards compatibility. + +Trivia tree is poor in structural composition. Let's describe the +exactprint invariant as for a given `inp` and a type `τ`, +`(print @τ . parse @τ) inp == inp`. This reads as "parsing `inp` as `τ` +and then printing it results to the same string". Some subnodes +satisfying the exactprint invariant doesn't guarantee that building from +these nodes will lead to a bigger node that also satisfy the exactprint +invariant. This is because a structural "off-by-one" (e.g.~applying one +unnecessary constructor or forgetting to unwrap one level of +`TriviaTree`) can lead to not finding any trivia. We can't tell if some +trivia is missing because the TriviaTree was constructed incorrectly, +deconstructed incorrectly, or if that node doesn't have trivia to begin +with. During implementation there was a lot of "dumping the AST to see +where I messed up" in the construction or elimination of `TriviaTree`. +The engeerning cost was too high. + +TriviaTree is also plagued with the `Newtype` idiom that Cabal uses +liberally. "Ala" parser methods in field grammar are often written in a +`f :: Type -> Type` context, and `f` is then instantiated to `Identity` +or some other wrapper type (e.g.~`SpecVersion` for `CabalSpecVersion`) +to redirect the `Parsec`/`Pretty` instance used. This doesn't work well +with the trivia tree model at all, which is designed to associate a data +with its surrounding trivia. We use the data to lookup in the trivia +tree. With newtypes, it is unsure whether we save/look up trivia with +the #emph[`pack`ed] data or the #emph[`unpack`ed] data, and any +incoherence will not be caught by the type checker but manifest as no +trivia during lookup. To make matters worse, field grammar uses +`Newtype` as type level parser/printer combinators. `List sep b a` is a +good example. Applying this `Newtype` changes the parsec to parse zero +or more `a`, using the `Newtype` `b`. Field grammar interacts with +newtypes over the `Newtype b a` class, not knowing the multiplicity of +the newtype used makes the lookup idea of trivia tree even more +convoluted to implement. + +In hope for more correctness while constrained to only make +backwards-compatible changes, Jappie proposed draw inspiration from +Barbie. + +=== "Barbie/Trees-that-grow" + +#link("https://hackage.haskell.org/package/barbies-2.1.1.0")[Barbie] is +a pattern that parameterizes a data declaration with a higher kinded +type parameter (commonly `f :: Type -> Type`). By leveraging this type +parameter, we can share the spine of the data type but have each leaf in +a different context. + +This is somewhat what we wanted: annotating each leaf. As for +backwards-compatibility, we wanted to use the `TypeFamilies` extension +to conditionally return the same type, keeping the unannotated type +completely identical to existing implementation. Also, not using trivia +trees frees us from looking up, so `Newtype`s no longer bother us +despite the types becoming really nasty. + +Here's an example of using this method to encode the +`targetBuildDepends :: MonoidalFieldAla Dependency` field of +`BuildInfo`. `Dependency` is a normal Cabal type. + +```haskell +-- | Toggle whether a GPD component has annotation or not. +data ParsingPhase + = {-| Concrete syntax tree -} Conc + | {-| Abstract syntax tree -} Abst + +-- Return identical type if not annotated. +type family IfConc (m :: ParsingPhase) (f :: Type -> Type) (a :: Type) where + IfConc Abst _ a = a + IfConc Conc f a = f a + +type MonoidalFieldAla (m :: ParsingPhase) (a :: Type) = + IfConc m List + ( IfConc m ( (,,) [Comment Position] BS.ByteString ) + ( IfConc m ((,) Positions) a + ) + ) +``` + +After refining the idea, it ended up being quite similar to the famous +#link("https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf")[trees that grow] +idiom used in GHC to implement ghc-exactprint. + +A bearable albeit major inconvenience is constraints. In the existing +codebase there are some adhoc transformations done for +backwards-compatibility. For example, license-file and license-files are +both parsed to a list of licenses, concatenated and then inserted into +`GenericPackageDescription`. To make this still work with trees that +grow annotation, it requires adding a constraint saying that the +annotated licenses still form a Monoid, making the already long +constraint tuple even longer. See +#link("https://github.com/haskell/cabal/blob/b498d6a911509e6dade136cfbeaad30ad9382b78/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L585-L605")[before] +and +#link("https://github.com/leana8959/cabal/blob/a91c3fe5d5f0f01c350cc938a8d0c8460d452031/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L552-L590")[after];. + +A notable problem is we lose the shape of the original +`[Field Position]`. Components of `GenericPackageDescription` don't know +the section they belong to, and each data don't know which `FieldLine` +of which `Field` they were originally parsed from. This was previously +not known because the limitation of trivia tree didn't allow us to go +this far. + +- Regarding losing the shape of the sections: + + In a Cabal file, it is possible to have trivia for each section as + well. The library stanza "library" is normalized to lower case in + Cabal, but to achieve 100% roundtrip, we need to be able to save the + original string (which I call #emph[cased name];). Worse, cabal + doesn't parse a simple component but a component wrapped in a + conditional tree `CondTree`. `Library` is represented in a suboptimal + way where non-conditional fields such as library name is nested within + `CondTree`. The stop-gap solution would be to insert a + `Maybe ByteString` that is the original cased name into the parsed + library only at the top level of the `CondTree`. + +- Regarding losing the shape of a field: + + `monoidalField` fields parses things that are monoids and merges them + during parsing. This means that the following two cabal fields have + exactly the same meaning. + + ```cabal + build-depends: foo, bar + ``` + + ```cabal + build-depends: foo + build-depends: bar + ``` + + In order to remedy this, we store the position of the field name + position (where "build-depends" occurs) as a trivia, in association + with each element of the dependency list. During pretty printing, it + suffices to group items of the dependency list by the field name + position to which they belong to reconstruct the grouping. The + positioning issue is also solved this way, two birds one stone. + + A variant such problem is the adhoc concatenation of different fields + for backwards-compatibility. `license-file` and and `license-files` + are both valid fields. In fact, they both accept zero or more than one + license file, and the final parse is the concatenation of the two + fields. This requires marking from which field the data comes + originated, so at printing time we can recover what was originally + written. + +These problems illustrate that while it is possible to implement +cabal-exactprint using `GenericPackageDescription` as CST, it is not a +good fit because it would require copying the information on all non +terminals of `[Field Position]` to all the leaves (i.e.~sections to +`GenericPackageDescription` components and monoidal fields that will be +merged to parsed `FieldLine`s). + +This lead to the development of a new family of attempts based on +transforming the `Field Position` directly, without going to and from +`GenericPackageDescription`. The latest attempt of this family is the +approach described in this proposal. + +=== Typed-Fields + +`Field ann` represents either a field with its string content or a +section that has many fields. We want to use this type as CST while +allowing typed modification that people were hoping for in the original +thread. + +The idea is to create a `TField ann` (typed field) data type that has a +constructor for each of the fields that cabal supports. Each of the +field will bear the value of the parsed type that its corresponding +field has. + +This was an interesting idea, but it would be a type that is only used +for cabal-exactprint. Field grammar finds a field by its string name. +While in `TField` we retain the original field name, the value we get +out of it will be typed. In other words, the type of the value we get +from a TField depends on which field name it is. This lead to opening +the box of pandora called dependent sum and dependent map, which was +soon closed due to the complexity it bears. It wouldn't be possible to +use template haskell in Cabal-syntax to generate `GEq` and `GOrdering` +instances due to it being a GHC boot library. Writing these instances by +hand will also be tedious due to the size of the pattern match being +quadratic to the size of the constructor, which is the amount of cabal +fields we support. + +== Backwards Compatibility / Migration + +Because we don't touch the field grammar infrastructure at all, we don't +forsee any backwards-compatibility issues. + +== Interested parties + +As outlined in +#link("https://github.com/haskell/cabal/issues/7544")[Exact-printer Mega-issue \#7544];, +this would benefit the functionality of Cabal itself many ways, namely +the following: + +- New command `cabal add` that adds a dependency automatically by + editing the cabal file. +- `cabal gen-bounds` can modify the bounds of a cabal file. +- `cabal format` can format a cabal file in a canonical way while + preserving comments. +- `cabal init` can leverage the "addition" part of the modification + framework and generate cabal files easily. + +It would also benefit existing programs that depend on Cabal: + +- #link("https://github.com/Bodigrim/cabal-add")[cabal-add] + + It has three strategies to add dependencies that are tried in + sequence. All of the strategies use parsed fields to guide stringy + manipulation directly within the source file. In comparison + `cabal-exactprint` will allow users to manipulate `[Field ann]` + instead. Or, even better, we should be able to implement cabal-add in + cabal directly. + +- #link("https://github.com/phadej/cabal-fmt")[cabal-fmt] + + It parses the cabal file twice: once with `readFields` from cabal, and + again with its own parser to find all the comments. This can be + simplified the new `readFieldsWithComments` in + #link("https://github.com/haskell/cabal/pull/11252")[\#11252];. + +== Implementation Notes + +#link("https://jappie.me")[Jappie];'s +#link("https://github.com/haskellfoundation/tech-proposals/pull/65")[previous proposal] +has been accepted and funded by the Haskell Foundation. Under Jappie and +the Haskell Foundation's funding since september 2025, I have tried to +implement and iterate the previous proposal. Due to the design evoving +drastically over time, this is the most up-to-date proposal describing +our ideas after refinding them after a year. + +I will continue to work on this myself under the funding of Jappie and +Haskell Foundation. + +== Open Questions +We are still investigating if describing it is possible or beneficial to +describe the modification API in terms of lens. + +== References + +- #link("https://dl.acm.org/doi/full/10.1145/3704910")[Biparsers: Exact Printing for Data Synchronisation] From a1909f837e1889dd536a8d09f9d7c1f9b1c757f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 23 Jul 2026 10:10:51 +0200 Subject: [PATCH 16/43] reverse conversion test --- proposals/cabal-exactprint.md | 571 ++++++++++++++++++++------------- proposals/cabal-exactprint.typ | 5 +- 2 files changed, 351 insertions(+), 225 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index ea7411d..c25b47e 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -1,6 +1,5 @@ # Cabal Exactprint - ## Summary blah blah @@ -11,54 +10,97 @@ blah blah ## Proposed Change -We propose to leverage the existing `Field ann` data type, as well as the `Parsec` and `Pretty` classes and their instances to implement -Cabal Exactprint. - -As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. -Currently Cabal doesn't store any of the comments. This is already implemented in [#11252](https://github.com/haskell/cabal/pull/11252) which is yet to be merged. +We propose to leverage the existing `Field ann` data type, as well as +the `Parsec` and `Pretty` classes and their instances to implement Cabal +Exactprint. + +As a preliminary task, we modify the cabal lexer and field parser's +definition to retain comments. Currently Cabal doesn't store any of the +comments. This is already implemented in +[#11252](https://github.com/haskell/cabal/pull/11252) which is yet to be +merged. + +Firstly, we implement exact printing from `[Field ann]`. That is, +`exactRenderFields . readFields = id` should hold, serving `[Field ann]` +as the concrete syntax tree (CST). We chose it as the CST for its +flexibility. As long as we respect its invariants during modification, +unchanged parts in the output should stay the same, and changed parts +should translate to local transformation in the output string. In our +current prototype, we are already able to roundtrip 119662 out of 194557 +cabal files of hackage (\~60%) with an implementation that is concise +and simple. To increase the percentage of successful roundtrip, we need +to detect CRLF/LF and exactprint accordingly; furthermore, we can't +figure out whether a whitespace was a tab or a space. These will require +changes to the lexer which we have previously done in +[#11252](https://github.com/haskell/cabal/pull/11252). + +Secondly, we implement a modification/addition/removal framework to +facilitate building modification functions. A notable feature request in +[Exact-printer Mega-issue +#7544](https://github.com/haskell/cabal/issues/7544) is about being able +to programmatically modify cabal files. With this mechanism, we expose a +typed way to modify cabal files. For example, translating an +endomorphism over `Version` to an endomorphism over `[FieldLines ann]`, +which allows the user to modify the `cabal-version` field while having +all the position validation already dealt with behind the scenes. + +We will use the `Parsec` and `Pretty` classes to implement the typed +modification framework. Each field in a Cabal file is represented by a +field name in association with some field lines. Upon modification, we +proceed with the following steps: + +- Should the field lines be non empty, join them into a single field + line `fl` with indentation and newlines. + +- Run the `Parsec` instance of a desired type `τ` on the joined field + lines `fl`, obtain data `p` which these field lines represent. -Firstly, we implement exact printing from `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold, serving `[Field ann]` as the concrete syntax tree (CST). -We chose it as the CST for its flexibility. As long as we respect its invariants during modification, -unchanged parts in the output should stay the same, and changed parts should translate to local transformation in the output string. -In our current prototype, we are already able to roundtrip 119662 out of 194557 cabal files of hackage (~60%) with an implementation that is concise and simple. -To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't figure out whether a whitespace was a tab or a space. -These will require changes to the lexer which we have previously done in [#11252](https://github.com/haskell/cabal/pull/11252). +- Apply user's transformation function `t` on `p`, obtaining `p'`. -Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. -A notable feature request in [Exact-printer Mega-issue #7544](https://github.com/haskell/cabal/issues/7544) is about being able to programmatically modify cabal files. -With this mechanism, we expose a typed way to modify cabal files. For example, translating an endomorphism over `Version` to an endomorphism over `[FieldLines ann]`, -which allows the user to modify the `cabal-version` field while having all the position validation already dealt with behind the scenes. +- Run the `Pretty` instance of `τ` on `p'` to obtain a new textual + representation `fl'`. -We will use the `Parsec` and `Pretty` classes to implement the typed modification framework. -Each field in a Cabal file is represented by a field name in association with some field lines. -Upon modification, we proceed with the following steps: +- - Should the field be multiple (e.g. `build-depends` or + `license-files`), For each item `it`, we swap out the old textual + represent with the new one, using the location of `it` provided by + the parser. This solves the problem of in-field trivia, such as + comma placement and redundant parenthesis in `build-depends`. -- Should the field lines be non empty, join them into a single field line `fl` with indentation and newlines. -- Run the `Parsec` instance of a desired type `τ` on the joined field lines `fl`, obtain data `p` which these field lines represent. -- Apply user's transformation function `t` on `p`, obtaining `p'`. -- Run the `Pretty` instance of `τ` on `p'` to obtain a new textual representation `fl'`. -- - Should the field be multiple (e.g. `build-depends` or `license-files`), - For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by the parser. - This solves the problem of in-field trivia, such as comma placement and redundant parenthesis in `build-depends`. - Otherwise, we replace the entire string. -- Traverse all fields that has been modified to correct lines that have been moved. - - If a field `f` is pushed below due to addition before `f`, we increment the line numbers of `f` and its following siblings accordingly. - - If a field `f` is pulled up due to removal before `f`, we can either do nothing (leaving empty lines before `f`) or decrement the line numbers of `f` and its following siblings accordingly. + +- Traverse all fields that has been modified to correct lines that have + been moved. + + - If a field `f` is pushed below due to addition before `f`, we + increment the line numbers of `f` and its following siblings + accordingly. + + - If a field `f` is pulled up due to removal before `f`, we can either + do nothing (leaving empty lines before `f`) or decrement the line + numbers of `f` and its following siblings accordingly. + - Modification is be a hybrid of addition and removal. + - Run modifications similar to this until no more is needed. -Exactprint and the modification framework can be implemented and tested independently. +Exactprint and the modification framework can be implemented and tested +independently. -To validate an exactprint implementation, we test the property `exactRenderFields . readFields = id` against Hackage; -to validate a modification framework implementation, we add golden tests for different cases to ensure that important invariants are preserved, -namely that `Position` of fields are not overlapping. +To validate an exactprint implementation, we test the property +`exactRenderFields . readFields = id` against Hackage; to validate a +modification framework implementation, we add golden tests for different +cases to ensure that important invariants are preserved, namely that +`Position` of fields are not overlapping. -We want to let user describe a single modification that we call `Edit` by specifying a focus and a transformation. -Here we add a new dependency `myNewDep` as an example. -This modification can be expressed in plain English as "within the section library with no arguments [^1], within the field `build-depends`, add (append) a `myNewDep." -In pseudo Haskell of the API we intend to build the aforementioned example modification can be described as: +We want to let user describe a single modification that we call `Edit` +by specifying a focus and a transformation. Here we add a new dependency +`myNewDep` as an example. This modification can be expressed in plain +English as "within the section library with no arguments [^1], within +the field `build-depends`, add (append) a \`myNewDep." In pseudo Haskell +of the API we intend to build the aforementioned example modification +can be described as: -```haskell +``` haskell appendDependency :: Edit appendDependency = ModifySection @@ -78,115 +120,157 @@ appendDependency = addFieldLinesListLike :: forall t. (Parsec t, Pretty t) => t -> ([FieldLine Position] -> [FieldLine Position]) ``` -The set of all the foci of a `Edit` tree describes a set of matching paths down the tree of fields. -At the leaf (in the above example, `AddField`) we help user build a function that modifies `[FieldLine Position]` -by providing `addFieldLinesListLike`. +The set of all the foci of a `Edit` tree describes a set of matching +paths down the tree of fields. At the leaf (in the above example, +`AddField`) we help user build a function that modifies +`[FieldLine Position]` by providing `addFieldLinesListLike`. -We strive to make the API flexible and will expose ways to modify `[Field Position]` directly. However we don't try to guarantee that this will always be correct. +We strive to make the API flexible and will expose ways to modify +`[Field Position]` directly. However we don't try to guarantee that this +will always be correct. ## Alternatives Considered -Below is an exhaustive list of the changes we tried in chronological order since september 2025 and what I learned from these attempts. - -- "Trivia-tree" [#11425 (proof of concept)](https://github.com/haskell/cabal/pull/11425) implements a untyped tree `TriviaTree` using existential type. - - With it, we can imtate the shape of a recursive type `τ` freely and construct the same shape but with annotation as nodes. - Constructing and destructing `τ` guides us to store/read annotations accordingly. - This has the benefit of not duplicating all types we want to annotate, but due to its existential type nature, - it is very easy to get things wrong and parsers/printers lose their roundtrip guarantee during composition. - -- "Barbie/Trees-that-grow" [#11690 (proof of concept)](https://github.com/haskell/cabal/pull/11690) tries to do the same thing as Trivia-tree in a typed way. - - We draw inspiration from ghc-exactprint and its trees that grow model, annotating data structurally on each extension point. - Reaching the end of the design space of this approach with just enough fields implemented to make two cabal files Hackage roundtrip 100%, - inherent problems of using `GenericPackageDescription` as CST to implement exactprint started to catch my eyes. - This is the first successful approach where syntactic roundtrip property of `Pretty`/`Parsec` are preserved with composition. - The details of the inherent problems will be mentioned later in details. - -From then, I started experimenting using `[Field Position]` as the CST to implement cabal-exactprint. - -- "typed-fields" [leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690). - - Rereading the original [cabal-exactprint meta thread](https://github.com/haskell/cabal/issues/7544), I realized that a big part of the - demand was to modify `[Field Position]` in a typed way which doesn't necessarily need GPD. - To allow typed modification in the fields, we extended the `Field` data type to have more constructor (the goal was one per known cabal field). - - This attempt proved that modifying or printing `[Field Position]` (or something isomorphic to it) is a lot easier. - The shape of a cabal file is not lost and better reflects what was originally written. - Also we would avoid threading everything through field grammar, which proved to be unwieldy. - - However, this resulted in the `FieldLine` bearing a too specific type for the field grammar and casting will be necessary, - rendering the specific type information of each field useless for later parsing. - It can only be a exactprint implementation and won't benefit other parts of Cabal-syntax. +Below is an exhaustive list of the changes we tried in chronological +order since september 2025 and what I learned from these attempts. + +- "Trivia-tree" [#11425 (proof of + concept)](https://github.com/haskell/cabal/pull/11425) implements a + untyped tree `TriviaTree` using existential type. + + With it, we can imtate the shape of a recursive type `τ` freely and + construct the same shape but with annotation as nodes. Constructing + and destructing `τ` guides us to store/read annotations accordingly. + This has the benefit of not duplicating all types we want to annotate, + but due to its existential type nature, it is very easy to get things + wrong and parsers/printers lose their roundtrip guarantee during + composition. + +- "Barbie/Trees-that-grow" [#11690 (proof of + concept)](https://github.com/haskell/cabal/pull/11690) tries to do the + same thing as Trivia-tree in a typed way. + + We draw inspiration from ghc-exactprint and its trees that grow model, + annotating data structurally on each extension point. Reaching the end + of the design space of this approach with just enough fields + implemented to make two cabal files Hackage roundtrip 100%, inherent + problems of using `GenericPackageDescription` as CST to implement + exactprint started to catch my eyes. This is the first successful + approach where syntactic roundtrip property of `Pretty`/`Parsec` are + preserved with composition. The details of the inherent problems will + be mentioned later in details. + +From then, I started experimenting using `[Field Position]` as the CST +to implement cabal-exactprint. + +- "typed-fields" + [leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690). + + Rereading the original [cabal-exactprint meta + thread](https://github.com/haskell/cabal/issues/7544), I realized that + a big part of the demand was to modify `[Field Position]` in a typed + way which doesn't necessarily need GPD. To allow typed modification in + the fields, we extended the `Field` data type to have more constructor + (the goal was one per known cabal field). + + This attempt proved that modifying or printing `[Field Position]` (or + something isomorphic to it) is a lot easier. The shape of a cabal file + is not lost and better reflects what was originally written. Also we + would avoid threading everything through field grammar, which proved + to be unwieldy. + + However, this resulted in the `FieldLine` bearing a too specific type + for the field grammar and casting will be necessary, rendering the + specific type information of each field useless for later parsing. It + can only be a exactprint implementation and won't benefit other parts + of Cabal-syntax. ### "Trivia-Tree" -`TriviaTree` is an open recursive type implemented using existential type. -+ To construct a `TriviaTree` node, the constructor can be applied on data `p` - along with its associated trivia, and optionally the trivia of `p`'s children. - -+ To deconstruct a `TriviaTree` node, a function is provided to try to retrieve the associated - trivia given some data `p`. We use the `Eq` instance to lookup in the trivia tree. This is - internally implemented with `Data.Map`. - - Should the trivia exist, it will be returned, along with the `TriviaTree` that are children to `p` - for further lookup using the data within `p`. - -As long as the construction and the deconstruction matches up, the trivia can be successfully -recovered. - -Inspired by _[Biparsers: Exact Printing for Data Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910)_, -trivia tree is passed around along the data. Each parser is extended to return a pair `(p, t)` -where `p` is the data parsed and `t` is the associated `TriviaTree`. The printer is extended to -receive `(p, t)` to print the data `p` with its associated `TriviaTree` `t`. - -This model was appealing because it would allow us to maintain the same amount of fields in each -constructor of a type that should support cabal-exactprint, supporting backwards compatibility. - -Trivia tree is poor in structural composition. -Let's describe the exactprint invariant as for a given `inp` and a type `τ`, `(print @τ . parse @τ) inp == inp`. -This reads as "parsing `inp` as `τ` and then printing it results to the same string". -Some subnodes satisfying the exactprint invariant -doesn't guarantee that building from these nodes will lead to a bigger node that also satisfy the -exactprint invariant. -This is because a structural "off-by-one" (e.g. applying one unnecessary constructor or forgetting to unwrap one level of `TriviaTree`) -can lead to not finding any trivia. We can't tell if some trivia is missing because the -TriviaTree was constructed incorrectly, deconstructed incorrectly, or if that node doesn't have trivia to begin with. -During implementation there was a lot of "dumping the AST to see where I messed up" -in the construction or elimination of `TriviaTree`. The engeerning cost was too high. - -TriviaTree is also plagued with the `Newtype` idiom that Cabal uses liberally. -"Ala" parser methods in field grammar are often written in a `f :: Type -> Type` context, and `f` is then -instantiated to `Identity` or some other wrapper type (e.g. `SpecVersion` for `CabalSpecVersion`) to -redirect the `Parsec`/`Pretty` instance used. -This doesn't work well with the trivia tree model at all, which is designed to associate a data with -its surrounding trivia. We use the data to lookup in the trivia tree. -With newtypes, it is unsure whether we save/look up trivia with the _`pack`ed_ data or the -_`unpack`ed_ data, and any incoherence will not be caught by the type checker but manifest as no trivia during lookup. -To make matters worse, field grammar uses `Newtype` as type level parser/printer combinators. `List sep b a` -is a good example. Applying this `Newtype` changes the parsec to parse zero or more `a`, using -the `Newtype` `b`. Field grammar interacts with newtypes over the `Newtype b a` class, not knowing -the multiplicity of the newtype used makes the lookup idea of trivia tree even more convoluted to -implement. - -In hope for more correctness while constrained to only make backwards-compatible changes, Jappie proposed draw inspiration from Barbie. +`TriviaTree` is an open recursive type implemented using existential +type. + To construct a `TriviaTree` node, the constructor can be applied +on data `p` along with its associated trivia, and optionally the trivia +of `p`'s children. + +- To deconstruct a `TriviaTree` node, a function is provided to try to + retrieve the associated trivia given some data `p`. We use the `Eq` + instance to lookup in the trivia tree. This is internally implemented + with `Data.Map`. + + Should the trivia exist, it will be returned, along with the + `TriviaTree` that are children to `p` for further lookup using the + data within `p`. + +As long as the construction and the deconstruction matches up, the +trivia can be successfully recovered. + +Inspired by *[Biparsers: Exact Printing for Data +Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910)*, trivia +tree is passed around along the data. Each parser is extended to return +a pair `(p, t)` where `p` is the data parsed and `t` is the associated +`TriviaTree`. The printer is extended to receive `(p, t)` to print the +data `p` with its associated `TriviaTree` `t`. + +This model was appealing because it would allow us to maintain the same +amount of fields in each constructor of a type that should support +cabal-exactprint, supporting backwards compatibility. + +Trivia tree is poor in structural composition. Let's describe the +exactprint invariant as for a given `inp` and a type `τ`, +`(print @τ . parse @τ) inp == inp`. This reads as "parsing `inp` as `τ` +and then printing it results to the same string". Some subnodes +satisfying the exactprint invariant doesn't guarantee that building from +these nodes will lead to a bigger node that also satisfy the exactprint +invariant. This is because a structural "off-by-one" (e.g. applying one +unnecessary constructor or forgetting to unwrap one level of +`TriviaTree`) can lead to not finding any trivia. We can't tell if some +trivia is missing because the TriviaTree was constructed incorrectly, +deconstructed incorrectly, or if that node doesn't have trivia to begin +with. During implementation there was a lot of "dumping the AST to see +where I messed up" in the construction or elimination of `TriviaTree`. +The engeerning cost was too high. + +TriviaTree is also plagued with the `Newtype` idiom that Cabal uses +liberally. "Ala" parser methods in field grammar are often written in a +`f :: Type -> Type` context, and `f` is then instantiated to `Identity` +or some other wrapper type (e.g. `SpecVersion` for `CabalSpecVersion`) +to redirect the `Parsec`/`Pretty` instance used. This doesn't work well +with the trivia tree model at all, which is designed to associate a data +with its surrounding trivia. We use the data to lookup in the trivia +tree. With newtypes, it is unsure whether we save/look up trivia with +the *`pack`ed* data or the *`unpack`ed* data, and any incoherence will +not be caught by the type checker but manifest as no trivia during +lookup. To make matters worse, field grammar uses `Newtype` as type +level parser/printer combinators. `List sep b a` is a good example. +Applying this `Newtype` changes the parsec to parse zero or more `a`, +using the `Newtype` `b`. Field grammar interacts with newtypes over the +`Newtype b a` class, not knowing the multiplicity of the newtype used +makes the lookup idea of trivia tree even more convoluted to implement. + +In hope for more correctness while constrained to only make +backwards-compatible changes, Jappie proposed draw inspiration from +Barbie. ### "Barbie/Trees-that-grow" -[Barbie](https://hackage.haskell.org/package/barbies-2.1.1.0) is a pattern that parameterizes a data declaration with a higher kinded -type parameter (commonly `f :: Type -> Type`). -By leveraging this type parameter, we can share the spine of the data type but have each leaf in a different context. +[Barbie](https://hackage.haskell.org/package/barbies-2.1.1.0) is a +pattern that parameterizes a data declaration with a higher kinded type +parameter (commonly `f :: Type -> Type`). By leveraging this type +parameter, we can share the spine of the data type but have each leaf in +a different context. -This is somewhat what we wanted: annotating each leaf. -As for backwards-compatibility, we wanted to use the `TypeFamilies` extension to conditionally -return the same type, keeping the unannotated type completely identical to existing implementation. -Also, not using trivia trees frees us from looking up, so `Newtype`s no longer bother us despite -the types becoming really nasty. +This is somewhat what we wanted: annotating each leaf. As for +backwards-compatibility, we wanted to use the `TypeFamilies` extension +to conditionally return the same type, keeping the unannotated type +completely identical to existing implementation. Also, not using trivia +trees frees us from looking up, so `Newtype`s no longer bother us +despite the types becoming really nasty. -Here's an example of using this method to encode the `targetBuildDepends :: MonoidalFieldAla Dependency` -field of `BuildInfo`. `Dependency` is a normal Cabal type. +Here's an example of using this method to encode the +`targetBuildDepends :: MonoidalFieldAla Dependency` field of +`BuildInfo`. `Dependency` is a normal Cabal type. -```haskell +``` haskell -- | Toggle whether a GPD component has annotation or not. data ParsingPhase = {-| Concrete syntax tree -} Conc @@ -205,137 +289,176 @@ type MonoidalFieldAla (m :: ParsingPhase) (a :: Type) = ) ``` -After refining the idea, it ended up being quite similar to the famous [trees that grow](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) idiom used in GHC to implement ghc-exactprint. - -A bearable albeit major inconvenience is constraints. -In the existing codebase there are some adhoc transformations done for backwards-compatibility. -For example, license-file and license-files are both parsed to a list of licenses, concatenated and -then inserted into `GenericPackageDescription`. -To make this still work with trees that grow annotation, it requires adding a -constraint saying that the annotated licenses still form a Monoid, making the -already long constraint tuple even longer. See [before](https://github.com/haskell/cabal/blob/b498d6a911509e6dade136cfbeaad30ad9382b78/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L585-L605) and [after](https://github.com/leana8959/cabal/blob/a91c3fe5d5f0f01c350cc938a8d0c8460d452031/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L552-L590). - -A notable problem is we lose the shape of the original `[Field Position]`. -Components of `GenericPackageDescription` don't know the section they belong -to, and each data don't know which `FieldLine` of which `Field` they were originally parsed -from. -This was previously not known because the limitation of trivia tree didn't allow us to go this -far. +After refining the idea, it ended up being quite similar to the famous +[trees that +grow](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) +idiom used in GHC to implement ghc-exactprint. + +A bearable albeit major inconvenience is constraints. In the existing +codebase there are some adhoc transformations done for +backwards-compatibility. For example, license-file and license-files are +both parsed to a list of licenses, concatenated and then inserted into +`GenericPackageDescription`. To make this still work with trees that +grow annotation, it requires adding a constraint saying that the +annotated licenses still form a Monoid, making the already long +constraint tuple even longer. See +[before](https://github.com/haskell/cabal/blob/b498d6a911509e6dade136cfbeaad30ad9382b78/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L585-L605) +and +[after](https://github.com/leana8959/cabal/blob/a91c3fe5d5f0f01c350cc938a8d0c8460d452031/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L552-L590). + +A notable problem is we lose the shape of the original +`[Field Position]`. Components of `GenericPackageDescription` don't know +the section they belong to, and each data don't know which `FieldLine` +of which `Field` they were originally parsed from. This was previously +not known because the limitation of trivia tree didn't allow us to go +this far. - Regarding losing the shape of the sections: - In a Cabal file, it is possible to have trivia for each section as well. - The library stanza "library" is normalized to lower case in Cabal, but - to achieve 100% roundtrip, we need to be able to save the original string - (which I call _cased name_). - Worse, cabal doesn't parse a simple component but a component wrapped in a conditional tree `CondTree`. - `Library` is represented in a suboptimal way where non-conditional fields such as library name is nested within `CondTree`. - The stop-gap solution would be to insert a `Maybe ByteString` that is the - original cased name into the parsed library only at the top level of the `CondTree`. + In a Cabal file, it is possible to have trivia for each section as + well. The library stanza "library" is normalized to lower case in + Cabal, but to achieve 100% roundtrip, we need to be able to save the + original string (which I call *cased name*). Worse, cabal doesn't + parse a simple component but a component wrapped in a conditional tree + `CondTree`. `Library` is represented in a suboptimal way where + non-conditional fields such as library name is nested within + `CondTree`. The stop-gap solution would be to insert a + `Maybe ByteString` that is the original cased name into the parsed + library only at the top level of the `CondTree`. - Regarding losing the shape of a field: - `monoidalField` fields parses things that are monoids and merges them during parsing. - This means that the following two cabal fields have exactly the same meaning. + `monoidalField` fields parses things that are monoids and merges them + during parsing. This means that the following two cabal fields have + exactly the same meaning. - ```cabal + ``` cabal build-depends: foo, bar + ``` - ```cabal + ``` cabal build-depends: foo - build-depends: bar + build-depends: bar + ``` - In order to remedy this, we store the position of the field name position (where "build-depends" - occurs) as a trivia, in association with each element of the dependency list. - During pretty printing, it suffices to group items of the dependency list by the field name position - to which they belong to reconstruct the grouping. The positioning issue is also solved this way, two - birds one stone. - - A variant such problem is the adhoc concatenation of different fields for - backwards-compatibility. - `license-file` and and `license-files` are both valid fields. In fact, they both accept zero or more - than one license file, and the final parse is the concatenation of the two fields. - This requires marking from which field the data comes originated, so at printing time - we can recover what was originally written. - -These problems illustrate that while it is possible to implement cabal-exactprint using `GenericPackageDescription` as CST, -it is not a good fit because it would require copying the information on all non terminals of -`[Field Position]` to all the leaves (i.e. sections to `GenericPackageDescription` components and -monoidal fields that will be merged to parsed `FieldLine`s). - -This lead to the development of a new family of attempts based on transforming the `Field Position` -directly, without going to and from `GenericPackageDescription`. -The latest attempt of this family is the approach described in this proposal. + In order to remedy this, we store the position of the field name + position (where "build-depends" occurs) as a trivia, in association + with each element of the dependency list. During pretty printing, it + suffices to group items of the dependency list by the field name + position to which they belong to reconstruct the grouping. The + positioning issue is also solved this way, two birds one stone. + + A variant such problem is the adhoc concatenation of different fields + for backwards-compatibility. `license-file` and and `license-files` + are both valid fields. In fact, they both accept zero or more than one + license file, and the final parse is the concatenation of the two + fields. This requires marking from which field the data comes + originated, so at printing time we can recover what was originally + written. + +These problems illustrate that while it is possible to implement +cabal-exactprint using `GenericPackageDescription` as CST, it is not a +good fit because it would require copying the information on all non +terminals of `[Field Position]` to all the leaves (i.e. sections to +`GenericPackageDescription` components and monoidal fields that will be +merged to parsed `FieldLine`s). + +This lead to the development of a new family of attempts based on +transforming the `Field Position` directly, without going to and from +`GenericPackageDescription`. The latest attempt of this family is the +approach described in this proposal. ### Typed-Fields -`Field ann` represents either a field with its string content or a section that has many fields. -We want to use this type as CST while allowing typed modification that people were hoping for in the original thread. - -The idea is to create a `TField ann` (typed field) data type that has a constructor for each of the fields that cabal supports. -Each of the field will bear the value of the parsed type that its corresponding field has. - -This was an interesting idea, but it would be a type that is only used for cabal-exactprint. -Field grammar finds a field by its string name. While in `TField` we retain the original field name, -the value we get out of it will be typed. In other words, the type of the value we get from a TField -depends on which field name it is. This lead to opening the box of pandora called dependent sum and -dependent map, which was soon closed due to the complexity it bears. It wouldn't be possible to use -template haskell in Cabal-syntax to generate `GEq` and `GOrdering` instances due to it being a GHC boot -library. Writing these instances by hand will also be tedious due to the size of the -pattern match being quadratic to the size of the constructor, which is the amount of cabal fields we -support. - +`Field ann` represents either a field with its string content or a +section that has many fields. We want to use this type as CST while +allowing typed modification that people were hoping for in the original +thread. + +The idea is to create a `TField ann` (typed field) data type that has a +constructor for each of the fields that cabal supports. Each of the +field will bear the value of the parsed type that its corresponding +field has. + +This was an interesting idea, but it would be a type that is only used +for cabal-exactprint. Field grammar finds a field by its string name. +While in `TField` we retain the original field name, the value we get +out of it will be typed. In other words, the type of the value we get +from a TField depends on which field name it is. This lead to opening +the box of pandora called dependent sum and dependent map, which was +soon closed due to the complexity it bears. It wouldn't be possible to +use template haskell in Cabal-syntax to generate `GEq` and `GOrdering` +instances due to it being a GHC boot library. Writing these instances by +hand will also be tedious due to the size of the pattern match being +quadratic to the size of the constructor, which is the amount of cabal +fields we support. ## Backwards Compatibility / Migration -Because we don't touch the field grammar infrastructure at all, we don't forsee any backwards-compatibility issues. +Because we don't touch the field grammar infrastructure at all, we don't +forsee any backwards-compatibility issues. ## Interested parties - +As outlined in [Exact-printer Mega-issue +#7544](https://github.com/haskell/cabal/issues/7544), this would benefit +the functionality of Cabal itself many ways, namely the following: -As outlined in [Exact-printer Mega-issue #7544](https://github.com/haskell/cabal/issues/7544), -this would benefit the functionality of Cabal itself many ways, namely the following: +- New command `cabal add` that adds a dependency automatically by + editing the cabal file. -- New command `cabal add` that adds a dependency automatically by editing the cabal file. - `cabal gen-bounds` can modify the bounds of a cabal file. -- `cabal format` can format a cabal file in a canonical way while preserving comments. - -- `cabal init` can leverage the "addition" part of the modification framework and generate cabal files easily. + +- `cabal format` can format a cabal file in a canonical way while + preserving comments. + +- `cabal init` can leverage the "addition" part of the modification + framework and generate cabal files easily. It would also benefit existing programs that depend on Cabal: - [cabal-add](https://github.com/Bodigrim/cabal-add) - It has three strategies to add dependencies that are tried in sequence. - All of the strategies use parsed fields to guide stringy manipulation directly within the source file. - In comparison `cabal-exactprint` will allow users to manipulate `[Field ann]` instead. Or, even better, we should be able to implement cabal-add in cabal directly. + It has three strategies to add dependencies that are tried in + sequence. All of the strategies use parsed fields to guide stringy + manipulation directly within the source file. In comparison + `cabal-exactprint` will allow users to manipulate `[Field ann]` + instead. Or, even better, we should be able to implement cabal-add in + cabal directly. - [cabal-fmt](https://github.com/phadej/cabal-fmt) - It parses the cabal file twice: once with `readFields` from cabal, and again with its own parser to find all the comments. - This can be simplified the new `readFieldsWithComments` in [#11252](https://github.com/haskell/cabal/pull/11252). + It parses the cabal file twice: once with `readFields` from cabal, and + again with its own parser to find all the comments. This can be + simplified the new `readFieldsWithComments` in + [#11252](https://github.com/haskell/cabal/pull/11252). ## Implementation Notes - - -[Jappie](https://jappie.me)'s [previous proposal](https://github.com/haskellfoundation/tech-proposals/pull/65) has been accepted and funded by the Haskell Foundation. -Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to implement and iterate the previous proposal. -Due to the design evoving drastically over time, this is the most up-to-date proposal describing our ideas after refinding them after a year. +[Jappie](https://jappie.me)'s [previous +proposal](https://github.com/haskellfoundation/tech-proposals/pull/65) +has been accepted and funded by the Haskell Foundation. Under Jappie and +the Haskell Foundation's funding since september 2025, I have tried to +implement and iterate the previous proposal. Due to the design evoving +drastically over time, this is the most up-to-date proposal describing +our ideas after refinding them after a year. -I will continue to work on this myself under the funding of Jappie and Haskell Foundation. +I will continue to work on this myself under the funding of Jappie and +Haskell Foundation. ## Open Questions -We are still investigating if describing it is possible or beneficial to describe the modification API in terms of lens. +We are still investigating if describing it is possible or beneficial to +describe the modification API in terms of lens. ## References -- [Biparsers: Exact Printing for Data Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910) - +- [Biparsers: Exact Printing for Data + Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910) -[^1]: In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, - and `library` is a section that can take a library name as a section argument. +[^1]: In cabal, sections can have arguments. If-else conditions are + actually sections where the condition is the single argument, and + `library` is a section that can take a library name as a section + argument. diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index f9d0634..92b1101 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -1,3 +1,5 @@ + + = Cabal Exactprint @@ -57,7 +59,8 @@ proceed with the following steps: - Apply user's transformation function `t` on `p`, obtaining `p'`. - Run the `Pretty` instance of `τ` on `p'` to obtain a new textual representation `fl'`. -- - Should the field be multiple (e.g.~`build-depends` or +- + - Should the field be multiple (e.g.~`build-depends` or `license-files`), For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by the parser. This solves the problem of in-field trivia, such as From a4618b2cd76ec7b7745521f523434134496bde61 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 23 Jul 2026 11:05:24 +0200 Subject: [PATCH 17/43] update links and reference them --- proposals/cabal-exactprint.md | 116 ++++++++++++++++++++------------ proposals/cabal-exactprint.typ | 119 +++++++++++++++++++++------------ 2 files changed, 150 insertions(+), 85 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index c25b47e..acc17a5 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -16,9 +16,9 @@ Exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. Currently Cabal doesn't store any of the -comments. This is already implemented in -[#11252](https://github.com/haskell/cabal/pull/11252) which is yet to be -merged. +comments. This is already implemented in [*Retain comments in field +parser #11252*](https://github.com/haskell/cabal/pull/11252) which is +yet to be merged. Firstly, we implement exact printing from `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold, serving `[Field ann]` @@ -31,15 +31,15 @@ cabal files of hackage (\~60%) with an implementation that is concise and simple. To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't figure out whether a whitespace was a tab or a space. These will require -changes to the lexer which we have previously done in -[#11252](https://github.com/haskell/cabal/pull/11252). +changes to the lexer which we have previously done in [*Retain comments +in field parser #11252*](https://github.com/haskell/cabal/pull/11252). Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in -[Exact-printer Mega-issue -#7544](https://github.com/haskell/cabal/issues/7544) is about being able -to programmatically modify cabal files. With this mechanism, we expose a -typed way to modify cabal files. For example, translating an +[*Exact-printer Mega-issue +#7544*](https://github.com/haskell/cabal/issues/7544) is about being +able to programmatically modify cabal files. With this mechanism, we +expose a typed way to modify cabal files. For example, translating an endomorphism over `Version` to an endomorphism over `[FieldLines ann]`, which allows the user to modify the `cabal-version` field while having all the position validation already dealt with behind the scenes. @@ -60,7 +60,7 @@ proceed with the following steps: - Run the `Pretty` instance of `τ` on `p'` to obtain a new textual representation `fl'`. -- - Should the field be multiple (e.g. `build-depends` or +- - Should the field be multiple (e.g. `build-depends` or `license-files`), For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by the parser. This solves the problem of in-field trivia, such as @@ -96,7 +96,7 @@ We want to let user describe a single modification that we call `Edit` by specifying a focus and a transformation. Here we add a new dependency `myNewDep` as an example. This modification can be expressed in plain English as "within the section library with no arguments [^1], within -the field `build-depends`, add (append) a \`myNewDep." In pseudo Haskell +the field `build-depends`, add (append) a `myNewDep`." In pseudo Haskell of the API we intend to build the aforementioned example modification can be described as: @@ -134,8 +134,8 @@ will always be correct. Below is an exhaustive list of the changes we tried in chronological order since september 2025 and what I learned from these attempts. -- "Trivia-tree" [#11425 (proof of - concept)](https://github.com/haskell/cabal/pull/11425) implements a +- [*Trivia Tree #11425 (proof of + concept)*](https://github.com/haskell/cabal/pull/11425) implements a untyped tree `TriviaTree` using existential type. With it, we can imtate the shape of a recursive type `τ` freely and @@ -146,9 +146,10 @@ order since september 2025 and what I learned from these attempts. wrong and parsers/printers lose their roundtrip guarantee during composition. -- "Barbie/Trees-that-grow" [#11690 (proof of - concept)](https://github.com/haskell/cabal/pull/11690) tries to do the - same thing as Trivia-tree in a typed way. +- [*Barbie/Trees-that-grow #11690 (proof of + concept)*](https://github.com/haskell/cabal/pull/11690) + tries to do the same + thing as Trivia-tree in a typed way. We draw inspiration from ghc-exactprint and its trees that grow model, annotating data structurally on each extension point. Reaching the end @@ -163,11 +164,11 @@ order since september 2025 and what I learned from these attempts. From then, I started experimenting using `[Field Position]` as the CST to implement cabal-exactprint. -- "typed-fields" - [leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690). +- [*typed-fields + leana8959/cabal/typed-fields*](https://github.com/haskell/cabal/pull/11690) - Rereading the original [cabal-exactprint meta - thread](https://github.com/haskell/cabal/issues/7544), I realized that + Rereading the original [*Exact-printer Mega-issue + #7544*](https://github.com/haskell/cabal/issues/7544), I realized that a big part of the demand was to modify `[Field Position]` in a typed way which doesn't necessarily need GPD. To allow typed modification in the fields, we extended the `Field` data type to have more constructor @@ -204,8 +205,8 @@ of `p`'s children. As long as the construction and the deconstruction matches up, the trivia can be successfully recovered. -Inspired by *[Biparsers: Exact Printing for Data -Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910)*, trivia +Inspired by [*Biparsers: Exact Printing for Data +Synchronisation*](https://dl.acm.org/doi/full/10.1145/3704910), trivia tree is passed around along the data. Each parser is extended to return a pair `(p, t)` where `p` is the data parsed and `t` is the associated `TriviaTree`. The printer is extended to receive `(p, t)` to print the @@ -215,7 +216,7 @@ This model was appealing because it would allow us to maintain the same amount of fields in each constructor of a type that should support cabal-exactprint, supporting backwards compatibility. -Trivia tree is poor in structural composition. Let's describe the +Trivia trees are poor in structural composition. Let's describe the exactprint invariant as for a given `inp` and a type `τ`, `(print @τ . parse @τ) inp == inp`. This reads as "parsing `inp` as `τ` and then printing it results to the same string". Some subnodes @@ -253,9 +254,9 @@ Barbie. ### "Barbie/Trees-that-grow" -[Barbie](https://hackage.haskell.org/package/barbies-2.1.1.0) is a -pattern that parameterizes a data declaration with a higher kinded type -parameter (commonly `f :: Type -> Type`). By leveraging this type +[*Barbie library*](https://hackage.haskell.org/package/barbies-2.1.1.0) +is a pattern that parameterizes a data declaration with a higher kinded +type parameter (commonly `f :: Type -> Type`). By leveraging this type parameter, we can share the spine of the data type but have each leaf in a different context. @@ -290,8 +291,8 @@ type MonoidalFieldAla (m :: ParsingPhase) (a :: Type) = ``` After refining the idea, it ended up being quite similar to the famous -[trees that -grow](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) +[*Trees that +Grow*](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) idiom used in GHC to implement ghc-exactprint. A bearable albeit major inconvenience is constraints. In the existing @@ -402,9 +403,10 @@ forsee any backwards-compatibility issues. ## Interested parties -As outlined in [Exact-printer Mega-issue -#7544](https://github.com/haskell/cabal/issues/7544), this would benefit -the functionality of Cabal itself many ways, namely the following: +As outlined in [*Exact-printer Mega-issue +#7544*](https://github.com/haskell/cabal/issues/7544), this would +benefit the functionality of Cabal itself many ways, namely the +following: - New command `cabal add` that adds a dependency automatically by editing the cabal file. @@ -419,7 +421,7 @@ the functionality of Cabal itself many ways, namely the following: It would also benefit existing programs that depend on Cabal: -- [cabal-add](https://github.com/Bodigrim/cabal-add) +- [*cabal-add*](https://github.com/Bodigrim/cabal-add) It has three strategies to add dependencies that are tried in sequence. All of the strategies use parsed fields to guide stringy @@ -428,22 +430,21 @@ It would also benefit existing programs that depend on Cabal: instead. Or, even better, we should be able to implement cabal-add in cabal directly. -- [cabal-fmt](https://github.com/phadej/cabal-fmt) +- [*cabal-fmt*](https://github.com/phadej/cabal-fmt) It parses the cabal file twice: once with `readFields` from cabal, and again with its own parser to find all the comments. This can be - simplified the new `readFieldsWithComments` in - [#11252](https://github.com/haskell/cabal/pull/11252). + simplified the new `readFieldsWithComments` in [*Retain comments in + field parser #11252*](https://github.com/haskell/cabal/pull/11252). ## Implementation Notes -[Jappie](https://jappie.me)'s [previous -proposal](https://github.com/haskellfoundation/tech-proposals/pull/65) -has been accepted and funded by the Haskell Foundation. Under Jappie and -the Haskell Foundation's funding since september 2025, I have tried to -implement and iterate the previous proposal. Due to the design evoving -drastically over time, this is the most up-to-date proposal describing -our ideas after refinding them after a year. +references.jappie-original-proposal.get-link has been accepted and +funded by the Haskell Foundation. Under Jappie and the Haskell +Foundation's funding since september 2025, I have tried to implement and +iterate the previous proposal. Due to the design evoving drastically +over time, this is the most up-to-date proposal describing our ideas +after refinding them after a year. I will continue to work on this myself under the funding of Jappie and Haskell Foundation. @@ -455,8 +456,35 @@ describe the modification API in terms of lens. ## References -- [Biparsers: Exact Printing for Data - Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910) +- [*Retain comments in field parser + #11252*](https://github.com/haskell/cabal/pull/11252) + +- [*Exact-printer Mega-issue + #7544*](https://github.com/haskell/cabal/issues/7544) + +- [*TriviaTree #11425*](https://github.com/haskell/cabal/pull/11425) + +- [*Barbie/Trees-that-grow + #11690*](https://github.com/haskell/cabal/pull/11690) + +- [*typed-fields + leana8959/cabal/typed-fields*](https://github.com/haskell/cabal/pull/11690) + +- [*Biparsers: Exact Printing for Data + Synchronisation*](https://dl.acm.org/doi/full/10.1145/3704910) + +- [*Barbie + library*](https://hackage.haskell.org/package/barbies-2.1.1.0) + +- [*Trees that + Grow*](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) + +- [*cabal-add*](https://github.com/Bodigrim/cabal-add) + +- [*cabal-fmt*](https://github.com/phadej/cabal-fmt) + +- [*Jappie's original Haskell Foundation Tech + Proposal*](https://github.com/haskellfoundation/tech-proposals/pull/65) [^1]: In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, and diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 92b1101..cc5b0f7 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -1,7 +1,54 @@ +// Pandoc doesn't seem to support bibliography files +#let mk-smartlink(url, name) = ( + get-link: link(url)[_#(name)_], + override-name: new-name => link(url)[_#(new-name)_], +) -= Cabal Exactprint +#let references = ( + comment-parser-pr: mk-smartlink( + "https://github.com/haskell/cabal/pull/11252", + )[Retain comments in field parser \#11252], + + exact-printer-mega-issue: mk-smartlink( + "https://github.com/haskell/cabal/issues/7544", + )[Exact-printer Mega-issue \#7544], + + trivia-tree: mk-smartlink( + "https://github.com/haskell/cabal/pull/11425", + )[TriviaTree \#11425], + + barbie-trees-that-grow: mk-smartlink( + "https://github.com/haskell/cabal/pull/11690", + )[Barbie/Trees-that-grow \#11690], + + typed-fields: mk-smartlink( + "https://github.com/haskell/cabal/pull/11690", + )[typed-fields leana8959/cabal/typed-fields], + + biparsers: mk-smartlink( + "https://dl.acm.org/doi/full/10.1145/3704910", + )[Biparsers: Exact Printing for Data Synchronisation], + + barbie-haskell-library: mk-smartlink( + "https://hackage.haskell.org/package/barbies-2.1.1.0", + )[Barbie library], + + trees-that-grow: mk-smartlink( + "https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf", + )[Trees that Grow], + cabal-add-project: mk-smartlink("https://github.com/Bodigrim/cabal-add")[cabal-add], + + cabal-fmt-project: mk-smartlink("https://github.com/phadej/cabal-fmt")[cabal-fmt], + + jappie-original-proposal: mk-smartlink( + "https://github.com/haskellfoundation/tech-proposals/pull/65", + )[Jappie's original Haskell Foundation Tech Proposal], +) + + += Cabal Exactprint == Summary @@ -19,9 +66,7 @@ Exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. Currently Cabal doesn't store any of the -comments. This is already implemented in -#link("https://github.com/haskell/cabal/pull/11252")[\#11252] which is -yet to be merged. +comments. This is already implemented in #references.comment-parser-pr.get-link which is yet to be merged. Firstly, we implement exact printing from `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold, serving `[Field ann]` @@ -34,12 +79,10 @@ cabal files of hackage (\~60%) with an implementation that is concise and simple. To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't figure out whether a whitespace was a tab or a space. These will require -changes to the lexer which we have previously done in -#link("https://github.com/haskell/cabal/pull/11252")[\#11252];. +changes to the lexer which we have previously done in #references.comment-parser-pr.get-link. Secondly, we implement a modification/addition/removal framework to -facilitate building modification functions. A notable feature request in -#link("https://github.com/haskell/cabal/issues/7544")[Exact-printer Mega-issue \#7544] +facilitate building modification functions. A notable feature request in #references.exact-printer-mega-issue.get-link is about being able to programmatically modify cabal files. With this mechanism, we expose a typed way to modify cabal files. For example, translating an endomorphism over `Version` to an endomorphism over @@ -60,7 +103,7 @@ proceed with the following steps: - Run the `Pretty` instance of `τ` on `p'` to obtain a new textual representation `fl'`. - - - Should the field be multiple (e.g.~`build-depends` or + - Should the field be multiple (e.g. `build-depends` or `license-files`), For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by the parser. This solves the problem of in-field trivia, such as @@ -90,10 +133,10 @@ We want to let user describe a single modification that we call `Edit` by specifying a focus and a transformation. Here we add a new dependency `myNewDep` as an example. This modification can be expressed in plain English as "within the section library with no arguments #footnote[In -cabal, sections can have arguments. If-else conditions are actually -sections where the condition is the single argument, and `library` is a -section that can take a library name as a section argument.];, within -the field `build-depends`, add (append) a \`myNewDep." In pseudo Haskell + cabal, sections can have arguments. If-else conditions are actually + sections where the condition is the single argument, and `library` is a + section that can take a library name as a section argument.], within +the field `build-depends`, add (append) a `myNewDep`." In pseudo Haskell of the API we intend to build the aforementioned example modification can be described as: @@ -131,8 +174,7 @@ will always be correct. Below is an exhaustive list of the changes we tried in chronological order since september 2025 and what I learned from these attempts. -- "Trivia-tree" - #link("https://github.com/haskell/cabal/pull/11425")[\#11425 (proof of concept)] +- #(references.trivia-tree.override-name)[Trivia Tree \#11425 (proof of concept)] implements a untyped tree `TriviaTree` using existential type. With it, we can imtate the shape of a recursive type `τ` freely and @@ -143,8 +185,10 @@ order since september 2025 and what I learned from these attempts. wrong and parsers/printers lose their roundtrip guarantee during composition. -- "Barbie/Trees-that-grow" - #link("https://github.com/haskell/cabal/pull/11690")[\#11690 (proof of concept)] +- #( + references.barbie-trees-that-grow.override-name + )[Barbie/Trees-that-grow \#11690 (proof of concept)] + #link("https://github.com/haskell/cabal/pull/11690")[] tries to do the same thing as Trivia-tree in a typed way. We draw inspiration from ghc-exactprint and its trees that grow model, @@ -160,11 +204,9 @@ order since september 2025 and what I learned from these attempts. From then, I started experimenting using `[Field Position]` as the CST to implement cabal-exactprint. -- "typed-fields" - #link("https://github.com/haskell/cabal/pull/11690")[leana8959/cabal/typed-fields];. +- #references.typed-fields.get-link - Rereading the original - #link("https://github.com/haskell/cabal/issues/7544")[cabal-exactprint meta thread];, + Rereading the original #references.exact-printer-mega-issue.get-link, I realized that a big part of the demand was to modify `[Field Position]` in a typed way which doesn't necessarily need GPD. To allow typed modification in the fields, we extended the `Field` @@ -184,7 +226,7 @@ to implement cabal-exactprint. of Cabal-syntax. === "Trivia-Tree" - + `TriviaTree` is an open recursive type implemented using existential type. + To construct a `TriviaTree` node, the constructor can be applied on data `p` along with its associated trivia, and optionally the trivia @@ -202,8 +244,7 @@ of `p`'s children. As long as the construction and the deconstruction matches up, the trivia can be successfully recovered. -Inspired by -#emph[#link("https://dl.acm.org/doi/full/10.1145/3704910")[Biparsers: Exact Printing for Data Synchronisation];];, +Inspired by #references.biparsers.get-link, trivia tree is passed around along the data. Each parser is extended to return a pair `(p, t)` where `p` is the data parsed and `t` is the associated `TriviaTree`. The printer is extended to receive `(p, t)` to @@ -213,7 +254,7 @@ This model was appealing because it would allow us to maintain the same amount of fields in each constructor of a type that should support cabal-exactprint, supporting backwards compatibility. -Trivia tree is poor in structural composition. Let's describe the +Trivia trees are poor in structural composition. Let's describe the exactprint invariant as for a given `inp` and a type `τ`, `(print @τ . parse @τ) inp == inp`. This reads as "parsing `inp` as `τ` and then printing it results to the same string". Some subnodes @@ -251,8 +292,8 @@ backwards-compatible changes, Jappie proposed draw inspiration from Barbie. === "Barbie/Trees-that-grow" - -#link("https://hackage.haskell.org/package/barbies-2.1.1.0")[Barbie] is + +#references.barbie-haskell-library.get-link is a pattern that parameterizes a data declaration with a higher kinded type parameter (commonly `f :: Type -> Type`). By leveraging this type parameter, we can share the spine of the data type but have each leaf in @@ -289,8 +330,7 @@ type MonoidalFieldAla (m :: ParsingPhase) (a :: Type) = ``` After refining the idea, it ended up being quite similar to the famous -#link("https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf")[trees that grow] -idiom used in GHC to implement ghc-exactprint. +#references.trees-that-grow.get-link idiom used in GHC to implement ghc-exactprint. A bearable albeit major inconvenience is constraints. In the existing codebase there are some adhoc transformations done for @@ -300,7 +340,9 @@ both parsed to a list of licenses, concatenated and then inserted into grow annotation, it requires adding a constraint saying that the annotated licenses still form a Monoid, making the already long constraint tuple even longer. See -#link("https://github.com/haskell/cabal/blob/b498d6a911509e6dade136cfbeaad30ad9382b78/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L585-L605")[before] +#link( + "https://github.com/haskell/cabal/blob/b498d6a911509e6dade136cfbeaad30ad9382b78/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L585-L605", +)[before] and #link("https://github.com/leana8959/cabal/blob/a91c3fe5d5f0f01c350cc938a8d0c8460d452031/Cabal-syntax/src/Distribution/PackageDescription/FieldGrammar.hs#L552-L590")[after];. @@ -398,8 +440,7 @@ forsee any backwards-compatibility issues. == Interested parties -As outlined in -#link("https://github.com/haskell/cabal/issues/7544")[Exact-printer Mega-issue \#7544];, +As outlined in #references.exact-printer-mega-issue.get-link, this would benefit the functionality of Cabal itself many ways, namely the following: @@ -413,7 +454,7 @@ the following: It would also benefit existing programs that depend on Cabal: -- #link("https://github.com/Bodigrim/cabal-add")[cabal-add] +- #references.cabal-add-project.get-link It has three strategies to add dependencies that are tried in sequence. All of the strategies use parsed fields to guide stringy @@ -422,18 +463,15 @@ It would also benefit existing programs that depend on Cabal: instead. Or, even better, we should be able to implement cabal-add in cabal directly. -- #link("https://github.com/phadej/cabal-fmt")[cabal-fmt] +- #references.cabal-fmt-project.get-link It parses the cabal file twice: once with `readFields` from cabal, and again with its own parser to find all the comments. This can be - simplified the new `readFieldsWithComments` in - #link("https://github.com/haskell/cabal/pull/11252")[\#11252];. + simplified the new `readFieldsWithComments` in #references.comment-parser-pr.get-link. == Implementation Notes -#link("https://jappie.me")[Jappie];'s -#link("https://github.com/haskellfoundation/tech-proposals/pull/65")[previous proposal] -has been accepted and funded by the Haskell Foundation. Under Jappie and +references.jappie-original-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to implement and iterate the previous proposal. Due to the design evoving drastically over time, this is the most up-to-date proposal describing @@ -447,5 +485,4 @@ We are still investigating if describing it is possible or beneficial to describe the modification API in terms of lens. == References - -- #link("https://dl.acm.org/doi/full/10.1145/3704910")[Biparsers: Exact Printing for Data Synchronisation] +#list(..references.values().map(x => x.get-link)) From a8cea29deab0dbe72a950399940f39fa7d606ea7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 23 Jul 2026 11:19:12 +0200 Subject: [PATCH 18/43] minor fixes --- proposals/cabal-exactprint.md | 31 ++++++++++++++++--------------- proposals/cabal-exactprint.typ | 19 +++++++++---------- 2 files changed, 25 insertions(+), 25 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index acc17a5..a56c648 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -11,8 +11,8 @@ blah blah ## Proposed Change We propose to leverage the existing `Field ann` data type, as well as -the `Parsec` and `Pretty` classes and their instances to implement Cabal -Exactprint. +the `Parsec` and `Pretty` classes and their instances to implement +cabal-exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. Currently Cabal doesn't store any of the @@ -30,9 +30,10 @@ current prototype, we are already able to roundtrip 119662 out of 194557 cabal files of hackage (\~60%) with an implementation that is concise and simple. To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't -figure out whether a whitespace was a tab or a space. These will require -changes to the lexer which we have previously done in [*Retain comments -in field parser #11252*](https://github.com/haskell/cabal/pull/11252). +figure out whether a whitespace was a tab or a space yet. These will +require changes to the lexer which we have previously done in [*Retain +comments in field parser +#11252*](https://github.com/haskell/cabal/pull/11252). Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in @@ -63,25 +64,26 @@ proceed with the following steps: - - Should the field be multiple (e.g. `build-depends` or `license-files`), For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by - the parser. This solves the problem of in-field trivia, such as - comma placement and redundant parenthesis in `build-depends`. + the parser. This solves the problem of in-field trivia by only + modifying the orignial field lines within a specific range that has + changed. - Otherwise, we replace the entire string. - Traverse all fields that has been modified to correct lines that have been moved. - - If a field `f` is pushed below due to addition before `f`, we + - If a field `f` is moved down due to addition before `f`, we increment the line numbers of `f` and its following siblings accordingly. - - If a field `f` is pulled up due to removal before `f`, we can either + - If a field `f` is moved up due to removal before `f`, we can either do nothing (leaving empty lines before `f`) or decrement the line numbers of `f` and its following siblings accordingly. - Modification is be a hybrid of addition and removal. -- Run modifications similar to this until no more is needed. +- Run modifications similar to this until no more is demanded. Exactprint and the modification framework can be implemented and tested independently. @@ -147,9 +149,8 @@ order since september 2025 and what I learned from these attempts. composition. - [*Barbie/Trees-that-grow #11690 (proof of - concept)*](https://github.com/haskell/cabal/pull/11690) - tries to do the same - thing as Trivia-tree in a typed way. + concept)*](https://github.com/haskell/cabal/pull/11690) tries to do + the same thing as Trivia-tree in a typed way. We draw inspiration from ghc-exactprint and its trees that grow model, annotating data structurally on each extension point. Reaching the end @@ -164,7 +165,7 @@ order since september 2025 and what I learned from these attempts. From then, I started experimenting using `[Field Position]` as the CST to implement cabal-exactprint. -- [*typed-fields +- [*Typed-fields leana8959/cabal/typed-fields*](https://github.com/haskell/cabal/pull/11690) Rereading the original [*Exact-printer Mega-issue @@ -467,7 +468,7 @@ describe the modification API in terms of lens. - [*Barbie/Trees-that-grow #11690*](https://github.com/haskell/cabal/pull/11690) -- [*typed-fields +- [*Typed-fields leana8959/cabal/typed-fields*](https://github.com/haskell/cabal/pull/11690) - [*Biparsers: Exact Printing for Data diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index cc5b0f7..73a2b35 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -1,5 +1,6 @@ // Pandoc doesn't seem to support bibliography files + #let mk-smartlink(url, name) = ( get-link: link(url)[_#(name)_], override-name: new-name => link(url)[_#(new-name)_], @@ -24,7 +25,7 @@ typed-fields: mk-smartlink( "https://github.com/haskell/cabal/pull/11690", - )[typed-fields leana8959/cabal/typed-fields], + )[Typed-fields leana8959/cabal/typed-fields], biparsers: mk-smartlink( "https://dl.acm.org/doi/full/10.1145/3704910", @@ -61,8 +62,7 @@ blah blah == Proposed Change We propose to leverage the existing `Field ann` data type, as well as -the `Parsec` and `Pretty` classes and their instances to implement Cabal -Exactprint. +the `Parsec` and `Pretty` classes and their instances to implement cabal-exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. Currently Cabal doesn't store any of the @@ -78,7 +78,7 @@ current prototype, we are already able to roundtrip 119662 out of 194557 cabal files of hackage (\~60%) with an implementation that is concise and simple. To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't -figure out whether a whitespace was a tab or a space. These will require +figure out whether a whitespace was a tab or a space yet. These will require changes to the lexer which we have previously done in #references.comment-parser-pr.get-link. Secondly, we implement a modification/addition/removal framework to @@ -106,19 +106,19 @@ proceed with the following steps: - Should the field be multiple (e.g. `build-depends` or `license-files`), For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by - the parser. This solves the problem of in-field trivia, such as - comma placement and redundant parenthesis in `build-depends`. + the parser. This solves the problem of in-field trivia by only modifying + the orignial field lines within a specific range that has changed. - Otherwise, we replace the entire string. - Traverse all fields that has been modified to correct lines that have been moved. - - If a field `f` is pushed below due to addition before `f`, we + - If a field `f` is moved down due to addition before `f`, we increment the line numbers of `f` and its following siblings accordingly. - - If a field `f` is pulled up due to removal before `f`, we can either + - If a field `f` is moved up due to removal before `f`, we can either do nothing (leaving empty lines before `f`) or decrement the line numbers of `f` and its following siblings accordingly. - Modification is be a hybrid of addition and removal. -- Run modifications similar to this until no more is needed. +- Run modifications similar to this until no more is demanded. Exactprint and the modification framework can be implemented and tested independently. @@ -188,7 +188,6 @@ order since september 2025 and what I learned from these attempts. - #( references.barbie-trees-that-grow.override-name )[Barbie/Trees-that-grow \#11690 (proof of concept)] - #link("https://github.com/haskell/cabal/pull/11690")[] tries to do the same thing as Trivia-tree in a typed way. We draw inspiration from ghc-exactprint and its trees that grow model, From 5edb543831f204968cf9b634075762cc4afd255c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 23 Jul 2026 11:52:29 +0200 Subject: [PATCH 19/43] add modivations and stuff --- proposals/cabal-exactprint.md | 76 ++++++++++++++++++++++++++++++++-- proposals/cabal-exactprint.typ | 67 ++++++++++++++++++++++++++++-- 2 files changed, 136 insertions(+), 7 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index a56c648..6f0f08f 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -1,12 +1,75 @@ # Cabal Exactprint +Note that this proposal builds on the earlier [*TWG (Technical Working +Group) +proposal*](https://github.com/haskellfoundation/tech-proposals/pull/65), +also succeeds [*Jappie's original Cabal +Proposal*](https://github.com/haskell/cabal-proposals/pull/5) to match +the updates in our approach. + ## Summary -blah blah +The Exact Printer project aims to develop a precise parsing and printing +tool for .cabal files in the cabal library. + +This will allow both cabal and other tools to introduce deltas into +cabal files through a typed API, simplifying the +modification/addition/removal of fields, without mangling the format, +structure or comments of users files. + +Furthermore it makes cabal authoritative on the cabal file format +allowing downstream users to use the provided printing functions and get +a stability guarantee. + +We define the parse-print idempotency to be `print . parse == id`, which +reads "parseing then printing is as if we've done nothing". We only +focus on ensuring this property to hold for valid cabal files, and we +not consider the braces syntax in this work. ## Motivation -blah blah +Cabal reads cabal package manifests in the cabal format (with the +extension .cabal). However, it is currently unable to modify it +loselessly. + +Here are some of the symptoms of this problem manifesting in different +ways through out the cabal CLI: + +- `cabal format` + + It should fix the indentation of your file and canonicalize some + fields. Instead, it also drops all your comments, all the imports are + merged in-place, elif in a conditional will be desugared to a nested + if in an else, etc. + +- `cabal add` + + Cabal should be able to add a dependency to a component. This can't be + implemented because modifying a portion of the cabal file's in-memory + representation mangles the entire cabal file, similar to + `cabal format`. drops all comments and merges imports. + +- Missing module declaration When a module exists but is not declared in + the cabal file, cabal can't add it for you. Again, because cabal would + mangle the cabal file if it tries to touch it. It can only tell you + that it's missing. Argh. + +- `cabal gen-bounds` + + Cabal is very helpful and can generate dependency constraints + ("bounds") for you. However, it just dumps them in the terminal, + because it can't modify the cabal file. + +Cabal is also not authoritative in this matter, many projects have been +created to do what cabal can't: + +- [*cabal-fmt*](https://github.com/phadej/cabal-fmt) + +- [*cabal-add*](https://github.com/Bodigrim/cabal-add) + +- [*hpack*](https://github.com/sol/hpack) + +- [*autopack*](https://github.com/kowainik/autopack) ## Proposed Change @@ -440,7 +503,7 @@ It would also benefit existing programs that depend on Cabal: ## Implementation Notes -references.jappie-original-proposal.get-link has been accepted and +references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to implement and iterate the previous proposal. Due to the design evoving drastically @@ -487,6 +550,13 @@ describe the modification API in terms of lens. - [*Jappie's original Haskell Foundation Tech Proposal*](https://github.com/haskellfoundation/tech-proposals/pull/65) +- [*Jappie's original Cabal + Proposal*](https://github.com/haskell/cabal-proposals/pull/5) + +- [*hpack*](https://github.com/sol/hpack) + +- [*autopack*](https://github.com/kowainik/autopack) + [^1]: In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, and `library` is a section that can take a library name as a section diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 73a2b35..b981c52 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -43,21 +43,80 @@ cabal-fmt-project: mk-smartlink("https://github.com/phadej/cabal-fmt")[cabal-fmt], - jappie-original-proposal: mk-smartlink( + jappie-original-twg-proposal: mk-smartlink( "https://github.com/haskellfoundation/tech-proposals/pull/65", )[Jappie's original Haskell Foundation Tech Proposal], + + jappie-original-cabal-proposal: mk-smartlink( + "https://github.com/haskell/cabal-proposals/pull/5", + )[Jappie's original Cabal Proposal], + + // TODO: read these references + hpack-project: mk-smartlink("https://github.com/sol/hpack")[hpack], + autopack-project: mk-smartlink("https://github.com/kowainik/autopack")[autopack], ) = Cabal Exactprint +Note that this proposal builds on the earlier +#(references.jappie-original-twg-proposal.override-name)[TWG (Technical Working Group) proposal], +also succeeds #references.jappie-original-cabal-proposal.get-link to match the updates in our +approach. + == Summary -blah blah +The Exact Printer project aims to develop a precise parsing and printing tool for .cabal files in the cabal library. + +This will allow both cabal and other tools to introduce deltas into cabal files through a typed API, +simplifying the modification/addition/removal of fields, without mangling the format, structure or comments of users files. + +Furthermore it makes cabal authoritative on the cabal file format +allowing downstream users to use the provided printing functions +and get a stability guarantee. + +We define the parse-print idempotency to be `print . parse == id`, which reads "parseing then +printing is as if we've done nothing". We only focus on ensuring this property to hold for valid +cabal files, and we not consider the braces syntax in this work. == Motivation -blah blah +Cabal reads cabal package manifests in the cabal format (with the extension +.cabal). However, it is currently unable to modify it loselessly. + +Here are some of the symptoms of this problem manifesting in different ways through out the cabal CLI: + +- `cabal format` + + It should fix the indentation of your file and canonicalize some fields. + Instead, it also drops all your comments, all the imports are merged in-place, elif in a + conditional will be desugared to a nested if in an else, etc. + +- `cabal add` + + Cabal should be able to add a dependency to a component. + This can't be implemented because modifying a portion of the cabal file's in-memory representation + mangles the entire cabal file, similar to `cabal format`. + drops all comments and merges imports. + +- Missing module declaration + When a module exists but is not declared in the cabal file, cabal can't add it for you. + Again, because cabal would mangle the cabal file if it tries to touch it. + It can only tell you that it's missing. Argh. + +- `cabal gen-bounds` + + Cabal is very helpful and can generate dependency constraints ("bounds") for you. + However, it just dumps them in the terminal, because it can't modify the cabal file. + + +Cabal is also not authoritative in this matter, many projects have been created to do what cabal +can't: + +- #references.cabal-fmt-project.get-link +- #references.cabal-add-project.get-link +- #references.hpack-project.get-link +- #references.autopack-project.get-link == Proposed Change @@ -470,7 +529,7 @@ It would also benefit existing programs that depend on Cabal: == Implementation Notes -references.jappie-original-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and +references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to implement and iterate the previous proposal. Due to the design evoving drastically over time, this is the most up-to-date proposal describing From db0826ffee28a0f6b842a4dbec3de9433c425f36 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 23 Jul 2026 11:53:29 +0200 Subject: [PATCH 20/43] fix typos --- proposals/cabal-exactprint.md | 10 +++++----- proposals/cabal-exactprint.typ | 6 +++--- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 6f0f08f..0a5c0c3 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -22,9 +22,9 @@ allowing downstream users to use the provided printing functions and get a stability guarantee. We define the parse-print idempotency to be `print . parse == id`, which -reads "parseing then printing is as if we've done nothing". We only -focus on ensuring this property to hold for valid cabal files, and we -not consider the braces syntax in this work. +reads "parsing then printing is as if we've done nothing". We only focus +on ensuring this property to hold for valid cabal files, and we not +consider the braces syntax in this work. ## Motivation @@ -128,7 +128,7 @@ proceed with the following steps: `license-files`), For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by the parser. This solves the problem of in-field trivia by only - modifying the orignial field lines within a specific range that has + modifying the original field lines within a specific range that has changed. - Otherwise, we replace the entire string. @@ -463,7 +463,7 @@ fields we support. ## Backwards Compatibility / Migration Because we don't touch the field grammar infrastructure at all, we don't -forsee any backwards-compatibility issues. +foresee any backwards-compatibility issues. ## Interested parties diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index b981c52..8195593 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -75,7 +75,7 @@ Furthermore it makes cabal authoritative on the cabal file format allowing downstream users to use the provided printing functions and get a stability guarantee. -We define the parse-print idempotency to be `print . parse == id`, which reads "parseing then +We define the parse-print idempotency to be `print . parse == id`, which reads "parsing then printing is as if we've done nothing". We only focus on ensuring this property to hold for valid cabal files, and we not consider the braces syntax in this work. @@ -166,7 +166,7 @@ proceed with the following steps: `license-files`), For each item `it`, we swap out the old textual represent with the new one, using the location of `it` provided by the parser. This solves the problem of in-field trivia by only modifying - the orignial field lines within a specific range that has changed. + the original field lines within a specific range that has changed. - Otherwise, we replace the entire string. - Traverse all fields that has been modified to correct lines that have been moved. @@ -494,7 +494,7 @@ fields we support. == Backwards Compatibility / Migration Because we don't touch the field grammar infrastructure at all, we don't -forsee any backwards-compatibility issues. +foresee any backwards-compatibility issues. == Interested parties From 355257b03f6b9d6a3c4a7f2514f6963b3702a7f1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 23 Jul 2026 11:55:25 +0200 Subject: [PATCH 21/43] homogeneize all cabal exactprint orthography --- proposals/cabal-exactprint.md | 32 ++++++++++++++++---------------- proposals/cabal-exactprint.typ | 18 +++++++++--------- 2 files changed, 25 insertions(+), 25 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 0a5c0c3..dca2e59 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -9,8 +9,8 @@ the updates in our approach. ## Summary -The Exact Printer project aims to develop a precise parsing and printing -tool for .cabal files in the cabal library. +The Cabal Exactprint project aims to develop a precise parsing and +printing tool for .cabal files in the cabal library. This will allow both cabal and other tools to introduce deltas into cabal files through a typed API, simplifying the @@ -74,8 +74,8 @@ created to do what cabal can't: ## Proposed Change We propose to leverage the existing `Field ann` data type, as well as -the `Parsec` and `Pretty` classes and their instances to implement -cabal-exactprint. +the `Parsec` and `Pretty` classes and their instances to implement Cabal +Exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. Currently Cabal doesn't store any of the @@ -83,7 +83,7 @@ comments. This is already implemented in [*Retain comments in field parser #11252*](https://github.com/haskell/cabal/pull/11252) which is yet to be merged. -Firstly, we implement exact printing from `[Field ann]`. That is, +Firstly, we implement exactprinting from `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold, serving `[Field ann]` as the concrete syntax tree (CST). We chose it as the CST for its flexibility. As long as we respect its invariants during modification, @@ -226,7 +226,7 @@ order since september 2025 and what I learned from these attempts. be mentioned later in details. From then, I started experimenting using `[Field Position]` as the CST -to implement cabal-exactprint. +to implement Cabal Exactprint. - [*Typed-fields leana8959/cabal/typed-fields*](https://github.com/haskell/cabal/pull/11690) @@ -277,8 +277,8 @@ a pair `(p, t)` where `p` is the data parsed and `t` is the associated data `p` with its associated `TriviaTree` `t`. This model was appealing because it would allow us to maintain the same -amount of fields in each constructor of a type that should support -cabal-exactprint, supporting backwards compatibility. +amount of fields in each constructor of a type that should support Cabal +Exactprint, supporting backwards compatibility. Trivia trees are poor in structural composition. Let's describe the exactprint invariant as for a given `inp` and a type `τ`, @@ -423,9 +423,9 @@ this far. originated, so at printing time we can recover what was originally written. -These problems illustrate that while it is possible to implement -cabal-exactprint using `GenericPackageDescription` as CST, it is not a -good fit because it would require copying the information on all non +These problems illustrate that while it is possible to implement Cabal +Exactprint using `GenericPackageDescription` as CST, it is not a good +fit because it would require copying the information on all non terminals of `[Field Position]` to all the leaves (i.e. sections to `GenericPackageDescription` components and monoidal fields that will be merged to parsed `FieldLine`s). @@ -448,7 +448,7 @@ field will bear the value of the parsed type that its corresponding field has. This was an interesting idea, but it would be a type that is only used -for cabal-exactprint. Field grammar finds a field by its string name. +for Cabal Exactprint. Field grammar finds a field by its string name. While in `TField` we retain the original field name, the value we get out of it will be typed. In other words, the type of the value we get from a TField depends on which field name it is. This lead to opening @@ -489,10 +489,10 @@ It would also benefit existing programs that depend on Cabal: It has three strategies to add dependencies that are tried in sequence. All of the strategies use parsed fields to guide stringy - manipulation directly within the source file. In comparison - `cabal-exactprint` will allow users to manipulate `[Field ann]` - instead. Or, even better, we should be able to implement cabal-add in - cabal directly. + manipulation directly within the source file. In comparison Cabal + Exactprint will allow users to manipulate `[Field ann]` instead. Or, + even better, we should be able to implement cabal-add in cabal + directly. - [*cabal-fmt*](https://github.com/phadej/cabal-fmt) diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 8195593..e3b90a9 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -51,7 +51,7 @@ "https://github.com/haskell/cabal-proposals/pull/5", )[Jappie's original Cabal Proposal], - // TODO: read these references + // TODO: read these references in details hpack-project: mk-smartlink("https://github.com/sol/hpack")[hpack], autopack-project: mk-smartlink("https://github.com/kowainik/autopack")[autopack], ) @@ -66,7 +66,7 @@ approach. == Summary -The Exact Printer project aims to develop a precise parsing and printing tool for .cabal files in the cabal library. +The Cabal Exactprint project aims to develop a precise parsing and printing tool for .cabal files in the cabal library. This will allow both cabal and other tools to introduce deltas into cabal files through a typed API, simplifying the modification/addition/removal of fields, without mangling the format, structure or comments of users files. @@ -121,13 +121,13 @@ can't: == Proposed Change We propose to leverage the existing `Field ann` data type, as well as -the `Parsec` and `Pretty` classes and their instances to implement cabal-exactprint. +the `Parsec` and `Pretty` classes and their instances to implement Cabal Exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. Currently Cabal doesn't store any of the comments. This is already implemented in #references.comment-parser-pr.get-link which is yet to be merged. -Firstly, we implement exact printing from `[Field ann]`. That is, +Firstly, we implement exactprinting from `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold, serving `[Field ann]` as the concrete syntax tree (CST). We chose it as the CST for its flexibility. As long as we respect its invariants during modification, @@ -260,7 +260,7 @@ order since september 2025 and what I learned from these attempts. be mentioned later in details. From then, I started experimenting using `[Field Position]` as the CST -to implement cabal-exactprint. +to implement Cabal Exactprint. - #references.typed-fields.get-link @@ -310,7 +310,7 @@ print the data `p` with its associated `TriviaTree` `t`. This model was appealing because it would allow us to maintain the same amount of fields in each constructor of a type that should support -cabal-exactprint, supporting backwards compatibility. +Cabal Exactprint, supporting backwards compatibility. Trivia trees are poor in structural composition. Let's describe the exactprint invariant as for a given `inp` and a type `τ`, @@ -455,7 +455,7 @@ this far. written. These problems illustrate that while it is possible to implement -cabal-exactprint using `GenericPackageDescription` as CST, it is not a +Cabal Exactprint using `GenericPackageDescription` as CST, it is not a good fit because it would require copying the information on all non terminals of `[Field Position]` to all the leaves (i.e.~sections to `GenericPackageDescription` components and monoidal fields that will be @@ -479,7 +479,7 @@ field will bear the value of the parsed type that its corresponding field has. This was an interesting idea, but it would be a type that is only used -for cabal-exactprint. Field grammar finds a field by its string name. +for Cabal Exactprint. Field grammar finds a field by its string name. While in `TField` we retain the original field name, the value we get out of it will be typed. In other words, the type of the value we get from a TField depends on which field name it is. This lead to opening @@ -517,7 +517,7 @@ It would also benefit existing programs that depend on Cabal: It has three strategies to add dependencies that are tried in sequence. All of the strategies use parsed fields to guide stringy manipulation directly within the source file. In comparison - `cabal-exactprint` will allow users to manipulate `[Field ann]` + Cabal Exactprint will allow users to manipulate `[Field ann]` instead. Or, even better, we should be able to implement cabal-add in cabal directly. From 19b47b62279bcf2dd8d063e4543e297d0133a8f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 24 Jul 2026 08:50:11 +0200 Subject: [PATCH 22/43] simple version --- proposals/cabal-exactprint.typ | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index e3b90a9..cc7c1b1 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -144,8 +144,8 @@ Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in #references.exact-printer-mega-issue.get-link is about being able to programmatically modify cabal files. With this mechanism, we expose a typed way to modify cabal files. For example, -translating an endomorphism over `Version` to an endomorphism over -`[FieldLines ann]`, which allows the user to modify the `cabal-version` +translating `SpecVersion -> SpecVersion` to `[FieldLines ann] -> [FieldLines ann]`, +which allows the user to modify the `cabal-version` field while having all the position validation already dealt with behind the scenes. From 050e438735fa68329986ba58e6c80cb8c4ce6232 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 24 Jul 2026 09:21:38 +0200 Subject: [PATCH 23/43] update description of the pain of GPD --- proposals/cabal-exactprint.md | 37 ++++++++++++++++++++++++++++++++-- proposals/cabal-exactprint.typ | 27 ++++++++++++++++++++++++- 2 files changed, 61 insertions(+), 3 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index dca2e59..74d0c55 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -103,8 +103,8 @@ facilitate building modification functions. A notable feature request in [*Exact-printer Mega-issue #7544*](https://github.com/haskell/cabal/issues/7544) is about being able to programmatically modify cabal files. With this mechanism, we -expose a typed way to modify cabal files. For example, translating an -endomorphism over `Version` to an endomorphism over `[FieldLines ann]`, +expose a typed way to modify cabal files. For example, translating +`SpecVersion -> SpecVersion` to `[FieldLines ann] -> [FieldLines ann]`, which allows the user to modify the `cabal-version` field while having all the position validation already dealt with behind the scenes. @@ -423,6 +423,39 @@ this far. originated, so at printing time we can recover what was originally written. +To demonstrate the added complexity of "losing the shape of +`[Field ann]`" casued by using `GenericPackageDescription`, we use the +previous definition of `MonoidalFieldAla` as example. It is the same as +the two following definition albeit generic in `ParsingPhase`. You can +see that in the `Abst`ract case, we maintain backwards-compatibility of +the type. + +``` haskell +type MonoidalFieldAlaConc a = [ ([Comment Position], BS.ByteString, (Positions, a)) ] +type MonoidalFieldAlaAbst a = a +``` + +From outside to inside, `MonoidalFieldAlaConc` represents the +bookkeeping of the following structure: + +- We maintain everything in a list to remember which field a + `build-depends` belongs to. This is because `build-depends` can be + merged. Each item in this list will be referred to as a *group*. + +- Each group has its associated comments because each group was + originally a list of field lines. + +- The ByteString here represents the original cased name of Fields. The + user could've written `BuIlD-DePenDs` and we would need to restore it + despite this string looks very funny. + +- `Positions` is a product type, describing the field's position this + group is associated to, as well as the position of the first field + line `a` was originally found when represented as a string. This is + the direct consequence of duplicating the Field's information to the + leaf, because the field itself is not representable in + `GenericPackageDescription`. + These problems illustrate that while it is possible to implement Cabal Exactprint using `GenericPackageDescription` as CST, it is not a good fit because it would require copying the information on all non diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index cc7c1b1..309da2b 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -416,7 +416,7 @@ this far. In a Cabal file, it is possible to have trivia for each section as well. The library stanza "library" is normalized to lower case in Cabal, but to achieve 100% roundtrip, we need to be able to save the - original string (which I call #emph[cased name];). Worse, cabal + original string (which I call _cased name_). Worse, cabal doesn't parse a simple component but a component wrapped in a conditional tree `CondTree`. `Library` is represented in a suboptimal way where non-conditional fields such as library name is nested within @@ -454,6 +454,31 @@ this far. originated, so at printing time we can recover what was originally written. +To demonstrate the added complexity of "losing the shape of `[Field ann]`" +casued by using `GenericPackageDescription`, we use +the previous definition of `MonoidalFieldAla` as example. +It is the same as the two following definition albeit generic in `ParsingPhase`. +You can see that in the `Abst`ract case, we maintain backwards-compatibility of the type. +```haskell +type MonoidalFieldAlaConc a = [ ([Comment Position], BS.ByteString, (Positions, a)) ] +type MonoidalFieldAlaAbst a = a +``` + +From outside to inside, `MonoidalFieldAlaConc` represents the bookkeeping of the following structure: + +- We maintain everything in a list to remember which field a `build-depends` belongs to. This is + because `build-depends` can be merged. Each item in this list will be referred to as a _group_. + +- Each group has its associated comments because each group was originally a list of field lines. + +- The ByteString here represents the original cased name of Fields. The user could've written + `BuIlD-DePenDs` and we would need to restore it despite this string looks very funny. + +- `Positions` is a product type, describing the field's position this group is associated to, as + well as the position of the first field line `a` was originally found when represented as a string. + This is the direct consequence of duplicating the Field's information to the leaf, because the + field itself is not representable in `GenericPackageDescription`. + These problems illustrate that while it is possible to implement Cabal Exactprint using `GenericPackageDescription` as CST, it is not a good fit because it would require copying the information on all non From 005303f68b6347d6f7bcf0ff097de57fa0b2cec7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 24 Jul 2026 09:39:46 +0200 Subject: [PATCH 24/43] it's actually called package description --- proposals/cabal-exactprint.md | 136 ++++++++++++++++++--------------- proposals/cabal-exactprint.typ | 68 ++++++++++------- 2 files changed, 117 insertions(+), 87 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 74d0c55..fbedd23 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -10,30 +10,31 @@ the updates in our approach. ## Summary The Cabal Exactprint project aims to develop a precise parsing and -printing tool for .cabal files in the cabal library. +printing tool for package descriptions in the cabal library. This will allow both cabal and other tools to introduce deltas into -cabal files through a typed API, simplifying the +package descriptions through a typed API, simplifying the modification/addition/removal of fields, without mangling the format, structure or comments of users files. -Furthermore it makes cabal authoritative on the cabal file format -allowing downstream users to use the provided printing functions and get -a stability guarantee. +Furthermore it makes cabal authoritative on the package description +format allowing downstream users to use the provided printing functions +and get a stability guarantee. We define the parse-print idempotency to be `print . parse == id`, which reads "parsing then printing is as if we've done nothing". We only focus -on ensuring this property to hold for valid cabal files, and we not -consider the braces syntax in this work. +on ensuring this property to hold for valid package descriptions, and we +don't consider the braces syntax in this work. ## Motivation -Cabal reads cabal package manifests in the cabal format (with the -extension .cabal). However, it is currently unable to modify it -loselessly. +Cabal builds packages by following stanzas written in [*package +descriptions*](https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions). +These files have the extension `.cabal`. Cabal is currently unable to +modify such files loselessly. -Here are some of the symptoms of this problem manifesting in different -ways through out the cabal CLI: +Here are some of the symptoms manifesting in different ways through out +the Cabal CLI: - `cabal format` @@ -45,20 +46,22 @@ ways through out the cabal CLI: - `cabal add` Cabal should be able to add a dependency to a component. This can't be - implemented because modifying a portion of the cabal file's in-memory - representation mangles the entire cabal file, similar to - `cabal format`. drops all comments and merges imports. + implemented because modifying a portion of the package description's + in-memory representation mangles the entire package description, + similar to `cabal format`. -- Missing module declaration When a module exists but is not declared in - the cabal file, cabal can't add it for you. Again, because cabal would - mangle the cabal file if it tries to touch it. It can only tell you - that it's missing. Argh. +- Missing module declaration + + When a module is needed but is not declared in the package + description, Cabal can't add it for you. Again, because cabal would + mangle the package description if it tries to touch it. It can only + tell you that it's missing. Argh. - `cabal gen-bounds` Cabal is very helpful and can generate dependency constraints ("bounds") for you. However, it just dumps them in the terminal, - because it can't modify the cabal file. + because it can't modify the package description. Cabal is also not authoritative in this matter, many projects have been created to do what cabal can't: @@ -90,28 +93,29 @@ flexibility. As long as we respect its invariants during modification, unchanged parts in the output should stay the same, and changed parts should translate to local transformation in the output string. In our current prototype, we are already able to roundtrip 119662 out of 194557 -cabal files of hackage (\~60%) with an implementation that is concise -and simple. To increase the percentage of successful roundtrip, we need -to detect CRLF/LF and exactprint accordingly; furthermore, we can't -figure out whether a whitespace was a tab or a space yet. These will -require changes to the lexer which we have previously done in [*Retain -comments in field parser +package descriptions of hackage (\~60%) with an implementation that is +concise and simple. To increase the percentage of successful roundtrip, +we need to detect CRLF/LF and exactprint accordingly; furthermore, we +can't figure out whether a whitespace was a tab or a space yet. These +will require changes to the lexer which we have previously done in +[*Retain comments in field parser #11252*](https://github.com/haskell/cabal/pull/11252). Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in [*Exact-printer Mega-issue #7544*](https://github.com/haskell/cabal/issues/7544) is about being -able to programmatically modify cabal files. With this mechanism, we -expose a typed way to modify cabal files. For example, translating -`SpecVersion -> SpecVersion` to `[FieldLines ann] -> [FieldLines ann]`, -which allows the user to modify the `cabal-version` field while having -all the position validation already dealt with behind the scenes. +able to programmatically modify package descriptions. With this +mechanism, we expose a typed way to modify package descriptions. For +example, translating `SpecVersion -> SpecVersion` to +`[FieldLines ann] -> [FieldLines ann]`, which allows the user to modify +the `cabal-version` field while having all the position validation +already dealt with behind the scenes. We will use the `Parsec` and `Pretty` classes to implement the typed -modification framework. Each field in a Cabal file is represented by a -field name in association with some field lines. Upon modification, we -proceed with the following steps: +modification framework. Each field in a package description is +represented by a field name in association with some field lines. Upon +modification, we proceed with the following steps: - Should the field lines be non empty, join them into a single field line `fl` with indentation and newlines. @@ -218,12 +222,12 @@ order since september 2025 and what I learned from these attempts. We draw inspiration from ghc-exactprint and its trees that grow model, annotating data structurally on each extension point. Reaching the end of the design space of this approach with just enough fields - implemented to make two cabal files Hackage roundtrip 100%, inherent - problems of using `GenericPackageDescription` as CST to implement - exactprint started to catch my eyes. This is the first successful - approach where syntactic roundtrip property of `Pretty`/`Parsec` are - preserved with composition. The details of the inherent problems will - be mentioned later in details. + implemented to make two package description from Hackage roundtrip + 100%, inherent problems of using `GenericPackageDescription` as CST to + implement exactprint started to catch my eyes. This is the first + successful approach where syntactic roundtrip property of + `Pretty`/`Parsec` are preserved with composition. The details of the + inherent problems will be mentioned later in details. From then, I started experimenting using `[Field Position]` as the CST to implement Cabal Exactprint. @@ -239,10 +243,10 @@ to implement Cabal Exactprint. (the goal was one per known cabal field). This attempt proved that modifying or printing `[Field Position]` (or - something isomorphic to it) is a lot easier. The shape of a cabal file - is not lost and better reflects what was originally written. Also we - would avoid threading everything through field grammar, which proved - to be unwieldy. + something isomorphic to it) is a lot easier. The shape of the field + syntax is not lost and better reflects what was originally written. + Also we would avoid threading everything through field grammar, which + proved to be unwieldy. However, this resulted in the `FieldLine` bearing a too specific type for the field grammar and casting will be necessary, rendering the @@ -380,13 +384,13 @@ this far. - Regarding losing the shape of the sections: - In a Cabal file, it is possible to have trivia for each section as - well. The library stanza "library" is normalized to lower case in - Cabal, but to achieve 100% roundtrip, we need to be able to save the - original string (which I call *cased name*). Worse, cabal doesn't - parse a simple component but a component wrapped in a conditional tree - `CondTree`. `Library` is represented in a suboptimal way where - non-conditional fields such as library name is nested within + In a package description, it is possible to have trivia for each + section as well. The library stanza "library" is normalized to lower + case in Cabal, but to achieve 100% roundtrip, we need to be able to + save the original string (which I call *cased name*). Worse, cabal + doesn't parse a simple component but a component wrapped in a + conditional tree `CondTree`. `Library` is represented in a suboptimal + way where non-conditional fields such as library name is nested within `CondTree`. The stop-gap solution would be to insert a `Maybe ByteString` that is the original cased name into the parsed library only at the top level of the `CondTree`. @@ -506,15 +510,15 @@ benefit the functionality of Cabal itself many ways, namely the following: - New command `cabal add` that adds a dependency automatically by - editing the cabal file. + editing the package description. -- `cabal gen-bounds` can modify the bounds of a cabal file. +- `cabal gen-bounds` can modify the bounds of a package description. -- `cabal format` can format a cabal file in a canonical way while - preserving comments. +- `cabal format` can format a package description in a canonical way + while preserving comments. - `cabal init` can leverage the "addition" part of the modification - framework and generate cabal files easily. + framework and generate package description easily. It would also benefit existing programs that depend on Cabal: @@ -529,10 +533,16 @@ It would also benefit existing programs that depend on Cabal: - [*cabal-fmt*](https://github.com/phadej/cabal-fmt) - It parses the cabal file twice: once with `readFields` from cabal, and - again with its own parser to find all the comments. This can be - simplified the new `readFieldsWithComments` in [*Retain comments in - field parser #11252*](https://github.com/haskell/cabal/pull/11252). + It parses the package description twice: once with `readFields` from + cabal, and again with its own parser to find all the comments. This + can be simplified the new `readFieldsWithComments` in [*Retain + comments in field parser + #11252*](https://github.com/haskell/cabal/pull/11252). + +This work would also simplify implementation of formatters or +modification tools operating on other formats using the same envelope +format, namely [*"project +descriptions"*](https://cabal.readthedocs.io/en/stable/cabal-project-description-file.html#project-description-cabal-project-file). ## Implementation Notes @@ -567,6 +577,12 @@ describe the modification API in terms of lens. - [*Typed-fields leana8959/cabal/typed-fields*](https://github.com/haskell/cabal/pull/11690) +- [*Cabal manual/Package + Descriptions*](https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions) + +- [*Cabal manual/Project + Descriptions*](https://cabal.readthedocs.io/en/stable/cabal-project-description-file.html#project-description-cabal-project-file) + - [*Biparsers: Exact Printing for Data Synchronisation*](https://dl.acm.org/doi/full/10.1145/3704910) diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 309da2b..1130486 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -27,6 +27,14 @@ "https://github.com/haskell/cabal/pull/11690", )[Typed-fields leana8959/cabal/typed-fields], + package-description-documentation: mk-smartlink( + "https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions", + )[Cabal manual/Package Descriptions], + + project-description-documentation: mk-smartlink( + "https://cabal.readthedocs.io/en/stable/cabal-project-description-file.html#project-description-cabal-project-file", + )[Cabal manual/Project Descriptions], + biparsers: mk-smartlink( "https://dl.acm.org/doi/full/10.1145/3704910", )[Biparsers: Exact Printing for Data Synchronisation], @@ -66,25 +74,27 @@ approach. == Summary -The Cabal Exactprint project aims to develop a precise parsing and printing tool for .cabal files in the cabal library. +The Cabal Exactprint project aims to develop a precise parsing and printing tool for package +descriptions in the cabal library. -This will allow both cabal and other tools to introduce deltas into cabal files through a typed API, +This will allow both cabal and other tools to introduce deltas into package descriptions through a typed API, simplifying the modification/addition/removal of fields, without mangling the format, structure or comments of users files. -Furthermore it makes cabal authoritative on the cabal file format -allowing downstream users to use the provided printing functions -and get a stability guarantee. +Furthermore it makes cabal authoritative on the package description format allowing downstream +users to use the provided printing functions and get a stability guarantee. We define the parse-print idempotency to be `print . parse == id`, which reads "parsing then printing is as if we've done nothing". We only focus on ensuring this property to hold for valid -cabal files, and we not consider the braces syntax in this work. +package descriptions, and we don't consider the braces syntax in this work. + +// TODO: cite braces syntax == Motivation -Cabal reads cabal package manifests in the cabal format (with the extension -.cabal). However, it is currently unable to modify it loselessly. +Cabal builds packages by following stanzas written in #(references.package-description-documentation.override-name)[package descriptions]. +These files have the extension `.cabal`. Cabal is currently unable to modify such files loselessly. -Here are some of the symptoms of this problem manifesting in different ways through out the cabal CLI: +Here are some of the symptoms manifesting in different ways through out the Cabal CLI: - `cabal format` @@ -95,19 +105,19 @@ Here are some of the symptoms of this problem manifesting in different ways thro - `cabal add` Cabal should be able to add a dependency to a component. - This can't be implemented because modifying a portion of the cabal file's in-memory representation - mangles the entire cabal file, similar to `cabal format`. - drops all comments and merges imports. + This can't be implemented because modifying a portion of the package description's in-memory representation + mangles the entire package description, similar to `cabal format`. - Missing module declaration - When a module exists but is not declared in the cabal file, cabal can't add it for you. - Again, because cabal would mangle the cabal file if it tries to touch it. + + When a module is needed but is not declared in the package description, Cabal can't add it for you. + Again, because cabal would mangle the package description if it tries to touch it. It can only tell you that it's missing. Argh. - `cabal gen-bounds` Cabal is very helpful and can generate dependency constraints ("bounds") for you. - However, it just dumps them in the terminal, because it can't modify the cabal file. + However, it just dumps them in the terminal, because it can't modify the package description. Cabal is also not authoritative in this matter, many projects have been created to do what cabal @@ -134,7 +144,7 @@ flexibility. As long as we respect its invariants during modification, unchanged parts in the output should stay the same, and changed parts should translate to local transformation in the output string. In our current prototype, we are already able to roundtrip 119662 out of 194557 -cabal files of hackage (\~60%) with an implementation that is concise +package descriptions of hackage (\~60%) with an implementation that is concise and simple. To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't figure out whether a whitespace was a tab or a space yet. These will require @@ -142,15 +152,15 @@ changes to the lexer which we have previously done in #references.comment-parser Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in #references.exact-printer-mega-issue.get-link -is about being able to programmatically modify cabal files. With this -mechanism, we expose a typed way to modify cabal files. For example, +is about being able to programmatically modify package descriptions. With this +mechanism, we expose a typed way to modify package descriptions. For example, translating `SpecVersion -> SpecVersion` to `[FieldLines ann] -> [FieldLines ann]`, which allows the user to modify the `cabal-version` field while having all the position validation already dealt with behind the scenes. We will use the `Parsec` and `Pretty` classes to implement the typed -modification framework. Each field in a Cabal file is represented by a +modification framework. Each field in a package description is represented by a field name in association with some field lines. Upon modification, we proceed with the following steps: @@ -252,7 +262,7 @@ order since september 2025 and what I learned from these attempts. We draw inspiration from ghc-exactprint and its trees that grow model, annotating data structurally on each extension point. Reaching the end of the design space of this approach with just enough fields - implemented to make two cabal files Hackage roundtrip 100%, inherent + implemented to make two package description from Hackage roundtrip 100%, inherent problems of using `GenericPackageDescription` as CST to implement exactprint started to catch my eyes. This is the first successful approach where syntactic roundtrip property of `Pretty`/`Parsec` are @@ -272,7 +282,7 @@ to implement Cabal Exactprint. field). This attempt proved that modifying or printing `[Field Position]` (or - something isomorphic to it) is a lot easier. The shape of a cabal file + something isomorphic to it) is a lot easier. The shape of the field syntax is not lost and better reflects what was originally written. Also we would avoid threading everything through field grammar, which proved to be unwieldy. @@ -413,7 +423,7 @@ this far. - Regarding losing the shape of the sections: - In a Cabal file, it is possible to have trivia for each section as + In a package description, it is possible to have trivia for each section as well. The library stanza "library" is normalized to lower case in Cabal, but to achieve 100% roundtrip, we need to be able to save the original string (which I call _cased name_). Worse, cabal @@ -528,12 +538,12 @@ this would benefit the functionality of Cabal itself many ways, namely the following: - New command `cabal add` that adds a dependency automatically by - editing the cabal file. -- `cabal gen-bounds` can modify the bounds of a cabal file. -- `cabal format` can format a cabal file in a canonical way while + editing the package description. +- `cabal gen-bounds` can modify the bounds of a package description. +- `cabal format` can format a package description in a canonical way while preserving comments. - `cabal init` can leverage the "addition" part of the modification - framework and generate cabal files easily. + framework and generate package description easily. It would also benefit existing programs that depend on Cabal: @@ -548,10 +558,14 @@ It would also benefit existing programs that depend on Cabal: - #references.cabal-fmt-project.get-link - It parses the cabal file twice: once with `readFields` from cabal, and + It parses the package description twice: once with `readFields` from cabal, and again with its own parser to find all the comments. This can be simplified the new `readFieldsWithComments` in #references.comment-parser-pr.get-link. + +This work would also simplify implementation of formatters or modification tools operating on +other formats using the same envelope format, namely #(references.project-description-documentation.override-name)["project descriptions"]. + == Implementation Notes references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and From 3bfa5a42c3987df02280834ad9c1adfe4dbfb563 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 24 Jul 2026 09:52:27 +0200 Subject: [PATCH 25/43] add details to each prior art, sort references by what they are --- proposals/cabal-exactprint.md | 56 ++++++++++++++++++---------------- proposals/cabal-exactprint.typ | 56 ++++++++++++++++++---------------- 2 files changed, 59 insertions(+), 53 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index fbedd23..39d8fdc 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -18,7 +18,7 @@ modification/addition/removal of fields, without mangling the format, structure or comments of users files. Furthermore it makes cabal authoritative on the package description -format allowing downstream users to use the provided printing functions +format, allowing downstream users to use the provided printing functions and get a stability guarantee. We define the parse-print idempotency to be `print . parse == id`, which @@ -45,10 +45,11 @@ the Cabal CLI: - `cabal add` - Cabal should be able to add a dependency to a component. This can't be - implemented because modifying a portion of the package description's - in-memory representation mangles the entire package description, - similar to `cabal format`. + Cabal should be able to add a dependency to a component. The Rust + language's cabal equivalent, `cargo`, does this just fine. This can't + be implemented in Cabal yet because modifying a portion of the package + description's in-memory representation mangles the entire package + description, similar to `cabal format`. - Missing module declaration @@ -66,13 +67,16 @@ the Cabal CLI: Cabal is also not authoritative in this matter, many projects have been created to do what cabal can't: -- [*cabal-fmt*](https://github.com/phadej/cabal-fmt) +- [*cabal-fmt*](https://github.com/phadej/cabal-fmt), a formatter. -- [*cabal-add*](https://github.com/Bodigrim/cabal-add) +- [*cabal-add*](https://github.com/Bodigrim/cabal-add), an non-official + implementation of `cabal add` command. -- [*hpack*](https://github.com/sol/hpack) +- [*hpack*](https://github.com/sol/hpack), an alternative to package + description. -- [*autopack*](https://github.com/kowainik/autopack) +- [*autopack*](https://github.com/kowainik/autopack), automatic module + descovery. ## Proposed Change @@ -566,9 +570,6 @@ describe the modification API in terms of lens. - [*Retain comments in field parser #11252*](https://github.com/haskell/cabal/pull/11252) -- [*Exact-printer Mega-issue - #7544*](https://github.com/haskell/cabal/issues/7544) - - [*TriviaTree #11425*](https://github.com/haskell/cabal/pull/11425) - [*Barbie/Trees-that-grow @@ -577,6 +578,23 @@ describe the modification API in terms of lens. - [*Typed-fields leana8959/cabal/typed-fields*](https://github.com/haskell/cabal/pull/11690) +- [*cabal-add*](https://github.com/Bodigrim/cabal-add) + +- [*cabal-fmt*](https://github.com/phadej/cabal-fmt) + +- [*hpack*](https://github.com/sol/hpack) + +- [*autopack*](https://github.com/kowainik/autopack) + +- [*Jappie's original Haskell Foundation Tech + Proposal*](https://github.com/haskellfoundation/tech-proposals/pull/65) + +- [*Jappie's original Cabal + Proposal*](https://github.com/haskell/cabal-proposals/pull/5) + +- [*Exact-printer Mega-issue + #7544*](https://github.com/haskell/cabal/issues/7544) + - [*Cabal manual/Package Descriptions*](https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions) @@ -592,20 +610,6 @@ describe the modification API in terms of lens. - [*Trees that Grow*](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) -- [*cabal-add*](https://github.com/Bodigrim/cabal-add) - -- [*cabal-fmt*](https://github.com/phadej/cabal-fmt) - -- [*Jappie's original Haskell Foundation Tech - Proposal*](https://github.com/haskellfoundation/tech-proposals/pull/65) - -- [*Jappie's original Cabal - Proposal*](https://github.com/haskell/cabal-proposals/pull/5) - -- [*hpack*](https://github.com/sol/hpack) - -- [*autopack*](https://github.com/kowainik/autopack) - [^1]: In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, and `library` is a section that can take a library name as a section diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 1130486..bf4604a 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -7,14 +7,11 @@ ) #let references = ( + // prior arts comment-parser-pr: mk-smartlink( "https://github.com/haskell/cabal/pull/11252", )[Retain comments in field parser \#11252], - exact-printer-mega-issue: mk-smartlink( - "https://github.com/haskell/cabal/issues/7544", - )[Exact-printer Mega-issue \#7544], - trivia-tree: mk-smartlink( "https://github.com/haskell/cabal/pull/11425", )[TriviaTree \#11425], @@ -27,6 +24,26 @@ "https://github.com/haskell/cabal/pull/11690", )[Typed-fields leana8959/cabal/typed-fields], + cabal-add-project: mk-smartlink("https://github.com/Bodigrim/cabal-add")[cabal-add], + cabal-fmt-project: mk-smartlink("https://github.com/phadej/cabal-fmt")[cabal-fmt], + hpack-project: mk-smartlink("https://github.com/sol/hpack")[hpack], + autopack-project: mk-smartlink("https://github.com/kowainik/autopack")[autopack], + + // proposals + jappie-original-twg-proposal: mk-smartlink( + "https://github.com/haskellfoundation/tech-proposals/pull/65", + )[Jappie's original Haskell Foundation Tech Proposal], + + jappie-original-cabal-proposal: mk-smartlink( + "https://github.com/haskell/cabal-proposals/pull/5", + )[Jappie's original Cabal Proposal], + + // misc + exact-printer-mega-issue: mk-smartlink( + "https://github.com/haskell/cabal/issues/7544", + )[Exact-printer Mega-issue \#7544], + + // documentation, references, papers package-description-documentation: mk-smartlink( "https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions", )[Cabal manual/Package Descriptions], @@ -46,22 +63,6 @@ trees-that-grow: mk-smartlink( "https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf", )[Trees that Grow], - - cabal-add-project: mk-smartlink("https://github.com/Bodigrim/cabal-add")[cabal-add], - - cabal-fmt-project: mk-smartlink("https://github.com/phadej/cabal-fmt")[cabal-fmt], - - jappie-original-twg-proposal: mk-smartlink( - "https://github.com/haskellfoundation/tech-proposals/pull/65", - )[Jappie's original Haskell Foundation Tech Proposal], - - jappie-original-cabal-proposal: mk-smartlink( - "https://github.com/haskell/cabal-proposals/pull/5", - )[Jappie's original Cabal Proposal], - - // TODO: read these references in details - hpack-project: mk-smartlink("https://github.com/sol/hpack")[hpack], - autopack-project: mk-smartlink("https://github.com/kowainik/autopack")[autopack], ) @@ -80,7 +81,7 @@ descriptions in the cabal library. This will allow both cabal and other tools to introduce deltas into package descriptions through a typed API, simplifying the modification/addition/removal of fields, without mangling the format, structure or comments of users files. -Furthermore it makes cabal authoritative on the package description format allowing downstream +Furthermore it makes cabal authoritative on the package description format, allowing downstream users to use the provided printing functions and get a stability guarantee. We define the parse-print idempotency to be `print . parse == id`, which reads "parsing then @@ -104,8 +105,9 @@ Here are some of the symptoms manifesting in different ways through out the Caba - `cabal add` - Cabal should be able to add a dependency to a component. - This can't be implemented because modifying a portion of the package description's in-memory representation + Cabal should be able to add a dependency to a component. The Rust language's cabal equivalent, + `cargo`, does this just fine. + This can't be implemented in Cabal yet because modifying a portion of the package description's in-memory representation mangles the entire package description, similar to `cabal format`. - Missing module declaration @@ -123,10 +125,10 @@ Here are some of the symptoms manifesting in different ways through out the Caba Cabal is also not authoritative in this matter, many projects have been created to do what cabal can't: -- #references.cabal-fmt-project.get-link -- #references.cabal-add-project.get-link -- #references.hpack-project.get-link -- #references.autopack-project.get-link +- #references.cabal-fmt-project.get-link, a formatter. +- #references.cabal-add-project.get-link, an non-official implementation of `cabal add` command. +- #references.hpack-project.get-link, an alternative to package description. +- #references.autopack-project.get-link, automatic module descovery. == Proposed Change From e1dc44f7b02b84294cb4e20fc84c800a8ad892a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 24 Jul 2026 09:54:01 +0200 Subject: [PATCH 26/43] don't italicize links --- proposals/cabal-exactprint.md | 143 ++++++++++++++++----------------- proposals/cabal-exactprint.typ | 4 +- 2 files changed, 72 insertions(+), 75 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 39d8fdc..8da82f0 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -1,10 +1,10 @@ # Cabal Exactprint -Note that this proposal builds on the earlier [*TWG (Technical Working +Note that this proposal builds on the earlier [TWG (Technical Working Group) -proposal*](https://github.com/haskellfoundation/tech-proposals/pull/65), -also succeeds [*Jappie's original Cabal -Proposal*](https://github.com/haskell/cabal-proposals/pull/5) to match +proposal](https://github.com/haskellfoundation/tech-proposals/pull/65), +also succeeds [Jappie's original Cabal +Proposal](https://github.com/haskell/cabal-proposals/pull/5) to match the updates in our approach. ## Summary @@ -28,8 +28,8 @@ don't consider the braces syntax in this work. ## Motivation -Cabal builds packages by following stanzas written in [*package -descriptions*](https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions). +Cabal builds packages by following stanzas written in [package +descriptions](https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions). These files have the extension `.cabal`. Cabal is currently unable to modify such files loselessly. @@ -67,15 +67,15 @@ the Cabal CLI: Cabal is also not authoritative in this matter, many projects have been created to do what cabal can't: -- [*cabal-fmt*](https://github.com/phadej/cabal-fmt), a formatter. +- [cabal-fmt](https://github.com/phadej/cabal-fmt), a formatter. -- [*cabal-add*](https://github.com/Bodigrim/cabal-add), an non-official +- [cabal-add](https://github.com/Bodigrim/cabal-add), an non-official implementation of `cabal add` command. -- [*hpack*](https://github.com/sol/hpack), an alternative to package +- [hpack](https://github.com/sol/hpack), an alternative to package description. -- [*autopack*](https://github.com/kowainik/autopack), automatic module +- [autopack](https://github.com/kowainik/autopack), automatic module descovery. ## Proposed Change @@ -86,9 +86,9 @@ Exactprint. As a preliminary task, we modify the cabal lexer and field parser's definition to retain comments. Currently Cabal doesn't store any of the -comments. This is already implemented in [*Retain comments in field -parser #11252*](https://github.com/haskell/cabal/pull/11252) which is -yet to be merged. +comments. This is already implemented in [Retain comments in field +parser #11252](https://github.com/haskell/cabal/pull/11252) which is yet +to be merged. Firstly, we implement exactprinting from `[Field ann]`. That is, `exactRenderFields . readFields = id` should hold, serving `[Field ann]` @@ -102,16 +102,16 @@ concise and simple. To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't figure out whether a whitespace was a tab or a space yet. These will require changes to the lexer which we have previously done in -[*Retain comments in field parser -#11252*](https://github.com/haskell/cabal/pull/11252). +[Retain comments in field parser +#11252](https://github.com/haskell/cabal/pull/11252). Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in -[*Exact-printer Mega-issue -#7544*](https://github.com/haskell/cabal/issues/7544) is about being -able to programmatically modify package descriptions. With this -mechanism, we expose a typed way to modify package descriptions. For -example, translating `SpecVersion -> SpecVersion` to +[Exact-printer Mega-issue +#7544](https://github.com/haskell/cabal/issues/7544) is about being able +to programmatically modify package descriptions. With this mechanism, we +expose a typed way to modify package descriptions. For example, +translating `SpecVersion -> SpecVersion` to `[FieldLines ann] -> [FieldLines ann]`, which allows the user to modify the `cabal-version` field while having all the position validation already dealt with behind the scenes. @@ -207,8 +207,8 @@ will always be correct. Below is an exhaustive list of the changes we tried in chronological order since september 2025 and what I learned from these attempts. -- [*Trivia Tree #11425 (proof of - concept)*](https://github.com/haskell/cabal/pull/11425) implements a +- [Trivia Tree #11425 (proof of + concept)](https://github.com/haskell/cabal/pull/11425) implements a untyped tree `TriviaTree` using existential type. With it, we can imtate the shape of a recursive type `τ` freely and @@ -219,9 +219,9 @@ order since september 2025 and what I learned from these attempts. wrong and parsers/printers lose their roundtrip guarantee during composition. -- [*Barbie/Trees-that-grow #11690 (proof of - concept)*](https://github.com/haskell/cabal/pull/11690) tries to do - the same thing as Trivia-tree in a typed way. +- [Barbie/Trees-that-grow #11690 (proof of + concept)](https://github.com/haskell/cabal/pull/11690) tries to do the + same thing as Trivia-tree in a typed way. We draw inspiration from ghc-exactprint and its trees that grow model, annotating data structurally on each extension point. Reaching the end @@ -236,11 +236,11 @@ order since september 2025 and what I learned from these attempts. From then, I started experimenting using `[Field Position]` as the CST to implement Cabal Exactprint. -- [*Typed-fields - leana8959/cabal/typed-fields*](https://github.com/haskell/cabal/pull/11690) +- [Typed-fields + leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690) - Rereading the original [*Exact-printer Mega-issue - #7544*](https://github.com/haskell/cabal/issues/7544), I realized that + Rereading the original [Exact-printer Mega-issue + #7544](https://github.com/haskell/cabal/issues/7544), I realized that a big part of the demand was to modify `[Field Position]` in a typed way which doesn't necessarily need GPD. To allow typed modification in the fields, we extended the `Field` data type to have more constructor @@ -277,8 +277,8 @@ of `p`'s children. As long as the construction and the deconstruction matches up, the trivia can be successfully recovered. -Inspired by [*Biparsers: Exact Printing for Data -Synchronisation*](https://dl.acm.org/doi/full/10.1145/3704910), trivia +Inspired by [Biparsers: Exact Printing for Data +Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910), trivia tree is passed around along the data. Each parser is extended to return a pair `(p, t)` where `p` is the data parsed and `t` is the associated `TriviaTree`. The printer is extended to receive `(p, t)` to print the @@ -326,8 +326,8 @@ Barbie. ### "Barbie/Trees-that-grow" -[*Barbie library*](https://hackage.haskell.org/package/barbies-2.1.1.0) -is a pattern that parameterizes a data declaration with a higher kinded +[Barbie library](https://hackage.haskell.org/package/barbies-2.1.1.0) is +a pattern that parameterizes a data declaration with a higher kinded type parameter (commonly `f :: Type -> Type`). By leveraging this type parameter, we can share the spine of the data type but have each leaf in a different context. @@ -363,8 +363,8 @@ type MonoidalFieldAla (m :: ParsingPhase) (a :: Type) = ``` After refining the idea, it ended up being quite similar to the famous -[*Trees that -Grow*](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) +[Trees that +Grow](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) idiom used in GHC to implement ghc-exactprint. A bearable albeit major inconvenience is constraints. In the existing @@ -508,10 +508,9 @@ foresee any backwards-compatibility issues. ## Interested parties -As outlined in [*Exact-printer Mega-issue -#7544*](https://github.com/haskell/cabal/issues/7544), this would -benefit the functionality of Cabal itself many ways, namely the -following: +As outlined in [Exact-printer Mega-issue +#7544](https://github.com/haskell/cabal/issues/7544), this would benefit +the functionality of Cabal itself many ways, namely the following: - New command `cabal add` that adds a dependency automatically by editing the package description. @@ -526,7 +525,7 @@ following: It would also benefit existing programs that depend on Cabal: -- [*cabal-add*](https://github.com/Bodigrim/cabal-add) +- [cabal-add](https://github.com/Bodigrim/cabal-add) It has three strategies to add dependencies that are tried in sequence. All of the strategies use parsed fields to guide stringy @@ -535,18 +534,17 @@ It would also benefit existing programs that depend on Cabal: even better, we should be able to implement cabal-add in cabal directly. -- [*cabal-fmt*](https://github.com/phadej/cabal-fmt) +- [cabal-fmt](https://github.com/phadej/cabal-fmt) It parses the package description twice: once with `readFields` from cabal, and again with its own parser to find all the comments. This - can be simplified the new `readFieldsWithComments` in [*Retain - comments in field parser - #11252*](https://github.com/haskell/cabal/pull/11252). + can be simplified the new `readFieldsWithComments` in [Retain comments + in field parser #11252](https://github.com/haskell/cabal/pull/11252). This work would also simplify implementation of formatters or modification tools operating on other formats using the same envelope -format, namely [*"project -descriptions"*](https://cabal.readthedocs.io/en/stable/cabal-project-description-file.html#project-description-cabal-project-file). +format, namely ["project +descriptions"](https://cabal.readthedocs.io/en/stable/cabal-project-description-file.html#project-description-cabal-project-file). ## Implementation Notes @@ -567,48 +565,47 @@ describe the modification API in terms of lens. ## References -- [*Retain comments in field parser - #11252*](https://github.com/haskell/cabal/pull/11252) +- [Retain comments in field parser + #11252](https://github.com/haskell/cabal/pull/11252) -- [*TriviaTree #11425*](https://github.com/haskell/cabal/pull/11425) +- [TriviaTree #11425](https://github.com/haskell/cabal/pull/11425) -- [*Barbie/Trees-that-grow - #11690*](https://github.com/haskell/cabal/pull/11690) +- [Barbie/Trees-that-grow + #11690](https://github.com/haskell/cabal/pull/11690) -- [*Typed-fields - leana8959/cabal/typed-fields*](https://github.com/haskell/cabal/pull/11690) +- [Typed-fields + leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690) -- [*cabal-add*](https://github.com/Bodigrim/cabal-add) +- [cabal-add](https://github.com/Bodigrim/cabal-add) -- [*cabal-fmt*](https://github.com/phadej/cabal-fmt) +- [cabal-fmt](https://github.com/phadej/cabal-fmt) -- [*hpack*](https://github.com/sol/hpack) +- [hpack](https://github.com/sol/hpack) -- [*autopack*](https://github.com/kowainik/autopack) +- [autopack](https://github.com/kowainik/autopack) -- [*Jappie's original Haskell Foundation Tech - Proposal*](https://github.com/haskellfoundation/tech-proposals/pull/65) +- [Jappie's original Haskell Foundation Tech + Proposal](https://github.com/haskellfoundation/tech-proposals/pull/65) -- [*Jappie's original Cabal - Proposal*](https://github.com/haskell/cabal-proposals/pull/5) +- [Jappie's original Cabal + Proposal](https://github.com/haskell/cabal-proposals/pull/5) -- [*Exact-printer Mega-issue - #7544*](https://github.com/haskell/cabal/issues/7544) +- [Exact-printer Mega-issue + #7544](https://github.com/haskell/cabal/issues/7544) -- [*Cabal manual/Package - Descriptions*](https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions) +- [Cabal manual/Package + Descriptions](https://cabal.readthedocs.io/en/stable/cabal-package-description-file.html#package-descriptions) -- [*Cabal manual/Project - Descriptions*](https://cabal.readthedocs.io/en/stable/cabal-project-description-file.html#project-description-cabal-project-file) +- [Cabal manual/Project + Descriptions](https://cabal.readthedocs.io/en/stable/cabal-project-description-file.html#project-description-cabal-project-file) -- [*Biparsers: Exact Printing for Data - Synchronisation*](https://dl.acm.org/doi/full/10.1145/3704910) +- [Biparsers: Exact Printing for Data + Synchronisation](https://dl.acm.org/doi/full/10.1145/3704910) -- [*Barbie - library*](https://hackage.haskell.org/package/barbies-2.1.1.0) +- [Barbie library](https://hackage.haskell.org/package/barbies-2.1.1.0) -- [*Trees that - Grow*](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) +- [Trees that + Grow](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) [^1]: In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, and diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index bf4604a..3c25928 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -2,8 +2,8 @@ #let mk-smartlink(url, name) = ( - get-link: link(url)[_#(name)_], - override-name: new-name => link(url)[_#(new-name)_], + get-link: link(url)[#name], + override-name: new-name => link(url)[#new-name], ) #let references = ( From f9b8420ac0a06bac0719e2eaf4c63c910915f640 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 24 Jul 2026 09:59:46 +0200 Subject: [PATCH 27/43] describe how to run the poc --- proposals/cabal-exactprint.md | 33 ++++++++++++++++++++++----------- proposals/cabal-exactprint.typ | 20 ++++++++++++++------ 2 files changed, 36 insertions(+), 17 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 8da82f0..dd438f4 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -97,13 +97,14 @@ flexibility. As long as we respect its invariants during modification, unchanged parts in the output should stay the same, and changed parts should translate to local transformation in the output string. In our current prototype, we are already able to roundtrip 119662 out of 194557 -package descriptions of hackage (\~60%) with an implementation that is -concise and simple. To increase the percentage of successful roundtrip, -we need to detect CRLF/LF and exactprint accordingly; furthermore, we -can't figure out whether a whitespace was a tab or a space yet. These -will require changes to the lexer which we have previously done in -[Retain comments in field parser -#11252](https://github.com/haskell/cabal/pull/11252). +package descriptions of hackage (\~60%) with [an implementation that is +concise and +simple](https://github.com/leana8959/cabal/tree/transform-fields) [^1]. +To increase the percentage of successful roundtrip, we need to detect +CRLF/LF and exactprint accordingly; furthermore, we can't figure out +whether a whitespace was a tab or a space yet. These will require +changes to the lexer which we have previously done in [Retain comments +in field parser #11252](https://github.com/haskell/cabal/pull/11252). Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in @@ -168,7 +169,7 @@ cases to ensure that important invariants are preserved, namely that We want to let user describe a single modification that we call `Edit` by specifying a focus and a transformation. Here we add a new dependency `myNewDep` as an example. This modification can be expressed in plain -English as "within the section library with no arguments [^1], within +English as "within the section library with no arguments [^2], within the field `build-depends`, add (append) a `myNewDep`." In pseudo Haskell of the API we intend to build the aforementioned example modification can be described as: @@ -237,7 +238,7 @@ From then, I started experimenting using `[Field Position]` as the CST to implement Cabal Exactprint. - [Typed-fields - leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690) + leana8959/cabal/typed-fields](https://github.com/leana8959/cabal/tree/typed-fields) Rereading the original [Exact-printer Mega-issue #7544](https://github.com/haskell/cabal/issues/7544), I realized that @@ -574,7 +575,7 @@ describe the modification API in terms of lens. #11690](https://github.com/haskell/cabal/pull/11690) - [Typed-fields - leana8959/cabal/typed-fields](https://github.com/haskell/cabal/pull/11690) + leana8959/cabal/typed-fields](https://github.com/leana8959/cabal/tree/typed-fields) - [cabal-add](https://github.com/Bodigrim/cabal-add) @@ -584,6 +585,9 @@ describe the modification API in terms of lens. - [autopack](https://github.com/kowainik/autopack) +- [Transform-fields + leana8959/cabal/transform-fields](https://github.com/leana8959/cabal/tree/transform-fields) + - [Jappie's original Haskell Foundation Tech Proposal](https://github.com/haskellfoundation/tech-proposals/pull/65) @@ -607,7 +611,14 @@ describe the modification API in terms of lens. - [Trees that Grow](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) -[^1]: In cabal, sections can have arguments. If-else conditions are +[^1]: To run this implementation, run the following in your terminal + + ``` bash + Cabal-tests:hackage-tests --test-option="field-roundtrip" + + ``` + +[^2]: In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, and `library` is a section that can take a library name as a section argument. diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 3c25928..b9675ca 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -7,7 +7,7 @@ ) #let references = ( - // prior arts + // implementations comment-parser-pr: mk-smartlink( "https://github.com/haskell/cabal/pull/11252", )[Retain comments in field parser \#11252], @@ -21,7 +21,7 @@ )[Barbie/Trees-that-grow \#11690], typed-fields: mk-smartlink( - "https://github.com/haskell/cabal/pull/11690", + "https://github.com/leana8959/cabal/tree/typed-fields", )[Typed-fields leana8959/cabal/typed-fields], cabal-add-project: mk-smartlink("https://github.com/Bodigrim/cabal-add")[cabal-add], @@ -29,6 +29,10 @@ hpack-project: mk-smartlink("https://github.com/sol/hpack")[hpack], autopack-project: mk-smartlink("https://github.com/kowainik/autopack")[autopack], + transform-fields: mk-smartlink( + "https://github.com/leana8959/cabal/tree/transform-fields", + )[Transform-fields leana8959/cabal/transform-fields], + // proposals jappie-original-twg-proposal: mk-smartlink( "https://github.com/haskellfoundation/tech-proposals/pull/65", @@ -146,10 +150,14 @@ flexibility. As long as we respect its invariants during modification, unchanged parts in the output should stay the same, and changed parts should translate to local transformation in the output string. In our current prototype, we are already able to roundtrip 119662 out of 194557 -package descriptions of hackage (\~60%) with an implementation that is concise -and simple. To increase the percentage of successful roundtrip, we need -to detect CRLF/LF and exactprint accordingly; furthermore, we can't -figure out whether a whitespace was a tab or a space yet. These will require +package descriptions of hackage (\~60%) with #(references.transform-fields.override-name)[an implementation that is concise and simple] #footnote[ + To run this implementation, run the following in your terminal + ```bash + Cabal-tests:hackage-tests --test-option="field-roundtrip" + ``` +]. +To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; +furthermore, we can't figure out whether a whitespace was a tab or a space yet. These will require changes to the lexer which we have previously done in #references.comment-parser-pr.get-link. Secondly, we implement a modification/addition/removal framework to From 234ca8dcd52bd82659e77d649349362abfb67130 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 24 Jul 2026 10:18:47 +0200 Subject: [PATCH 28/43] minor corrections --- proposals/cabal-exactprint.md | 19 ++++++++++--------- proposals/cabal-exactprint.typ | 12 ++++++------ 2 files changed, 16 insertions(+), 15 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index dd438f4..04253f4 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -135,15 +135,15 @@ modification, we proceed with the following steps: - - Should the field be multiple (e.g. `build-depends` or `license-files`), For each item `it`, we swap out the old textual - represent with the new one, using the location of `it` provided by - the parser. This solves the problem of in-field trivia by only + representation with the new one, using the location of `it` provided + by the parser. This solves the problem of in-field trivia by only modifying the original field lines within a specific range that has changed. - Otherwise, we replace the entire string. -- Traverse all fields that has been modified to correct lines that have - been moved. +- Traverse all fields that has been modified to correct line numbers + that have been moved. - If a field `f` is moved down due to addition before `f`, we increment the line numbers of `f` and its following siblings @@ -170,7 +170,7 @@ We want to let user describe a single modification that we call `Edit` by specifying a focus and a transformation. Here we add a new dependency `myNewDep` as an example. This modification can be expressed in plain English as "within the section library with no arguments [^2], within -the field `build-depends`, add (append) a `myNewDep`." In pseudo Haskell +the field `build-depends`, add (append) `myNewDep`." In pseudo Haskell of the API we intend to build the aforementioned example modification can be described as: @@ -200,8 +200,9 @@ paths down the tree of fields. At the leaf (in the above example, `[FieldLine Position]` by providing `addFieldLinesListLike`. We strive to make the API flexible and will expose ways to modify -`[Field Position]` directly. However we don't try to guarantee that this -will always be correct. +`[Field Position]` directly. We don't try to guarantee the correctness +of this escape hatch, however we provide validation functions to catch +problems. ## Alternatives Considered @@ -436,8 +437,8 @@ To demonstrate the added complexity of "losing the shape of `[Field ann]`" casued by using `GenericPackageDescription`, we use the previous definition of `MonoidalFieldAla` as example. It is the same as the two following definition albeit generic in `ParsingPhase`. You can -see that in the `Abst`ract case, we maintain backwards-compatibility of -the type. +see that in the `Abst`ract case, we maintain backwards-compatibility at +type level. ``` haskell type MonoidalFieldAlaConc a = [ ([Comment Position], BS.ByteString, (Positions, a)) ] diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index b9675ca..90a7ff3 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -184,11 +184,11 @@ proceed with the following steps: - - Should the field be multiple (e.g. `build-depends` or `license-files`), For each item `it`, we swap out the old textual - represent with the new one, using the location of `it` provided by + representation with the new one, using the location of `it` provided by the parser. This solves the problem of in-field trivia by only modifying the original field lines within a specific range that has changed. - Otherwise, we replace the entire string. -- Traverse all fields that has been modified to correct lines that have +- Traverse all fields that has been modified to correct line numbers that have been moved. - If a field `f` is moved down due to addition before `f`, we increment the line numbers of `f` and its following siblings @@ -215,7 +215,7 @@ English as "within the section library with no arguments #footnote[In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, and `library` is a section that can take a library name as a section argument.], within -the field `build-depends`, add (append) a `myNewDep`." In pseudo Haskell +the field `build-depends`, add (append) `myNewDep`." In pseudo Haskell of the API we intend to build the aforementioned example modification can be described as: @@ -245,8 +245,8 @@ paths down the tree of fields. At the leaf (in the above example, `[FieldLine Position]` by providing `addFieldLinesListLike`. We strive to make the API flexible and will expose ways to modify -`[Field Position]` directly. However we don't try to guarantee that this -will always be correct. +`[Field Position]` directly. We don't try to guarantee the correctness of this +escape hatch, however we provide validation functions to catch problems. == Alternatives Considered @@ -478,7 +478,7 @@ To demonstrate the added complexity of "losing the shape of `[Field ann]`" casued by using `GenericPackageDescription`, we use the previous definition of `MonoidalFieldAla` as example. It is the same as the two following definition albeit generic in `ParsingPhase`. -You can see that in the `Abst`ract case, we maintain backwards-compatibility of the type. +You can see that in the `Abst`ract case, we maintain backwards-compatibility at type level. ```haskell type MonoidalFieldAlaConc a = [ ([Comment Position], BS.ByteString, (Positions, a)) ] type MonoidalFieldAlaAbst a = a From db2bc4089e2baaef9b7fafae2a0c0329449ae1b2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 24 Jul 2026 10:30:40 +0200 Subject: [PATCH 29/43] fix typos --- proposals/cabal-exactprint.md | 4 ++-- proposals/cabal-exactprint.typ | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 04253f4..49a73df 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -76,7 +76,7 @@ created to do what cabal can't: description. - [autopack](https://github.com/kowainik/autopack), automatic module - descovery. + discovery. ## Proposed Change @@ -434,7 +434,7 @@ this far. written. To demonstrate the added complexity of "losing the shape of -`[Field ann]`" casued by using `GenericPackageDescription`, we use the +`[Field ann]`" caused by using `GenericPackageDescription`, we use the previous definition of `MonoidalFieldAla` as example. It is the same as the two following definition albeit generic in `ParsingPhase`. You can see that in the `Abst`ract case, we maintain backwards-compatibility at diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 90a7ff3..767b958 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -132,7 +132,7 @@ can't: - #references.cabal-fmt-project.get-link, a formatter. - #references.cabal-add-project.get-link, an non-official implementation of `cabal add` command. - #references.hpack-project.get-link, an alternative to package description. -- #references.autopack-project.get-link, automatic module descovery. +- #references.autopack-project.get-link, automatic module discovery. == Proposed Change @@ -475,7 +475,7 @@ this far. written. To demonstrate the added complexity of "losing the shape of `[Field ann]`" -casued by using `GenericPackageDescription`, we use +caused by using `GenericPackageDescription`, we use the previous definition of `MonoidalFieldAla` as example. It is the same as the two following definition albeit generic in `ParsingPhase`. You can see that in the `Abst`ract case, we maintain backwards-compatibility at type level. From dfb96dc42683e8a8d7f82b4887be0560c87323c2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 24 Jul 2026 11:24:22 +0200 Subject: [PATCH 30/43] pin proof of concept at a rev --- proposals/cabal-exactprint.md | 15 ++++++++++----- proposals/cabal-exactprint.typ | 11 +++++++---- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 49a73df..99349e2 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -612,12 +612,17 @@ describe the modification API in terms of lens. - [Trees that Grow](https://www.cs.tufts.edu/comp/150FP/archive/simon-peyton-jones/trees-that-grow.pdf) -[^1]: To run this implementation, run the following in your terminal +[^1]: To run this implementation to see for yourself: - ``` bash - Cabal-tests:hackage-tests --test-option="field-roundtrip" - - ``` + - Clone the branch at this commit + + + - Run the following in your terminal + + ``` bash + Cabal-tests:hackage-tests --test-option="field-roundtrip" + + ``` [^2]: In cabal, sections can have arguments. If-else conditions are actually sections where the condition is the single argument, and diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 767b958..a0a2fbd 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -151,10 +151,13 @@ unchanged parts in the output should stay the same, and changed parts should translate to local transformation in the output string. In our current prototype, we are already able to roundtrip 119662 out of 194557 package descriptions of hackage (\~60%) with #(references.transform-fields.override-name)[an implementation that is concise and simple] #footnote[ - To run this implementation, run the following in your terminal - ```bash - Cabal-tests:hackage-tests --test-option="field-roundtrip" - ``` + To run this implementation to see for yourself: + + - Clone the branch at this commit https://github.com/haskell/cabal/commit/00a79443390c3a28c911c91b2d8c2f432e09c05c + - Run the following in your terminal + ```bash + Cabal-tests:hackage-tests --test-option="field-roundtrip" + ``` ]. To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't figure out whether a whitespace was a tab or a space yet. These will require From f9bbcc7cdfd942fff1a8f0887755c74deaeceee4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Mon, 27 Jul 2026 09:28:42 +0200 Subject: [PATCH 31/43] minor fixes --- proposals/cabal-exactprint.md | 29 ++++++++++++++++++----------- proposals/cabal-exactprint.typ | 23 ++++++++++++----------- 2 files changed, 30 insertions(+), 22 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 99349e2..a7b66b3 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -100,11 +100,12 @@ current prototype, we are already able to roundtrip 119662 out of 194557 package descriptions of hackage (\~60%) with [an implementation that is concise and simple](https://github.com/leana8959/cabal/tree/transform-fields) [^1]. -To increase the percentage of successful roundtrip, we need to detect -CRLF/LF and exactprint accordingly; furthermore, we can't figure out -whether a whitespace was a tab or a space yet. These will require -changes to the lexer which we have previously done in [Retain comments -in field parser #11252](https://github.com/haskell/cabal/pull/11252). +The review will also be easy thanks to the small diff. To increase the +percentage of successful roundtrip, we need to detect CRLF/LF and +exactprint accordingly; furthermore, we can't figure out whether a +whitespace was a tab or a space yet. These will require changes to the +lexer which we have previously done in [Retain comments in field parser +#11252](https://github.com/haskell/cabal/pull/11252). Secondly, we implement a modification/addition/removal framework to facilitate building modification functions. A notable feature request in @@ -163,8 +164,8 @@ independently. To validate an exactprint implementation, we test the property `exactRenderFields . readFields = id` against Hackage; to validate a modification framework implementation, we add golden tests for different -cases to ensure that important invariants are preserved, namely that -`Position` of fields are not overlapping. +cases to ensure that important invariants of `[Field ann]` are +preserved, namely that `Position` of fields are not overlapping. We want to let user describe a single modification that we call `Edit` by specifying a focus and a transformation. Here we add a new dependency @@ -452,8 +453,8 @@ bookkeeping of the following structure: `build-depends` belongs to. This is because `build-depends` can be merged. Each item in this list will be referred to as a *group*. -- Each group has its associated comments because each group was - originally a list of field lines. +- Each group (which was a field) has its associated comments from the + original fieldlines. - The ByteString here represents the original cased name of Fields. The user could've written `BuIlD-DePenDs` and we would need to restore it @@ -543,6 +544,10 @@ It would also benefit existing programs that depend on Cabal: can be simplified the new `readFieldsWithComments` in [Retain comments in field parser #11252](https://github.com/haskell/cabal/pull/11252). +- Eventually + [haskell-language-server](https://github.com/haskell/haskell-language-server) + will be able to generate code actions to modify cabal files. + This work would also simplify implementation of formatters or modification tools operating on other formats using the same envelope format, namely ["project @@ -553,9 +558,9 @@ descriptions"](https://cabal.readthedocs.io/en/stable/cabal-project-description- references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to implement and -iterate the previous proposal. Due to the design evoving drastically +iterate on the previous proposal. Due to the design evoving drastically over time, this is the most up-to-date proposal describing our ideas -after refinding them after a year. +after refining them after a year. I will continue to work on this myself under the funding of Jappie and Haskell Foundation. @@ -586,6 +591,8 @@ describe the modification API in terms of lens. - [autopack](https://github.com/kowainik/autopack) +- [haskell-language-server](https://github.com/haskell/haskell-language-server) + - [Transform-fields leana8959/cabal/transform-fields](https://github.com/leana8959/cabal/tree/transform-fields) diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index a0a2fbd..3218e4b 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -1,11 +1,11 @@ -// Pandoc doesn't seem to support bibliography files - +// This source file is used to generate ./cabal-exactprint.md using pandoc, +// so the references are kept in sync in a coherent format. +// Pandoc doesn't seem to support bibliography files #let mk-smartlink(url, name) = ( get-link: link(url)[#name], override-name: new-name => link(url)[#new-name], ) - #let references = ( // implementations comment-parser-pr: mk-smartlink( @@ -28,6 +28,7 @@ cabal-fmt-project: mk-smartlink("https://github.com/phadej/cabal-fmt")[cabal-fmt], hpack-project: mk-smartlink("https://github.com/sol/hpack")[hpack], autopack-project: mk-smartlink("https://github.com/kowainik/autopack")[autopack], + hls-project: mk-smartlink("https://github.com/haskell/haskell-language-server")[haskell-language-server], transform-fields: mk-smartlink( "https://github.com/leana8959/cabal/tree/transform-fields", @@ -92,8 +93,6 @@ We define the parse-print idempotency to be `print . parse == id`, which reads " printing is as if we've done nothing". We only focus on ensuring this property to hold for valid package descriptions, and we don't consider the braces syntax in this work. -// TODO: cite braces syntax - == Motivation Cabal builds packages by following stanzas written in #(references.package-description-documentation.override-name)[package descriptions]. @@ -158,7 +157,7 @@ package descriptions of hackage (\~60%) with #(references.transform-fields.overr ```bash Cabal-tests:hackage-tests --test-option="field-roundtrip" ``` -]. +]. The review will also be easy thanks to the small diff. To increase the percentage of successful roundtrip, we need to detect CRLF/LF and exactprint accordingly; furthermore, we can't figure out whether a whitespace was a tab or a space yet. These will require changes to the lexer which we have previously done in #references.comment-parser-pr.get-link. @@ -208,8 +207,8 @@ independently. To validate an exactprint implementation, we test the property `exactRenderFields . readFields = id` against Hackage; to validate a modification framework implementation, we add golden tests for different -cases to ensure that important invariants are preserved, namely that -`Position` of fields are not overlapping. +cases to ensure that important invariants of `[Field ann]` are preserved, +namely that `Position` of fields are not overlapping. We want to let user describe a single modification that we call `Edit` by specifying a focus and a transformation. Here we add a new dependency @@ -492,7 +491,7 @@ From outside to inside, `MonoidalFieldAlaConc` represents the bookkeeping of the - We maintain everything in a list to remember which field a `build-depends` belongs to. This is because `build-depends` can be merged. Each item in this list will be referred to as a _group_. -- Each group has its associated comments because each group was originally a list of field lines. +- Each group (which was a field) has its associated comments from the original fieldlines. - The ByteString here represents the original cased name of Fields. The user could've written `BuIlD-DePenDs` and we would need to restore it despite this string looks very funny. @@ -575,6 +574,8 @@ It would also benefit existing programs that depend on Cabal: again with its own parser to find all the comments. This can be simplified the new `readFieldsWithComments` in #references.comment-parser-pr.get-link. +- Eventually #references.hls-project.get-link will be able to generate + code actions to modify cabal files. This work would also simplify implementation of formatters or modification tools operating on other formats using the same envelope format, namely #(references.project-description-documentation.override-name)["project descriptions"]. @@ -583,9 +584,9 @@ other formats using the same envelope format, namely #(references.project-descri references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to -implement and iterate the previous proposal. Due to the design evoving +implement and iterate on the previous proposal. Due to the design evoving drastically over time, this is the most up-to-date proposal describing -our ideas after refinding them after a year. +our ideas after refining them after a year. I will continue to work on this myself under the funding of Jappie and Haskell Foundation. From 7b7316bea37ad4f1cd21595abfbdb25e603014ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Mon, 27 Jul 2026 11:24:38 +0200 Subject: [PATCH 32/43] fix link --- proposals/cabal-exactprint.md | 13 +++++++------ proposals/cabal-exactprint.typ | 2 +- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index a7b66b3..1d713bb 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -555,12 +555,13 @@ descriptions"](https://cabal.readthedocs.io/en/stable/cabal-project-description- ## Implementation Notes -references.jappie-original-twg-proposal.get-link has been accepted and -funded by the Haskell Foundation. Under Jappie and the Haskell -Foundation's funding since september 2025, I have tried to implement and -iterate on the previous proposal. Due to the design evoving drastically -over time, this is the most up-to-date proposal describing our ideas -after refining them after a year. +[Jappie's original Haskell Foundation Tech +Proposal](https://github.com/haskellfoundation/tech-proposals/pull/65) +has been accepted and funded by the Haskell Foundation. Under Jappie and +the Haskell Foundation's funding since september 2025, I have tried to +implement and iterate on the previous proposal. Due to the design +evoving drastically over time, this is the most up-to-date proposal +describing our ideas after refining them after a year. I will continue to work on this myself under the funding of Jappie and Haskell Foundation. diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 3218e4b..396dc21 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -582,7 +582,7 @@ other formats using the same envelope format, namely #(references.project-descri == Implementation Notes -references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and +#references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and the Haskell Foundation's funding since september 2025, I have tried to implement and iterate on the previous proposal. Due to the design evoving drastically over time, this is the most up-to-date proposal describing From 674fd99042f9fd26569af2fae03de92f040c181f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Mon, 27 Jul 2026 11:26:00 +0200 Subject: [PATCH 33/43] update funding info --- proposals/cabal-exactprint.typ | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 396dc21..f9d14a9 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -582,14 +582,13 @@ other formats using the same envelope format, namely #(references.project-descri == Implementation Notes -#references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. Under Jappie and -the Haskell Foundation's funding since september 2025, I have tried to +#references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. +Under the Haskell Foundation's funding since september 2025, I have tried to implement and iterate on the previous proposal. Due to the design evoving drastically over time, this is the most up-to-date proposal describing our ideas after refining them after a year. -I will continue to work on this myself under the funding of Jappie and -Haskell Foundation. +I will continue to work on this myself under the funding of the Haskell Foundation. == Open Questions We are still investigating if describing it is possible or beneficial to From 73fe65b7daffe54fe928d2dbb3724d3a36d1f8ba Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Mon, 27 Jul 2026 11:27:40 +0200 Subject: [PATCH 34/43] update compatibility description --- proposals/cabal-exactprint.md | 13 +++++++------ proposals/cabal-exactprint.typ | 5 +++-- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 1d713bb..3d96b8c 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -506,8 +506,9 @@ fields we support. ## Backwards Compatibility / Migration -Because we don't touch the field grammar infrastructure at all, we don't -foresee any backwards-compatibility issues. +We are extending the parser and implementing a modification framework. +The changes are local to the parser, we don't foresee any +backwards-compatibility issues. ## Interested parties @@ -557,14 +558,14 @@ descriptions"](https://cabal.readthedocs.io/en/stable/cabal-project-description- [Jappie's original Haskell Foundation Tech Proposal](https://github.com/haskellfoundation/tech-proposals/pull/65) -has been accepted and funded by the Haskell Foundation. Under Jappie and -the Haskell Foundation's funding since september 2025, I have tried to +has been accepted and funded by the Haskell Foundation. Under the +Haskell Foundation's funding since september 2025, I have tried to implement and iterate on the previous proposal. Due to the design evoving drastically over time, this is the most up-to-date proposal describing our ideas after refining them after a year. -I will continue to work on this myself under the funding of Jappie and -Haskell Foundation. +I will continue to work on this myself under the funding of the Haskell +Foundation. ## Open Questions diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index f9d14a9..dbce1f5 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -540,8 +540,9 @@ fields we support. == Backwards Compatibility / Migration -Because we don't touch the field grammar infrastructure at all, we don't -foresee any backwards-compatibility issues. +We are extending the parser and implementing a modification framework. +The changes are local to the parser, we don't foresee any backwards-compatibility issues. + == Interested parties From 72e4ddd560486c55d4bbc83b850435d8402d557a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Tue, 28 Jul 2026 09:54:19 +0200 Subject: [PATCH 35/43] clearify and add todo --- proposals/cabal-exactprint.md | 68 +++++++++++++++++---------- proposals/cabal-exactprint.typ | 84 +++++++++++++++++++++++++--------- 2 files changed, 107 insertions(+), 45 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 3d96b8c..6c06f2e 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -123,40 +123,60 @@ modification framework. Each field in a package description is represented by a field name in association with some field lines. Upon modification, we proceed with the following steps: -- Should the field lines be non empty, join them into a single field - line `fl` with indentation and newlines. +1. Should the field lines be non empty, join them into a single field + line `fl` with indentation and newlines. -- Run the `Parsec` instance of a desired type `τ` on the joined field - lines `fl`, obtain data `p` which these field lines represent. +2. Run the `Parsec` instance of a desired type `τ` on the joined field + lines `fl`, obtain data `p` which these field lines represent. -- Apply user's transformation function `t` on `p`, obtaining `p'`. +3. Apply user's transformation function `t` on `p`, obtaining `p'`. -- Run the `Pretty` instance of `τ` on `p'` to obtain a new textual - representation `fl'`. +4. Run the `Pretty` instance of `τ` on `p'` to obtain a new textual + representation `fl'`. -- - Should the field be multiple (e.g. `build-depends` or - `license-files`), For each item `it`, we swap out the old textual - representation with the new one, using the location of `it` provided - by the parser. This solves the problem of in-field trivia by only - modifying the original field lines within a specific range that has - changed. +5. We replace the entire `fl` with `fl'`. - - Otherwise, we replace the entire string. +6. Traverse all fields that has been modified to correct line numbers + that have been moved. -- Traverse all fields that has been modified to correct line numbers - that have been moved. + 1. If a field `f` is moved down due to addition before `f`, we + increment the line numbers of `f` and its following siblings + accordingly. - - If a field `f` is moved down due to addition before `f`, we - increment the line numbers of `f` and its following siblings - accordingly. + 2. If a field `f` is moved up due to removal before `f`, we can + either do nothing (leaving empty lines before `f`) or decrement + the line numbers of `f` and its following siblings accordingly. - - If a field `f` is moved up due to removal before `f`, we can either - do nothing (leaving empty lines before `f`) or decrement the line - numbers of `f` and its following siblings accordingly. + 3. Modification is be a hybrid of addition and removal. - - Modification is be a hybrid of addition and removal. +7. Run modifications similar to this until no more is demanded. -- Run modifications similar to this until no more is demanded. +The above steps would not allow modifying list like values such as +`[Dependency]`, especially that cabal allows having more than one comma +separated item in a list to be on the same line, there's no bijection +between fieldline and the individual item parsed. + +To achieve precise and local change to a single item in a list, we +extend the algorithm in the following ways, using `Dependency` an +example: + +1. Instead of parsing `Located [Dependency]` where we obtain a single + location the start of the entire list, we parse + `[Located Dependency]` to obtain a location on each item. + +2. Considering the user might only want to modify an item, the user's + transformation function can be typed as + `Dependency -> Maybe Dependency` where `Maybe` indicates that the + dependency should be rerendered. + +3. We only operate on changed `Dependency`. Given the location of a + `Dependency` and the its transformed counterpart, refer to its + source position and swap out the old representation with the new + representation. + +The extended algorithm doesn't cover the use case of adding new +dependencies to the front or the end of the dependency list, or sorting. +To do so, one can use the original algorithm for single values. Exactprint and the modification framework can be implemented and tested independently. diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index dbce1f5..2657da3 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -24,11 +24,19 @@ "https://github.com/leana8959/cabal/tree/typed-fields", )[Typed-fields leana8959/cabal/typed-fields], - cabal-add-project: mk-smartlink("https://github.com/Bodigrim/cabal-add")[cabal-add], - cabal-fmt-project: mk-smartlink("https://github.com/phadej/cabal-fmt")[cabal-fmt], + cabal-add-project: mk-smartlink( + "https://github.com/Bodigrim/cabal-add", + )[cabal-add], + cabal-fmt-project: mk-smartlink( + "https://github.com/phadej/cabal-fmt", + )[cabal-fmt], hpack-project: mk-smartlink("https://github.com/sol/hpack")[hpack], - autopack-project: mk-smartlink("https://github.com/kowainik/autopack")[autopack], - hls-project: mk-smartlink("https://github.com/haskell/haskell-language-server")[haskell-language-server], + autopack-project: mk-smartlink( + "https://github.com/kowainik/autopack", + )[autopack], + hls-project: mk-smartlink( + "https://github.com/haskell/haskell-language-server", + )[haskell-language-server], transform-fields: mk-smartlink( "https://github.com/leana8959/cabal/tree/transform-fields", @@ -176,30 +184,62 @@ modification framework. Each field in a package description is represented by a field name in association with some field lines. Upon modification, we proceed with the following steps: -- Should the field lines be non empty, join them into a single field ++ Should the field lines be non empty, join them into a single field line `fl` with indentation and newlines. -- Run the `Parsec` instance of a desired type `τ` on the joined field + ++ Run the `Parsec` instance of a desired type `τ` on the joined field lines `fl`, obtain data `p` which these field lines represent. -- Apply user's transformation function `t` on `p`, obtaining `p'`. -- Run the `Pretty` instance of `τ` on `p'` to obtain a new textual + ++ Apply user's transformation function `t` on `p`, obtaining `p'`. + ++ Run the `Pretty` instance of `τ` on `p'` to obtain a new textual representation `fl'`. -- - - Should the field be multiple (e.g. `build-depends` or - `license-files`), For each item `it`, we swap out the old textual - representation with the new one, using the location of `it` provided by - the parser. This solves the problem of in-field trivia by only modifying - the original field lines within a specific range that has changed. - - Otherwise, we replace the entire string. -- Traverse all fields that has been modified to correct line numbers that have + ++ We replace the entire `fl` with `fl'`. + ++ Traverse all fields that has been modified to correct line numbers that have been moved. - - If a field `f` is moved down due to addition before `f`, we + + If a field `f` is moved down due to addition before `f`, we increment the line numbers of `f` and its following siblings accordingly. - - If a field `f` is moved up due to removal before `f`, we can either + + If a field `f` is moved up due to removal before `f`, we can either do nothing (leaving empty lines before `f`) or decrement the line numbers of `f` and its following siblings accordingly. - - Modification is be a hybrid of addition and removal. -- Run modifications similar to this until no more is demanded. + + Modification is be a hybrid of addition and removal. + ++ Run modifications similar to this until no more is demanded. + ++ Validate that the new data is parsable and parses to the transformed value. + +The above steps would not allow modifying list like values such as `[Dependency]`, especially +that cabal allows having more than one comma separated item in a list to be on the same line, +there's no bijection between fieldline and the individual item parsed. + +To achieve precise and local change to a single item in a list, +we extend the algorithm in the following ways, using `Dependency` an example: + ++ Instead of parsing `Located [Dependency]` where we obtain a single location the start of the + entire list, we parse `[Located Dependency]` to obtain a location on each item. + ++ Considering the user might only want to modify an item, the user's transformation function + can be typed as `Dependency -> Maybe Dependency` where `Maybe` indicates that the dependency + should be rerendered. + ++ We only operate on changed `Dependency`. + Given the location of a `Dependency` and the its transformed counterpart, refer to its source position + and swap out the old representation with the new representation. + Each change of an item is hence local and composable. + +The extended algorithm doesn't cover the use case of adding new dependencies to the front +or the end of the dependency list, or sorting the list. +To do so, one can use the original algorithm for single values. +// TODO: this would format the entire list, but at the same time if you were to sort +// where sometimes there are more than one on a singleline, you might as well just mangle the entire +// field. +// Note that the rest of the file will still be untouched. The change will be scoped to a Field +// instead of a fieldline item. +// `foo, bar +// baz` Exactprint and the modification framework can be implemented and tested independently. @@ -255,7 +295,9 @@ escape hatch, however we provide validation functions to catch problems. Below is an exhaustive list of the changes we tried in chronological order since september 2025 and what I learned from these attempts. -- #(references.trivia-tree.override-name)[Trivia Tree \#11425 (proof of concept)] +- #( + references.trivia-tree.override-name + )[Trivia Tree \#11425 (proof of concept)] implements a untyped tree `TriviaTree` using existential type. With it, we can imtate the shape of a recursive type `τ` freely and From d1e1ee57d3521d763d82e8c1e50af2a8425daf90 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 29 Jul 2026 11:42:53 +0200 Subject: [PATCH 36/43] give concrete code examples --- proposals/cabal-exactprint.typ | 137 ++++++++++++++++++++++++++++++--- 1 file changed, 128 insertions(+), 9 deletions(-) diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 2657da3..ad1d2ea 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -179,8 +179,8 @@ which allows the user to modify the `cabal-version` field while having all the position validation already dealt with behind the scenes. -We will use the `Parsec` and `Pretty` classes to implement the typed -modification framework. Each field in a package description is represented by a +To implement this we use existing building blocks. `Pretty` and `Parsec` instance already exist. +Each field in a package description is represented by a field name in association with some field lines. Upon modification, we proceed with the following steps: @@ -233,13 +233,132 @@ we extend the algorithm in the following ways, using `Dependency` an example: The extended algorithm doesn't cover the use case of adding new dependencies to the front or the end of the dependency list, or sorting the list. To do so, one can use the original algorithm for single values. -// TODO: this would format the entire list, but at the same time if you were to sort -// where sometimes there are more than one on a singleline, you might as well just mangle the entire -// field. -// Note that the rest of the file will still be untouched. The change will be scoped to a Field -// instead of a fieldline item. -// `foo, bar -// baz` + +=== Proposed API + +Below are parts of the proposed API, and some example usages of it. + +```haskell +-- | Build a @[FieldLine Position]@ modification function given a function @a -> a@, parsed as @b@. +modifyValueAtomAla + :: forall (b :: Type) (a :: Type) + . ( Newtype b a + , Parsec b + , Pretty b + ) + => (a -> Maybe a) -- ^ Nothing prevents a new render. + -> ([FieldLine Position] -> FieldLine Position)) +modifyValueAtomAla = {- Implementation of the algorithm for single value. -} + +-- | Build a @[FieldLine Position]@ modification function given a function @a -> Maybe a@, parsed as @List sep b a@. +modifyValueList + :: forall (sep :: Type) (b :: Type) (a :: Type) + . ( Newtype (List sep b (Located a)) (Located a) + , Parsec (List sep b (Located a)) + , Pretty (List sep b (Located a)) + ) + => (a -> Maybe a) -- ^ Nothing prevents a new render. + -> ([FieldLine Position] -> [FieldLine Position]) +modifyValueList = {- Implementation of the extended algorithm for multiple values. -} + +addValueList + :: forall (sep :: Type) (b :: Type) (a :: Type) + . ( Newtype (List sep b (Located a)) (Located a) + , Parsec (List sep b (Located a)) + , Pretty (List sep b (Located a)) + ) + => InsertPosition -- ^ prepend or append + -> a + -> ([FieldLine Position] -> [FieldLine Position]) +addValueList = {- Parse and use the source location to insert a value at desired location. -} + +removeValueList + :: forall (sep :: Type) (b :: Type) (a :: Type) + . ( Newtype (List sep b (Located a)) (Located a) + , Parsec (List sep b (Located a)) + , Pretty (List sep b (Located a)) + ) + -> (a -> Bool) + -> ([FieldLine Position] -> [FieldLine Position]) +removeValueList = {- Parse, if the predicate is met, remove the value from the list. -} +``` + +=== Example usages + +The following examples operate on this cabal build-depends field. + +```cabal +build-depends: + base > 4 && < 5, text > 2.0.4 + -- interleaved comments + , containers > 0.8 +``` + +Example: modify the bound a dependency within some field lines, can be generalized to cabal gen-bounds. + +```haskell +setBaseVersionTo :: Version -> ([FieldLine Position] -> [FieldLine Position]) +setBaseVersionTo targetVersion = modifyValueList @CommaVSep @Identity @Dependency $ \case + (Depedency pname _ libs) | pname == mkPackageName "base" -> Just (Depedency pname targetVersion libs) + _ -> Nothing +``` + +```cabal +build-depends: + base > 4.8, text > 2.0.4 + -- interleaved comments + , containers > 0.8 +``` + +Example: append a new dependency, can be generalized to cabal add. +```haskell +addNewDependency :: Dependency -> ([FieldLine Position] -> [FieldLine Position]) +addNewDependency = addValueList @CommaVSep @Identity @Dependency Prepend +``` + +```cabal +build-depends: + foo, + base > 4 && < 5, text > 2.0.4 + -- interleaved comments + , containers > 0.8 +``` + +Example: remove a dependency +```haskell +removeDependency + :: (Dependency -> Bool) + -> ([FieldLine Position] -> [FieldLine Position]) +removeDependency = removeValueList @CommaVSep @Identity @Depedency +``` + +```cabal +-- Remove `base` +build-depends: + text > 2.0.4 + -- interleaved comments + , containers > 0.8 +``` + +Example: sort the dependencies. +We treat the entire dependency list as an atom, and all in-field-lines trivia are lost. +The comments are not moved to the closest item. See open question on comment handling. +```haskell +sortDependency + :: (Dependency -> Dependency -> Ord) + -> ([FieldLine Position] -> [FieldLine Position]) +sortDependency cmp = modifyValueAtomAla @(List CommaVSep @Identity) @Dependency $ \deps -> + Just (sortBy cmp deps) +``` + +```cabal +-- Sort by ascending package name. +build-depends: + base > 4 && < 5, + containers > 0.8, + -- interleaved comments + text > 2.0.4, +``` Exactprint and the modification framework can be implemented and tested independently. From 0964b86864036dbfd76ddca1bb8707ab6c51b964 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 29 Jul 2026 11:54:20 +0200 Subject: [PATCH 37/43] document open questions --- proposals/cabal-exactprint.typ | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index ad1d2ea..ec8d2f3 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -342,6 +342,7 @@ build-depends: Example: sort the dependencies. We treat the entire dependency list as an atom, and all in-field-lines trivia are lost. +In-field-lines trivia are also lost because the entire list is rerendered. The comments are not moved to the closest item. See open question on comment handling. ```haskell sortDependency @@ -354,7 +355,7 @@ sortDependency cmp = modifyValueAtomAla @(List CommaVSep @Identity) @Dependency ```cabal -- Sort by ascending package name. build-depends: - base > 4 && < 5, + base > 4 && < 5, containers > 0.8, -- interleaved comments text > 2.0.4, @@ -756,5 +757,33 @@ I will continue to work on this myself under the funding of the Haskell Foundati We are still investigating if describing it is possible or beneficial to describe the modification API in terms of lens. +- Whitespaces + + Cabal allow leading spaces to be ` ` (plain whitespace) or `\t` (tab). + We would like to know if it is possible to enforce the usage of plain whitespace across all cabal files. + There is an existing todo comment to enforce the use of plain whitespace in field indentation in field lexer. + + Trailing whitespaces and lines with only whitespaces are also lost in the current exactprint implementation. + To restore them, they need to be saved. This would entail more modification to the lexer and field parser. + We want to know if it's feasible to drop them. + On a related note, git can be configured to detect trailing whitespaces and warn the user, or + automatically remove them. + +- Line endings + + On a windows machine, lines are ended with CRLF instead of LF. It shouldn't be hard to detect if a + cabal file uses one or the other. + However, we want to discuss on what to do regarding mixed line endings. + +- Sorting + + It is possible to sort a cabal field using the proposed API. + The loss of trivia is local to the field, but it has some problems: + - Comments will stay where they were originally. + - In-field-lines trivia will be lost. + + Sorting is more of a formatter feature, which exactprint doesn't try to perfect. We want to know + if the current implementation is satisfactory. + == References #list(..references.values().map(x => x.get-link)) From eb11c41ba512ed712f1a701c3261e91d59fa6bfb Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Wed, 29 Jul 2026 12:02:31 +0200 Subject: [PATCH 38/43] generate markdown --- proposals/cabal-exactprint.md | 181 +++++++++++++++++++++++++++++++++- 1 file changed, 176 insertions(+), 5 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 6c06f2e..1b4eeb0 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -118,8 +118,8 @@ translating `SpecVersion -> SpecVersion` to the `cabal-version` field while having all the position validation already dealt with behind the scenes. -We will use the `Parsec` and `Pretty` classes to implement the typed -modification framework. Each field in a package description is +To implement this we use existing building blocks. `Pretty` and `Parsec` +instance already exist. Each field in a package description is represented by a field name in association with some field lines. Upon modification, we proceed with the following steps: @@ -151,6 +151,9 @@ modification, we proceed with the following steps: 7. Run modifications similar to this until no more is demanded. +8. Validate that the new data is parsable and parses to the transformed + value. + The above steps would not allow modifying list like values such as `[Dependency]`, especially that cabal allows having more than one comma separated item in a list to be on the same line, there's no bijection @@ -172,11 +175,144 @@ example: 3. We only operate on changed `Dependency`. Given the location of a `Dependency` and the its transformed counterpart, refer to its source position and swap out the old representation with the new - representation. + representation. Each change of an item is hence local and + composable. The extended algorithm doesn't cover the use case of adding new -dependencies to the front or the end of the dependency list, or sorting. -To do so, one can use the original algorithm for single values. +dependencies to the front or the end of the dependency list, or sorting +the list. To do so, one can use the original algorithm for single +values. + +### Proposed API + +Below are parts of the proposed API, and some example usages of it. + +``` haskell +-- | Build a @[FieldLine Position]@ modification function given a function @a -> a@, parsed as @b@. +modifyValueAtomAla + :: forall (b :: Type) (a :: Type) + . ( Newtype b a + , Parsec b + , Pretty b + ) + => (a -> Maybe a) -- ^ Nothing prevents a new render. + -> ([FieldLine Position] -> FieldLine Position)) +modifyValueAtomAla = {- Implementation of the algorithm for single value. -} + +-- | Build a @[FieldLine Position]@ modification function given a function @a -> Maybe a@, parsed as @List sep b a@. +modifyValueList + :: forall (sep :: Type) (b :: Type) (a :: Type) + . ( Newtype (List sep b (Located a)) (Located a) + , Parsec (List sep b (Located a)) + , Pretty (List sep b (Located a)) + ) + => (a -> Maybe a) -- ^ Nothing prevents a new render. + -> ([FieldLine Position] -> [FieldLine Position]) +modifyValueList = {- Implementation of the extended algorithm for multiple values. -} + +addValueList + :: forall (sep :: Type) (b :: Type) (a :: Type) + . ( Newtype (List sep b (Located a)) (Located a) + , Parsec (List sep b (Located a)) + , Pretty (List sep b (Located a)) + ) + => InsertPosition -- ^ prepend or append + -> a + -> ([FieldLine Position] -> [FieldLine Position]) +addValueList = {- Parse and use the source location to insert a value at desired location. -} + +removeValueList + :: forall (sep :: Type) (b :: Type) (a :: Type) + . ( Newtype (List sep b (Located a)) (Located a) + , Parsec (List sep b (Located a)) + , Pretty (List sep b (Located a)) + ) + -> (a -> Bool) + -> ([FieldLine Position] -> [FieldLine Position]) +removeValueList = {- Parse, if the predicate is met, remove the value from the list. -} +``` + +### Example usages + +The following examples operate on this cabal build-depends field. + +``` cabal +build-depends: + base > 4 && < 5, text > 2.0.4 + -- interleaved comments + , containers > 0.8 +``` + +Example: modify the bound a dependency within some field lines, can be +generalized to cabal gen-bounds. + +``` haskell +setBaseVersionTo :: Version -> ([FieldLine Position] -> [FieldLine Position]) +setBaseVersionTo targetVersion = modifyValueList @CommaVSep @Identity @Dependency $ \case + (Depedency pname _ libs) | pname == mkPackageName "base" -> Just (Depedency pname targetVersion libs) + _ -> Nothing +``` + +``` cabal +build-depends: + base > 4.8, text > 2.0.4 + -- interleaved comments + , containers > 0.8 +``` + +Example: append a new dependency, can be generalized to cabal add. + +``` haskell +addNewDependency :: Dependency -> ([FieldLine Position] -> [FieldLine Position]) +addNewDependency = addValueList @CommaVSep @Identity @Dependency Prepend +``` + +``` cabal +build-depends: + foo, + base > 4 && < 5, text > 2.0.4 + -- interleaved comments + , containers > 0.8 +``` + +Example: remove a dependency + +``` haskell +removeDependency + :: (Dependency -> Bool) + -> ([FieldLine Position] -> [FieldLine Position]) +removeDependency = removeValueList @CommaVSep @Identity @Depedency +``` + +``` cabal +-- Remove `base` +build-depends: + text > 2.0.4 + -- interleaved comments + , containers > 0.8 +``` + +Example: sort the dependencies. We treat the entire dependency list as +an atom, and all in-field-lines trivia are lost. In-field-lines trivia +are also lost because the entire list is rerendered. The comments are +not moved to the closest item. See open question on comment handling. + +``` haskell +sortDependency + :: (Dependency -> Dependency -> Ord) + -> ([FieldLine Position] -> [FieldLine Position]) +sortDependency cmp = modifyValueAtomAla @(List CommaVSep @Identity) @Dependency $ \deps -> + Just (sortBy cmp deps) +``` + +``` cabal +-- Sort by ascending package name. +build-depends: + base > 4 && < 5, + containers > 0.8, + -- interleaved comments + text > 2.0.4, +``` Exactprint and the modification framework can be implemented and tested independently. @@ -592,6 +728,41 @@ Foundation. We are still investigating if describing it is possible or beneficial to describe the modification API in terms of lens. +- Whitespaces + + Cabal allow leading spaces to be ` ` (plain whitespace) or `\t` (tab). + We would like to know if it is possible to enforce the usage of plain + whitespace across all cabal files. There is an existing todo comment + to enforce the use of plain whitespace in field indentation in field + lexer. + + Trailing whitespaces and lines with only whitespaces are also lost in + the current exactprint implementation. To restore them, they need to + be saved. This would entail more modification to the lexer and field + parser. We want to know if it's feasible to drop them. On a related + note, git can be configured to detect trailing whitespaces and warn + the user, or automatically remove them. + +- Line endings + + On a windows machine, lines are ended with CRLF instead of LF. It + shouldn't be hard to detect if a cabal file uses one or the other. + However, we want to discuss on what to do regarding mixed line + endings. + +- Sorting + + It is possible to sort a cabal field using the proposed API. The loss + of trivia is local to the field, but it has some problems: + + - Comments will stay where they were originally. + + - In-field-lines trivia will be lost. + + Sorting is more of a formatter feature, which exactprint doesn't try + to perfect. We want to know if the current implementation is + satisfactory. + ## References - [Retain comments in field parser From e4b5ce744ae84972970eb9e99bd585efe83231dd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 30 Jul 2026 08:10:38 +0200 Subject: [PATCH 39/43] use coherent name across the examples --- proposals/cabal-exactprint.typ | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index ec8d2f3..627db76 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -395,16 +395,14 @@ appendDependency = -- Focus on a field, create it should it not exist. (hasFieldName "build-depends") -- Inject a new dependency into the list of dependencies. - (addFieldLinesListLike @Dependency myNewDep) - --- A helper function that adds a given thing into a list of 'FieldLine Position'. -addFieldLinesListLike :: forall t. (Parsec t, Pretty t) => t -> ([FieldLine Position] -> [FieldLine Position]) + -- Defined in previous example using 'addValueList'. + (addNewDependency @Dependency myNewDep) ``` The set of all the foci of a `Edit` tree describes a set of matching paths down the tree of fields. At the leaf (in the above example, `AddField`) we help user build a function that modifies -`[FieldLine Position]` by providing `addFieldLinesListLike`. +`[FieldLine Position]` by providing `addValueList`. We strive to make the API flexible and will expose ways to modify `[Field Position]` directly. We don't try to guarantee the correctness of this From 3d085961cc2bdbbc2ad071886e2708712660a576 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 30 Jul 2026 08:59:44 +0200 Subject: [PATCH 40/43] add description about project files --- proposals/cabal-exactprint.typ | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 627db76..fdafbfe 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -741,6 +741,10 @@ It would also benefit existing programs that depend on Cabal: This work would also simplify implementation of formatters or modification tools operating on other formats using the same envelope format, namely #(references.project-description-documentation.override-name)["project descriptions"]. +We have tested this existing implementation with project files within the haskell/cabal repo +itself, and obtained 55 roundtrip out of 60 files. The failures are mostly due to line-ending +errors. + == Implementation Notes #references.jappie-original-twg-proposal.get-link has been accepted and funded by the Haskell Foundation. @@ -764,6 +768,11 @@ describe the modification API in terms of lens. Trailing whitespaces and lines with only whitespaces are also lost in the current exactprint implementation. To restore them, they need to be saved. This would entail more modification to the lexer and field parser. We want to know if it's feasible to drop them. + + By subsituting tab with spaces in package description throughout hackage before parsing, we + made another 3724 files roundtrip. In other words, these files are only failing due to their tabs + being converted to spaces. That is 1.9141 percent of hackage. + On a related note, git can be configured to detect trailing whitespaces and warn the user, or automatically remove them. From 2e49e41fed76e803c53c8f4aa3e89579c7d8b301 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Thu, 30 Jul 2026 08:59:56 +0200 Subject: [PATCH 41/43] regenerate markdown --- proposals/cabal-exactprint.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 1b4eeb0..5d38aee 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -345,16 +345,14 @@ appendDependency = -- Focus on a field, create it should it not exist. (hasFieldName "build-depends") -- Inject a new dependency into the list of dependencies. - (addFieldLinesListLike @Dependency myNewDep) - --- A helper function that adds a given thing into a list of 'FieldLine Position'. -addFieldLinesListLike :: forall t. (Parsec t, Pretty t) => t -> ([FieldLine Position] -> [FieldLine Position]) + -- Defined in previous example using 'addValueList'. + (addNewDependency @Dependency myNewDep) ``` The set of all the foci of a `Edit` tree describes a set of matching paths down the tree of fields. At the leaf (in the above example, `AddField`) we help user build a function that modifies -`[FieldLine Position]` by providing `addFieldLinesListLike`. +`[FieldLine Position]` by providing `addValueList`. We strive to make the API flexible and will expose ways to modify `[Field Position]` directly. We don't try to guarantee the correctness @@ -710,6 +708,10 @@ modification tools operating on other formats using the same envelope format, namely ["project descriptions"](https://cabal.readthedocs.io/en/stable/cabal-project-description-file.html#project-description-cabal-project-file). +We have tested this existing implementation with project files within +the haskell/cabal repo itself, and obtained 55 roundtrip out of 60 +files. The failures are mostly due to line-ending errors. + ## Implementation Notes [Jappie's original Haskell Foundation Tech @@ -739,9 +741,15 @@ describe the modification API in terms of lens. Trailing whitespaces and lines with only whitespaces are also lost in the current exactprint implementation. To restore them, they need to be saved. This would entail more modification to the lexer and field - parser. We want to know if it's feasible to drop them. On a related - note, git can be configured to detect trailing whitespaces and warn - the user, or automatically remove them. + parser. We want to know if it's feasible to drop them. + + By subsituting tab with spaces in package description throughout + hackage before parsing, we made another 3724 files roundtrip. In other + words, these files are only failing due to their tabs being converted + to spaces. That is 1.9141 percent of hackage. + + On a related note, git can be configured to detect trailing + whitespaces and warn the user, or automatically remove them. - Line endings From 033db6796777967c05337547335b433883d4ec2e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Tue, 4 Aug 2026 09:57:41 +0200 Subject: [PATCH 42/43] remove extra parenthesis, fix type --- proposals/cabal-exactprint.md | 2 +- proposals/cabal-exactprint.typ | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 5d38aee..3311e1b 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -196,7 +196,7 @@ modifyValueAtomAla , Pretty b ) => (a -> Maybe a) -- ^ Nothing prevents a new render. - -> ([FieldLine Position] -> FieldLine Position)) + -> ([FieldLine Position] -> [FieldLine Position]) modifyValueAtomAla = {- Implementation of the algorithm for single value. -} -- | Build a @[FieldLine Position]@ modification function given a function @a -> Maybe a@, parsed as @List sep b a@. diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index fdafbfe..613ec04 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -247,7 +247,7 @@ modifyValueAtomAla , Pretty b ) => (a -> Maybe a) -- ^ Nothing prevents a new render. - -> ([FieldLine Position] -> FieldLine Position)) + -> ([FieldLine Position] -> [FieldLine Position]) modifyValueAtomAla = {- Implementation of the algorithm for single value. -} -- | Build a @[FieldLine Position]@ modification function given a function @a -> Maybe a@, parsed as @List sep b a@. From 5088cfd72c09f3655f8827680d44d659456c65dc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=C3=A9ana?= Date: Fri, 7 Aug 2026 10:33:18 +0200 Subject: [PATCH 43/43] add few more open questions --- proposals/cabal-exactprint.md | 26 ++++++++++++++++++++++++++ proposals/cabal-exactprint.typ | 23 +++++++++++++++++++++++ 2 files changed, 49 insertions(+) diff --git a/proposals/cabal-exactprint.md b/proposals/cabal-exactprint.md index 3311e1b..5780b53 100644 --- a/proposals/cabal-exactprint.md +++ b/proposals/cabal-exactprint.md @@ -771,6 +771,32 @@ describe the modification API in terms of lens. to perfect. We want to know if the current implementation is satisfactory. +- Granularity of modification + + We propose functions to insertion/removal/modify list-like field + content for any separator that Cabal supports (e.g. `license-files`, + `build-depends`, etc), as well as function to modify single element + field content (e.g. `cabal-version`). + + Should insertion and removal allow user to insert to arbitrary + position into the list? Are simple prepend/append (for the case of + insertion) enough ? + +- "Prettiness" of list modification + + This is of course subjective, but it can still somewhat be discussed. + [ The current implementation + ](https://github.com/leana8959/cabal/blob/5c3c0eb445937e52f9b9c87e8d998cec283990f1/Cabal-tests/tests/ParserTests.hs#L168-L363) + generally follows the idea that the new item should look like it's + formatted, without touching existing items before or after. To make + the matter simpler (and by consequence more correct) I do not try to + prepend a leading comma even if the user had it, for example. + + As an example of subjectively bad formatting, it is possible to + append/prepend on the same line as long as the old item is correctly + separated from the new one. If one were to chain this operation many + times, the string will become very long, albeit being exact. + ## References - [Retain comments in field parser diff --git a/proposals/cabal-exactprint.typ b/proposals/cabal-exactprint.typ index 613ec04..2624211 100644 --- a/proposals/cabal-exactprint.typ +++ b/proposals/cabal-exactprint.typ @@ -792,5 +792,28 @@ describe the modification API in terms of lens. Sorting is more of a formatter feature, which exactprint doesn't try to perfect. We want to know if the current implementation is satisfactory. +- Granularity of modification + + We propose functions to insertion/removal/modify list-like field content for any separator that Cabal + supports (e.g. `license-files`, `build-depends`, etc), as well as function to modify single + element field content (e.g. `cabal-version`). + + Should insertion and removal allow user to insert to arbitrary position into the list? + Are simple prepend/append (for the case of insertion) enough ? + +- "Prettiness" of list modification + + This is of course subjective, but it can still somewhat be discussed. + #link("https://github.com/leana8959/cabal/blob/5c3c0eb445937e52f9b9c87e8d998cec283990f1/Cabal-tests/tests/ParserTests.hs#L168-L363")[ + The current implementation ] generally follows the idea that the new item should look like it's + formatted, without touching existing items before or after. + To make the matter simpler (and by consequence more correct) I do not try to prepend a leading + comma even if the user had it, for example. + + As an example of subjectively bad formatting, it is possible to append/prepend on the same line as + long as the old item is correctly separated from the new one. If one were to chain this operation + many times, the string will become very long, albeit being exact. + + == References #list(..references.values().map(x => x.get-link))