feat: Implement source mapping for elements and enhance inline parsing

docs/api.rst

@@ -52,6 +52,9 @@ Inline Elements
 Helper Classes
 --------------
 
+.. autoclass:: marko.element.Element
+    :members:
+
 .. autoclass:: marko.source.Source
     :members:
 

docs/usage.rst

@@ -113,6 +113,48 @@ The parsed document is an Abstract Syntax Tree (AST) that you can traverse and m
 
 For more details about element types, see :ref:`elements`.
 
+Source Positions
+----------------
+
+Each element in the AST carries the positions of its corresponding text in the
+source, which is useful for implementing features like syntax highlighters.
+
+Every element has a :attr:`~marko.element.Element.source_span` attribute, a
+``(start, end)`` tuple of indices into the source text, and the convenience
+properties :attr:`~marko.element.Element.start_pos` and
+:attr:`~marko.element.Element.end_pos`. The positions are ``None`` when not
+available (e.g. for elements produced by extensions that don't record them)::
+
+    from marko import Markdown
+
+    text = "[Google](https://google.com)"
+    doc = Markdown().parse(text + "\n")
+    paragraph = doc.children[0]
+    link = paragraph.children[0]
+    print(link.source_span)          # (0, 28)
+    print(text[link.start_pos:link.end_pos])  # [Google](https://google.com)
+
+For inline elements, the parts of the span that are pure syntax (e.g. the
+markers of emphasis, the backticks of a code span, the brackets of a link) are
+exposed via :attr:`~marko.element.Element.syntax_spans`, while the content
+spans are covered by the element's children::
+
+    print(link.syntax_spans)  # [(0, 1), (7, 28)] -> "[" and "](https://google.com)"
+    print(link.children[0].source_span)  # (1, 7) -> "Google"
+
+For :class:`~marko.inline.Link` and :class:`~marko.inline.Image` elements, the
+destination and title are also exposed separately via the ``dest_span`` and
+``title_span`` attributes::
+
+    print(link.dest_span)  # (9, 27) -> "https://google.com"
+
+.. note::
+
+   Positions are indices into the *normalized* source text: line terminators
+   (``\r\n``, ``\r``) are normalized to ``\n`` before parsing, and U+0000 is
+   replaced with U+FFFD. For plain LF text the positions map directly to the
+   original string.
+
 Custom Rendering
 ----------------
 

examples/demo.md

@@ -0,0 +1,88 @@
+# Marko Syntax Highlighting Demo
+
+This document demonstrates how Marko's source mapping info can be used to
+build a **syntax highlighter**. Every token below is colored by its element
+type, using the `source_span` and `syntax_spans` attributes.
+
+## Headings & emphasis
+
+### Level three *with emphasis*
+
+#### Level four with `inline code`
+
+Setext heading
+==============
+
+Second level
+------------
+
+Text with *emphasis*, **strong emphasis**, ***both***, and ~~strikethrough~~
+(GFM). Escaped characters like \*stars\* and \# hashes stay literal.
+
+## Links, images & autolinks
+
+Here is [a link](https://example.com), one [with a title](https://example.com "The title"),
+and a [reference link][ref]. An image: ![Marko logo](https://example.com/logo.png "Logo").
+Autolinks like <https://example.org> and email <me@example.com> work too.
+
+[ref]: https://example.com/ref "Reference"
+
+## Code
+
+Inline code: use `marko.parse(text)` to get the AST, then check
+`element.source_span`.
+
+```python
+import marko
+
+doc = marko.parse("# Hello")
+heading = doc.children[0]
+print(heading.source_span)  # (0, 10)
+```
+
+    An indented code block looks like this one,
+    and keeps its own color.
+
+## Block quotes
+
+> A block quote with *styled* content.
+> It spans multiple lines.
+>
+> > And can be nested.
+
+## Lists
+
+- unordered item with `code`
+- another item
+  - nested item one
+  - nested item two
+- [x] a completed task
+- [ ] a pending task
+
+1. first ordered item
+2. second ordered item
+   1. nested ordered
+
+## Tables (GFM)
+
+| Name   | Price | Quantity |
+| ------ | ----- | -------- |
+| Apple  | $1    | 10       |
+| Banana | $0.5  | 3        |
+
+## Other blocks
+
+---
+
+Some <span style="color: red">raw HTML</span> inline, and a block below:
+
+<div class="note">
+  This is an HTML block.
+</div>
+
+> [!NOTE]
+> GFM alert boxes are supported too.
+
+A paragraph with a hard line break:  
+and a soft line break
+right here.