-
Notifications
You must be signed in to change notification settings - Fork 249
Roam: Fix table conversion and code block formatting #506
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # CLAUDE.md | ||
|
|
||
| This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. | ||
|
|
||
| ## Project Overview | ||
|
|
||
| Obsidian Importer is an Obsidian plugin that imports notes from various formats (Notion, Evernote, Apple Notes, OneNote, Google Keep, Bear, Roam, CSV, HTML, etc.) into Obsidian vaults. | ||
|
|
||
| ## Commands | ||
|
|
||
| - **Build**: `npm run build` (runs `tsc -skipLibCheck` then esbuild production bundle) | ||
| - **Dev**: `npm run dev` (esbuild watch mode with inline sourcemaps) | ||
| - **Lint**: `npm run lint` (eslint with `--fix`, uses flat config) | ||
|
|
||
| There is no automated test suite. The `tests/` directory contains fixture data for manual testing. Verify changes by building and testing the plugin in Obsidian. | ||
|
|
||
| ## Architecture | ||
|
|
||
| ### Plugin Entry Point | ||
|
|
||
| `src/main.ts` — defines `ImporterPlugin` (extends Obsidian `Plugin`), `ImporterModal` (the UI), and `ImportContext` (progress tracking during imports). All importers are registered in the `importers` map inside `ImporterPlugin`. | ||
|
|
||
| ### Importer Pattern | ||
|
|
||
| Every importer extends `FormatImporter` (`src/format-importer.ts`): | ||
|
|
||
| 1. **`init()`** — adds UI settings to `this.modal.contentEl` (file chooser, output location, format-specific options) | ||
| 2. **`showTemplateConfiguration(ctx, container)`** — optional second config screen (e.g., CSV column mapping). Returns `true`/`false`/`null`. | ||
| 3. **`import(ctx: ImportContext)`** — performs the actual import. Report progress via `ctx.reportNoteSuccess()`, `ctx.reportAttachmentSuccess()`, `ctx.reportSkipped()`, `ctx.reportFailed()`, `ctx.reportProgress()`. | ||
|
|
||
| Format implementations live in `src/formats/` — each is a file or directory. | ||
|
|
||
| ### Registering a New Importer | ||
|
|
||
| Add entry to the `importers` object in `src/main.ts`: | ||
| ```ts | ||
| 'format-id': { | ||
| name: 'Display Name', | ||
| optionText: 'Dropdown text (.ext)', | ||
| helpPermalink: 'import/format-id', | ||
| importer: YourImporterClass, | ||
| } | ||
| ``` | ||
|
|
||
| ### Key Modules | ||
|
|
||
| - `src/filesystem.ts` — file I/O abstraction (`PickedFile`, `NodePickedFile`, `WebPickedFile`). Must use this instead of direct Node.js `fs`/`path` imports. | ||
| - `src/template.ts` — template system for structured data (CSV, etc.). Substitutes `{{fieldName}}` placeholders and generates YAML frontmatter. | ||
| - `src/util.ts` — filename sanitization, text utilities. | ||
| - `src/zip.ts` — zip file handling via `@zip.js/zip.js`. | ||
|
|
||
| ### Cross-Platform Compatibility | ||
|
|
||
| The plugin runs on desktop (Electron/Node.js) and mobile (web). Node.js modules must be soft-imported: | ||
| ```ts | ||
| import type * as NodeModuleName from 'node:modulename'; | ||
| const modulename: typeof NodeModuleName = Platform.isDesktopApp ? window.require('node:modulename') : null; | ||
| ``` | ||
|
|
||
| Some importers (e.g., Apple Notes) are desktop-only and set `this.notAvailable = true` on unsupported platforms. | ||
|
|
||
| ## Code Style | ||
|
|
||
| - TypeScript only, strict mode | ||
| - Tabs for indentation, single quotes, semicolons | ||
| - Stroustrup brace style | ||
| - Unused function args are allowed (prefixed with `_` or not) | ||
| - `any` type is permitted | ||
| - Minimal dependencies — avoid heavy libraries | ||
| - Avoid concurrency (sequential processing to prevent memory issues with large vaults) | ||
| - esbuild bundles to single `main.js`; `obsidian`, `electron`, and `codemirror` are external |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -15,6 +15,9 @@ const imageRegex = /https:\/\/firebasestorage(.*?)\?alt(.*?)\)/; | |
| const binaryRegex = /https:\/\/firebasestorage(.*?)\?alt(.*?)/; | ||
|
|
||
| const blockRefRegex = /(?<=\(\()\b(.*?)\b(?=\)\))/g; | ||
| const roamTableRe = /^\{\{(\[\[)?table(\]\])?\}\}$/i; | ||
| const codeBlockLangs = ['clojure', 'css', 'elixir', 'html', 'plain text', 'python', 'ruby', 'swift', 'typescript', 'jsx', 'yaml', 'json', 'json-ld', 'rust', 'r', 'shell', 'php', 'java', 'c#', 'c', 'c\\+\\+', 'objective-c', 'go', 'kotlin', 'sql', 'haskell', 'scala', 'commonlisp', 'solidity', 'julia', 'sparql', 'turtle', 'lua', 'dart', 'latex', 'markdown', 'xml', 'toml', 'vb', 'vbscript', 'javascript']; | ||
| const codeBlockLangRe = new RegExp('```(' + codeBlockLangs.join('|') + ')\\n', 'gi'); | ||
|
|
||
| export class RoamJSONImporter extends FormatImporter { | ||
| downloadAttachments: boolean = false; | ||
|
|
@@ -248,6 +251,13 @@ export class RoamJSONImporter extends FormatImporter { | |
| } | ||
|
|
||
| private async roamMarkupScrubber(graphFolder: string, attachmentsFolder: string, blockText: string, skipDownload: boolean = false): Promise<string> { | ||
| // Strip language tags from code blocks (```javascript\n → ```\n) | ||
| blockText = blockText.replace(codeBlockLangRe, '```\n'); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why are these being stripped? Obsidian supports these. |
||
| // Normalize code fences: ensure closing ``` is on its own line | ||
| blockText = blockText.replace(/([^\n])```/g, '$1\n```'); | ||
| // Ensure opening ``` followed by content has a newline after it | ||
| blockText = blockText.replace(/```([^\n])/g, '```\n$1'); | ||
|
|
||
| // Remove roam-specific components | ||
| blockText = blockText.replace(roamSpecificMarkupRe, ''); | ||
|
|
||
|
|
@@ -286,7 +296,7 @@ export class RoamJSONImporter extends FormatImporter { | |
| blockText = await this.downloadFirebaseFile(blockText, attachmentsFolder); | ||
| } | ||
| } | ||
| // blockText = blockText.replaceAll("{{[[table]]}}", ""); | ||
| // table conversion is handled in jsonToMarkdown via convertRoamTable | ||
| // blockText = blockText.replaceAll("{{[[kanban]]}}", ""); | ||
| // blockText = blockText.replaceAll("{{mermaid}}", ""); | ||
| // blockText = blockText.replaceAll("{{[[mermaid]]}}", ""); | ||
|
|
@@ -339,15 +349,25 @@ export class RoamJSONImporter extends FormatImporter { | |
| this.oldestTimestamp = createdTimestamp; | ||
| } | ||
|
|
||
| if ('string' in json && json.string) { | ||
| const prefix = json.heading ? '#'.repeat(json.heading) + ' ' : ''; | ||
| const scrubbed = await this.roamMarkupScrubber(graphFolder, attachmentsFolder, json.string); | ||
| markdown.push(`${isChild ? indent + '* ' : indent}${prefix}${scrubbed}`); | ||
| if ('string' in json && json.string && roamTableRe.test(json.string.trim()) && json.children) { | ||
| markdown.push(await this.convertRoamTable(graphFolder, attachmentsFolder, json, indent)); | ||
| } | ||
| else { | ||
| if ('string' in json && json.string) { | ||
| const prefix = json.heading ? '#'.repeat(json.heading) + ' ' : ''; | ||
| const scrubbed = await this.roamMarkupScrubber(graphFolder, attachmentsFolder, json.string); | ||
| const linePrefix = isChild ? indent + '* ' : indent; | ||
| const continuationIndent = isChild ? indent + ' ' : indent; | ||
| const indented = scrubbed.contains('\n') | ||
| ? scrubbed.split('\n').map((line, i) => i === 0 ? line : continuationIndent + line).join('\n') | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should avoid prepending whitespace to blank lines. You also don't need to the check for newlines. |
||
| : scrubbed; | ||
| markdown.push(`${linePrefix}${prefix}${indented}`); | ||
| } | ||
|
|
||
| if (json.children) { | ||
| for (const child of json.children) { | ||
| markdown.push(await this.jsonToMarkdown(graphFolder, attachmentsFolder, child, indent + ' ', true, '', this.oldestTimestamp, this.newestTimestamp)); | ||
| if (json.children) { | ||
| for (const child of json.children) { | ||
| markdown.push(await this.jsonToMarkdown(graphFolder, attachmentsFolder, child, indent + ' ', true, '', this.oldestTimestamp, this.newestTimestamp)); | ||
| } | ||
| } | ||
| } | ||
|
|
||
|
|
@@ -387,6 +407,51 @@ export class RoamJSONImporter extends FormatImporter { | |
| return markdown.join('\n'); | ||
| } | ||
|
|
||
| private async convertRoamTable( | ||
| graphFolder: string, | ||
| attachmentsFolder: string, | ||
| json: RoamPage | RoamBlock, | ||
| indent: string | ||
| ): Promise<string> { | ||
| const rows = json.children || []; | ||
| if (rows.length === 0) return ''; | ||
|
|
||
| // Extract cells from each row by walking the linear first-child chain | ||
| const tableData: string[][] = []; | ||
| for (const row of rows) { | ||
| const cells: string[] = []; | ||
| let current: RoamBlock | undefined = row; | ||
| while (current) { | ||
| const scrubbed = await this.roamMarkupScrubber(graphFolder, attachmentsFolder, current.string || ''); | ||
| cells.push(scrubbed.replace(/\|/g, '\\|')); | ||
| current = current.children?.[0]; | ||
| } | ||
| tableData.push(cells); | ||
| } | ||
|
|
||
| // Determine max columns and pad shorter rows | ||
| const numCols = Math.max(...tableData.map(r => r.length)); | ||
| for (const row of tableData) { | ||
| while (row.length < numCols) { | ||
| row.push(''); | ||
| } | ||
| } | ||
|
|
||
| // Build pipe table | ||
| const lines: string[] = []; | ||
| // Header row | ||
| lines.push(indent + '| ' + tableData[0].join(' | ') + ' |'); | ||
| // Separator | ||
| lines.push(indent + '| ' + tableData[0].map(() => '---').join(' | ') + ' |'); | ||
| // Data rows | ||
| for (let i = 1; i < tableData.length; i++) { | ||
| lines.push(indent + '| ' + tableData[i].join(' | ') + ' |'); | ||
| } | ||
|
|
||
| // Blank line before table for clean rendering | ||
| return '\n' + lines.join('\n'); | ||
| } | ||
|
|
||
| private async modifySourceBlockString(markdownPages: Map<string, string>, sourceBlock: BlockInfo, graphFolder: string, sourceBlockUID: string) { | ||
| if (!sourceBlock.blockString.endsWith('^' + sourceBlockUID)) { | ||
| const sourceBlockFilePath = `${graphFolder}/${sourceBlock.pageName}.md`; | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm confused why this is needed. Why are these tags being removed?