Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/en/api/02-resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,19 @@ OpenViking supports various resource types, categorized by functionality:

| Type | Extensions | Description |
|------|------------|-------------|
| PDF | `.pdf` | Supports local parsing and MinerU API conversion |
| PDF | `.pdf` | Uses pdf-inspector for page text, headings, tables, and reading order; pdfplumber extracts images after each page's text. Pages that require OCR are kept with explicit warnings |
| Markdown | `.md`, `.markdown`, `.mdown`, `.mkd` | Native support, extracts structure and stores in segments |
| HTML | `.html`, `.htm` | Cleans navigation/ads and extracts content, converts to Markdown |
| Word | `.docx` | Extracts text, headings, tables and converts to Markdown |
| Word | `.doc`, `.docx` | Uses AnyDoc to preserve text, headings, tables, and embedded image positions in Markdown |
| Plain Text | `.txt`, `.text` | Direct import and processing |
| EPUB | `.epub` | E-book format, supports ebooklib or manual extraction |
| EPUB | `.epub` | Uses AnyDoc to convert the e-book structure and embedded images to Markdown |

**Spreadsheets & Presentations**

| Type | Extensions | Description |
|------|------------|-------------|
| Excel | `.xlsx`, `.xls`, `.xlsm` | Supports new and legacy Excel formats, converts to Markdown tables by worksheet |
| PowerPoint | `.pptx` | Extracts content by slide, supports extracting notes |
| Excel | `.xlsx`, `.xls`, `.xlsm` | Uses AnyDoc to convert worksheets to Markdown |
| PowerPoint | `.pptx` | Uses AnyDoc to preserve slide content and embedded image positions; speaker notes stay in the same resource under `Speaker Notes` sections |

**Code**

Expand Down
4 changes: 1 addition & 3 deletions docs/en/configuration/01-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,6 @@ Parsers live under `parsers`:
"audio": {},
"video": {},
"markdown": {},
"excel": {},
"html": {},
"text": {},
"directory": {},
Expand All @@ -357,8 +356,7 @@ Parsers live under `parsers`:
| `code` | Repository file types, ignore rules, and network safety |
| `image` | Image understanding and OCR |
| `audio`, `video` | Audio/video parsing |
| `markdown`, `html`, `text` | Text document chunking |
| `excel` | Workbook parsing and chunking |
| `markdown`, `html`, `text` | Text document chunking; AnyDoc-backed Office and EPUB files reuse the `markdown` sectioning settings |
| `directory` | Directory scanning and ignore rules |
| `feishu` | Feishu/Lark access and parsing |
| `webfeed` | Sitemap, RSS, and Atom ingestion |
Expand Down
10 changes: 5 additions & 5 deletions docs/zh/api/02-resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,18 @@ OpenViking 支持多种资源类型,按照功能分类如下:
文档类
| 类型 | 扩展名 | 说明 |
|------|--------|------|
| PDF | `.pdf` | 支持本地解析和 MinerU API 转换 |
| PDF | `.pdf` | 由 pdf-inspector 提取分页文本、标题、表格和阅读顺序,pdfplumber 在每页正文后提取图片;需要 OCR 的页面会保留并给出明确警告 |
| Markdown | `.md`, `.markdown`, `.mdown`, `.mkd` | 原生支持,会提取结构并分段存储 |
| HTML | `.html`, `.htm` | 清理导航/广告后提取内容,转换为 Markdown |
| Word | `.docx` | 提取文本、标题、表格并转换为 Markdown |
| Word | `.doc`, `.docx` | 通过 AnyDoc 将文本、标题、表格和嵌入图片的原始位置转换为 Markdown |
| 纯文本 | `.txt`, `.text` | 直接导入处理 |
| EPUB | `.epub` | 电子书格式,支持 ebooklib 或手动提取 |
| EPUB | `.epub` | 通过 AnyDoc 将电子书结构和嵌入图片转换为 Markdown |

表格类
| 类型 | 扩展名 | 说明 |
|------|--------|------|
| Excel | `.xlsx`, `.xls`, `.xlsm` | 支持新版和老版 Excel,按工作表转换为 Markdown 表格 |
| PowerPoint | `.pptx` | 按幻灯片提取内容,支持提取备注 |
| Excel | `.xlsx`, `.xls`, `.xlsm` | 通过 AnyDoc 将工作表转换为 Markdown |
| PowerPoint | `.pptx` | 通过 AnyDoc 保留幻灯片内容和嵌入图片位置;演讲者备注保留在同一资源的 `Speaker Notes` 分节中 |

代码类
| 类型 | 资源名 | 说明 |
Expand Down
4 changes: 1 addition & 3 deletions docs/zh/configuration/01-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,6 @@ Provider 和密钥管理配置见[加密指南](../guides/08-encryption.md)。
"audio": {},
"video": {},
"markdown": {},
"excel": {},
"html": {},
"text": {},
"directory": {},
Expand All @@ -357,8 +356,7 @@ Provider 和密钥管理配置见[加密指南](../guides/08-encryption.md)。
| `code` | 代码仓库文件类型、忽略规则和安全限制 |
| `image` | 图片理解和 OCR |
| `audio`、`video` | 音视频内容解析 |
| `markdown`、`html`、`text` | 文本文档分段 |
| `excel` | Excel 工作表解析与分段 |
| `markdown`、`html`、`text` | 文本文档分段;由 AnyDoc 解析的 Office 和 EPUB 文件复用 `markdown` 分段配置 |
| `directory` | 目录扫描和忽略规则 |
| `feishu` | 飞书文档访问与解析 |
| `webfeed` | Sitemap、RSS 和 Atom 导入 |
Expand Down
4 changes: 0 additions & 4 deletions examples/ov.conf.example
Original file line number Diff line number Diff line change
Expand Up @@ -223,14 +223,10 @@
},
"parsers": {
"pdf": {
"strategy": "auto",
"max_content_length": 100000,
"max_section_size": 4000,
"section_size_flexibility": 0.3,
"max_section_chars": 6000,
"mineru_endpoint": "https://mineru.example.com/api/v1",
"mineru_api_key": "{your-mineru-api-key}",
"mineru_timeout": 300.0,
},
"code": {
"github_raw_domain": "raw.githubusercontent.com",
Expand Down
58 changes: 58 additions & 0 deletions openviking/parse/image_validation.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
"""Shared acceptance policy for images embedded in parsed documents."""

import io
from pathlib import Path

from openviking_cli.utils.logger import get_logger

logger = get_logger(__name__)

IMAGE_MIN_SIDE = 14
IMAGE_MIN_PIXELS = 196
IMAGE_MAX_PIXELS = 36_000_000
IMAGE_MIN_ASPECT_RATIO = 1 / 150
IMAGE_MAX_ASPECT_RATIO = 150
IMAGE_MAX_FILE_BYTES = 10 * 1024 * 1024


def is_valid_image(image_bytes: bytes, source_path: Path) -> bool:
"""Return whether an extracted image is safe and useful to ingest."""
if len(image_bytes) > IMAGE_MAX_FILE_BYTES:
logger.warning(f"[ImageValidation] Image exceeds 10MB, skipping: {source_path}")
return False

try:
from PIL import Image

with Image.open(io.BytesIO(image_bytes)) as image:
width, height = image.size
except Exception as exc:
logger.warning(
f"[ImageValidation] Cannot read image dimensions, skipping {source_path}: {exc}"
)
return False

if width <= IMAGE_MIN_SIDE or height <= IMAGE_MIN_SIDE:
logger.warning(
f"[ImageValidation] Image side too small ({width}x{height}), skipping: {source_path}"
)
return False

pixels = width * height
if pixels < IMAGE_MIN_PIXELS or pixels > IMAGE_MAX_PIXELS:
logger.warning(
f"[ImageValidation] Image pixel count out of range ({pixels}), skipping: {source_path}"
)
return False

aspect_ratio = width / height
if aspect_ratio < IMAGE_MIN_ASPECT_RATIO or aspect_ratio > IMAGE_MAX_ASPECT_RATIO:
logger.warning(
f"[ImageValidation] Image aspect ratio out of range ({aspect_ratio:.4f}), "
f"skipping: {source_path}"
)
return False

return True
10 changes: 2 additions & 8 deletions openviking/parse/parsers/__init__.py
Original file line number Diff line number Diff line change
@@ -1,26 +1,20 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0

from .anydoc import AnyDocParser
from .base_parser import BaseParser
from .epub import EPubParser
from .excel import ExcelParser
from .html import HTMLParser
from .markdown import MarkdownParser
from .pdf import PDFParser
from .powerpoint import PowerPointParser
from .text import TextParser
from .word import WordParser
from .zip_parser import ZipParser

__all__ = [
"AnyDocParser",
"BaseParser",
"EPubParser",
"ExcelParser",
"HTMLParser",
"MarkdownParser",
"PDFParser",
"PowerPointParser",
"TextParser",
"WordParser",
"ZipParser",
]
Loading
Loading