Skip to content

[LLM:Feature] Add Agentic Flow to MNN LLM Chat - #4512

Open
huangzhengxiang wants to merge 11 commits into
alibaba:masterfrom
Embedded-AI-Systems:agent
Open

[LLM:Feature] Add Agentic Flow to MNN LLM Chat#4512
huangzhengxiang wants to merge 11 commits into
alibaba:masterfrom
Embedded-AI-Systems:agent

Conversation

@huangzhengxiang

@huangzhengxiang huangzhengxiang commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds an optional Agent mode to MNN LLM Chat while preserving the existing normal chat flow. Agent mode can use host-provided tools for web search,built-in browser reading, Python execution, workspace files, memory, and skill updates. Normal sessions remain backward compatible and continue using the original path when Agent mode is not enabled.

Changes

  • Added per-session Agent mode selection for new conversations.
  • Persisted session-level Agent metadata with database migration and backward compatibility for existing sessions.
  • Added an agentic loop in ChatPresenter:
    • Parses model-emitted system_calls.
    • Executes tool calls step by step.
    • Feeds observations back into the same conversation.
    • Supports visible progress, continuation, budget limits, and stop control.
  • Added robust output parsing for imperfect model JSON:
    • fenced JSON blocks
    • single tool objects
    • nested system_calls arrays
    • loose python_exec recovery
  • Added built-in tools:
    • web_search
    • browser_url
    • python_exec
    • get_current_time
  • Added lightweight Python runtime integration through Chaquopy.
  • Added workspace-aware Python helpers for file generation, including Excel output.
  • Added generated file detection and attachment handling for chat replies.
  • Added an in-app workspace file browser from chat history.
  • Added memory and skill persistence with filtering/compaction to avoid noisy one-off updates.
  • Updated native session handling to better support prompt cache/KV-cache continuation for Agent turns.
  • Added token/performance display for assistant messages.
  • Added docs for Agent design, built-in browser, built-in Python, memory/skill, development notes, and future iteration.
  • Ignored local Android/CMake build artifacts such as .cxx/ and app/standard/.

Notes

  • Existing non-Agent conversations should continue to use the old chat path.
  • Agent mode is selected when creating a new conversation and then persists with that session.
  • Python file outputs are resolved inside the app workspace and surfaced back into chat when possible.
  • The workspace browser now uses the app's internal file browser instead of relying on system file-manager folder intents.

Testing

  • Manual device testing was used during development for:
    • Agent mode selection
    • web/browser tool calls
    • Python execution
    • Excel file generation
    • generated file return in chat
    • workspace file browsing
    • session restore/history behavior

Module

MNN Chat

Type

  • Feature
  • Bugfix
  • Perf
  • Refact
  • Style
  • Doc
  • Test
  • Chore

Checklist

  • Commit message follows [Module:Type] Description format
  • Code compiles without errors
  • Tested on relevant platform(s)
  • No unrelated format or style changes included

@wangzhaode wangzhaode self-assigned this Jun 11, 2026
@wangzhaode
wangzhaode requested a review from Juude June 11, 2026 01:40
wangzhaode
wangzhaode reviewed Jun 11, 2026
View reviewed changes

@wangzhaode wangzhaode left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

这个 PR 为 Android MNN LLM Chat 添加了 Agent 模式，包括 web search、Python 执行、浏览器读取等工具调用能力。代码完全在 apps/Android/MnnLlmChat/ 目录下，不影响 MNN 核心引擎。

需要关注的点

  1. Chaquopy 依赖：引入了 com.chaquo.python 插件（Python on Android），会显著增加 APK 体积。确认是否只在特定 flavor 中启用，以及是否有无 Python 的降级方案。

  2. 安全性：python_exec 工具在设备上执行模型生成的 Python 代码，存在安全风险。建议添加沙箱限制（禁止网络访问、文件系统访问范围限制等）。

  3. Agent loop 限制：确认是否有最大迭代次数限制防止无限循环（budget limits 在描述中提到了，很好）。

  4. 数据库迁移：新增 Agent 相关字段需要确保向后兼容（老版本数据库升级不崩溃）。

  5. build.gradle 格式：从 build.gradle.kts（Kotlin DSL）变成了 build.gradle（Groovy），这是有意的重构还是冲突？请确认。

  6. C++ JNI 层改动：llm_session.cpp 和 llm_mnn_jni.cpp 有少量改动，需要确认不影响现有 chat 流程。

  7. 文档完善 ✅：docs/ 目录下有完整的设计文档。

建议

  • 等 @Juude 的 review 完成后再决定合入
  • 确认格式检查失败的原因并修复
  • 本 PR 不涉及引擎层改动，主要需要 Android 应用侧的测试验证

@Juude

Juude commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

apps/Android/MnnLlmChat/docs/Agent.gif 这个 gif 太大了，可以找个在线裁剪平台裁剪下，然后邮件发我下，我上传到 cdn

@huangzhengxiang

huangzhengxiang commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

apps/Android/MnnLlmChat/docs/Agent.gif 这个 gif 太大了，可以找个在线裁剪平台裁剪下，然后邮件发我下，我上传到 cdn

发你的gmail邮箱了，压缩完2.8MB

@Juude

Juude commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

apps/Android/MnnLlmChat/docs/Agent.gif 这个 gif 太大了，可以找个在线裁剪平台裁剪下，然后邮件发我下，我上传到 cdn

发你的gmail邮箱了，压缩完2.8MB

可以直接用这个 cdn 地址： https://meta.alicdn.com/data/mnn/assets/Agent.gif

@huangzhengxiang

huangzhengxiang commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

apps/Android/MnnLlmChat/docs/Agent.gif 这个 gif 太大了，可以找个在线裁剪平台裁剪下，然后邮件发我下，我上传到 cdn

发你的gmail邮箱了，压缩完2.8MB

可以直接用这个 cdn 地址： https://meta.alicdn.com/data/mnn/assets/Agent.gif

已更新，把原有gif删了。

@wangzhaode

wangzhaode commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

感谢提交这个功能，Agent 模式的整体框架、按会话默认关闭、工具调用预算和数据库迁移都设计得比较完整，而且修改主要位于 Android App，不影响 MNN 核心引擎。

目前合入官方 App 前还需要处理几个安全和发布问题：

  1. 当前 python_exec 还不具备可靠的工作区隔离。我们用现有代码做了最小验证，允许的 urllib 可以读取 Agent 工作区外的文件；同时模型生成的代码会自动执行。建议先将该能力放到 internal/experimental flavor，或至少在执行前向用户展示代码并明确确认。
  2. browser_url 目前只检查 HTTP/HTTPS 协议，请增加对 localhost、私网、链路本地地址的拦截，并在每次重定向后重新校验目标地址。
  3. Chaquopy、NumPy、Pandas 和 OpenPyXL 当前会进入所有 flavor，请考虑只在 Agent flavor 启用，并提供开启前后的 APK 体积 A/B。
  4. 请补充 standard、googleplay 的构建验证，并修复现有 commit message 和 clang-format 检查。

完成上述安全边界后，这个功能可以继续推进。谢谢这次比较完整的实现和文档！

@wangzhaode wangzhaode added the awaiting contributor Waiting for contributor to address review comments or rebase label Aug 7, 2026
@wangzhaode

wangzhaode commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

感谢继续修改。将 NumPy、Pandas 和 OpenPyXL 限制到 Agent flavor，并修复 changed-lines 格式问题，这两项已经解决了之前反馈的一部分。

目前还有两个安全问题会阻塞官方 App 合入：

  1. python_exec 仍然会自动执行模型生成的代码，也没有可靠限制 Python 只能访问 Agent workspace。请至少在执行前向用户展示完整代码并明确确认；如果无法提供可靠隔离，建议仅在 internal/experimental 构建中启用。
  2. browser_url 仍然只验证 http/https scheme，没有阻止 localhost、私网和链路本地地址，也没有在重定向后重新验证目标地址。请补充 DNS/IP 校验，并对每次 redirect 重新执行相同检查。

另外，目前只是把 pip 包限制到了 Agent flavor，Chaquopy/Python runtime 本身仍由 app 模块全局启用。请确认 standard 和 googleplay APK 是否仍包含 Python runtime，并提供 Agent 开启前后的 APK 体积 A/B，以及 standard、googleplay、agent 三个 flavor 的基本构建结果。完成这些安全边界后再合并会更稳妥，谢谢！

@huangzhengxiang
Merge remote-tracking branch 'alibaba/master' into agent
aadc00d 
# Conflicts:
#	transformers/llm/engine/include/llm/llm.hpp
#	transformers/llm/engine/src/sampler.hpp
@huangzhengxiang
[LLM:Style] Fix formatting after upstream merge
5056f23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Juude Juude Awaiting requested review from Juude
@wangzhaode wangzhaode Awaiting requested review from wangzhaode

At least 1 approving review is required to merge this pull request.

Assignees

@wangzhaode wangzhaode

Labels

awaiting contributor Waiting for contributor to address review comments or rebase

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@huangzhengxiang @Juude @wangzhaode