diff --git a/.gitignore b/.gitignore index 9db9486cd3..660991a405 100644 --- a/.gitignore +++ b/.gitignore @@ -17,6 +17,7 @@ # Java class files *.class +**/.cxx # Generated files bin/ diff --git a/apps/Android/MnnLlmChat/README.md b/apps/Android/MnnLlmChat/README.md index 7fb4339a3f..a0d2406889 100644 --- a/apps/Android/MnnLlmChat/README.md +++ b/apps/Android/MnnLlmChat/README.md @@ -35,6 +35,34 @@ This is our full multimodal language model (LLM) Android app !!!warning!!! This version has been tested exclusively on the OnePlus 13 and Xiaomi 14 Ultra, Due to the demanding performance requirements of large language models (LLMs), many budget or low-spec devices may experience issues such as slow inference speeds, application instability, or even failure to run entirely. and its stability on other devices cannot be guaranteed. If you encounter any issues, please feel free to open an issue for assistance. +## Agentic Capability + +The Android app includes an optional session-level Agent mode inspired by the ActMe project: + ++ 核心技术来源 / Core technical source: `https://github.com/huangzhengxiang/ActMe.git` ++ Normal chat and Agent chat are different conversation modes. The mode is selected when a conversation is created and is persisted with that session. ++ Existing history created before this feature is treated as normal chat. ++ Agent mode runs a lightweight local agentic loop: model planning, `system_calls` parsing, visible tool execution, tool observations, and final answer generation. ++ Current MnnLlmChat tool scope is adapted from ActMe but excludes ADB: `get_current_time`, `web_search`, `browser_url`, and `python_exec`. ++ Built-in Python includes Excel helpers plus `numpy`, `pandas`, and `openpyxl` for local data analysis. ++ Agent mode persists lightweight Memory and Skill updates in the MnnLlmChat chat database and injects them into future Agent prompts. ++ Local ChatActivity conversations rely on the live native MNN session for KV/prompt-cache reuse. Each turn submits only new input; stored chat history is for UI display. Reloading an old session does not currently restore a persisted prompt cache. + +

+ Icon +

+ +Design notes: + ++ [Agentic loop design](./docs/agentic_design.md) ++ [Built-in browser design](./docs/builtin_browser_design.md) ++ [Built-in Python design](./docs/builtin_python_design.md) ++ [Skill and memory design](./docs/skill_memory_design.md) ++ [Development notes](./docs/development_notes.md) ++ [Iteration roadmap](./docs/iteration_roadmap.md) + +ADB control is intentionally not included in this MnnLlmChat port. + # Development + Prepare diff --git a/apps/Android/MnnLlmChat/app/.gitignore b/apps/Android/MnnLlmChat/app/.gitignore index 2170eb1a55..1eb1e71d07 100644 --- a/apps/Android/MnnLlmChat/app/.gitignore +++ b/apps/Android/MnnLlmChat/app/.gitignore @@ -1,4 +1,6 @@ /build +/.cxx +/standard google-services.json src/standard/google-services.json diff --git a/apps/Android/MnnLlmChat/app/build.gradle b/apps/Android/MnnLlmChat/app/build.gradle index d6a923a292..6b4b6e0ff4 100644 --- a/apps/Android/MnnLlmChat/app/build.gradle +++ b/apps/Android/MnnLlmChat/app/build.gradle @@ -2,6 +2,7 @@ plugins { id 'com.android.application' id 'org.jetbrains.kotlin.android' id 'org.jetbrains.kotlin.plugin.serialization' version '2.1.21' + id 'com.chaquo.python' } def enableFirebase = (project.findProperty("ENABLE_FIREBASE")?.toString()?.toBoolean() ?: false) @@ -127,7 +128,7 @@ android { } } } - + productFlavors { standard { dimension "store" @@ -138,6 +139,11 @@ android { buildConfigField "boolean", "IS_GOOGLE_PLAY_BUILD", "true" versionNameSuffix ".gp" } + agent { + dimension "store" + buildConfigField "boolean", "IS_GOOGLE_PLAY_BUILD", "false" + versionNameSuffix ".agent" + } } @@ -183,6 +189,21 @@ android { } } +chaquopy { + defaultConfig { + version "3.11" + } + productFlavors { + getByName("agent") { + pip { + install "openpyxl==3.1.5" + install "numpy" + install "pandas" + } + } + } +} + dependencies { // https://developer.android.com/jetpack/androidx/releases/compose diff --git a/apps/Android/MnnLlmChat/app/src/main/AndroidManifest.xml b/apps/Android/MnnLlmChat/app/src/main/AndroidManifest.xml index a58352a418..5ac3a05d62 100644 --- a/apps/Android/MnnLlmChat/app/src/main/AndroidManifest.xml +++ b/apps/Android/MnnLlmChat/app/src/main/AndroidManifest.xml @@ -46,6 +46,7 @@ android:name="com.alibaba.mnnllm.android.chat.ChatActivity" android:configChanges="orientation|screenSize" android:exported="true" + android:screenOrientation="portrait" android:windowSoftInputMode="adjustResize"> @@ -75,7 +76,8 @@ + android:exported="true" + android:screenOrientation="portrait"> @@ -86,7 +88,8 @@ android:name=".mainsettings.MainSettingsActivity" android:configChanges="orientation|screenSize" android:exported="true" - android:label="@string/settings"> + android:label="@string/settings" + android:screenOrientation="portrait"> @@ -97,7 +100,8 @@ android:name=".mainsettings.StorageManagementActivity" android:configChanges="orientation|screenSize" android:exported="false" - android:label="@string/storage_management" /> + android:label="@string/storage_management" + android:screenOrientation="portrait" /> diff --git a/apps/Android/MnnLlmChat/app/src/main/cpp/llm_mnn_jni.cpp b/apps/Android/MnnLlmChat/app/src/main/cpp/llm_mnn_jni.cpp index 2bcbbd5ad2..2a0dee5ef2 100644 --- a/apps/Android/MnnLlmChat/app/src/main/cpp/llm_mnn_jni.cpp +++ b/apps/Android/MnnLlmChat/app/src/main/cpp/llm_mnn_jni.cpp @@ -167,6 +167,7 @@ JNIEXPORT jobject JNICALL Java_com_alibaba_mnnllm_android_llm_LlmSession_submitN if (!onProgressMethod) { MNN_DEBUG("ProgressListener onProgress method not found."); } + int64_t input_len = llm->CountInputTokens(input_str); auto *context = llm->Response(input_str, [&, progressListener, onProgressMethod]( const std::string &response, bool is_eop) { if (progressListener && onProgressMethod) { @@ -185,6 +186,16 @@ JNIEXPORT jobject JNICALL Java_com_alibaba_mnnllm_android_llm_LlmSession_submitN int64_t audio_time = 0; int64_t prefill_time = 0; int64_t decode_time = 0; + if (context == nullptr) { + env->ReleaseStringUTFChars(inputStr, input_str); + jclass hashMapClass = env->FindClass("java/util/HashMap"); + jmethodID hashMapInit = env->GetMethodID(hashMapClass, "", "()V"); + jmethodID putMethod = + env->GetMethodID(hashMapClass, "put", "(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;"); + jobject hashMap = env->NewObject(hashMapClass, hashMapInit); + env->CallObjectMethod(hashMap, putMethod, env->NewStringUTF("error"), env->NewStringUTF("Generation failed")); + return hashMap; + } prompt_len += context->prompt_len; decode_len += context->gen_seq_len; vision_time += context->vision_us; @@ -198,6 +209,10 @@ JNIEXPORT jobject JNICALL Java_com_alibaba_mnnllm_android_llm_LlmSession_submitN jobject hashMap = env->NewObject(hashMapClass, hashMapInit); // Add metrics to the HashMap + env->CallObjectMethod(hashMap, putMethod, env->NewStringUTF("input_len"), + env->NewObject(env->FindClass("java/lang/Long"), + env->GetMethodID(env->FindClass("java/lang/Long"), "", "(J)V"), + input_len)); env->CallObjectMethod(hashMap, putMethod, env->NewStringUTF("prompt_len"), env->NewObject(env->FindClass("java/lang/Long"), env->GetMethodID(env->FindClass("java/lang/Long"), @@ -222,6 +237,7 @@ JNIEXPORT jobject JNICALL Java_com_alibaba_mnnllm_android_llm_LlmSession_submitN env->NewObject(env->FindClass("java/lang/Long"), env->GetMethodID(env->FindClass("java/lang/Long"), "", "(J)V"), decode_time)); + env->ReleaseStringUTFChars(inputStr, input_str); return hashMap; } diff --git a/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp b/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp index 0638f1733c..7677f06a8c 100644 --- a/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp +++ b/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.cpp @@ -289,16 +289,22 @@ const MNN::Transformer::LlmContext * LlmSession::Response(const std::string &pro history_.emplace_back("user", getUserString(prompt.c_str(), false, is_r1_)); #endif MNN_DEBUG("submitNative history count %zu", history_.size()); - - // Generate full prompt string using Prompt::applyTemplate + + prompt_string_for_debug.clear(); + response_string_for_debug.clear(); + + // Generate a logical prompt preview for debugging. The engine may still use + // prompt cache internally and prefill only the delta; see the PERF log below. std::string full_prompt_text; for (auto & it : history_) { full_prompt_text += it.second; - prompt_string_for_debug += it.second; } - - MNN_DEBUG("submitNative prompt_string_for_debug count %s max_new_tokens_:%d", prompt_string_for_debug.c_str(), max_new_tokens_); - + prompt_string_for_debug = full_prompt_text; + + std::string prompt_preview = full_prompt_text.substr(0, 2048); + MNN_DEBUG("submitNative logical_prompt chars=%zu preview=%s max_new_tokens_:%d", full_prompt_text.size(), + prompt_preview.c_str(), max_new_tokens_); + // Check for multimodal content in the full prompt auto multimodal_result = processMultimodalPrompt(full_prompt_text); restoreAndroidSteppingStatusIfNeeded(llm_); @@ -361,10 +367,12 @@ const MNN::Transformer::LlmContext * LlmSession::Response(const std::string &pro float decode_s = context->decode_us / 1e6f; float prefill_tps = (prefill_s > 0) ? context->prompt_len / prefill_s : 0; float decode_tps = (decode_s > 0) ? context->gen_seq_len / decode_s : 0; - MNN_DEBUG("PERF | prefill: %d tok in %.2fs (%.1f t/s) | decode: %d tok in %.2fs (%.1f t/s) | history: %zu msgs", - context->prompt_len, prefill_s, prefill_tps, - context->gen_seq_len, decode_s, decode_tps, - history_.size()); + bool prompt_cache_enabled = current_config_.value("prompt_cache", false); + MNN_DEBUG( + "PERF | prompt_cache: %d | actual_prefill: %d tok in %.2fs (%.1f t/s) | decode: %d tok in %.2fs (%.1f t/s) | " + "history: %zu msgs", + prompt_cache_enabled ? 1 : 0, context->prompt_len, prefill_s, prefill_tps, context->gen_seq_len, decode_s, + decode_tps, history_.size()); return context; } @@ -403,6 +411,13 @@ void LlmSession::SetMaxNewTokens(int i) { max_new_tokens_ = i; } +int LlmSession::CountInputTokens(const std::string& prompt) const { + if (llm_ == nullptr) { + return 0; + } + return static_cast(llm_->tokenizer_encode(prompt).size()); +} + void LlmSession::setSystemPrompt(std::string system_prompt) { system_prompt_= std::move(system_prompt); if (history_.size() > 1) { diff --git a/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.h b/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.h index d87bebc065..47c02833c1 100644 --- a/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.h +++ b/apps/Android/MnnLlmChat/app/src/main/cpp/llm_session.h @@ -39,6 +39,8 @@ class LlmSession { Response(const std::string &prompt, const std::function &on_progress); void SetMaxNewTokens(int i); + int CountInputTokens(const std::string& prompt) const; + void setSystemPrompt(std::string system_prompt); void SetAssistantPrompt(const std::string& assistant_prompt); diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/MnnLlmApplication.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/MnnLlmApplication.kt index 5694de8633..6493edcb4b 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/MnnLlmApplication.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/MnnLlmApplication.kt @@ -6,6 +6,7 @@ import android.app.Application import com.alibaba.mls.api.ApplicationProvider import com.alibaba.mls.api.download.ModelDownloadManager import com.alibaba.mnnllm.android.update.UpdateChecker +import com.alibaba.mnnllm.android.agent.AgenticPythonEngine import com.alibaba.mnnllm.android.utils.CrashUtil import com.alibaba.mnnllm.android.utils.CurrentActivityTracker import com.alibaba.mnnllm.android.utils.TimberConfig @@ -21,6 +22,7 @@ class MnnLlmApplication : Application() { override fun onCreate() { super.onCreate() ApplicationProvider.set(this) + AgenticPythonEngine.initialize(this) UpdateChecker.registerDownloadReceiver(applicationContext) CrashUtil.init(this) instance = this diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgentWorkspaceFileBrowser.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgentWorkspaceFileBrowser.kt new file mode 100644 index 0000000000..99af74cf70 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgentWorkspaceFileBrowser.kt @@ -0,0 +1,222 @@ +package com.alibaba.mnnllm.android.agent + +import android.content.Context +import android.content.Intent +import android.view.ViewGroup +import android.widget.LinearLayout +import android.widget.ScrollView +import android.widget.TextView +import android.widget.Toast +import androidx.core.content.FileProvider +import com.alibaba.mnnllm.android.R +import com.google.android.material.dialog.MaterialAlertDialogBuilder +import java.io.File +import java.util.Locale + +object AgentWorkspaceFileBrowser { + fun show(context: Context) { + val workspace = AgenticPythonEngine.workspaceDir(context) + showDirectory(context, workspace, workspace) + } + + private fun showDirectory(context: Context, workspace: File, directory: File) { + val children = listDirectoryChildren(directory) + if (children.isEmpty()) { + MaterialAlertDialogBuilder(context) + .setTitle(directoryTitle(workspace, directory)) + .setMessage(R.string.workspace_files_empty) + .setPositiveButton(android.R.string.ok, null) + .show() + return + } + + val list = LinearLayout(context).apply { + orientation = LinearLayout.VERTICAL + setPadding(24, 8, 24, 8) + } + if (directory.canonicalFile != workspace.canonicalFile) { + list.addView(createParentRow(context, workspace, directory.parentFile ?: workspace)) + } + children.forEach { file -> + list.addView(createFileRow(context, workspace, file)) + } + + val scroll = ScrollView(context).apply { + addView( + list, + ViewGroup.LayoutParams( + ViewGroup.LayoutParams.MATCH_PARENT, + ViewGroup.LayoutParams.WRAP_CONTENT + ) + ) + } + + MaterialAlertDialogBuilder(context) + .setTitle(directoryTitle(workspace, directory)) + .setView(scroll) + .setNegativeButton(android.R.string.cancel, null) + .show() + } + + private fun createFileRow(context: Context, workspace: File, file: File): TextView { + val relativePath = runCatching { + file.relativeTo(workspace).path.replace("\\", "/") + }.getOrElse { file.name } + return TextView(context).apply { + text = buildString { + append(if (file.isDirectory) "[Folder] " else "[File] ") + append(relativePath) + append("\n") + append(if (file.isDirectory) "${file.listFiles()?.size ?: 0} items" else formatFileSize(file.length())) + } + textSize = 14f + setPadding(0, 14, 0, 14) + layoutParams = LinearLayout.LayoutParams( + ViewGroup.LayoutParams.MATCH_PARENT, + ViewGroup.LayoutParams.WRAP_CONTENT + ) + setOnClickListener { + if (file.isDirectory) { + showDirectory(context, workspace, file) + } else { + showFile(context, workspace, file) + } + } + } + } + + private fun createParentRow(context: Context, workspace: File, parent: File): TextView { + return TextView(context).apply { + text = "[Folder] .." + textSize = 14f + setPadding(0, 14, 0, 14) + layoutParams = LinearLayout.LayoutParams( + ViewGroup.LayoutParams.MATCH_PARENT, + ViewGroup.LayoutParams.WRAP_CONTENT + ) + setOnClickListener { showDirectory(context, workspace, parent) } + } + } + + private fun listDirectoryChildren(directory: File): List { + if (!directory.exists() || !directory.isDirectory) return emptyList() + return directory.listFiles().orEmpty() + .sortedWith(compareByDescending { it.isDirectory }.thenBy { it.name.lowercase(Locale.US) }) + } + + private fun directoryTitle(workspace: File, directory: File): String { + val relativePath = runCatching { + directory.relativeTo(workspace).path.replace("\\", "/") + }.getOrElse { "" } + return if (relativePath.isBlank() || relativePath == ".") { + "Workspace files" + } else { + relativePath + } + } + + private fun showFile(context: Context, workspace: File, file: File) { + val relativePath = runCatching { + file.relativeTo(workspace).path.replace("\\", "/") + }.getOrElse { file.name } + val message = if (canPreviewText(file.name)) { + readTextPreview(file) + } else { + "No built-in preview for this file type.\n\nPath: $relativePath\nSize: ${formatFileSize(file.length())}" + } + val text = TextView(context).apply { + this.text = message + textSize = 13f + setPadding(24, 8, 24, 8) + } + val scroll = ScrollView(context).apply { + addView( + text, + ViewGroup.LayoutParams( + ViewGroup.LayoutParams.MATCH_PARENT, + ViewGroup.LayoutParams.WRAP_CONTENT + ) + ) + } + MaterialAlertDialogBuilder(context) + .setTitle(relativePath) + .setView(scroll) + .setNegativeButton(android.R.string.cancel, null) + .setPositiveButton("Open with app") { _, _ -> openFileWithExternalApp(context, file) } + .show() + } + + private fun openFileWithExternalApp(context: Context, file: File) { + try { + val uri = FileProvider.getUriForFile( + context, + context.packageName + ".fileprovider", + file + ) + val intent = Intent(Intent.ACTION_VIEW).apply { + setDataAndType(uri, guessMimeType(file.name)) + addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION) + } + context.startActivity(Intent.createChooser(intent, file.name)) + } catch (e: Exception) { + Toast.makeText(context, R.string.workspace_file_open_failed, Toast.LENGTH_SHORT).show() + } + } + + private fun canPreviewText(name: String): Boolean { + val lower = name.lowercase(Locale.US) + return lower.endsWith(".txt") || + lower.endsWith(".md") || + lower.endsWith(".csv") || + lower.endsWith(".json") || + lower.endsWith(".py") || + lower.endsWith(".log") || + lower.endsWith(".xml") || + lower.endsWith(".html") || + lower.endsWith(".htm") + } + + private fun readTextPreview(file: File): String { + return try { + val limit = 128 * 1024 + val buffer = ByteArray(limit + 1) + val count = file.inputStream().use { input -> + input.read(buffer) + } + if (count <= 0) { + return "" + } + val truncated = count > limit + val text = buffer.copyOfRange(0, minOf(count, limit)).toString(Charsets.UTF_8) + if (truncated) { + "$text\n\n... truncated ..." + } else { + text + } + } catch (_: Exception) { + "Failed to read this file." + } + } + + private fun guessMimeType(name: String): String { + val lower = name.lowercase(Locale.US) + return when { + lower.endsWith(".xlsx") -> "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + lower.endsWith(".xls") -> "application/vnd.ms-excel" + lower.endsWith(".csv") -> "text/csv" + lower.endsWith(".pdf") -> "application/pdf" + lower.endsWith(".txt") -> "text/plain" + lower.endsWith(".json") -> "application/json" + lower.endsWith(".png") -> "image/png" + lower.endsWith(".jpg") || lower.endsWith(".jpeg") -> "image/jpeg" + else -> "*/*" + } + } + + private fun formatFileSize(bytes: Long): String { + if (bytes < 1024) return "$bytes B" + val kb = bytes / 1024.0 + if (kb < 1024) return String.format(Locale.US, "%.1f KB", kb) + return String.format(Locale.US, "%.1f MB", kb / 1024.0) + } +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticOutputParser.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticOutputParser.kt new file mode 100644 index 0000000000..d84f4187c5 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticOutputParser.kt @@ -0,0 +1,309 @@ +// Copyright (c) 2024 Alibaba Group Holding Limited All rights reserved. + +package com.alibaba.mnnllm.android.agent + +import com.google.gson.Gson +import com.google.gson.JsonArray +import com.google.gson.JsonElement +import com.google.gson.JsonObject +import com.google.gson.JsonParser +import com.google.gson.JsonSyntaxException + +object AgenticOutputParser { + private val gson = Gson() + private val fencedJsonRegex = Regex("```(?:json)?\\s*([\\s\\S]*?)```", RegexOption.IGNORE_CASE) + private val pythonExecLooseRegex = Regex( + """"type"\s*:\s*"python_exec"[\s\S]*?"code"\s*:\s*"((?:\\.|[^"\\])*)"""", + RegexOption.IGNORE_CASE + ) + private val toolTypes = setOf( + "get_current_time", + "web_search", + "browser_url", + "python_exec", + "run_python", + "python" + ) + + fun parse(text: String): AgenticResponse? { + for (candidate in candidates(text)) { + val response = parseCandidate(candidate) ?: parseLooseToolCandidate(candidate) ?: continue + if (response.systemCalls.isNullOrEmpty() && !response.reply.isNullOrBlank()) { + val nested = parseNestedReply(response.reply) + if (nested != null) { + return nested + } + } + if (response.reply != null || + !response.systemCalls.isNullOrEmpty() || + !response.memoryUpdates.isNullOrEmpty() || + !response.skillUpdates.isNullOrEmpty() + ) { + return response + } + } + return null + } + + fun extractToolCalls(text: String): List { + return parse(text)?.systemCalls.orEmpty() + } + + private fun candidates(text: String): List { + val result = mutableListOf() + result.add(text.trim()) + fencedJsonRegex.findAll(text).forEach { match -> + result.add(match.groupValues[1].trim()) + } + extractFirstJsonObject(text)?.let { result.add(it) } + return result.distinct().filter { it.isNotBlank() } + } + + private fun parseCandidate(candidate: String): AgenticResponse? { + return try { + val element = JsonParser.parseString(candidate) + when { + element.isJsonArray -> parseToolArray(element.asJsonArray) + element.isJsonObject -> { + val jsonObject = element.asJsonObject + parseSingleToolObject(jsonObject) + ?: parseResponseObject(jsonObject) + } + else -> null + } + } catch (_: JsonSyntaxException) { + null + } catch (_: IllegalStateException) { + null + } + } + + private fun parseLooseToolCandidate(candidate: String): AgenticResponse? { + val match = pythonExecLooseRegex.find(candidate) ?: return null + val code = unescapeJsonString(match.groupValues[1]) + if (code.isBlank()) return null + val input = Regex(""""input"\s*:\s*"((?:\\.|[^"\\])*)"""") + .find(candidate) + ?.groupValues + ?.getOrNull(1) + ?.let(::unescapeJsonString) + .orEmpty() + val timeoutMs = Regex(""""timeout_ms"\s*:\s*(\d+)""") + .find(candidate) + ?.groupValues + ?.getOrNull(1) + ?.toLongOrNull() + val outputFiles = Regex(""""(?:output_files|generated_files|expected_outputs|files)"\s*:\s*\[([\s\S]*?)]""") + .find(candidate) + ?.groupValues + ?.getOrNull(1) + ?.let { raw -> + Regex(""""((?:\\.|[^"\\])*)"""").findAll(raw) + .map { unescapeJsonString(it.groupValues[1]) } + .filter { it.isNotBlank() } + .toList() + } + return AgenticResponse( + reply = "", + systemCalls = listOf( + AgentSystemCall( + type = "python_exec", + code = code, + input = input, + timeoutMs = timeoutMs, + outputFiles = outputFiles + ) + ) + ) + } + + private fun unescapeJsonString(value: String): String { + return runCatching { + gson.fromJson("\"$value\"", String::class.java) + }.getOrElse { + value.replace("\\n", "\n") + .replace("\\t", "\t") + .replace("\\\"", "\"") + .replace("\\\\", "\\") + } + } + + private fun parseResponseObject(jsonObject: JsonObject): AgenticResponse? { + return AgenticResponse( + reply = jsonObject.stringValue("reply"), + memoryUpdates = parseMemoryUpdates(jsonObject.get("memory_updates")), + skillUpdates = parseSkillUpdates(jsonObject.get("skill_updates")), + systemCalls = parseSystemCalls(jsonObject.get("system_calls")) + ) + } + + private fun parseSingleToolObject(jsonObject: JsonObject): AgenticResponse? { + if (jsonObject.has("system_calls") || + jsonObject.has("reply") || + jsonObject.has("memory_updates") || + jsonObject.has("skill_updates") + ) { + return null + } + + val explicitType = jsonObject.get("type")?.takeIf { it.isJsonPrimitive }?.asString?.trim() + val inferredType = explicitType?.takeIf { it in toolTypes } ?: inferToolType(jsonObject) + if (inferredType.isNullOrBlank()) { + return null + } + + val normalized = jsonObject.deepCopy() + normalized.addProperty("type", inferredType) + val call = gson.fromJson(normalized, AgentSystemCall::class.java) + return AgenticResponse(reply = "", systemCalls = listOf(call)) + } + + private fun parseToolArray(jsonArray: JsonArray): AgenticResponse? { + val calls = parseToolCallsFromArray(jsonArray) + return calls.takeIf { it.isNotEmpty() }?.let { + AgenticResponse(reply = "", systemCalls = it) + } + } + + private fun parseSystemCalls(element: JsonElement?): List? { + if (element == null || element.isJsonNull) return null + return when { + element.isJsonArray -> parseToolCallsFromArray(element.asJsonArray) + element.isJsonObject -> parseSingleToolObject(element.asJsonObject)?.systemCalls.orEmpty() + element.isJsonPrimitive && element.asJsonPrimitive.isString -> + parse(element.asString)?.systemCalls.orEmpty() + else -> emptyList() + }.takeIf { it.isNotEmpty() } + } + + private fun parseToolCallsFromArray(jsonArray: JsonArray): List { + return jsonArray.flatMap { element -> + when { + element.isJsonObject -> + parseSingleToolObject(element.asJsonObject)?.systemCalls.orEmpty() + element.isJsonArray -> + parseToolCallsFromArray(element.asJsonArray) + element.isJsonPrimitive && element.asJsonPrimitive.isString -> + parse(element.asString)?.systemCalls.orEmpty() + else -> + emptyList() + } + } + } + + private fun parseMemoryUpdates(element: JsonElement?): List? { + val array = element?.takeIf { it.isJsonArray }?.asJsonArray ?: return null + return array.mapNotNull { item -> + when { + item.isJsonPrimitive && item.asJsonPrimitive.isString -> + AgentMemoryUpdate(category = "general", content = item.asString) + item.isJsonObject -> { + val obj = item.asJsonObject + AgentMemoryUpdate( + category = obj.stringValue("category") ?: "general", + content = obj.stringValue("content") ?: obj.stringValue("text") + ) + } + else -> null + } + }.takeIf { it.isNotEmpty() } + } + + private fun parseSkillUpdates(element: JsonElement?): List? { + val array = element?.takeIf { it.isJsonArray }?.asJsonArray ?: return null + return array.mapNotNull { item -> + when { + item.isJsonPrimitive && item.asJsonPrimitive.isString -> + AgentSkillUpdate(description = item.asString) + item.isJsonObject -> { + val obj = item.asJsonObject + AgentSkillUpdate( + name = obj.stringValue("name"), + description = obj.stringValue("description") ?: obj.stringValue("content"), + triggerKeywords = parseStringList(obj.get("trigger_keywords")), + actionTemplate = obj.stringValue("action_template") + ) + } + else -> null + } + }.takeIf { it.isNotEmpty() } + } + + private fun inferToolType(jsonObject: JsonObject): String? { + return when { + jsonObject.has("code") -> "python_exec" + jsonObject.has("url") -> "browser_url" + jsonObject.has("query") -> "web_search" + else -> null + } + } + + private fun parseStringList(element: JsonElement?): List? { + if (element == null || element.isJsonNull) return null + return when { + element.isJsonArray -> element.asJsonArray.mapNotNull { + it.takeIf { value -> value.isJsonPrimitive }?.asString + } + element.isJsonPrimitive && element.asJsonPrimitive.isString -> + element.asString.split(',', ';').map { it.trim() }.filter { it.isNotBlank() } + else -> null + } + } + + private fun JsonObject.stringValue(name: String): String? { + val element = get(name) ?: return null + if (!element.isJsonPrimitive || !element.asJsonPrimitive.isString) return null + return element.asString.takeIf { it.isNotBlank() } + } + + private fun parseNestedReply(reply: String): AgenticResponse? { + for (candidate in candidates(reply)) { + val nested = parseCandidate(candidate) ?: continue + if (nested.reply != null || + !nested.systemCalls.isNullOrEmpty() || + !nested.memoryUpdates.isNullOrEmpty() || + !nested.skillUpdates.isNullOrEmpty() + ) { + return nested + } + } + return null + } + + private fun extractFirstJsonObject(text: String): String? { + val start = text.indexOf('{') + if (start < 0) return null + + var depth = 0 + var inString = false + var escaping = false + + for (i in start until text.length) { + val ch = text[i] + if (escaping) { + escaping = false + continue + } + if (ch == '\\' && inString) { + escaping = true + continue + } + if (ch == '"') { + inString = !inString + continue + } + if (inString) continue + + when (ch) { + '{' -> depth++ + '}' -> { + depth-- + if (depth == 0) { + return text.substring(start, i + 1) + } + } + } + } + return null + } +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticPrompts.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticPrompts.kt new file mode 100644 index 0000000000..e305753919 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticPrompts.kt @@ -0,0 +1,136 @@ +// Copyright (c) 2024 Alibaba Group Holding Limited All rights reserved. + +package com.alibaba.mnnllm.android.agent + +data class AgentToolSpec( + val name: String, + val description: String, + val inputSchema: String +) + +data class AgentLoopBudget( + val maxPasses: Int, + val maxToolCalls: Int, + val maxBrowserCalls: Int, + val maxPythonCalls: Int +) + +object AgenticPrompts { + val defaultLoopBudget = AgentLoopBudget( + maxPasses = 6, + maxToolCalls = 12, + maxBrowserCalls = 8, + maxPythonCalls = 6 + ) + + val supportedTools = listOf( + AgentToolSpec( + name = "get_current_time", + description = "Return the current local date, time, timezone, weekday, and epoch milliseconds.", + inputSchema = """{"type":"get_current_time"}""" + ), + AgentToolSpec( + name = "browser_url", + description = "Open or read a URL through the in-app browser layer and return page text, title, url, and status.", + inputSchema = """{"url":"https://example.com","goal":"what to verify on this page"}""" + ), + AgentToolSpec( + name = "web_search", + description = "Search the web and return candidate result titles, snippets, and URLs for follow-up browser_url calls.", + inputSchema = """{"type":"web_search","query":"search query"}""" + ), + AgentToolSpec( + name = "python_exec", + description = "Run bounded Python 3.11 code in the app sandbox for calculation, data analysis, py_compile checks, script storage, and Excel read/write helpers. Available packages include numpy, pandas, and openpyxl.", + inputSchema = """{"type":"python_exec","code":"print(1+1)","input":"optional text or JSON","timeout_ms":15000,"output_files":[]}""" + ) + ) + + const val outputContract: String = """ +Return either a final user reply or a JSON tool request. Tool requests must be valid JSON: +{ + "reply": "", + "memory_updates": [], + "skill_updates": [], + "system_calls": [ + { + "type": "get_current_time | web_search | browser_url | python_exec", + "query": "for web_search", + "url": "for browser_url", + "code": "for python_exec", + "input": "optional input for python_exec", + "timeout_ms": 15000, + "output_files": ["workspace-relative paths of files this call creates, such as sample.xlsx"] + } + ] +} + +Rules: +- Prefer verifying fresh, factual, or high-impact claims with web_search and browser_url. +- Use browser_url to inspect primary sources, not only search snippets. +- Use get_current_time when exact current time matters. +- Use python_exec for math, statistics, parsing, table-like data processing, py_compile checks, reusable scripts, and Excel work. +- In python_exec, available packages include numpy, pandas, openpyxl, and standard-library modules such as csv, json, statistics, and collections. +- In python_exec, available helpers include read_excel(path), write_excel(filename, sheets), save_script(name, source), load_script(name), list_scripts(), run_script(name), compile_script(name), emit(value), and set_result(value). +- Python code must directly perform the requested action in the submitted script; do not only define functions without calling them. +- Python relative file paths are resolved inside the app's writable agent workspace. +- For creating simple Excel files, prefer write_excel(filename, sheets) over pandas so files are saved inside the app workspace and start quickly. +- When python_exec creates files, fill output_files with every generated workspace-relative filename/path. +- Files created or modified by python_exec are automatically returned to the chat as attachments. +- Do not import network/process/filesystem control modules such as requests, socket, subprocess, os, or sys. Use browser_url/web_search for network access and the provided workspace helpers for files. +- memory_updates are only for durable user preferences/facts. Do not store task status, generated file summaries, examples, apologies, or one-off results. +- skill_updates are only for reusable procedures with a short name, triggers, and action_template. Do not store one-off task descriptions. +- Keep memory/skill updates short: memory content <= 168 chars, skill action_template <= 350 chars. +- Do not expose raw tool JSON to the user in the final answer. +- If a tool fails, try another route when useful, then explain only the user-relevant limitation. +- Stop requesting tools when the current evidence is enough or the app reports that the loop budget is exhausted. +""" + + fun buildIdentityMemory(modelName: String, modelId: String): String { + return buildString { + appendLine("Host: MNN Chat, local on-device MNN LLM runtime.") + appendLine("Model: ${modelName.ifBlank { "unknown" }} (${modelId.ifBlank { "unknown" }}).") + appendLine("Tools are executed by the host app after valid system_calls JSON.") + }.trim() + } + + fun buildSystemPrompt(memoryBlock: String = "", skillBlock: String = ""): String { + return buildString { + appendLine("You are MNN Chat's on-device assistant.") + appendLine("Use host tools via valid system_calls JSON when they improve accuracy or perform requested actions.") + appendLine("For current web facts, search, browse, or URL inspection, request web_search/browser_url instead of saying you lack internet access.") + appendLine("Answer directly when tools are unnecessary.") + appendLine() + appendLine("Available tools:") + supportedTools.forEach { tool -> + appendLine("- ${tool.name}: ${tool.description}") + appendLine(" schema: ${tool.inputSchema}") + } + if (memoryBlock.isNotBlank()) { + appendLine() + appendLine("Relevant memory:") + appendLine(memoryBlock) + } + if (skillBlock.isNotBlank()) { + appendLine() + appendLine("Relevant skills:") + appendLine(skillBlock) + } + appendLine() + appendLine(outputContract.trim()) + } + } + + fun buildSystemPromptForModel( + modelName: String, + modelId: String, + memoryBlock: String = "", + skillBlock: String = "" + ): String { + val identityBlock = buildIdentityMemory(modelName, modelId) + val mergedMemory = listOf(identityBlock, memoryBlock) + .filter { it.isNotBlank() } + .joinToString(separator = "\n\n") + return buildSystemPrompt(memoryBlock = mergedMemory, skillBlock = skillBlock) + } +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticProtocol.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticProtocol.kt new file mode 100644 index 0000000000..66ce7a8273 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticProtocol.kt @@ -0,0 +1,53 @@ +// Copyright (c) 2024 Alibaba Group Holding Limited All rights reserved. + +package com.alibaba.mnnllm.android.agent + +import com.google.gson.annotations.SerializedName + +data class AgenticResponse( + val reply: String? = null, + @SerializedName("memory_updates") val memoryUpdates: List? = null, + @SerializedName("skill_updates") val skillUpdates: List? = null, + @SerializedName("system_calls") val systemCalls: List? = null +) { + fun hasToolCalls(): Boolean = !systemCalls.isNullOrEmpty() + fun visibleReply(): String = reply.orEmpty() +} + +data class AgentSystemCall( + val type: String? = null, + val query: String? = null, + val url: String? = null, + val code: String? = null, + val input: String? = null, + @SerializedName("timeout_ms") val timeoutMs: Long? = null, + @SerializedName("input_files") val inputFiles: List? = null, + @SerializedName("expected_outputs") val expectedOutputs: List? = null, + @SerializedName("output_files") val outputFiles: List? = null, + @SerializedName("generated_files") val generatedFiles: List? = null, + val files: List? = null +) + +data class AgentMemoryUpdate( + val category: String? = null, + val content: String? = null +) + +data class AgentSkillUpdate( + val name: String? = null, + val description: String? = null, + @SerializedName("trigger_keywords") val triggerKeywords: List? = null, + @SerializedName("action_template") val actionTemplate: String? = null +) + +data class AgentToolObservation( + val type: String, + val status: String, + val title: String? = null, + @SerializedName("final_url") val finalUrl: String? = null, + val text: String? = null, + val stdout: String? = null, + val stderr: String? = null, + @SerializedName("output_files") val outputFiles: List? = null, + val error: String? = null +) diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticPythonEngine.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticPythonEngine.kt new file mode 100644 index 0000000000..9b4a50f7d8 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticPythonEngine.kt @@ -0,0 +1,111 @@ +package com.alibaba.mnnllm.android.agent + +import android.content.Context +import android.util.Log +import com.chaquo.python.Python +import com.chaquo.python.android.AndroidPlatform +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.withContext +import kotlinx.serialization.Serializable +import kotlinx.serialization.decodeFromString +import kotlinx.serialization.json.Json +import kotlinx.serialization.json.JsonElement +import java.io.File + +object AgenticPythonEngine { + private const val TAG = "AgenticPythonEngine" + private const val MAX_CODE_CHARS = 12_000 + private const val MAX_INPUT_CHARS = 24_000 + private const val MAX_OUTPUT_CHARS = 24_000 + private const val DEFAULT_TIMEOUT_MS = 15_000L + private const val MAX_TIMEOUT_MS = 30_000L + + private val json = Json { ignoreUnknownKeys = true } + + @Volatile + private var initialized = false + + @Volatile + private var appContext: Context? = null + + fun initialize(context: Context) { + appContext = context.applicationContext + } + + fun workspaceDir(context: Context? = appContext): File { + val base = requireNotNull(context?.applicationContext ?: appContext) { + "Python workspace requires initialized context." + } + return File(base.filesDir, "agent_workspace").apply { mkdirs() } + } + + private fun ensureStarted(): Boolean { + if (initialized) return true + synchronized(this) { + if (initialized) return true + val context = appContext ?: return false + if (!Python.isStarted()) { + Python.start(AndroidPlatform(context)) + } + initialized = true + Log.i(TAG, "python runtime initialized") + } + return true + } + + suspend fun execute(code: String, input: String = "", timeoutMs: Long = DEFAULT_TIMEOUT_MS): AgenticPythonResult { + val trimmedCode = code.take(MAX_CODE_CHARS) + val trimmedInput = input.take(MAX_INPUT_CHARS) + val boundedTimeout = timeoutMs.coerceIn(1_000L, MAX_TIMEOUT_MS) + return withContext(Dispatchers.Default) { + if (!ensureStarted()) { + return@withContext AgenticPythonResult( + ok = false, + error = "Python runtime is not initialized." + ) + } + runCatching { + val module = Python.getInstance().getModule("agent_python") + val raw = module.callAttr( + "run_code", + trimmedCode, + trimmedInput, + boundedTimeout, + workspaceDir().absolutePath + ).toString() + json.decodeFromString(raw) + }.getOrElse { error -> + Log.e(TAG, "python_exec failed: ${error.message}", error) + AgenticPythonResult(ok = false, error = error.message ?: error::class.java.simpleName) + }.trimmed() + } + } + + private fun AgenticPythonResult.trimmed(): AgenticPythonResult { + return copy( + stdout = stdout.take(MAX_OUTPUT_CHARS), + stderr = stderr.take(MAX_OUTPUT_CHARS), + error = error.take(MAX_OUTPUT_CHARS) + ) + } +} + +@Serializable +data class AgenticPythonResult( + val ok: Boolean = false, + val stdout: String = "", + val stderr: String = "", + val error: String = "", + val result: JsonElement? = null, + val files: List = emptyList(), + val elapsed_ms: Int = 0 +) + +@Serializable +data class AgenticPythonFile( + val name: String = "", + val path: String = "", + val relative_path: String = "", + val mime_type: String = "", + val size_bytes: Long = 0L +) diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticToolExecutor.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticToolExecutor.kt new file mode 100644 index 0000000000..39865fb6d4 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticToolExecutor.kt @@ -0,0 +1,531 @@ +// Copyright (c) 2024 Alibaba Group Holding Limited All rights reserved. + +package com.alibaba.mnnllm.android.agent + +import android.util.Base64 +import android.util.Log +import com.alibaba.mnnllm.android.chat.model.ChatFileAttachment +import kotlinx.coroutines.CancellationException +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.currentCoroutineContext +import kotlinx.coroutines.ensureActive +import kotlinx.coroutines.withContext +import kotlinx.coroutines.withTimeoutOrNull +import java.io.BufferedReader +import java.io.File +import java.io.InputStream +import java.io.InputStreamReader +import java.net.HttpURLConnection +import java.net.URL +import java.net.URLDecoder +import java.net.URLEncoder +import java.time.LocalDateTime +import java.time.ZoneId +import java.time.format.DateTimeFormatter +import java.time.format.TextStyle +import java.util.Locale +import java.util.zip.GZIPInputStream + +object AgenticToolExecutor { + private const val TAG = "AgenticToolExecutor" + private const val MAX_SEARCH_RESULTS = 5 + private const val MAX_PAGE_CHARS = 16_000 + private const val USER_AGENT = + "Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0 Mobile Safari/537.36" + private val fileReferenceRegex = Regex( + """(?i)(?:file://)?/?[\p{L}\p{N}_ .\-/\\%]+?\.(?:xlsx|xls|csv|pdf|txt|json|png|jpe?g|py)""" + ) + + data class ToolStepEvent( + val type: Type, + val title: String, + val detail: String = "" + ) { + enum class Type { STARTED, FINISHED, FAILED } + } + + data class ToolExecutionResult( + val observations: String, + val generatedFiles: List = emptyList() + ) + + private data class SearchHit( + val title: String, + val url: String, + val snippet: String + ) + + suspend fun execute( + calls: List, + onStep: suspend (ToolStepEvent) -> Unit = {} + ): ToolExecutionResult { + val results = mutableListOf() + val generatedFiles = mutableListOf() + for (call in calls) { + currentCoroutineContext().ensureActive() + executeOne(call, onStep)?.let { result -> + results.add(result.observation) + generatedFiles.addAll(result.generatedFiles) + } + } + return ToolExecutionResult( + observations = results.joinToString("\n---\n"), + generatedFiles = generatedFiles + ) + } + + fun resolveMentionedWorkspaceFiles(text: String): List { + if (text.isBlank()) return emptyList() + val references = fileReferenceRegex.findAll(text) + .map { it.value } + .toList() + return resolveWorkspaceFileReferences(references) + } + + fun resolveWorkspaceFileReferences(references: List): List { + if (references.isEmpty()) return emptyList() + return references.mapNotNull { resolveWorkspaceFileReference(it) } + .distinctBy { it.path } + } + + private data class SingleToolResult( + val observation: String, + val generatedFiles: List = emptyList() + ) + + private suspend fun executeOne( + call: AgentSystemCall, + onStep: suspend (ToolStepEvent) -> Unit + ): SingleToolResult? { + val title = toolTitle(call) + val detail = toolDetail(call) + onStep(ToolStepEvent(ToolStepEvent.Type.STARTED, title, detail)) + return try { + val result = when (normalizeType(call.type)) { + "get_current_time" -> SingleToolResult(executeGetCurrentTime()) + "web_search" -> SingleToolResult(executeWebSearch(call.query.orEmpty())) + "browser_url", "browse_url", "web_browse", "open_url" -> + SingleToolResult(executeBrowseUrl(call.url.orEmpty().ifBlank { call.query.orEmpty() })) + "python_exec", "run_python", "python" -> executePython(call) + else -> SingleToolResult("[TOOL_ERROR] Unknown system call: ${call.type}") + } + onStep(ToolStepEvent(ToolStepEvent.Type.FINISHED, title, summarizeToolResult(result.observation))) + result + } catch (e: CancellationException) { + throw e + } catch (e: Exception) { + val message = e.message ?: e::class.java.simpleName + onStep(ToolStepEvent(ToolStepEvent.Type.FAILED, title, message)) + SingleToolResult("[TOOL_ERROR] $title failed: $message") + } + } + + private fun toolTitle(call: AgentSystemCall): String { + return when (normalizeType(call.type)) { + "get_current_time" -> "Get current time" + "web_search" -> "Web search" + "browser_url", "browse_url", "web_browse", "open_url" -> "Read web page" + "python_exec", "run_python", "python" -> "Run Python" + else -> "Run tool" + } + } + + private fun toolDetail(call: AgentSystemCall): String { + return when (normalizeType(call.type)) { + "web_search" -> call.query.orEmpty() + "browser_url", "browse_url", "web_browse", "open_url" -> call.url.orEmpty().ifBlank { call.query.orEmpty() } + "python_exec", "run_python", "python" -> + call.code.orEmpty().lineSequence().firstOrNull()?.trim().orEmpty().ifBlank { "python_exec" } + else -> call.type.orEmpty() + }.take(240) + } + + private fun summarizeToolResult(result: String): String { + return when { + result.contains("[SEARCH_RESULT]") -> "Search completed, ${result.length} chars" + result.contains("[BROWSE_RESULT]") -> "Page read, ${result.length} chars" + result.contains("[PYTHON_RESULT]") -> "Python completed, ${result.length} chars" + result.contains("[PYTHON_ERROR]") -> "Python failed" + result.contains("[TIME_RESULT]") -> "Time read" + result.contains("[TOOL_ERROR]") || + result.contains("[SEARCH_ERROR]") || + result.contains("[BROWSE_ERROR]") -> "Tool failed" + else -> "Done, ${result.length} chars" + } + } + + private fun normalizeType(type: String?): String { + return type.orEmpty().trim().lowercase(Locale.US) + } + + private fun executeGetCurrentTime(): String { + val zone = ZoneId.systemDefault() + val now = LocalDateTime.now(zone) + val formatted = now.format(DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss")) + val weekday = now.dayOfWeek.getDisplayName(TextStyle.FULL, Locale.getDefault()) + val epochMs = now.atZone(zone).toInstant().toEpochMilli() + return """ + [TIME_RESULT] + datetime: $formatted + weekday: $weekday + timezone: ${zone.id} + epoch_ms: $epochMs + """.trimIndent() + } + + private suspend fun executePython(call: AgentSystemCall): SingleToolResult { + val code = call.code.orEmpty().ifBlank { call.query.orEmpty() } + if (code.isBlank()) return SingleToolResult("[PYTHON_ERROR] Empty Python code.") + val result = AgenticPythonEngine.execute( + code = code, + input = call.input.orEmpty(), + timeoutMs = normalizePythonTimeout(code, call.timeoutMs) + ) + val files = result.files.mapNotNull { file -> + if (file.path.isBlank()) { + null + } else { + ChatFileAttachment( + name = file.name.ifBlank { java.io.File(file.path).name }, + path = file.path, + mimeType = file.mime_type, + sizeBytes = file.size_bytes + ) + } + } + resolveDeclaredOutputFiles(call) + val observation = buildString { + appendLine(if (result.ok) "[PYTHON_RESULT]" else "[PYTHON_ERROR]") + appendLine("elapsed_ms: ${result.elapsed_ms}") + if (result.stdout.isNotBlank()) { + appendLine("stdout:") + appendLine(result.stdout.trimEnd()) + } + if (result.result != null) { + appendLine("result:") + appendLine(result.result.toString()) + } + if (result.stderr.isNotBlank()) { + appendLine("stderr:") + appendLine(result.stderr.trimEnd()) + } + if (result.error.isNotBlank()) { + appendLine("error:") + appendLine(result.error.trimEnd()) + } + if (files.isNotEmpty()) { + appendLine("files:") + files.forEach { file -> + appendLine("- ${file.name} | ${file.path} | ${file.mimeType} | ${file.sizeBytes} bytes") + } + } + }.trim() + return SingleToolResult(observation, files) + } + + private fun resolveDeclaredOutputFiles(call: AgentSystemCall): List { + val references = buildList { + addAll(call.outputFiles.orEmpty()) + addAll(call.generatedFiles.orEmpty()) + addAll(call.expectedOutputs.orEmpty()) + addAll(call.files.orEmpty()) + } + return resolveWorkspaceFileReferences(references) + } + + private fun resolveWorkspaceFileReference(rawReference: String): ChatFileAttachment? { + val workspace = runCatching { AgenticPythonEngine.workspaceDir().canonicalFile }.getOrNull() + ?: return null + val decoded = runCatching { URLDecoder.decode(rawReference.trim(), "UTF-8") } + .getOrElse { rawReference.trim() } + .trim('"', '\'', '`', ' ', '\n', '\r', '\t', '(', ')', '[', ']', '<', '>') + if (decoded.isBlank()) return null + + val withoutScheme = decoded + .removePrefix("file://localhost") + .removePrefix("file://") + .replace('\\', '/') + + val candidates = mutableListOf() + val direct = File(withoutScheme) + if (direct.isAbsolute) { + candidates += direct + candidates += File(workspace, withoutScheme.trimStart('/')) + } else { + candidates += File(workspace, withoutScheme) + } + candidates += File(workspace, File(withoutScheme).name) + + val found = candidates.asSequence() + .mapNotNull { candidate -> + runCatching { candidate.canonicalFile }.getOrNull() + } + .firstOrNull { candidate -> + candidate.isFile && candidate.path.startsWith(workspace.path + File.separator) + } ?: findWorkspaceFileByName(workspace, File(withoutScheme).name) + ?: return null + + return ChatFileAttachment( + name = found.name, + path = found.absolutePath, + mimeType = guessMimeType(found.name), + sizeBytes = found.length() + ) + } + + private fun findWorkspaceFileByName(workspace: File, name: String): File? { + if (name.isBlank() || !workspace.exists()) return null + return workspace.walkTopDown() + .onEnter { dir -> dir.name != "__pycache__" && dir.name != "python" } + .filter { it.isFile && it.name == name } + .maxByOrNull { it.lastModified() } + ?.canonicalFile + ?.takeIf { it.path.startsWith(workspace.path + File.separator) } + } + + private fun normalizePythonTimeout(code: String, requestedTimeoutMs: Long?): Long { + val requested = requestedTimeoutMs ?: 15_000L + val needsDataPackageWarmup = listOf( + "import pandas", + "from pandas", + "import numpy", + "from numpy", + "import openpyxl", + "from openpyxl", + "read_excel(", + "write_excel(", + ".to_excel(" + ).any { code.contains(it) } + val minimum = if (needsDataPackageWarmup) 15_000L else 1_000L + return requested.coerceAtLeast(minimum).coerceAtMost(30_000L) + } + + private fun guessMimeType(name: String): String { + val lower = name.lowercase(Locale.US) + return when { + lower.endsWith(".xlsx") -> "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + lower.endsWith(".xls") -> "application/vnd.ms-excel" + lower.endsWith(".csv") -> "text/csv" + lower.endsWith(".pdf") -> "application/pdf" + lower.endsWith(".txt") -> "text/plain" + lower.endsWith(".json") -> "application/json" + lower.endsWith(".png") -> "image/png" + lower.endsWith(".jpg") || lower.endsWith(".jpeg") -> "image/jpeg" + lower.endsWith(".py") -> "text/x-python" + else -> "application/octet-stream" + } + } + + private suspend fun executeWebSearch(query: String): String = withContext(Dispatchers.IO) { + if (query.isBlank()) return@withContext "[SEARCH_ERROR] Empty query." + val result = withTimeoutOrNull(12_000L) { + searchBing(query) + } + if (result.isNullOrEmpty()) { + "[SEARCH_ERROR] No search results were returned." + } else { + buildString { + appendLine("[SEARCH_RESULT]") + appendLine("query: $query") + appendLine("source: Bing") + result.take(MAX_SEARCH_RESULTS).forEachIndexed { index, hit -> + appendLine("${index + 1}. ${hit.title}") + appendLine(" url: ${hit.url}") + if (hit.snippet.isNotBlank()) appendLine(" snippet: ${hit.snippet}") + } + }.trim() + } + } + + private fun searchBing(query: String): List { + val encoded = URLEncoder.encode(query, "UTF-8") + val url = URL("https://www.bing.com/search?q=$encoded&form=QBRE&pq=$encoded&qs=n&sp=-1&lq=0") + Log.d(TAG, "BING-REQ: $url") + val conn = openHttp(url) + val html = try { + readResponseBody(conn) + } finally { + conn.disconnect() + } + val results = parseBingHtml(html) + Log.d(TAG, "BING-RESULTS: ${results.size}") + return results + } + + private suspend fun executeBrowseUrl(rawUrl: String): String = withContext(Dispatchers.IO) { + val normalized = normalizeHttpUrl(rawUrl) + ?: return@withContext "[BROWSE_ERROR] Invalid URL; only http/https pages are supported." + val text = withTimeoutOrNull(15_000L) { + fetchUrlText(normalized) + } + if (text.isNullOrBlank()) { + "[BROWSE_ERROR] Page content could not be read." + } else { + "[BROWSE_RESULT]\n$text" + } + } + + private fun fetchUrlText(url: String): String { + val parsed = URL(url) + val conn = openHttp(parsed) + val html = try { + readResponseBody(conn) + } finally { + conn.disconnect() + } + val title = extractHtmlTitle(html) + val text = htmlToReadableText(html) + return buildString { + appendLine("final_url: $url") + if (title.isNotBlank()) appendLine("title: $title") + appendLine("text:") + append(text.take(MAX_PAGE_CHARS)) + } + } + + private fun openHttp(url: URL): HttpURLConnection { + return (url.openConnection() as HttpURLConnection).apply { + connectTimeout = 8_000 + readTimeout = 10_000 + instanceFollowRedirects = true + setRequestProperty("User-Agent", USER_AGENT) + setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") + setRequestProperty("Accept-Language", "zh-CN,zh;q=0.9,en;q=0.8") + setRequestProperty("Accept-Encoding", "gzip") + setRequestProperty("Cache-Control", "no-cache") + } + } + + private fun readResponseBody(conn: HttpURLConnection): String { + val stream = runCatching { + if (conn.responseCode >= 400) conn.errorStream else conn.inputStream + }.getOrNull() ?: return "" + val input = if (conn.contentEncoding.equals("gzip", ignoreCase = true)) { + GZIPInputStream(stream) + } else { + stream + } + return input.use { readText(it, responseCharset(conn.contentType)) } + } + + private fun readText(input: InputStream, charset: String): String { + return BufferedReader(InputStreamReader(input, charset)).use { it.readText() } + } + + private fun responseCharset(contentType: String?): String { + val match = Regex("charset=([^;]+)", RegexOption.IGNORE_CASE).find(contentType.orEmpty()) + return match?.groupValues?.getOrNull(1)?.trim('"', '\'', ' ')?.ifBlank { null } ?: "UTF-8" + } + + private fun parseBingHtml(html: String): List { + val results = mutableListOf() + val itemRegex = Regex("]*class=\"[^\"]*b_algo[^\"]*\"[\\s\\S]*?", RegexOption.IGNORE_CASE) + for (item in itemRegex.findAll(html)) { + if (results.size >= MAX_SEARCH_RESULTS) break + val block = item.value + val link = Regex("]*>[\\s\\S]*?]*href=\"([^\"]+)\"[^>]*>([\\s\\S]*?)", RegexOption.IGNORE_CASE) + .find(block) + val url = normalizeBingHref(link?.groupValues?.getOrNull(1).orEmpty()) + val title = stripAll(link?.groupValues?.getOrNull(2).orEmpty()) + val snippet = Regex("]*>([\\s\\S]*?)

", RegexOption.IGNORE_CASE) + .find(block) + ?.groupValues + ?.getOrNull(1) + ?.let(::stripAll) + .orEmpty() + if (title.isNotBlank() && url.isNotBlank()) { + results.add(SearchHit(title, url, snippet)) + } + } + return results + } + + private fun normalizeBingHref(raw: String): String { + val decoded = htmlDecode(raw) + return if (decoded.contains("/ck/a?", ignoreCase = true) && decoded.contains("u=")) { + val u = Regex("[?&]u=([^&]+)").find(decoded)?.groupValues?.getOrNull(1) + if (u.isNullOrBlank()) decoded else decodeBingRedirectUrl(u) ?: decoded + } else { + decoded + } + } + + private fun decodeBingRedirectUrl(raw: String): String? { + val decoded = runCatching { URLDecoder.decode(raw, "UTF-8") }.getOrNull() ?: return null + if (decoded.startsWith("http://", ignoreCase = true) || decoded.startsWith("https://", ignoreCase = true)) { + return decoded + } + if (decoded.startsWith("a1") && decoded.length > 2) { + return runCatching { + String( + Base64.decode(decoded.substring(2), Base64.URL_SAFE or Base64.NO_PADDING or Base64.NO_WRAP), + Charsets.UTF_8 + ) + }.getOrNull()?.takeIf { + it.startsWith("http://", ignoreCase = true) || it.startsWith("https://", ignoreCase = true) + } + } + return null + } + + private fun normalizeHttpUrl(rawUrl: String): String? { + val trimmed = rawUrl.trim() + if (trimmed.isBlank()) return null + val withScheme = if ( + trimmed.startsWith("http://", ignoreCase = true) || + trimmed.startsWith("https://", ignoreCase = true) + ) { + trimmed + } else { + "https://$trimmed" + } + val parsed = runCatching { URL(withScheme) }.getOrNull() ?: return null + val protocol = parsed.protocol.lowercase(Locale.US) + if (protocol != "http" && protocol != "https") return null + if (parsed.host.isNullOrBlank()) return null + return parsed.toString() + } + + private fun extractHtmlTitle(html: String): String { + return Regex("]*>([\\s\\S]*?)", RegexOption.IGNORE_CASE) + .find(html) + ?.groupValues + ?.getOrNull(1) + ?.let(::stripAll) + .orEmpty() + } + + private fun htmlToReadableText(html: String): String { + val body = Regex("]*>([\\s\\S]*?)", RegexOption.IGNORE_CASE) + .find(html) + ?.groupValues + ?.getOrNull(1) + ?: html + val cleaned = body + .replace(Regex("", RegexOption.IGNORE_CASE), " ") + .replace(Regex("", RegexOption.IGNORE_CASE), " ") + .replace(Regex("", RegexOption.IGNORE_CASE), " ") + .replace(Regex("", RegexOption.IGNORE_CASE), "\n") + .replace(Regex("", RegexOption.IGNORE_CASE), "\n") + return stripAll(cleaned) + } + + private fun stripAll(html: String?): String { + if (html.isNullOrBlank()) return "" + return html + .replace(Regex("<[^>]+>"), " ") + .let(::htmlDecode) + .replace(Regex("[\\t\\x0B\\f\\r ]+"), " ") + .replace(Regex("\\n\\s*\\n+"), "\n") + .trim() + } + + private fun htmlDecode(text: String): String { + return text + .replace("&", "&") + .replace("<", "<") + .replace(">", ">") + .replace(""", "\"") + .replace("'", "'") + .replace(" ", " ") + } +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatActivity.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatActivity.kt index 09582cece8..667a30fbd7 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatActivity.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatActivity.kt @@ -31,6 +31,9 @@ import com.alibaba.mnnllm.android.chat.chatlist.ChatListComponent import com.alibaba.mnnllm.android.chat.chatlist.ChatViewHolders import com.alibaba.mnnllm.android.chat.input.ChatInputComponent import com.alibaba.mnnllm.android.chat.model.ChatDataItem +import com.alibaba.mnnllm.android.chat.model.ChatFileAttachment +import com.alibaba.mnnllm.android.chat.model.ChatDataManager +import com.alibaba.mnnllm.android.chat.model.ChatDatabaseHelper import com.alibaba.mnnllm.android.databinding.ActivityChatBinding import com.alibaba.mnnllm.android.llm.AudioDataListener import com.alibaba.mnnllm.android.llm.LlmSession @@ -50,6 +53,7 @@ import com.alibaba.mnnllm.android.chat.voice.VoiceModelMarketBottomSheet import com.alibaba.mnnllm.android.modelist.ModelItemWrapper import com.alibaba.mnnllm.android.utils.CrashReportContext import com.alibaba.mnnllm.android.utils.ConfigInfoDialog +import com.google.android.material.dialog.MaterialAlertDialogBuilder import kotlinx.coroutines.MainScope import kotlinx.coroutines.flow.MutableStateFlow import kotlinx.coroutines.flow.filter @@ -96,6 +100,7 @@ class ChatActivity : AppCompatActivity() { private lateinit var chatPresenter: ChatPresenter private var chatInputModule: ChatInputComponent? = null lateinit var chatListComponent: ChatListComponent + private var agentModeEnabled: Boolean = false // Real-time audio playback settings private var isRealTimePlayback = true @@ -137,6 +142,19 @@ class ChatActivity : AppCompatActivity() { startMockStream() return } + if (shouldPromptInitialSessionMode()) { + showNewSessionModeDialog( + onSelected = { agentEnabled -> + intent.putExtra(ChatRouter.EXTRA_AGENT_MODE, agentEnabled) + setupSession() + initializeVoiceModelsChecker() + }, + onCancel = { + finish() + } + ) + return + } this.setupSession() initializeVoiceModelsChecker() } @@ -163,6 +181,7 @@ class ChatActivity : AppCompatActivity() { } chatPresenter = ChatPresenter(this, modelName, modelId) + chatPresenter.setAgentEnabled(agentModeEnabled) setChatPresenter(chatPresenter) chatInputModule = ChatInputComponent(this, binding, modelId, modelName) setupChatListComponent() @@ -210,12 +229,26 @@ class ChatActivity : AppCompatActivity() { private fun setupSession() { chatSession = chatPresenter.createSession() sessionId = chatSession!!.sessionId + val restoredMode = intent.getStringExtra("chatSessionId")?.let { + ChatDataManager.getInstance(this).getSessionMode(it) + } ?: if (intent.getBooleanExtra(ChatRouter.EXTRA_AGENT_MODE, false)) { + ChatDatabaseHelper.SESSION_MODE_AGENT + } else { + ChatDatabaseHelper.SESSION_MODE_NORMAL + } + agentModeEnabled = restoredMode == ChatDatabaseHelper.SESSION_MODE_AGENT + chatPresenter.setAgentEnabled(agentModeEnabled) CrashReportContext.setCurrentModel(modelId, sessionId) onSessionCreated() Log.d(TAG, "current SessionId: $sessionId") chatPresenter.load() } + private fun shouldPromptInitialSessionMode(): Boolean { + val isNewSession = intent.getStringExtra("chatSessionId").isNullOrEmpty() + return isNewSession && !isDiffusion && !isMockStreamSession && !intent.hasExtra(ChatRouter.EXTRA_AGENT_MODE) + } + private fun shouldStartMockStream(): Boolean { if (!BuildConfig.DEBUG) { return false @@ -598,18 +631,60 @@ class ChatActivity : AppCompatActivity() { private fun handleNewSession() { - if (!isGenerating) { - currentUserMessage = null - if (chatListComponent.reset()) { - Toast.makeText(this, R.string.new_conversation_started, Toast.LENGTH_LONG).show() + if (isGenerating) { + Toast.makeText(this, "Cannot Create New Session when generating", Toast.LENGTH_LONG).show() + return + } + if (isDiffusion) { + startNewSession(agentEnabled = false) + return + } + showNewSessionModeDialog( + onSelected = { agentEnabled: Boolean -> + startNewSession(agentEnabled) } - this.sessionName = null - chatPresenter.reset{newSessionId -> - sessionId = newSessionId - CrashReportContext.setCurrentModel(modelId, sessionId) + ) + } + + private fun showNewSessionModeDialog( + onSelected: (Boolean) -> Unit, + onCancel: (() -> Unit)? = null + ) { + MaterialAlertDialogBuilder(this) + .setTitle(R.string.new_conversation_mode_title) + .setMessage(R.string.new_conversation_mode_message) + .setNegativeButton(R.string.new_conversation_normal_mode) { _, _ -> + onSelected(false) + } + .setPositiveButton(R.string.new_conversation_agent_mode) { _, _ -> + onSelected(true) + } + .setOnCancelListener { + onCancel?.invoke() } + .show() + } + + private fun setCurrentSessionMode(agentEnabled: Boolean, showToast: Boolean = true) { + agentModeEnabled = agentEnabled + chatPresenter.setAgentEnabled(agentEnabled) + if (!showToast) return + val messageRes = if (agentEnabled) { + R.string.new_agent_conversation_started } else { - Toast.makeText(this, "Cannot Create New Session when generating", Toast.LENGTH_LONG).show() + R.string.new_conversation_started + } + Toast.makeText(this, messageRes, Toast.LENGTH_SHORT).show() + } + + private fun startNewSession(agentEnabled: Boolean) { + currentUserMessage = null + chatListComponent.reset() + this.sessionName = null + chatPresenter.reset { newSessionId -> + sessionId = newSessionId + setCurrentSessionMode(agentEnabled) + CrashReportContext.setCurrentModel(modelId, sessionId) } } @@ -734,6 +809,15 @@ class ChatActivity : AppCompatActivity() { chatListComponent.updateAssistantResponse(chatDataItem) } + fun onAgentStatus(status: String) { + val chatDataItem = chatListComponent.recentItem ?: return + chatDataItem.thinkingText = "" + chatDataItem.displayText = status + chatDataItem.text = status + chatDataItem.loading = true + chatListComponent.updateAssistantResponse(chatDataItem) + } + fun onDiffusionGenerateProgress(progress: String?, diffusionDestPath: String?) { val chatDataItem = chatListComponent.recentItem if (chatDataItem == null) { @@ -799,10 +883,23 @@ class ChatActivity : AppCompatActivity() { } else { // Normal success case - set response if available val response = benchMarkResult["response"] as? String - if (!response.isNullOrEmpty() && recentItem.text.isNullOrEmpty()) { + val shouldReplaceDisplay = benchMarkResult["replace_display"] as? Boolean == true + if (!response.isNullOrEmpty() && (recentItem.text.isNullOrEmpty() || shouldReplaceDisplay)) { recentItem.text = response recentItem.displayText = response } + if (shouldReplaceDisplay) { + val agentSteps = benchMarkResult["agent_steps"] as? String + if (!agentSteps.isNullOrBlank()) { + recentItem.thinkingText = agentSteps + recentItem.thinkingFinishedTime = 0L + } + } + @Suppress("UNCHECKED_CAST") + val generatedFiles = benchMarkResult["generated_files"] as? List + if (!generatedFiles.isNullOrEmpty()) { + recentItem.generatedFiles = generatedFiles + } } recentItem.benchmarkInfo = ModelUtils.generateBenchMarkString(benchMarkResult) @@ -945,6 +1042,7 @@ class ChatActivity : AppCompatActivity() { this.sessionName = null chatPresenter.reset { newSessionId -> sessionId = newSessionId + setCurrentSessionMode(false, showToast = false) CrashReportContext.setCurrentModel(modelId, sessionId) // Create voice chat fragment with the new session val voiceChatFragment = VoiceChatFragment.newInstance(modelName, modelId!!, chatPresenter) diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatPresenter.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatPresenter.kt index ab51818560..b152dadc13 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatPresenter.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatPresenter.kt @@ -11,7 +11,12 @@ import com.alibaba.mnnllm.android.llm.ChatService import com.alibaba.mnnllm.android.llm.ChatSession import com.alibaba.mnnllm.api.openai.di.ServiceLocator import com.alibaba.mnnllm.api.openai.manager.ServerEventManager +import com.alibaba.mnnllm.android.agent.AgenticPrompts +import com.alibaba.mnnllm.android.agent.AgenticOutputParser +import com.alibaba.mnnllm.android.agent.AgenticToolExecutor +import com.alibaba.mnnllm.android.agent.AgentSystemCall import com.alibaba.mnnllm.android.chat.model.ChatDataItem +import com.alibaba.mnnllm.android.chat.model.ChatFileAttachment import com.alibaba.mnnllm.android.chat.model.ChatDataManager import com.alibaba.mnnllm.android.chat.chatlist.ChatViewHolders import com.alibaba.mnnllm.android.llm.GenerateProgressListener @@ -25,6 +30,8 @@ import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.SupervisorJob import kotlinx.coroutines.async import kotlinx.coroutines.cancel +import kotlinx.coroutines.currentCoroutineContext +import kotlinx.coroutines.ensureActive import kotlinx.coroutines.launch import java.text.DateFormat import java.util.Random @@ -46,6 +53,7 @@ class ChatPresenter( private val presenterScope = CoroutineScope(SupervisorJob() + Dispatchers.IO) private var generateListener:GenerateListener? = null private val additionalListeners = mutableListOf() + private var agentEnabled: Boolean = false /** * Get LLM session instance @@ -67,6 +75,25 @@ class ChatPresenter( fun getSessionId(): String? { return sessionId } + + fun setAgentEnabled(enabled: Boolean) { + agentEnabled = enabled + sessionId?.let { + chatDataManager?.updateSessionMode( + it, + if (enabled) { + com.alibaba.mnnllm.android.chat.model.ChatDatabaseHelper.SESSION_MODE_AGENT + } else { + com.alibaba.mnnllm.android.chat.model.ChatDatabaseHelper.SESSION_MODE_NORMAL + } + ) + } + applySystemPromptForCurrentMode() + } + + fun isAgentEnabled(): Boolean { + return agentEnabled + } /** * Add an additional generate listener for multi-UI updates @@ -153,6 +180,7 @@ class ChatPresenter( } else { chatSession.load() } + applySystemPromptForCurrentMode() chatActivity.lifecycleScope.launch { chatActivity.onLoadingChanged(false) } @@ -191,8 +219,29 @@ class ChatPresenter( } } + private fun applySystemPromptForCurrentMode() { + val llmSession = getLlmSession() ?: return + val currentModelId = chatActivity.modelId ?: modelId + val currentModelName = chatActivity.modelName.ifBlank { modelName } + val systemPrompt = if (agentEnabled) { + AgenticPrompts.buildSystemPromptForModel( + modelName = currentModelName, + modelId = currentModelId, + memoryBlock = chatDataManager?.buildAgentMemoryBlock().orEmpty(), + skillBlock = chatDataManager?.buildAgentSkillBlock().orEmpty() + ) + } else { + ModelConfig.loadConfig(currentModelId)?.systemPrompt + ?: ModelConfig.defaultConfig.systemPrompt + ?: "You are a helpful assistant." + } + llmSession.updateSystemPrompt(systemPrompt) + Log.d(TAG, "Agent mode ${if (agentEnabled) "enabled" else "disabled"} for model=$currentModelId") + } + private fun submitDiffusionRequest(input: String, userData: ChatDataItem): HashMap { - val prompt = resolveDiffusionPrompt(input, modelId) + val currentModelId = chatActivity.modelId ?: modelId + val prompt = resolveDiffusionPrompt(input, currentModelId) val diffusionDestPath = FileUtils.generateDestDiffusionFilePath( chatActivity, sessionId!! @@ -201,7 +250,7 @@ class ChatPresenter( FileUtils.getPathForUri(it) } ?: "" - val config = ModelConfig.loadConfig(modelId) + val config = ModelConfig.loadConfig(currentModelId) val steps = config?.diffusionSteps ?: ModelConfig.defaultConfig.diffusionSteps ?: 20 val seed = if (config?.diffusionSeed != null && config.diffusionSeed!! != -1L) { config.diffusionSeed!!.toInt() @@ -231,16 +280,18 @@ class ChatPresenter( ) } - private fun submitLlmRequest(prompt:String): HashMap { + private fun submitLlmRequest(prompt:String, emitProgress: Boolean = true): HashMap { val generateResultProcessor = GenerateResultProcessor() generateResultProcessor.generateBegin() val result = chatSession.generate(prompt, mapOf(), object: GenerateProgressListener { override fun onProgress(progress: String?): Boolean { generateResultProcessor.process(progress) - chatActivity.lifecycleScope.launch { - this@ChatPresenter.generateListener?.onLlmGenerateProgress(progress, generateResultProcessor) - additionalListeners.forEach { it.onLlmGenerateProgress(progress, generateResultProcessor) } + if (emitProgress) { + chatActivity.lifecycleScope.launch { + this@ChatPresenter.generateListener?.onLlmGenerateProgress(progress, generateResultProcessor) + additionalListeners.forEach { it.onLlmGenerateProgress(progress, generateResultProcessor) } + } } if (stopGenerating) { Log.d(TAG, "stopGenerating requested") @@ -252,14 +303,210 @@ class ChatPresenter( return result } - private fun submitRequest(input: String, userData: ChatDataItem): HashMap { + private suspend fun submitAgenticLlmRequest(input: String): HashMap { + val statusLog = StringBuilder() + val searchedQueries = mutableSetOf() + val visitedUrls = mutableSetOf() + val executedPythonKeys = mutableSetOf() + var totalToolCalls = 0 + var totalBrowserCalls = 0 + var totalPythonCalls = 0 + val aggregateMetrics = HashMap() + val generatedFiles = mutableListOf() + val budget = AgenticPrompts.defaultLoopBudget + + fun appendStatus(line: String) { + statusLog.appendLine(line) + emitAgentStatus(statusLog.toString().trimEnd()) + } + + fun ensureNotStopped() { + if (stopGenerating) { + throw kotlinx.coroutines.CancellationException("Agent generation stopped") + } + } + + appendStatus("[Agent] 规划中...") + ensureNotStopped() + applySystemPromptForCurrentMode() + var llmResult = submitLlmRequest(input, emitProgress = false) + mergeLlmMetrics(aggregateMetrics, llmResult) + var raw = llmResult["response"] as? String ?: "" + var parsed = AgenticOutputParser.parse(raw) + + for (pass in 1..budget.maxPasses) { + currentCoroutineContext().ensureActive() + ensureNotStopped() + val calls = parsed?.systemCalls.orEmpty() + if (calls.isEmpty()) break + + appendStatus("[Agent] 第 $pass 轮计划：${calls.size} 个工具调用") + val executableCalls = mutableListOf() + for (call in calls) { + val type = call.type.orEmpty().trim().lowercase() + val isSearch = type == "web_search" + val isBrowse = type == "browser_url" || type == "browse_url" || type == "web_browse" || type == "open_url" + val isPython = type == "python_exec" || type == "run_python" || type == "python" + val key = when { + isSearch -> call.query.orEmpty().trim().lowercase() + isBrowse -> call.url.orEmpty().ifBlank { call.query.orEmpty() }.trim().lowercase() + isPython -> type + ":" + call.code.orEmpty().take(500) + ":" + call.input.orEmpty().take(500) + else -> type + } + val duplicate = (isSearch && key in searchedQueries) || + (isBrowse && key in visitedUrls) || + (isPython && key in executedPythonKeys) + val budgetBlocked = totalToolCalls >= budget.maxToolCalls || + (isBrowse && totalBrowserCalls >= budget.maxBrowserCalls) || + (isPython && totalPythonCalls >= budget.maxPythonCalls) + when { + duplicate -> appendStatus("[Agent] 跳过重复调用：$key") + budgetBlocked -> appendStatus("[Agent] 跳过工具调用：预算已用尽") + else -> { + if (isSearch) searchedQueries += key + if (isBrowse) { + visitedUrls += key + totalBrowserCalls += 1 + } + if (isPython) { + executedPythonKeys += key + totalPythonCalls += 1 + } + totalToolCalls += 1 + executableCalls += call + } + } + } + + if (executableCalls.isEmpty()) break + + ensureNotStopped() + val toolExecutionResult = AgenticToolExecutor.execute(executableCalls) { event -> + when (event.type) { + AgenticToolExecutor.ToolStepEvent.Type.STARTED -> + appendStatus("[Agent] ${event.title}：${event.detail}") + AgenticToolExecutor.ToolStepEvent.Type.FINISHED -> + appendStatus("[Agent] ${event.title}完成：${event.detail}") + AgenticToolExecutor.ToolStepEvent.Type.FAILED -> + appendStatus("[Agent] ${event.title}失败：${event.detail}") + } + } + generatedFiles.addAll(toolExecutionResult.generatedFiles) + + val remaining = budget.maxToolCalls - totalToolCalls + appendStatus("[Agent] 已获得工具结果，继续推理...") + ensureNotStopped() + val continuationInput = buildString { + appendLine("system_calls 执行结果：") + appendLine(toolExecutionResult.observations) + appendLine() + if (remaining > 0) { + appendLine("Tool budget: remaining=$remaining. Continue from the existing conversation context and KV cache. Do not restate or re-send the original user question unless needed for the final answer. You may continue calling get_current_time, web_search, browser_url, or python_exec if useful. Prefer primary sources, use Python for calculation/data work, and avoid duplicate queries, URLs, or code. Return final reply when the answer is sufficiently supported.") + } else { + appendLine("Tool budget: remaining=0. Do not call more tools. Continue from the existing conversation context and return the best final reply from the available results.") + } + } + llmResult = submitLlmRequest(continuationInput, emitProgress = false) + mergeLlmMetrics(aggregateMetrics, llmResult) + raw = llmResult["response"] as? String ?: "" + parsed = AgenticOutputParser.parse(raw) + } + + if (parsed?.hasToolCalls() == true && parsed?.reply.isNullOrBlank()) { + appendStatus("[Agent] 工具预算结束，整理最终回答...") + ensureNotStopped() + val finalOnlyPrompt = buildString { + appendLine("Tool budget is exhausted. Do not call tools again. Continue from the existing conversation context and KV cache. Return the best final user-facing reply based on the observations already in this conversation. Do not expose JSON.") + } + llmResult = submitLlmRequest(finalOnlyPrompt, emitProgress = false) + mergeLlmMetrics(aggregateMetrics, llmResult) + raw = llmResult["response"] as? String ?: "" + parsed = AgenticOutputParser.parse(raw) + } + + persistAgentUpdates(parsed) + val localSkillHints = chatDataManager?.runLocalAgentSkills(input).orEmpty() + val baseReply = parsed?.reply?.takeIf { it.isNotBlank() } ?: raw + val finalReply = if (localSkillHints.isBlank()) { + baseReply + } else { + "$baseReply\n\n$localSkillHints" + } + generatedFiles.addAll(AgenticToolExecutor.resolveMentionedWorkspaceFiles(finalReply)) + applySystemPromptForCurrentMode() + return HashMap().apply { + putAll(aggregateMetrics) + put("response", finalReply) + put("replace_display", true) + put("agent_steps", statusLog.toString().trimEnd()) + if (generatedFiles.isNotEmpty()) { + put("generated_files", generatedFiles.distinctBy { it.path }) + } + } + } + + private fun mergeLlmMetrics(target: HashMap, source: HashMap) { + listOf("input_len", "prompt_len", "decode_len", "prefill_time", "decode_time", "vision_time", "audio_time").forEach { key -> + val sourceValue = metricAsLong(source[key]) + if (sourceValue > 0L) { + target[key] = metricAsLong(target[key]) + sourceValue + } + } + } + + private fun metricAsLong(value: Any?): Long { + return when (value) { + is Long -> value + is Int -> value.toLong() + is Number -> value.toLong() + is String -> value.toLongOrNull() ?: 0L + else -> 0L + } + } + + private fun persistAgentUpdates(parsed: com.alibaba.mnnllm.android.agent.AgenticResponse?) { + val manager = chatDataManager ?: return + parsed?.memoryUpdates.orEmpty().forEach { update -> + manager.upsertAgentMemory(update.category, update.content, source = "agent") + } + parsed?.skillUpdates.orEmpty().forEach { update -> + manager.upsertAgentSkill( + name = update.name, + description = update.description, + triggerKeywords = update.triggerKeywords, + actionTemplate = update.actionTemplate + ) + } + } + + private fun emitAgentStatus(status: String) { + chatActivity.lifecycleScope.launch { + this@ChatPresenter.generateListener?.onAgentStatus(status) + additionalListeners.forEach { it.onAgentStatus(status) } + } + } + + private suspend fun submitRequest(input: String, userData: ChatDataItem): HashMap { stopGenerating = false val benchMarkResult = try { - if (ModelTypeUtils.isDiffusionModel(this.modelName)) { + val currentModelName = chatActivity.modelName.ifBlank { modelName } + if (ModelTypeUtils.isDiffusionModel(currentModelName)) { submitDiffusionRequest(input, userData) + } else if (agentEnabled && getLlmSession() != null) { + submitAgenticLlmRequest(input) } else { submitLlmRequest(input) } + } catch (e: kotlinx.coroutines.CancellationException) { + if (!stopGenerating) { + throw e + } + Log.d(TAG, "Agent generation cancelled", e) + HashMap().apply { + put("error", true) + put("message", "已停止") + put("response", "已停止") + } } catch (e: Exception) { Log.e(TAG, "Error during generation request", e) // Create a basic error result to ensure onGenerateFinished is called @@ -278,7 +525,15 @@ class ChatPresenter( } private fun updateSession(sessionId: String, modelId: String?, sessionName: String) { - chatDataManager!!.addOrUpdateSession(sessionId, modelId) + chatDataManager!!.addOrUpdateSession( + sessionId, + modelId, + if (agentEnabled) { + com.alibaba.mnnllm.android.chat.model.ChatDatabaseHelper.SESSION_MODE_AGENT + } else { + com.alibaba.mnnllm.android.chat.model.ChatDatabaseHelper.SESSION_MODE_NORMAL + } + ) chatDataManager!!.updateSessionName(this.sessionId!!, this.sessionName) } @@ -292,7 +547,7 @@ class ChatPresenter( try { if (this.sessionName.isNullOrEmpty()) { this.sessionName = SessionUtils.generateSessionName(userData) - updateSession(sessionId!!, modelId, sessionName!!) + updateSession(sessionId!!, chatActivity.modelId ?: modelId, sessionName!!) } // Always save user input to database first @@ -369,6 +624,12 @@ class ChatPresenter( chatActivity.onGenerateFinished(benchMarkResult) } } + + override fun onAgentStatus(status: String) { + chatActivity.lifecycleScope.launch { + chatActivity.onAgentStatus(status) + } + } } /** @@ -454,6 +715,7 @@ class ChatPresenter( !(newSession as com.alibaba.mnnllm.android.llm.LlmSession).isModelLoaded()) { newSession.load() } + applySystemPromptForCurrentMode() chatActivity.lifecycleScope.launch { onSwitchComplete(newSession) chatActivity.onLoadingChanged(false) @@ -538,5 +800,6 @@ class ChatPresenter( fun onGenerateStart() fun onGenerateFinished(benchMarkResult: HashMap) fun onLlmGenerateProgress(progress: String?, generateResultProcessor: GenerateResultProcessor) + fun onAgentStatus(status: String) {} } } diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatRouter.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatRouter.kt index 5ded7baa4b..ae245fd9d6 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatRouter.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/ChatRouter.kt @@ -120,12 +120,20 @@ class ProgressDialog( object ChatRouter { + const val EXTRA_AGENT_MODE = "agentMode" + internal fun resolveDiffusionDir(configFilePath: String): String { val path = File(configFilePath) return if (path.isDirectory) configFilePath else (path.parent ?: configFilePath) } - fun startRun(context: Context, modelIdParam: String, destModelDir:String?, sessionId: String?) { + fun startRun( + context: Context, + modelIdParam: String, + destModelDir:String?, + sessionId: String?, + agentMode: Boolean? = null + ) { Log.d(TAG, "startRun modelIdParam: $modelIdParam destModelDir: $destModelDir sessionId: $sessionId") val isDiffusion = ModelTypeUtils.isDiffusionModel(modelIdParam) var modelId:String? = modelIdParam @@ -144,7 +152,7 @@ object ChatRouter { if (ModelTypeUtils.isQnnModel(modelId)) { Log.d(TAG, "QNN model detected: $modelId") if (QnnModule.deviceSupported()) { - checkAndDownloadQnnLibs(context, modelId, destModelDir, sessionId, isDiffusion) + checkAndDownloadQnnLibs(context, modelId, destModelDir, sessionId, isDiffusion, agentMode) } else { Log.w(TAG, "QNN model detected but device does not support QNN acceleration") Toast.makeText(context, context.getString(R.string.qnn_device_not_supported), Toast.LENGTH_LONG).show() @@ -153,10 +161,17 @@ object ChatRouter { } // Continue with normal flow for non-QNN models - proceedToStartChat(context, modelId, destModelDir, sessionId, isDiffusion) + proceedToStartChat(context, modelId, destModelDir, sessionId, isDiffusion, agentMode) } - private fun checkAndDownloadQnnLibs(context: Context, modelId: String, destModelDir: String?, sessionId: String?, isDiffusion: Boolean) { + private fun checkAndDownloadQnnLibs( + context: Context, + modelId: String, + destModelDir: String?, + sessionId: String?, + isDiffusion: Boolean, + agentMode: Boolean? + ) { // Check if QNN libs are already copied MainScope().launch { try { @@ -174,7 +189,7 @@ object ChatRouter { if (loadSuccess) { Log.d(TAG, "QNN libraries loaded successfully, proceeding to start chat") - proceedToStartChat(context, modelId, destModelDir, sessionId, isDiffusion) + proceedToStartChat(context, modelId, destModelDir, sessionId, isDiffusion, agentMode) } else { Log.e(TAG, "Failed to load QNN libraries") Toast.makeText(context, context.getString(R.string.qnn_libs_load_failed), Toast.LENGTH_LONG).show() @@ -183,7 +198,7 @@ object ChatRouter { } // Show confirmation dialog - showQnnDownloadConfirmationDialog(context, modelId, destModelDir, sessionId, isDiffusion) + showQnnDownloadConfirmationDialog(context, modelId, destModelDir, sessionId, isDiffusion, agentMode) } catch (e: Exception) { Log.e(TAG, "Error checking QNN libs status", e) @@ -192,12 +207,19 @@ object ChatRouter { } } - private fun showQnnDownloadConfirmationDialog(context: Context, modelId: String, destModelDir: String?, sessionId: String?, isDiffusion: Boolean) { + private fun showQnnDownloadConfirmationDialog( + context: Context, + modelId: String, + destModelDir: String?, + sessionId: String?, + isDiffusion: Boolean, + agentMode: Boolean? + ) { val dialog = MaterialAlertDialogBuilder(context) .setTitle(context.getString(R.string.qnn_libs_download_title)) .setMessage(context.getString(R.string.qnn_libs_download_message)) .setPositiveButton(context.getString(R.string.download)) { _, _ -> - downloadQnnLibsAndStartChat(context, modelId, destModelDir, sessionId, isDiffusion) + downloadQnnLibsAndStartChat(context, modelId, destModelDir, sessionId, isDiffusion, agentMode) } .setNegativeButton(context.getString(R.string.cancel)) { dialog, _ -> dialog.dismiss() @@ -236,7 +258,14 @@ object ChatRouter { return dialog } - private fun downloadQnnLibsAndStartChat(context: Context, modelId: String, destModelDir: String?, sessionId: String?, isDiffusion: Boolean) { + private fun downloadQnnLibsAndStartChat( + context: Context, + modelId: String, + destModelDir: String?, + sessionId: String?, + isDiffusion: Boolean, + agentMode: Boolean? + ) { val progressDialog = ProgressDialog( context = context, modelId = null // We'll set this up in the coroutine @@ -269,7 +298,7 @@ object ChatRouter { if (loadSuccess) { Log.d(TAG, "QNN libraries loaded successfully, starting chat") - proceedToStartChat(context, modelId, destModelDir, sessionId, isDiffusion) + proceedToStartChat(context, modelId, destModelDir, sessionId, isDiffusion, agentMode) } else { Log.e(TAG, "Failed to load QNN libraries") Toast.makeText(context, context.getString(R.string.qnn_libs_load_failed), Toast.LENGTH_LONG).show() @@ -287,7 +316,14 @@ object ChatRouter { } } - private fun proceedToStartChat(context: Context, modelId: String, destModelDir: String?, sessionId: String?, isDiffusion: Boolean) { + private fun proceedToStartChat( + context: Context, + modelId: String, + destModelDir: String?, + sessionId: String?, + isDiffusion: Boolean, + agentMode: Boolean? + ) { val downloadManager = ModelDownloadManager.getInstance(context) if (isStopDownloadOnChatEnabled(context)) { downloadManager.pauseAllDownloads() @@ -305,6 +341,7 @@ object ChatRouter { Log.d(TAG, "isDiffusion: ${isDiffusion}, configFilePath: $configFilePath") val intent = Intent(context, ChatActivity::class.java) intent.putExtra("chatSessionId", sessionId) + agentMode?.let { intent.putExtra(EXTRA_AGENT_MODE, it) } if (isDiffusion) { // For diffusion models, pass the directory path, not the config file path val diffusionDir = resolveDiffusionDir(configFilePath) diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/chatlist/ChatViewHolders.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/chatlist/ChatViewHolders.kt index 68372ea611..ed4abef44a 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/chatlist/ChatViewHolders.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/chatlist/ChatViewHolders.kt @@ -21,6 +21,7 @@ import com.alibaba.mnnllm.android.R import com.alibaba.mnnllm.android.chat.ChatActivity import com.alibaba.mnnllm.android.chat.PromptUtils import com.alibaba.mnnllm.android.chat.model.ChatDataItem +import com.alibaba.mnnllm.android.chat.model.ChatFileAttachment import com.alibaba.mnnllm.android.chat.SelectTextActivity import com.alibaba.mnnllm.android.chat.chatlist.VideoPlayerComponent import com.alibaba.mnnllm.android.utils.ClipboardUtils @@ -37,6 +38,9 @@ import io.noties.markwon.ext.latex.JLatexMathTheme import io.noties.markwon.ext.tables.TablePlugin import io.noties.markwon.inlineparser.MarkwonInlineParserPlugin import com.google.android.material.dialog.MaterialAlertDialogBuilder +import com.google.android.material.color.MaterialColors +import androidx.core.content.FileProvider +import java.io.File import java.util.Locale object ChatViewHolders { @@ -188,6 +192,8 @@ object ChatViewHolders { private val imageGenerated: ImageView = view.findViewById(R.id.image_generated) + private val generatedFilesLayout: LinearLayout = + view.findViewById(R.id.ll_generated_files) // Action buttons private val actionButtonsLayout: LinearLayout = view.findViewById(R.id.ll_action_buttons) @@ -333,6 +339,7 @@ object ChatViewHolders { imageGenerated.setImageURI(data.imageUri) } shareImageButton.visibility = if (data.imageUri != null) View.VISIBLE else View.GONE + bindGeneratedFiles(data) return } @@ -360,6 +367,7 @@ object ChatViewHolders { if (data.imageUri != null) { imageGenerated.setImageURI(data.imageUri) } + bindGeneratedFiles(data) val drawableId = ModelUtils.getDrawableId(modelName) headerIcon.setImageResource(if (drawableId > 0) drawableId else R.drawable.ic_launcher) imageGenerated.tag = data @@ -374,6 +382,82 @@ object ChatViewHolders { replayAudioButton.tag = data shareImageButton.tag = data } + + private fun bindGeneratedFiles(data: ChatDataItem) { + val files = data.generatedFiles.orEmpty().filter { it.path.isNotBlank() } + generatedFilesLayout.removeAllViews() + generatedFilesLayout.visibility = if (files.isEmpty()) View.GONE else View.VISIBLE + files.forEach { file -> + val item = TextView(itemView.context).apply { + text = buildString { + append("[File] ") + append(file.name.ifBlank { File(file.path).name }) + if (file.sizeBytes > 0) { + append(" (") + append(formatFileSize(file.sizeBytes)) + append(")") + } + } + textSize = 14f + setTextColor(MaterialColors.getColor(itemView, com.google.android.material.R.attr.colorPrimary)) + setPadding(16, 10, 16, 10) + maxLines = 2 + ellipsize = TextUtils.TruncateAt.MIDDLE + setOnClickListener { openGeneratedFile(file) } + } + val params = LinearLayout.LayoutParams( + LinearLayout.LayoutParams.WRAP_CONTENT, + LinearLayout.LayoutParams.WRAP_CONTENT + ).apply { + topMargin = 6 + } + generatedFilesLayout.addView(item, params) + } + } + + private fun openGeneratedFile(file: ChatFileAttachment) { + try { + val localFile = File(file.path) + if (!localFile.exists()) { + Toast.makeText(itemView.context, "File not found", Toast.LENGTH_SHORT).show() + return + } + val uri = FileProvider.getUriForFile( + itemView.context, + itemView.context.packageName + ".fileprovider", + localFile + ) + val mimeType = file.mimeType.ifBlank { guessMimeType(localFile.name) } + val intent = Intent(Intent.ACTION_VIEW).apply { + setDataAndType(uri, mimeType) + addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION) + } + itemView.context.startActivity(Intent.createChooser(intent, localFile.name)) + } catch (e: Exception) { + Log.e(TAG, "Failed to open generated file: ${file.path}", e) + Toast.makeText(itemView.context, "No app can open this file", Toast.LENGTH_SHORT).show() + } + } + + private fun guessMimeType(name: String): String { + val lower = name.lowercase(Locale.US) + return when { + lower.endsWith(".xlsx") -> "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + lower.endsWith(".xls") -> "application/vnd.ms-excel" + lower.endsWith(".csv") -> "text/csv" + lower.endsWith(".pdf") -> "application/pdf" + lower.endsWith(".txt") -> "text/plain" + lower.endsWith(".json") -> "application/json" + else -> "*/*" + } + } + + private fun formatFileSize(bytes: Long): String { + if (bytes < 1024) return "$bytes B" + val kb = bytes / 1024.0 + if (kb < 1024) return String.format(Locale.US, "%.1f KB", kb) + return String.format(Locale.US, "%.1f MB", kb / 1024.0) + } private fun updateThinkingView(data: ChatDataItem, context: android.content.Context) { val showThinking = data.showThinking diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/AgentMemoryItem.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/AgentMemoryItem.kt new file mode 100644 index 0000000000..852594bde5 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/AgentMemoryItem.kt @@ -0,0 +1,9 @@ +package com.alibaba.mnnllm.android.chat.model + +data class AgentMemoryItem( + val id: Long = 0L, + val category: String, + val content: String, + val source: String = "agent", + val updatedAt: Long = System.currentTimeMillis() +) diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/AgentSkillItem.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/AgentSkillItem.kt new file mode 100644 index 0000000000..2ff7e81c03 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/AgentSkillItem.kt @@ -0,0 +1,11 @@ +package com.alibaba.mnnllm.android.chat.model + +data class AgentSkillItem( + val id: Long = 0L, + val name: String, + val description: String, + val triggerKeywords: String, + val actionTemplate: String, + val enabled: Boolean = true, + val createdAt: Long = System.currentTimeMillis() +) diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDataItem.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDataItem.kt index d0d6932ec6..530e302364 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDataItem.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDataItem.kt @@ -7,6 +7,13 @@ import com.alibaba.mnnllm.android.chat.chatlist.AudioPlayerComponent import com.alibaba.mnnllm.android.chat.chatlist.ChatViewHolders import java.io.File +data class ChatFileAttachment( + var name: String = "", + var path: String = "", + var mimeType: String = "", + var sizeBytes: Long = 0L +) + class ChatDataItem { var loading: Boolean = false var forceShowLoadingWithText: Boolean = false @@ -37,6 +44,8 @@ class ChatDataItem { @JvmField var benchmarkInfo: String? = null + var generatedFiles: List? = null + var displayText: String? = null get() = field?:"" diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDataManager.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDataManager.kt index b1e91df757..5f5be54066 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDataManager.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDataManager.kt @@ -15,11 +15,17 @@ class ChatDataManager private constructor(context: Context) { init { ensureLastChatTimeColumn() + ensureSessionModeColumn() + ensureAgentTables() fixAllMissingLastChatTimes() } - fun addOrUpdateSession(sessionId: String, modelId: String?) { - Log.d(TAG, "addOrUpdateSession: sessionId: $sessionId modelId: $modelId") + fun addOrUpdateSession( + sessionId: String, + modelId: String?, + sessionMode: String? = null + ) { + Log.d(TAG, "addOrUpdateSession: sessionId: $sessionId modelId: $modelId sessionMode: $sessionMode") val db = dbHelper.writableDatabase val cursor = db.query( @@ -35,6 +41,9 @@ class ChatDataManager private constructor(context: Context) { val values = ContentValues() values.put(ChatDatabaseHelper.COLUMN_SESSION_ID, sessionId) values.put(ChatDatabaseHelper.COLUMN_MODEL_ID, modelId) + if (!sessionMode.isNullOrBlank()) { + values.put(ChatDatabaseHelper.COLUMN_SESSION_MODE, normalizeSessionMode(sessionMode)) + } if (exists) { // Don't update lastChatTime when just updating session info @@ -46,11 +55,249 @@ class ChatDataManager private constructor(context: Context) { } else { // Set initial lastChatTime to 0 for new sessions values.put(ChatDatabaseHelper.COLUMN_LAST_CHAT_TIME, 0L) + if (sessionMode.isNullOrBlank()) { + values.put(ChatDatabaseHelper.COLUMN_SESSION_MODE, ChatDatabaseHelper.SESSION_MODE_NORMAL) + } db.insert(ChatDatabaseHelper.TABLE_SESSION, null, values) } db.close() } + fun updateSessionMode(sessionId: String, sessionMode: String) { + val db = dbHelper.writableDatabase + val values = ContentValues() + values.put(ChatDatabaseHelper.COLUMN_SESSION_MODE, normalizeSessionMode(sessionMode)) + db.update( + ChatDatabaseHelper.TABLE_SESSION, + values, + ChatDatabaseHelper.COLUMN_SESSION_ID + "=?", + arrayOf(sessionId) + ) + db.close() + } + + @SuppressLint("Range") + fun getSessionMode(sessionId: String): String { + val db = dbHelper.readableDatabase + return try { + val cursor = db.query( + ChatDatabaseHelper.TABLE_SESSION, + arrayOf(ChatDatabaseHelper.COLUMN_SESSION_MODE), + ChatDatabaseHelper.COLUMN_SESSION_ID + "=?", + arrayOf(sessionId), null, null, null + ) + val mode = if (cursor.moveToFirst()) { + cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_SESSION_MODE)) + } else { + null + } + cursor.close() + normalizeSessionMode(mode) + } catch (e: Exception) { + ChatDatabaseHelper.SESSION_MODE_NORMAL + } finally { + db.close() + } + } + + @SuppressLint("Range") + fun getAgentMemories(limit: Int = 40): List { + val list = mutableListOf() + val db = dbHelper.readableDatabase + try { + val cursor = db.query( + ChatDatabaseHelper.TABLE_AGENT_MEMORY, + null, + null, + null, + null, + null, + ChatDatabaseHelper.COLUMN_AGENT_MEMORY_UPDATED_AT + " DESC", + limit.coerceAtLeast(1).toString() + ) + while (cursor.moveToNext()) { + list += AgentMemoryItem( + id = cursor.getLong(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_MEMORY_ID)), + category = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_MEMORY_CATEGORY)).orEmpty(), + content = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_MEMORY_CONTENT)).orEmpty(), + source = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_MEMORY_SOURCE)).orEmpty(), + updatedAt = cursor.getLong(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_MEMORY_UPDATED_AT)) + ) + } + cursor.close() + } catch (e: Exception) { + Log.e(TAG, "getAgentMemories failed", e) + } finally { + db.close() + } + return list + } + + @SuppressLint("Range") + fun getEnabledAgentSkills(): List { + val list = mutableListOf() + val db = dbHelper.readableDatabase + try { + val cursor = db.query( + ChatDatabaseHelper.TABLE_AGENT_SKILL, + null, + ChatDatabaseHelper.COLUMN_AGENT_SKILL_ENABLED + "=1", + null, + null, + null, + ChatDatabaseHelper.COLUMN_AGENT_SKILL_CREATED_AT + " DESC" + ) + while (cursor.moveToNext()) { + list += AgentSkillItem( + id = cursor.getLong(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_SKILL_ID)), + name = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_SKILL_NAME)).orEmpty(), + description = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_SKILL_DESCRIPTION)).orEmpty(), + triggerKeywords = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_SKILL_TRIGGER_KEYWORDS)).orEmpty(), + actionTemplate = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_SKILL_ACTION_TEMPLATE)).orEmpty(), + enabled = cursor.getInt(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_SKILL_ENABLED)) == 1, + createdAt = cursor.getLong(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_AGENT_SKILL_CREATED_AT)) + ) + } + cursor.close() + } catch (e: Exception) { + Log.e(TAG, "getEnabledAgentSkills failed", e) + } finally { + db.close() + } + return list + } + + fun upsertAgentMemory(category: String?, content: String?, source: String = "agent") { + val safeCategory = compactAgentText(category, 48) + val safeContent = compactAgentText(content, MEMORY_CONTENT_MAX_CHARS) + if (safeCategory.isBlank() || safeContent.isBlank()) return + if (!isUsefulAgentMemory(safeCategory, safeContent)) return + val db = dbHelper.writableDatabase + try { + val values = ContentValues().apply { + put(ChatDatabaseHelper.COLUMN_AGENT_MEMORY_CATEGORY, safeCategory) + put(ChatDatabaseHelper.COLUMN_AGENT_MEMORY_CONTENT, safeContent) + put(ChatDatabaseHelper.COLUMN_AGENT_MEMORY_SOURCE, source) + put(ChatDatabaseHelper.COLUMN_AGENT_MEMORY_UPDATED_AT, System.currentTimeMillis()) + } + db.insert(ChatDatabaseHelper.TABLE_AGENT_MEMORY, null, values) + } catch (e: Exception) { + Log.e(TAG, "upsertAgentMemory failed", e) + } finally { + db.close() + } + } + + fun upsertAgentSkill( + name: String?, + description: String?, + triggerKeywords: List?, + actionTemplate: String? + ) { + val safeName = compactAgentText(name, 64) + val safeTemplate = compactAgentText(actionTemplate, SKILL_ACTION_MAX_CHARS) + if (safeName.isBlank() || safeTemplate.isBlank()) return + if (!isUsefulAgentSkill(safeName, safeTemplate)) return + val db = dbHelper.writableDatabase + try { + val values = ContentValues().apply { + put(ChatDatabaseHelper.COLUMN_AGENT_SKILL_NAME, safeName) + put(ChatDatabaseHelper.COLUMN_AGENT_SKILL_DESCRIPTION, compactAgentText(description, MEMORY_CONTENT_MAX_CHARS)) + put(ChatDatabaseHelper.COLUMN_AGENT_SKILL_TRIGGER_KEYWORDS, JSONArray(triggerKeywords.orEmpty().map { compactAgentText(it, 32) }.filter { it.isNotBlank() }).toString()) + put(ChatDatabaseHelper.COLUMN_AGENT_SKILL_ACTION_TEMPLATE, safeTemplate) + put(ChatDatabaseHelper.COLUMN_AGENT_SKILL_ENABLED, 1) + put(ChatDatabaseHelper.COLUMN_AGENT_SKILL_CREATED_AT, System.currentTimeMillis()) + } + db.replace(ChatDatabaseHelper.TABLE_AGENT_SKILL, null, values) + } catch (e: Exception) { + Log.e(TAG, "upsertAgentSkill failed", e) + } finally { + db.close() + } + } + + fun buildAgentMemoryBlock(limit: Int = 20): String { + return getAgentMemories(limit) + .filter { it.content.isNotBlank() } + .filter { isUsefulAgentMemory(it.category, it.content) } + .joinToString("\n") { + "- ${compactAgentText(it.category, 48)}: ${compactAgentText(it.content, MEMORY_CONTENT_MAX_CHARS)}" + } + } + + fun buildAgentSkillBlock(): String { + return getEnabledAgentSkills() + .filter { isUsefulAgentSkill(it.name, it.actionTemplate) } + .joinToString("\n\n") { skill -> + buildString { + appendLine("- ${compactAgentText(skill.name, 64)}: ${compactAgentText(skill.description, MEMORY_CONTENT_MAX_CHARS)}") + appendLine(" triggers: ${compactAgentText(skill.triggerKeywords, MEMORY_CONTENT_MAX_CHARS)}") + append(" action: ${compactAgentText(skill.actionTemplate, SKILL_ACTION_MAX_CHARS)}") + } + } + } + + fun runLocalAgentSkills(userInput: String): String { + val normalizedInput = userInput.lowercase() + val outputs = mutableListOf() + for (skill in getEnabledAgentSkills()) { + val triggers = parseTriggerKeywords(skill.triggerKeywords) + val matched = triggers.any { keyword -> + keyword.isNotBlank() && normalizedInput.contains(keyword.lowercase()) + } + if (matched) { + outputs += "【${skill.name}】${skill.actionTemplate}" + } + } + return outputs.joinToString("\n") + } + + private fun compactAgentText(value: String?, maxChars: Int): String { + val compacted = value.orEmpty() + .replace(Regex("\\s+"), " ") + .trim() + if (compacted.length <= maxChars) return compacted + return compacted.take(maxChars).trimEnd() + "..." + } + + private fun isUsefulAgentMemory(category: String?, content: String?): Boolean { + val text = "${category.orEmpty()} ${content.orEmpty()}".trim() + if (text.length < 8) return false + if (looksLikeTaskNoise(text)) return false + val lower = text.lowercase() + return listOf( + "用户", "偏好", "喜欢", "不喜欢", "希望", "默认", "记住", "姓名", "学校", "课程", "专业", + "preference", "prefer", "user", "remember", "default", "likes", "dislikes" + ).any { lower.contains(it.lowercase()) } + } + + private fun isUsefulAgentSkill(name: String?, actionTemplate: String?): Boolean { + val text = "${name.orEmpty()} ${actionTemplate.orEmpty()}".trim() + if (text.length < 12) return false + if (looksLikeTaskNoise(text)) return false + return actionTemplate.orEmpty().contains( + Regex("搜索|浏览|读取|分析|生成|保存|检查|search|browse|read|analy[sz]e|generate|save|check", RegexOption.IGNORE_CASE) + ) + } + + private fun looksLikeTaskNoise(text: String): Boolean { + val lower = text.lowercase() + return listOf( + "已准备", "已经准备", "已经修正", "成功创建", "用于展示", "希望这个例子", "随时告诉我", + "现在我已经", "我将为您", "请稍等", "示例数据", "sample_data", "sample.xlsx", + "prepared", "successfully created", "example data", "ready to", "feel free" + ).any { lower.contains(it.lowercase()) } + } + + private fun parseTriggerKeywords(raw: String): List { + return runCatching { + val array = JSONArray(raw) + (0 until array.length()).map { array.optString(it) } + }.getOrElse { + raw.split(',', '，', ';', '；').map { it.trim() } + }.filter { it.isNotBlank() } + } + fun addChatData(sessionId: String?, chatDataItem: ChatDataItem) { if (sessionId.isNullOrEmpty()) { Log.e(TAG, "addChatData: sessionId is null or empty") @@ -252,7 +499,8 @@ class ChatDataManager private constructor(context: Context) { ChatDatabaseHelper.COLUMN_SESSION_ID, ChatDatabaseHelper.COLUMN_MODEL_ID, ChatDatabaseHelper.COLUMN_SESSION_NAME, - ChatDatabaseHelper.COLUMN_LAST_CHAT_TIME + ChatDatabaseHelper.COLUMN_LAST_CHAT_TIME, + ChatDatabaseHelper.COLUMN_SESSION_MODE ), null, null, null, null, ChatDatabaseHelper.COLUMN_LAST_CHAT_TIME + " DESC" ) @@ -282,7 +530,12 @@ class ChatDataManager private constructor(context: Context) { } catch (e: Exception) { 0L // Fallback for when column doesn't exist } - list.add(SessionItem(sid, mid, name, lastChatTime)) + val sessionMode = try { + cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_SESSION_MODE)) + } catch (e: Exception) { + ChatDatabaseHelper.SESSION_MODE_NORMAL + } + list.add(SessionItem(sid, mid, name, lastChatTime, normalizeSessionMode(sessionMode))) } cursor.close() } @@ -452,7 +705,8 @@ class ChatDataManager private constructor(context: Context) { arrayOf( ChatDatabaseHelper.COLUMN_SESSION_ID, ChatDatabaseHelper.COLUMN_MODEL_ID, - ChatDatabaseHelper.COLUMN_SESSION_NAME + ChatDatabaseHelper.COLUMN_SESSION_NAME, + ChatDatabaseHelper.COLUMN_SESSION_MODE ), ChatDatabaseHelper.COLUMN_MODEL_ID + "=?", arrayOf(modelId), null, null, @@ -463,7 +717,12 @@ class ChatDataManager private constructor(context: Context) { val sid = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_SESSION_ID)) val mid = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_MODEL_ID)) val name = cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_SESSION_NAME)) - list.add(SessionItem(sid, mid, name)) + val sessionMode = try { + cursor.getString(cursor.getColumnIndex(ChatDatabaseHelper.COLUMN_SESSION_MODE)) + } catch (e: Exception) { + ChatDatabaseHelper.SESSION_MODE_NORMAL + } + list.add(SessionItem(sid, mid, name, sessionMode = normalizeSessionMode(sessionMode))) } cursor.close() db.close() @@ -576,6 +835,58 @@ class ChatDataManager private constructor(context: Context) { } } + private fun ensureSessionModeColumn() { + val db = dbHelper.writableDatabase + try { + db.rawQuery("SELECT ${ChatDatabaseHelper.COLUMN_SESSION_MODE} FROM ${ChatDatabaseHelper.TABLE_SESSION} LIMIT 1", null)?.close() + } catch (e: Exception) { + try { + db.execSQL( + "ALTER TABLE ${ChatDatabaseHelper.TABLE_SESSION} " + + "ADD COLUMN ${ChatDatabaseHelper.COLUMN_SESSION_MODE} TEXT DEFAULT '${ChatDatabaseHelper.SESSION_MODE_NORMAL}'" + ) + } catch (e2: Exception) { + // Ignore if already exists + } + } finally { + db.close() + } + } + + private fun ensureAgentTables() { + val db = dbHelper.writableDatabase + try { + db.execSQL( + "CREATE TABLE IF NOT EXISTS ${ChatDatabaseHelper.TABLE_AGENT_MEMORY} (" + + "${ChatDatabaseHelper.COLUMN_AGENT_MEMORY_ID} INTEGER PRIMARY KEY AUTOINCREMENT, " + + "${ChatDatabaseHelper.COLUMN_AGENT_MEMORY_CATEGORY} TEXT, " + + "${ChatDatabaseHelper.COLUMN_AGENT_MEMORY_CONTENT} TEXT, " + + "${ChatDatabaseHelper.COLUMN_AGENT_MEMORY_SOURCE} TEXT DEFAULT 'agent', " + + "${ChatDatabaseHelper.COLUMN_AGENT_MEMORY_UPDATED_AT} INTEGER)" + ) + db.execSQL( + "CREATE TABLE IF NOT EXISTS ${ChatDatabaseHelper.TABLE_AGENT_SKILL} (" + + "${ChatDatabaseHelper.COLUMN_AGENT_SKILL_ID} INTEGER PRIMARY KEY AUTOINCREMENT, " + + "${ChatDatabaseHelper.COLUMN_AGENT_SKILL_NAME} TEXT UNIQUE, " + + "${ChatDatabaseHelper.COLUMN_AGENT_SKILL_DESCRIPTION} TEXT, " + + "${ChatDatabaseHelper.COLUMN_AGENT_SKILL_TRIGGER_KEYWORDS} TEXT, " + + "${ChatDatabaseHelper.COLUMN_AGENT_SKILL_ACTION_TEMPLATE} TEXT, " + + "${ChatDatabaseHelper.COLUMN_AGENT_SKILL_ENABLED} INTEGER DEFAULT 1, " + + "${ChatDatabaseHelper.COLUMN_AGENT_SKILL_CREATED_AT} INTEGER)" + ) + } finally { + db.close() + } + } + + private fun normalizeSessionMode(sessionMode: String?): String { + return if (sessionMode == ChatDatabaseHelper.SESSION_MODE_AGENT) { + ChatDatabaseHelper.SESSION_MODE_AGENT + } else { + ChatDatabaseHelper.SESSION_MODE_NORMAL + } + } + @SuppressLint("Range") private fun fixMissingLastChatTime(sessionId: String): Long { val db = dbHelper.writableDatabase @@ -649,6 +960,9 @@ class ChatDataManager private constructor(context: Context) { companion object { private var sInstance: ChatDataManager? = null private const val TAG = "ChatDataManager" + private const val MEMORY_CONTENT_MAX_CHARS = 168 + private const val SKILL_ACTION_MAX_CHARS = 350 + @JvmStatic fun getInstance(context: Context): ChatDataManager { synchronized(ChatDataManager::class.java) { diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDatabaseHelper.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDatabaseHelper.kt index 7ba78a05f4..d3a2b0e9ad 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDatabaseHelper.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDatabaseHelper.kt @@ -12,6 +12,8 @@ class ChatDatabaseHelper(context: Context?) : db.execSQL(CREATE_TABLE_SESSION) db.execSQL(CREATE_TABLE_CHAT) db.execSQL(CREATE_TABLE_DOWNLOAD_HISTORY) + db.execSQL(CREATE_TABLE_AGENT_MEMORY) + db.execSQL(CREATE_TABLE_AGENT_SKILL) } override fun onUpgrade(db: SQLiteDatabase, oldVersion: Int, newVersion: Int) { @@ -60,16 +62,34 @@ class ChatDatabaseHelper(context: Context?) : // Column might already exist, ignore } } + if (oldVersion < 9) { + try { + db.execSQL("ALTER TABLE $TABLE_SESSION ADD COLUMN $COLUMN_SESSION_MODE TEXT DEFAULT '$SESSION_MODE_NORMAL'") + } catch (e: Exception) { + // Column might already exist, ignore + } + } + if (oldVersion < 10) { + try { + db.execSQL(CREATE_TABLE_AGENT_MEMORY) + db.execSQL(CREATE_TABLE_AGENT_SKILL) + } catch (e: Exception) { + // Tables might already exist, ignore + } + } } companion object { private const val DB_NAME = "chat.db" - private const val DB_VERSION = 8 + private const val DB_VERSION = 10 const val TABLE_SESSION: String = "Session" const val COLUMN_SESSION_ID: String = "sessionId" const val COLUMN_MODEL_ID: String = "modelId" const val COLUMN_SESSION_NAME: String = "name" const val COLUMN_LAST_CHAT_TIME: String = "lastChatTime" + const val COLUMN_SESSION_MODE: String = "sessionMode" + const val SESSION_MODE_NORMAL: String = "normal" + const val SESSION_MODE_AGENT: String = "agent" const val TABLE_CHAT: String = "ChatData" const val COLUMN_ID: String = "_id" @@ -95,12 +115,29 @@ class ChatDatabaseHelper(context: Context?) : const val COLUMN_MODEL_PATH: String = "modelPath" const val COLUMN_MODEL_TYPE: String = "modelType" + const val TABLE_AGENT_MEMORY: String = "AgentMemory" + const val COLUMN_AGENT_MEMORY_ID: String = "_id" + const val COLUMN_AGENT_MEMORY_CATEGORY: String = "category" + const val COLUMN_AGENT_MEMORY_CONTENT: String = "content" + const val COLUMN_AGENT_MEMORY_SOURCE: String = "source" + const val COLUMN_AGENT_MEMORY_UPDATED_AT: String = "updatedAt" + + const val TABLE_AGENT_SKILL: String = "AgentSkill" + const val COLUMN_AGENT_SKILL_ID: String = "_id" + const val COLUMN_AGENT_SKILL_NAME: String = "name" + const val COLUMN_AGENT_SKILL_DESCRIPTION: String = "description" + const val COLUMN_AGENT_SKILL_TRIGGER_KEYWORDS: String = "triggerKeywords" + const val COLUMN_AGENT_SKILL_ACTION_TEMPLATE: String = "actionTemplate" + const val COLUMN_AGENT_SKILL_ENABLED: String = "enabled" + const val COLUMN_AGENT_SKILL_CREATED_AT: String = "createdAt" + private const val CREATE_TABLE_SESSION = "CREATE TABLE IF NOT EXISTS " + TABLE_SESSION + " (" + COLUMN_SESSION_ID + " TEXT PRIMARY KEY, " + COLUMN_MODEL_ID + " TEXT," + COLUMN_SESSION_NAME + " TEXT," + - COLUMN_LAST_CHAT_TIME + " INTEGER DEFAULT 0)" + COLUMN_LAST_CHAT_TIME + " INTEGER DEFAULT 0," + + COLUMN_SESSION_MODE + " TEXT DEFAULT '" + SESSION_MODE_NORMAL + "')" private const val CREATE_TABLE_CHAT = "CREATE TABLE IF NOT EXISTS " + TABLE_CHAT + " (" + @@ -125,5 +162,23 @@ class ChatDatabaseHelper(context: Context?) : COLUMN_DOWNLOAD_TIME + " INTEGER, " + COLUMN_MODEL_PATH + " TEXT, " + COLUMN_MODEL_TYPE + " TEXT DEFAULT 'LLM')" + + private const val CREATE_TABLE_AGENT_MEMORY = "CREATE TABLE IF NOT EXISTS " + + TABLE_AGENT_MEMORY + " (" + + COLUMN_AGENT_MEMORY_ID + " INTEGER PRIMARY KEY AUTOINCREMENT, " + + COLUMN_AGENT_MEMORY_CATEGORY + " TEXT, " + + COLUMN_AGENT_MEMORY_CONTENT + " TEXT, " + + COLUMN_AGENT_MEMORY_SOURCE + " TEXT DEFAULT 'agent', " + + COLUMN_AGENT_MEMORY_UPDATED_AT + " INTEGER)" + + private const val CREATE_TABLE_AGENT_SKILL = "CREATE TABLE IF NOT EXISTS " + + TABLE_AGENT_SKILL + " (" + + COLUMN_AGENT_SKILL_ID + " INTEGER PRIMARY KEY AUTOINCREMENT, " + + COLUMN_AGENT_SKILL_NAME + " TEXT UNIQUE, " + + COLUMN_AGENT_SKILL_DESCRIPTION + " TEXT, " + + COLUMN_AGENT_SKILL_TRIGGER_KEYWORDS + " TEXT, " + + COLUMN_AGENT_SKILL_ACTION_TEMPLATE + " TEXT, " + + COLUMN_AGENT_SKILL_ENABLED + " INTEGER DEFAULT 1, " + + COLUMN_AGENT_SKILL_CREATED_AT + " INTEGER)" } } diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/SessionItem.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/SessionItem.kt index 29b7919619..58723e4943 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/SessionItem.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/chat/model/SessionItem.kt @@ -5,4 +5,8 @@ package com.alibaba.mnnllm.android.chat.model class SessionItem(@JvmField val sessionId: String, @JvmField val modelId: String, @JvmField var title: String, - @JvmField val lastChatTime: Long = 0L) \ No newline at end of file + @JvmField val lastChatTime: Long = 0L, + @JvmField val sessionMode: String = ChatDatabaseHelper.SESSION_MODE_NORMAL) { + val isAgentMode: Boolean + get() = sessionMode == ChatDatabaseHelper.SESSION_MODE_AGENT +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/history/ChatHistoryFragment.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/history/ChatHistoryFragment.kt index 9bbdb64272..890971f409 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/history/ChatHistoryFragment.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/history/ChatHistoryFragment.kt @@ -13,6 +13,7 @@ import androidx.recyclerview.widget.LinearLayoutManager import androidx.recyclerview.widget.RecyclerView import com.alibaba.mnnllm.android.main.MainActivity import com.alibaba.mnnllm.android.R +import com.alibaba.mnnllm.android.agent.AgentWorkspaceFileBrowser import com.alibaba.mnnllm.android.chat.ChatRouter import com.alibaba.mnnllm.android.chat.model.ChatDataManager import com.alibaba.mnnllm.android.chat.model.ChatDataManager.Companion.getInstance @@ -34,6 +35,9 @@ class ChatHistoryFragment : Fragment() { val view = inflater.inflate(R.layout.fragment_historylist, container, false) chatListRecyclerView = view.findViewById(R.id.chat_history_recycler_view) textNoHistory = view.findViewById(R.id.text_no_history) + view.findViewById(R.id.button_workspace_files).setOnClickListener { + AgentWorkspaceFileBrowser.show(requireContext()) + } chatListRecyclerView.setLayoutManager( LinearLayoutManager( context, diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/llm/LlmSession.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/llm/LlmSession.kt index 9aee91ab72..ac82c1fe36 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/llm/LlmSession.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/llm/LlmSession.kt @@ -5,6 +5,7 @@ package com.alibaba.mnnllm.android.llm; import android.util.Log import com.alibaba.mnnllm.android.llm.ChatService.Companion.provide +import com.alibaba.mnnllm.android.chat.chatlist.ChatViewHolders import com.alibaba.mnnllm.android.chat.model.ChatDataItem import com.alibaba.mnnllm.android.modelsettings.ModelConfig import com.alibaba.mnnllm.android.model.ModelTypeUtils @@ -12,9 +13,7 @@ import com.alibaba.mnnllm.android.modelsettings.ModelConfig.Companion.getExtraCo import com.google.gson.Gson import timber.log.Timber import java.io.File -import java.util.stream.Collectors import kotlin.concurrent.Volatile -import android.util.Pair import com.alibaba.mnnllm.android.utils.MmapUtils import android.content.Context import android.app.ActivityManager @@ -47,11 +46,14 @@ class LlmSession ( private var isQnn = false + private var pendingSystemPrompt: String? = null + override fun getHistory(): List?{ return savedHistory } override fun setHistory(history: List?) { + savedHistory = history } override fun load() { @@ -60,16 +62,6 @@ class LlmSession ( isQnn = ModelTypeUtils.isQnnModel(modelId) checkAndMergeSplitFiles() - var historyStringList: List? = null - val currentHistory = this.savedHistory - if (!currentHistory.isNullOrEmpty()) { - historyStringList = - currentHistory.stream() - .map { obj: ChatDataItem -> obj.text } - .filter { obj: String? -> obj != null } - .map { obj: String? -> obj!! } - .collect(Collectors.toList()) - } val config = if (useCustomConfig) { ModelConfig.loadMergedConfig(configPath, getExtraConfigFile(modelId))!! } else { @@ -94,19 +86,23 @@ class LlmSession ( if (backendType != null) { llmConfig.backendType = backendType } + llmConfig.promptCache = true + pendingSystemPrompt?.let { + llmConfig.systemPrompt = it + } if (isQnn) { llmConfig.visualModel = "visual_qnn_${QnnModule.modelMiddleName()}.mnn" } Log.d(TAG, "MNN_DEBUG load initNative") nativePtr = initNative( - configPath, - historyStringList, - if (llmConfig != null) { - Gson().toJson(llmConfig) - } else { - "{}" - }, - Gson().toJson(configMap) + configPath, + buildNativeHistoryForRestore(), + if (llmConfig != null) { + Gson().toJson(llmConfig) + } else { + "{}" + }, + Gson().toJson(configMap) ) Log.d(TAG, "MNN_DEBUG load initNative end") modelLoading = false @@ -119,6 +115,34 @@ class LlmSession ( } } + private fun buildNativeHistoryForRestore(): List? { + val history = savedHistory.orEmpty() + .filter { it.type == ChatViewHolders.USER || it.type == ChatViewHolders.ASSISTANT } + .mapNotNull { item -> + val text = item.text?.takeIf { it.isNotBlank() } + ?: item.displayText?.takeIf { it.isNotBlank() } + text?.let { item.type to it } + } + if (history.isEmpty()) return null + + val restored = mutableListOf>() + var lastType: Int? = null + var totalChars = 0 + for ((type, text) in history.asReversed()) { + if (type == lastType) continue + val trimmed = text.take(4_000) + if (totalChars + trimmed.length > 24_000) break + restored.add(type to trimmed) + totalChars += trimmed.length + lastType = type + if (restored.size >= 40) break + } + val ordered = restored.asReversed() + .dropWhile { it.first != ChatViewHolders.USER } + .dropLastWhile { it.first == ChatViewHolders.USER } + return ordered.map { it.second }.takeIf { it.isNotEmpty() } + } + /** * Check if the model is successfully loaded and ready for inference */ @@ -276,7 +300,10 @@ class LlmSession ( } fun updateSystemPrompt(systemPrompt: String) { - updateSystemPromptNative(nativePtr, systemPrompt) + pendingSystemPrompt = systemPrompt + if (nativePtr != 0L) { + updateSystemPromptNative(nativePtr, systemPrompt) + } } override fun updateThinking(thinking: Boolean) { @@ -382,6 +409,9 @@ class LlmSession ( } fun getSystemPrompt(): String? { + if (nativePtr == 0L) { + return pendingSystemPrompt + } return getSystemPromptNative(nativePtr) } @@ -470,4 +500,4 @@ class LlmSession ( callback: com.alibaba.mnnllm.android.benchmark.BenchmarkCallback ): com.alibaba.mnnllm.android.benchmark.BenchmarkResult -} \ No newline at end of file +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/model/ModelUtils.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/model/ModelUtils.kt index 3a68978ebb..5ea6a37966 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/model/ModelUtils.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/model/ModelUtils.kt @@ -120,14 +120,15 @@ object ModelUtils { if (metrics.containsKey("total_timeus")) { return generateDiffusionBenchMarkString(metrics) } - val promptLen = metrics.getOrDefault("prompt_len", 0L) as Long - val decodeLen = metrics.getOrDefault("decode_len", 0L) as Long - val prefillTimeUs = metrics.getOrDefault("prefill_time", 0L) as Long - val decodeTimeUs = metrics.getOrDefault("decode_time", 0L) as Long - var visionTimeUs = if (metrics.containsKey("vision_time")) metrics["vision_time"] as Long else 0L - var audioTimeUs = if (metrics.containsKey("audio_time")) metrics["audio_time"] as Long else 0L + val promptLen = metricLong(metrics, "prompt_len") + val inputLen = metricLong(metrics, "input_len").takeIf { it > 0L } ?: promptLen + val decodeLen = metricLong(metrics, "decode_len") + val prefillTimeUs = metricLong(metrics, "prefill_time") + val decodeTimeUs = metricLong(metrics, "decode_time") + var visionTimeUs = metricLong(metrics, "vision_time") + var audioTimeUs = metricLong(metrics, "audio_time") if (promptLen == 0L || decodeLen == 0L) { - return "generateBenchMarkString error" + return "" } // Calculate speeds in tokens per second var totalPrefillTimeUs = prefillTimeUs + visionTimeUs + audioTimeUs @@ -135,15 +136,26 @@ object ModelUtils { if ((totalPrefillTimeUs > 0)) (promptLen / (totalPrefillTimeUs / 1000000.0)) else 0.0 val decodeSpeed = if ((decodeTimeUs > 0)) (decodeLen / (decodeTimeUs / 1000000.0)) else 0.0 return String.format( - "Prefill: %.2fs, %d tokens, %.2f tokens/s \nDecode: %.2fs, %d tokens, %.2f tokens/s", + "Input: %d tokens\nPrefill: %.2fs, %d tokens, %.2f tokens/s \nDecode: %.2fs, %d tokens, %.2f tokens/s", + inputLen, totalPrefillTimeUs.toFloat() / 1000000, promptLen, promptSpeed, decodeTimeUs.toFloat() / 1000000,decodeLen, decodeSpeed, ) } + private fun metricLong(metrics: HashMap, key: String): Long { + return when (val value = metrics[key]) { + is Long -> value + is Int -> value.toLong() + is Number -> value.toLong() + is String -> value.toLongOrNull() ?: 0L + else -> 0L + } + } + @SuppressLint("DefaultLocale") fun generateDiffusionBenchMarkString(metrics: HashMap): String { - val totalDuration = metrics["total_timeus"] as Long * 1.0 / 1000000.0 + val totalDuration = metricLong(metrics, "total_timeus") * 1.0 / 1000000.0 return String.format("Generate time: %.2f s", totalDuration) } @@ -299,4 +311,4 @@ object ModelUtils { private const val TAG = "ModelUtils" -} \ No newline at end of file +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/ModelItemHolder.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/ModelItemHolder.kt index 34e0d3a405..bd95151e4a 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/ModelItemHolder.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/ModelItemHolder.kt @@ -5,16 +5,17 @@ import android.view.MenuItem import android.view.View import android.view.View.OnLongClickListener import android.widget.TextView -import android.widget.Toast -import androidx.appcompat.app.AlertDialog import androidx.appcompat.app.AppCompatActivity import androidx.appcompat.widget.PopupMenu +import androidx.recyclerview.widget.LinearLayoutManager import androidx.recyclerview.widget.RecyclerView import com.alibaba.mls.api.ModelItem import com.alibaba.mls.api.download.ModelDownloadManager import com.alibaba.mls.api.download.DownloadInfo import com.alibaba.mls.api.download.DownloadState import com.alibaba.mnnllm.android.R +import com.alibaba.mnnllm.android.chat.ChatRouter +import com.alibaba.mnnllm.android.chat.model.ChatDataManager import com.alibaba.mnnllm.android.model.ModelTypeUtils import com.alibaba.mnnllm.android.model.ModelUtils import com.alibaba.mnnllm.android.modelsettings.SettingsBottomSheetFragment @@ -22,6 +23,7 @@ import com.alibaba.mnnllm.android.modelsettings.DiffusionSettingsBottomSheetFrag import com.alibaba.mnnllm.android.utils.DialogUtils import com.alibaba.mnnllm.android.widgets.ModelAvatarView import com.alibaba.mnnllm.android.widgets.TagsLayout +import com.google.android.material.dialog.MaterialAlertDialogBuilder import kotlinx.coroutines.MainScope import kotlinx.coroutines.launch import java.text.SimpleDateFormat @@ -181,40 +183,52 @@ class ModelItemHolder( } } } else if (item.itemId == R.id.menu_show_model_info) { - // Show model info directly val context = v.context - val info = StringBuilder() - - info.append("Model Name: ${modelItem.modelName ?: modelItem.modelId}\n\n") - - // Show storage path - val storagePath = modelItem.localPath - info.append("Storage Location:\n$storagePath\n\n") - - // Show size - val sizeInfo = currentModelWrapper?.let { - ModelListItemUiStateFactory.getFormattedFileSize(it, modelDownloadManager) - } ?: "Unknown" - info.append("Size: ${if (sizeInfo.isNotEmpty()) sizeInfo else "Unknown"}\n") - - // Show last chat time - if ((currentModelWrapper?.lastChatTime ?: 0) > 0) { - val dateFormat = SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.getDefault()) - info.append("Last Chat Time: ${dateFormat.format(Date(currentModelWrapper!!.lastChatTime))}\n") + val dialogView = View.inflate(context, R.layout.dialog_model_info, null) + val dateFormat = SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.getDefault()) + val tvModelName = dialogView.findViewById(R.id.tv_model_name) + val tvDownloadTime = dialogView.findViewById(R.id.tv_download_time) + val tvLastChatTime = dialogView.findViewById(R.id.tv_last_chat_time) + val rvChatSessions = dialogView.findViewById(R.id.rv_chat_sessions) + val tvNoChatHistory = dialogView.findViewById(R.id.tv_no_chat_history) + + tvModelName.text = modelItem.modelName ?: modelItem.modelId + val downloadTime = currentModelWrapper?.downloadTime ?: 0L + tvDownloadTime.text = if (downloadTime > 0) { + dateFormat.format(Date(downloadTime)) } else { - info.append("Last Chat Time: Never\n") + context.getString(R.string.never_chatted) } - - if ((currentModelWrapper?.downloadTime ?: 0) > 0) { - val dateFormat = SimpleDateFormat("yyyy-MM-dd HH:mm:ss", Locale.getDefault()) - info.append("Downloaded Time: ${dateFormat.format(Date(currentModelWrapper!!.downloadTime))}\n") + val lastChatTime = currentModelWrapper?.lastChatTime ?: 0L + tvLastChatTime.text = if (lastChatTime > 0) { + dateFormat.format(Date(lastChatTime)) + } else { + context.getString(R.string.never_chatted) } - - AlertDialog.Builder(context) + + val sessions = modelItem.modelId?.let { + ChatDataManager.getInstance(context).getSessionsForModel(it) + }.orEmpty() + val dialog = MaterialAlertDialogBuilder(context) .setTitle(R.string.menu_show_model_info_title) - .setMessage(info.toString()) + .setView(dialogView) .setPositiveButton(android.R.string.ok, null) - .show() + .create() + + if (sessions.isEmpty()) { + rvChatSessions.visibility = View.GONE + tvNoChatHistory.visibility = View.VISIBLE + } else { + rvChatSessions.visibility = View.VISIBLE + tvNoChatHistory.visibility = View.GONE + rvChatSessions.layoutManager = LinearLayoutManager(context) + rvChatSessions.adapter = SessionAdapter(sessions) { session -> + dialog.dismiss() + ChatRouter.startRun(context, session.modelId, null, session.sessionId) + } + } + + dialog.show() } else if (item.itemId == R.id.menu_update_model) { // Handle update action modelItemListener.onItemUpdate(modelItem) diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/ModelListFragment.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/ModelListFragment.kt index 51b14b5896..719f815ab5 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/ModelListFragment.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/ModelListFragment.kt @@ -418,7 +418,6 @@ class ModelListFragment : Fragment(), ModelListContract.View, Searchable { modelName = modelMarketItem.modelName, modelSize = modelMarketItem.sizeB, onConfirm = { - // User confirmed, proceed with running the model ChatRouter.startRun(requireContext(), modelId!!, destPath, null) }, onCancel = { @@ -426,7 +425,6 @@ class ModelListFragment : Fragment(), ModelListContract.View, Searchable { } ) } else { - // Model is not large or size info not available, run directly ChatRouter.startRun(requireContext(), modelId!!, destPath, null) } } diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/SessionAdapter.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/SessionAdapter.kt index 615613f468..5da59e38f4 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/SessionAdapter.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelist/SessionAdapter.kt @@ -9,7 +9,10 @@ import androidx.recyclerview.widget.RecyclerView import com.alibaba.mnnllm.android.R import com.alibaba.mnnllm.android.chat.model.SessionItem -class SessionAdapter(private val sessions: List) : +class SessionAdapter( + private val sessions: List, + private val onSessionClick: ((SessionItem) -> Unit)? = null +) : RecyclerView.Adapter() { class SessionViewHolder(itemView: View) : RecyclerView.ViewHolder(itemView) { @@ -35,7 +38,10 @@ class SessionAdapter(private val sessions: List) : holder.tvSessionName.text = displayName holder.tvSessionId.text = "Session ID: ${session.sessionId}" + holder.itemView.setOnClickListener { + onSessionClick?.invoke(session) + } } override fun getItemCount(): Int = sessions.size -} \ No newline at end of file +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelsettings/ModelConfig.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelsettings/ModelConfig.kt index 56337bb604..61518124e3 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelsettings/ModelConfig.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelsettings/ModelConfig.kt @@ -281,7 +281,7 @@ data class ModelConfig( precision = "low", memory = "", systemPrompt = "You are a helpful assistant.", - promptCache = false, + promptCache = true, samplerType = "", mixedSamplers = mutableListOf("topK", "topP", "minP", "temperature"), temperature = 0.6f, diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelsettings/SettingsBottomSheetFragment.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelsettings/SettingsBottomSheetFragment.kt index 0c5e41f93b..1376ea4626 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelsettings/SettingsBottomSheetFragment.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/android/modelsettings/SettingsBottomSheetFragment.kt @@ -107,7 +107,7 @@ class SettingsBottomSheetFragment : BaseSettingsBottomSheetFragment() { currentConfig.useMmap = isChecked } // Prompt cache toggle - binding.promptCacheToggle.isChecked = currentConfig.promptCache ?: false + binding.promptCacheToggle.isChecked = currentConfig.promptCache ?: true binding.promptCacheToggle.setOnCheckedChangeListener { _, isChecked -> currentConfig.promptCache = isChecked } @@ -551,7 +551,7 @@ class SettingsBottomSheetFragment : BaseSettingsBottomSheetFragment() { } else if (currentConfig.promptCache != loadedConfig.promptCache) { needSaveConfig = true val llmSession = chatSession as? com.alibaba.mnnllm.android.llm.LlmSession - llmSession?.updateConfig("""{"prompt_cache": ${currentConfig.promptCache ?: false}}""") + llmSession?.updateConfig("""{"prompt_cache": ${currentConfig.promptCache ?: true}}""") needRecreate = false } else if (currentConfig.useMmap != loadedConfig.useMmap) { needSaveConfig = true @@ -582,9 +582,9 @@ class SettingsBottomSheetFragment : BaseSettingsBottomSheetFragment() { updateSamplerSettingsVisibility() chatSession?.updateSystemPrompt(currentConfig.systemPrompt ?: defaultConfig.systemPrompt ?: "") chatSession?.updateMaxNewTokens(currentConfig.maxNewTokens ?: defaultConfig.maxNewTokens ?: 2048) - binding.promptCacheToggle.isChecked = currentConfig.promptCache ?: false + binding.promptCacheToggle.isChecked = currentConfig.promptCache ?: true val llmSession = chatSession as? com.alibaba.mnnllm.android.llm.LlmSession - llmSession?.updateConfig("""{"prompt_cache": ${currentConfig.promptCache ?: false}}""") + llmSession?.updateConfig("""{"prompt_cache": ${currentConfig.promptCache ?: true}}""") } override fun onDestroyView() { diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/handlers/ResponseHandler.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/handlers/ResponseHandler.kt index df37c56be3..913bbc387b 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/handlers/ResponseHandler.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/handlers/ResponseHandler.kt @@ -35,7 +35,7 @@ class ResponseHandler { /** * processwithhistorymessagestreamingresponse * * @param call Ktorapplicationcallcontext * @param history historymessagelist * @param traceId traceID*/ suspend fun handleStreamResponseWithFullHistory( call: ApplicationCall, - history: List>, + history: List>, traceId: String ) { val responseMetadata = createResponseMetadata() @@ -175,7 +175,7 @@ class ResponseHandler { /** * processwithhistorymessagenon-streamingresponse * * @param call Ktorapplicationcallcontext * @param history historymessagelist*/ suspend fun handleNonStreamResponseWithFullHistory( call: ApplicationCall, - history: List> + history: List> ) { val fullResponse = StringBuilder() val responseMetadata = createResponseMetadata() @@ -228,7 +228,7 @@ class ResponseHandler { /** * processnon-streaminggenerate*/ private fun processNonStreamGeneration( llmSession: LlmSession, - history: List>, + history: List>, fullResponse: StringBuilder ): HashMap { return llmSession.submitFullHistory(history, object : GenerateProgressListener { @@ -248,4 +248,4 @@ class ResponseHandler { val usage = chatResponseFormatter.createUsageFromMetrics(metrics) return chatResponseFormatter.createChatCompletionResponse(responseMetadata.responseId, responseMetadata.created, content, usage) } -} \ No newline at end of file +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/logging/ChatLogger.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/logging/ChatLogger.kt index ce658d071e..701ed6deb1 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/logging/ChatLogger.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/logging/ChatLogger.kt @@ -32,7 +32,7 @@ class ChatLogger { } /** * recordconvertafterhistorymessage*/ - fun logTransformedHistory(traceId: String, unifiedHistory: List>) { + fun logTransformedHistory(traceId: String, unifiedHistory: List>) { Timber.tag(TAG_TRANSFORM).d("[$traceId] 转换后的统一历史消息: $unifiedHistory") } @@ -88,4 +88,4 @@ class ChatLogger { fun logInfo(traceId: String, message: String) { Timber.tag(TAG_REQUEST).i("[$traceId] $message") } -} \ No newline at end of file +} diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/services/AnthropicMessagesService.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/services/AnthropicMessagesService.kt index c5c1181269..0cd0219583 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/services/AnthropicMessagesService.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/services/AnthropicMessagesService.kt @@ -122,7 +122,7 @@ class AnthropicMessagesService { private suspend fun handleNonStreamResponse( call: ApplicationCall, - history: List>, + history: List>, model: String, llmSession: LlmSession ) { @@ -147,7 +147,7 @@ class AnthropicMessagesService { private suspend fun handleStreamResponse( call: ApplicationCall, - history: List>, + history: List>, model: String, traceId: String, llmSession: LlmSession diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/utils/MessageTransformer.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/utils/MessageTransformer.kt index 1826f27509..0c8b5fd275 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/utils/MessageTransformer.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/network/utils/MessageTransformer.kt @@ -8,7 +8,6 @@ import PreprocessedAudioContent import PreprocessedFileContent import PreprocessedVideoContent import TextContent -import android.util.Pair import com.alibaba.mnnllm.android.llm.LlmSession import com.alibaba.mnnllm.android.model.ModelTypeUtils import com.alibaba.mnnllm.api.openai.network.processors.MnnImageProcessor diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/runtime/DefaultLlmRuntimeController.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/runtime/DefaultLlmRuntimeController.kt index 579c6de47c..aee533c5b7 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/runtime/DefaultLlmRuntimeController.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/runtime/DefaultLlmRuntimeController.kt @@ -24,17 +24,23 @@ object DefaultLlmRuntimeController : LlmRuntimeController { ): EnsureSessionResult { synchronized(lock) { val currentSession = activeSession + val resolvedSessionId = sessionId ?: "service_runtime_${System.currentTimeMillis()}" if ( currentSession != null && RuntimeSessionReusePolicy.shouldReuse( forceReload = forceReload, activeModelId = activeModelId, requestedModelId = modelId, + activeSessionId = currentSession.sessionId, + requestedSessionId = resolvedSessionId, isSessionLoaded = currentSession.isModelLoaded(), activeUseAppConfig = activeUseAppConfig, requestedUseAppConfig = useAppConfig ) ) { + if (historyList != null) { + currentSession.setHistory(historyList) + } return EnsureSessionResult( success = true, session = currentSession, @@ -57,7 +63,6 @@ object DefaultLlmRuntimeController : LlmRuntimeController { ) val modelName = ModelUtils.getModelName(modelId) ?: modelId - val resolvedSessionId = sessionId ?: "service_runtime_${System.currentTimeMillis()}" return runCatching { val chatSession = ChatService.provide().createSession( diff --git a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/runtime/RuntimeSessionReusePolicy.kt b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/runtime/RuntimeSessionReusePolicy.kt index 8f94cec864..f5c0d20049 100644 --- a/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/runtime/RuntimeSessionReusePolicy.kt +++ b/apps/Android/MnnLlmChat/app/src/main/java/com/alibaba/mnnllm/api/openai/runtime/RuntimeSessionReusePolicy.kt @@ -5,12 +5,15 @@ internal object RuntimeSessionReusePolicy { forceReload: Boolean, activeModelId: String?, requestedModelId: String, + activeSessionId: String?, + requestedSessionId: String?, isSessionLoaded: Boolean, activeUseAppConfig: Boolean, requestedUseAppConfig: Boolean ): Boolean { if (forceReload) return false if (activeModelId != requestedModelId) return false + if (activeSessionId != requestedSessionId) return false if (activeUseAppConfig != requestedUseAppConfig) return false return isSessionLoaded } diff --git a/apps/Android/MnnLlmChat/app/src/main/python/agent_python.py b/apps/Android/MnnLlmChat/app/src/main/python/agent_python.py new file mode 100644 index 0000000000..cb476baba8 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/python/agent_python.py @@ -0,0 +1,436 @@ +import builtins +import contextlib +import io +import json +import os +import py_compile +import sys +import time +import traceback +from os.path import abspath, commonpath, isabs, join, realpath + + +ALLOWED_IMPORT_ROOTS = { + "base64", + "bisect", + "calendar", + "collections", + "csv", + "dateutil", + "datetime", + "decimal", + "et_xmlfile", + "fractions", + "functools", + "hashlib", + "heapq", + "html", + "io", + "itertools", + "json", + "math", + "numpy", + "operator", + "openpyxl", + "pandas", + "pytz", + "random", + "re", + "statistics", + "string", + "textwrap", + "time", + "tzdata", + "urllib", + "xml", +} + +DENIED_IMPORT_ROOTS = { + "ctypes", + "multiprocessing", + "os", + "pathlib", + "shutil", + "socket", + "subprocess", + "sys", +} + + +def run_code(code, input_text="", timeout_ms=15000, workspace_dir=""): + start = time.monotonic() + deadline = start + max(1, min(int(timeout_ms or 15000), 30000)) / 1000.0 + stdout = io.StringIO() + stderr = io.StringIO() + state = {"result": None, "has_result": False} + + def check_deadline(): + if time.monotonic() > deadline: + raise TimeoutError("python_exec exceeded timeout") + + def trace_func(frame, event, arg): + check_deadline() + return trace_func + + original_import = builtins.__import__ + + def safe_import(name, globals=None, locals=None, fromlist=(), level=0): + root = name.split(".", 1)[0] + caller_file = str((globals or {}).get("__file__", "")) + is_package_internal = caller_file and not caller_file.startswith("<") and not caller_file.endswith("agent_python.py") + if is_package_internal: + return original_import(name, globals, locals, fromlist, level) + if root in DENIED_IMPORT_ROOTS or root not in ALLOWED_IMPORT_ROOTS: + raise ImportError("module is not allowed: " + name) + return original_import(name, globals, locals, fromlist, level) + + def set_result(value): + state["result"] = value + state["has_result"] = True + + workspace_root = realpath(workspace_dir) if workspace_dir else "" + files_before = _snapshot_workspace_files(workspace_root) + + def validate_workspace_path(path): + if not path: + raise PermissionError("empty path") + raw_path = str(path) + if workspace_root and not isabs(raw_path): + raw_path = join(workspace_root, raw_path) + full = realpath(abspath(raw_path)) + if not workspace_root or commonpath([workspace_root, full]) != workspace_root: + raise PermissionError("path is outside python workspace: " + str(path)) + return full + + def safe_open(file, mode="r", *args, **kwargs): + return open(validate_workspace_path(file), mode, *args, **kwargs) + + def read_excel(path, max_rows=200, max_sheets=10, values_only=True): + full_path = validate_workspace_path(path) + return _read_excel(full_path, max_rows=max_rows, max_sheets=max_sheets, values_only=values_only) + + def write_excel(filename, sheets): + full_path = validate_workspace_path(filename) + return _write_excel(full_path, sheets) + + def script_path(name): + safe_name = _safe_script_name(name) + script_dir = validate_workspace_path("python") + os.makedirs(script_dir, exist_ok=True) + return validate_workspace_path(join("python", safe_name)) + + def save_script(name, source): + full_path = script_path(name) + os.makedirs(os.path.dirname(full_path), exist_ok=True) + with open(full_path, "w", encoding="utf-8") as f: + f.write(str(source or "")) + return {"path": full_path, "bytes": len(str(source or "").encode("utf-8"))} + + def load_script(name): + full_path = script_path(name) + with open(full_path, "r", encoding="utf-8") as f: + return {"path": full_path, "source": f.read()} + + def list_scripts(): + script_dir = validate_workspace_path("python") + os.makedirs(script_dir, exist_ok=True) + scripts = [] + for root, _, files in os.walk(script_dir): + for filename in files: + if not filename.endswith(".py"): + continue + full_path = join(root, filename) + rel = os.path.relpath(full_path, script_dir).replace("\\", "/") + scripts.append({"name": rel, "path": full_path, "bytes": os.path.getsize(full_path)}) + return sorted(scripts, key=lambda item: item["name"]) + + def run_script(name): + full_path = script_path(name) + with open(full_path, "r", encoding="utf-8") as f: + source = f.read() + compiled_script = compile(source, full_path, "exec") + exec(compiled_script, globals_dict, globals_dict) + return state["result"] if state["has_result"] else globals_dict.get("result", None) + + def compile_script(name): + full_path = script_path(name) + try: + py_compile.compile(full_path, doraise=True) + return {"ok": True, "path": full_path, "error": ""} + except py_compile.PyCompileError as exc: + return {"ok": False, "path": full_path, "error": str(exc)} + + safe_builtins = { + "abs": abs, + "all": all, + "any": any, + "bool": bool, + "bytes": bytes, + "chr": chr, + "dict": dict, + "divmod": divmod, + "enumerate": enumerate, + "filter": filter, + "float": float, + "format": format, + "hash": hash, + "hex": hex, + "int": int, + "isinstance": isinstance, + "issubclass": issubclass, + "len": len, + "list": list, + "map": map, + "max": max, + "min": min, + "next": next, + "ord": ord, + "pow": pow, + "print": print, + "range": range, + "repr": repr, + "reversed": reversed, + "round": round, + "set": set, + "slice": slice, + "sorted": sorted, + "str": str, + "sum": sum, + "tuple": tuple, + "zip": zip, + "Exception": Exception, + "ValueError": ValueError, + "TypeError": TypeError, + "RuntimeError": RuntimeError, + "TimeoutError": TimeoutError, + "__import__": safe_import, + "open": safe_open, + } + + globals_dict = { + "__builtins__": safe_builtins, + "__name__": "__agent_python__", + "input_text": input_text or "", + "workspace_dir": workspace_root, + "read_excel": read_excel, + "write_excel": write_excel, + "save_script": save_script, + "load_script": load_script, + "list_scripts": list_scripts, + "run_script": run_script, + "compile_script": compile_script, + "emit": set_result, + "set_result": set_result, + } + + try: + globals_dict["input_json"] = json.loads(input_text) if input_text else None + except Exception: + globals_dict["input_json"] = None + + ok = True + error = "" + previous_cwd = os.getcwd() + sys.settrace(trace_func) + try: + if workspace_root: + os.makedirs(workspace_root, exist_ok=True) + os.chdir(workspace_root) + compiled = compile(code or "", "", "exec") + with contextlib.redirect_stdout(stdout), contextlib.redirect_stderr(stderr): + exec(compiled, globals_dict, globals_dict) + except Exception: + ok = False + error = traceback.format_exc(limit=6) + finally: + sys.settrace(None) + try: + os.chdir(previous_cwd) + except Exception: + pass + + result = state["result"] if state["has_result"] else globals_dict.get("result", None) + generated_files = _collect_generated_files(workspace_root, files_before) + return json.dumps( + { + "ok": ok, + "stdout": stdout.getvalue(), + "stderr": stderr.getvalue(), + "error": error, + "result": _json_safe(result), + "files": generated_files, + "elapsed_ms": int((time.monotonic() - start) * 1000), + }, + ensure_ascii=False, + ) + + +def _json_safe(value): + try: + json.dumps(value) + return value + except Exception: + return repr(value) + + +def _snapshot_workspace_files(workspace_root): + snapshot = {} + if not workspace_root or not os.path.isdir(workspace_root): + return snapshot + for root, dirs, files in os.walk(workspace_root): + dirs[:] = [d for d in dirs if d != "__pycache__"] + for filename in files: + full_path = realpath(join(root, filename)) + try: + rel_path = os.path.relpath(full_path, workspace_root).replace("\\", "/") + stat = os.stat(full_path) + snapshot[rel_path] = (stat.st_mtime_ns, stat.st_size) + except Exception: + continue + return snapshot + + +def _guess_mime_type(path): + lower = path.lower() + if lower.endswith(".xlsx"): + return "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + if lower.endswith(".xls"): + return "application/vnd.ms-excel" + if lower.endswith(".csv"): + return "text/csv" + if lower.endswith(".pdf"): + return "application/pdf" + if lower.endswith(".txt"): + return "text/plain" + if lower.endswith(".json"): + return "application/json" + if lower.endswith(".png"): + return "image/png" + if lower.endswith(".jpg") or lower.endswith(".jpeg"): + return "image/jpeg" + return "application/octet-stream" + + +def _collect_generated_files(workspace_root, before): + if not workspace_root or not os.path.isdir(workspace_root): + return [] + after = _snapshot_workspace_files(workspace_root) + changed = [] + for rel_path, stat in after.items(): + if rel_path.startswith("python/") or "/__pycache__/" in rel_path: + continue + if before.get(rel_path) == stat: + continue + full_path = realpath(join(workspace_root, rel_path)) + changed.append( + { + "name": os.path.basename(full_path), + "path": full_path, + "relative_path": rel_path, + "mime_type": _guess_mime_type(full_path), + "size_bytes": stat[1], + } + ) + changed.sort(key=lambda item: item["relative_path"]) + return changed[:20] + + +def _read_excel(path, max_rows=200, max_sheets=10, values_only=True): + from datetime import date, datetime, time as dt_time + from decimal import Decimal + from openpyxl import load_workbook + + workbook = load_workbook(path, read_only=True, data_only=True) + try: + sheets = [] + for sheet_name in workbook.sheetnames[: max(1, int(max_sheets or 10))]: + ws = workbook[sheet_name] + rows = [] + for row_index, row in enumerate(ws.iter_rows(values_only=values_only), start=1): + if row_index > max(1, int(max_rows or 200)): + break + rows.append([_cell_value(v) for v in row]) + sheets.append( + { + "name": sheet_name, + "max_row": ws.max_row, + "max_column": ws.max_column, + "rows_returned": len(rows), + "rows": rows, + } + ) + return {"path": path, "sheet_count": len(workbook.sheetnames), "sheets": sheets} + finally: + workbook.close() + + +def _cell_value(value): + if value is None or isinstance(value, (str, int, float, bool)): + return value + if isinstance(value, Decimal): + return float(value) + if hasattr(value, "isoformat"): + return value.isoformat() + return str(value) + + +def _safe_script_name(name): + raw = str(name or "").strip().replace("\\", "/") + if not raw: + raise ValueError("script name is empty") + parts = [part for part in raw.split("/") if part not in ("", ".", "..")] + safe_parts = [] + for part in parts: + cleaned = "".join(ch if ch.isalnum() or ch in "._-" else "_" for ch in part) + if cleaned: + safe_parts.append(cleaned) + if not safe_parts: + raise ValueError("invalid script name") + filename = "/".join(safe_parts) + if not filename.endswith(".py"): + filename += ".py" + return filename + + +def _write_excel(path, sheets): + from openpyxl import Workbook + + workbook = Workbook() + default_sheet = workbook.active + workbook.remove(default_sheet) + + if isinstance(sheets, dict): + iterable = sheets.items() + elif isinstance(sheets, list): + iterable = [] + for index, item in enumerate(sheets, start=1): + if isinstance(item, dict): + iterable.append((item.get("name") or ("Sheet" + str(index)), item.get("rows") or [])) + else: + iterable.append(("Sheet" + str(index), item)) + else: + raise TypeError("sheets must be a dict or list") + + sheet_count = 0 + row_count = 0 + for raw_name, rows in iterable: + name = str(raw_name or "Sheet")[:31] + ws = workbook.create_sheet(title=name) + sheet_count += 1 + for row in rows or []: + if isinstance(row, dict): + values = list(row.values()) + elif isinstance(row, (list, tuple)): + values = list(row) + else: + values = [row] + ws.append([_cell_value(v) for v in values]) + row_count += 1 + + if sheet_count == 0: + workbook.create_sheet(title="Sheet1") + sheet_count = 1 + + workbook.save(path) + workbook.close() + return {"path": path, "sheet_count": sheet_count, "row_count": row_count} diff --git a/apps/Android/MnnLlmChat/app/src/main/res/drawable/ic_folder_24dp.xml b/apps/Android/MnnLlmChat/app/src/main/res/drawable/ic_folder_24dp.xml new file mode 100644 index 0000000000..d218e33199 --- /dev/null +++ b/apps/Android/MnnLlmChat/app/src/main/res/drawable/ic_folder_24dp.xml @@ -0,0 +1,9 @@ + + + diff --git a/apps/Android/MnnLlmChat/app/src/main/res/layout/fragment_historylist.xml b/apps/Android/MnnLlmChat/app/src/main/res/layout/fragment_historylist.xml index 230d154a58..7a49c17b3a 100644 --- a/apps/Android/MnnLlmChat/app/src/main/res/layout/fragment_historylist.xml +++ b/apps/Android/MnnLlmChat/app/src/main/res/layout/fragment_historylist.xml @@ -22,10 +22,29 @@ android:layout_width="match_parent" android:layout_height="match_parent" android:layout_below="@id/history_title" + android:layout_above="@id/button_workspace_files" android:visibility="visible" android:overScrollMode="never" android:clipToPadding="false" android:paddingBottom="8dp" /> + + + - \ No newline at end of file + diff --git a/apps/Android/MnnLlmChat/app/src/main/res/layout/item_holder_assistant.xml b/apps/Android/MnnLlmChat/app/src/main/res/layout/item_holder_assistant.xml index af432d6135..71c84a7b02 100644 --- a/apps/Android/MnnLlmChat/app/src/main/res/layout/item_holder_assistant.xml +++ b/apps/Android/MnnLlmChat/app/src/main/res/layout/item_holder_assistant.xml @@ -95,12 +95,23 @@ android:background="#00f" tools:src="@drawable/testtest"/> + + 按住说话 开启新的会话 已开启新的会话 + 已开启新的 Agent 会话 + 开启新的会话 + 选择这个会话是否启用 Agent 能力。普通聊天会保持原有路径。 + 普通聊天 + Agent 模式 拍照 图片 音频(wav) @@ -52,6 +57,9 @@ 模型列表 模型市场 对话历史 + 工作区文件 + 暂无工作区文件。 + 没有应用可以打开此文件。 Chat History 历史被删除 已生成图片： diff --git a/apps/Android/MnnLlmChat/app/src/main/res/values-zh-rTW/strings.xml b/apps/Android/MnnLlmChat/app/src/main/res/values-zh-rTW/strings.xml index 849d924ea1..1b7f016e0c 100644 --- a/apps/Android/MnnLlmChat/app/src/main/res/values-zh-rTW/strings.xml +++ b/apps/Android/MnnLlmChat/app/src/main/res/values-zh-rTW/strings.xml @@ -35,6 +35,11 @@ 按住說話 開啟新對話 已開啟新對話 + 已開啟新的 Agent 對話 + 開啟新對話 + 選擇這個對話是否啟用 Agent 能力。普通聊天會保持既有路徑。 + 普通聊天 + Agent 模式 拍照 圖片 音訊 (wav) @@ -51,6 +56,9 @@ 模型清單 模型市集 對話歷史 + 工作區檔案 + 暫無工作區檔案。 + 沒有應用可以開啟此檔案。 對話歷史 歷史記錄已刪除 圖片已生成： diff --git a/apps/Android/MnnLlmChat/app/src/main/res/values/strings.xml b/apps/Android/MnnLlmChat/app/src/main/res/values/strings.xml index 8323784be6..3731a9fb84 100644 --- a/apps/Android/MnnLlmChat/app/src/main/res/values/strings.xml +++ b/apps/Android/MnnLlmChat/app/src/main/res/values/strings.xml @@ -36,6 +36,11 @@ Press To Talk Start New Conversation New conversation started + New Agent conversation started + Start new conversation + Choose whether this conversation should use Agent capabilities. Normal chat keeps the existing path. + Normal chat + Agent mode Photo Image Audio(wav) @@ -53,6 +58,9 @@ Models Models Market History + Workspace files + No workspace files yet. + No app can open this file. Chat History History Delete Success Here\'s the generated image diff --git a/apps/Android/MnnLlmChat/app/src/test/java/com/alibaba/mnnllm/api/openai/runtime/RuntimeSessionReusePolicyTest.kt b/apps/Android/MnnLlmChat/app/src/test/java/com/alibaba/mnnllm/api/openai/runtime/RuntimeSessionReusePolicyTest.kt index 7811294cf6..4b688942ff 100644 --- a/apps/Android/MnnLlmChat/app/src/test/java/com/alibaba/mnnllm/api/openai/runtime/RuntimeSessionReusePolicyTest.kt +++ b/apps/Android/MnnLlmChat/app/src/test/java/com/alibaba/mnnllm/api/openai/runtime/RuntimeSessionReusePolicyTest.kt @@ -13,6 +13,8 @@ class RuntimeSessionReusePolicyTest { forceReload = false, activeModelId = "same-model", requestedModelId = "same-model", + activeSessionId = "same-session", + requestedSessionId = "same-session", isSessionLoaded = false, activeUseAppConfig = false, requestedUseAppConfig = false @@ -27,6 +29,8 @@ class RuntimeSessionReusePolicyTest { forceReload = false, activeModelId = "same-model", requestedModelId = "same-model", + activeSessionId = "same-session", + requestedSessionId = "same-session", isSessionLoaded = true, activeUseAppConfig = true, requestedUseAppConfig = true @@ -41,6 +45,8 @@ class RuntimeSessionReusePolicyTest { forceReload = false, activeModelId = "same-model", requestedModelId = "same-model", + activeSessionId = "same-session", + requestedSessionId = "same-session", isSessionLoaded = true, activeUseAppConfig = false, requestedUseAppConfig = true @@ -48,6 +54,22 @@ class RuntimeSessionReusePolicyTest { ) } + @Test + fun shouldReuse_returnsFalse_whenSessionIdChanges() { + assertFalse( + RuntimeSessionReusePolicy.shouldReuse( + forceReload = false, + activeModelId = "same-model", + requestedModelId = "same-model", + activeSessionId = "old-session", + requestedSessionId = "new-session", + isSessionLoaded = true, + activeUseAppConfig = true, + requestedUseAppConfig = true + ) + ) + } + @Test fun shouldExposeActiveSession_returnsFalse_whenSessionIsNotLoaded() { assertFalse(RuntimeSessionReusePolicy.shouldExposeActiveSession(isSessionLoaded = false)) diff --git a/apps/Android/MnnLlmChat/build.gradle b/apps/Android/MnnLlmChat/build.gradle index cf5301a8ed..61f03807b5 100644 --- a/apps/Android/MnnLlmChat/build.gradle +++ b/apps/Android/MnnLlmChat/build.gradle @@ -1,8 +1,9 @@ // Top-level build file where you can add configuration options common to all sub-projects/modules. plugins { -id 'com.android.application' version '8.7.3' apply false +id 'com.android.application' version '8.9.2' apply false id 'org.jetbrains.kotlin.jvm' version '2.1.21' id 'org.jetbrains.kotlin.android' version '2.1.21' apply false +id 'com.chaquo.python' version '17.0.0' apply false id 'com.google.gms.google-services' version '4.4.4' apply false id 'com.google.firebase.crashlytics' version '3.0.3' apply false diff --git a/apps/Android/MnnLlmChat/docs/Agent.gif b/apps/Android/MnnLlmChat/docs/Agent.gif new file mode 100644 index 0000000000..45beede1c3 Binary files /dev/null and b/apps/Android/MnnLlmChat/docs/Agent.gif differ diff --git a/apps/Android/MnnLlmChat/docs/agentic_design.md b/apps/Android/MnnLlmChat/docs/agentic_design.md new file mode 100644 index 0000000000..43ad41f2a0 --- /dev/null +++ b/apps/Android/MnnLlmChat/docs/agentic_design.md @@ -0,0 +1,268 @@ +# Agentic Design for MnnLlmChat + +This document describes the lightweight Agent mode added to `MnnLlmChat`. + +核心技术来源 / Core technical source: `https://github.com/huangzhengxiang/ActMe.git` + +The design is adapted from ActMe's mobile agent architecture, especially: + +- JSON-based `system_calls`. +- A multi-pass agentic loop. +- Visible tool execution steps. +- Tool-call parsing fallbacks. +- Conservative mobile budgets and cancellation behavior. + +MnnLlmChat remains an on-device local-model app. Agent mode does not change the model runtime into a cloud model. It lets the host app execute a small set of tools requested by the local model. + +## Conversation Modes + +Normal chat and Agent chat are session-level modes. + +The mode is selected when a conversation is created: + +- `normal`: use the original single-pass chat path. +- `agent`: use the agentic loop. + +The selected mode is persisted in the `Session` table as `sessionMode`. + +```text +normal session -> ChatPresenter.submitLlmRequest(...) +agent session -> ChatPresenter.submitAgenticLlmRequest(...) +``` + +The mode is not a temporary UI toggle. Reopening a history item restores that session's own mode. + +## Backward Compatibility + +Database compatibility is handled by `ChatDatabaseHelper` and `ChatDataManager`. + +Schema change: + +```text +DB_VERSION: 8 -> 9 +Session.sessionMode TEXT DEFAULT 'normal' +``` + +Upgrade behavior: + +- Existing databases are migrated with `ALTER TABLE Session ADD COLUMN sessionMode TEXT DEFAULT 'normal'`. +- `ChatDataManager.ensureSessionModeColumn()` also checks and adds the column defensively. +- Old conversations without `sessionMode` are treated as `normal`. +- History queries fall back to older column sets when needed. + +This supports upgrading from older app versions. Downgrading from a version that has opened DB v9 back to an older APK is not guaranteed, because the old code does not know DB v9. + +## Current Tool Scope + +The current MnnLlmChat port intentionally exposes only tools that are implemented in this app: + +- `get_current_time` +- `web_search` +- `browser_url` +- `python_exec` + +Python, Skill, and Memory are active in this MnnLlmChat port. ADB remains part of the fuller ActMe design and is intentionally not advertised here. + +## Loop Shape + +The Agent mode loop is: + +```text +user message +-> local model planning pass +-> parse JSON/system_calls +-> execute tools visibly +-> append observations to continuation input +-> local model continuation pass +-> repeat until final reply, stop request, or budget exhaustion +``` + +Tool steps are shown in the assistant message while the run is active: + +```text +[Agent] 规划中... +[Agent] 第 1 轮计划：1 个工具调用 +[Agent] 联网搜索：中国银行 积存金 价格 +[Agent] 联网搜索完成：搜索完成，... +[Agent] 已获得工具结果，继续推理... +``` + +When the run finishes, the visible message body is replaced by the final answer. The step log is kept in the thinking/process area. + +## KV Cache and Prompt Cache Semantics + +MnnLlmChat uses the native MNN `LlmSession` as the source of truth for live conversation context. + +For the ChatActivity local-chat path: + +- Each user turn submits only the newly arrived input to `LlmSession.generate(...)`. +- Database chat history is used for UI/history display, not for reconstructing the prompt on every turn. +- The native session keeps `keep_history=true`, maintains its in-memory `history_`, calls `llm_->response(history_, ...)`, and then calls `llm_->syncPromptCache(history_)`. +- As long as the same native `llm_` instance stays alive, MNN can reuse the in-memory prompt/KV cache for already processed context. + +Reloading a session is different: + +- Releasing or reloading creates a new native `llm_` instance. +- This app does not currently persist prompt cache or KV cache to disk. +- Reopening an old conversation restores visible database history and may pass a bounded alternating user/assistant history list into native initialization for useful conversational context. +- This cold restore is not the same as restoring the old native prompt cache. +- If persistent prompt-cache restore is required, it needs explicit native support for saving and loading cache state by `sessionId`. + +The API server compatibility path is separate. OpenAI/Anthropic-style stateless calls may still use `submitFullHistory(...)` because those requests carry complete message history by protocol design. + +## Prompt Contract + +`AgenticPrompts.kt` tells the model: + +- It is running inside MNN Chat. +- The current model is local/on-device. +- The local model itself cannot directly access the network. +- The host app can execute tools requested through `system_calls`. +- In Agent mode, search/browse requests should produce tool calls instead of refusal text. + +Expected output shape: + +```json +{ + "reply": "", + "memory_updates": [], + "skill_updates": [], + "system_calls": [ + { + "type": "web_search", + "query": "query" + }, + { + "type": "python_exec", + "code": "write_excel('sample.xlsx', {'Sheet1': [['Name'], ['Alice']]})", + "timeout_ms": 15000, + "output_files": ["sample.xlsx"] + } + ] +} +``` + +For final answers, `reply` should contain the user-facing text and `system_calls` should be empty or omitted. + +## Parsing Fallbacks + +`AgenticOutputParser.kt` accepts: + +- strict JSON object +- fenced JSON block +- JSON object embedded in surrounding text +- single tool object, such as `{"type":"python_exec","code":"..."}` +- `system_calls` as either an array or a single object +- nested agent JSON inside the `reply` field +- string arrays for `memory_updates` and `skill_updates` +- loose `python_exec` extraction when malformed JSON still contains a recoverable `type` and `code` + +The goal is to avoid showing raw dictionaries to users when a local model formats tool calls imperfectly. + +## Tool Execution + +`AgenticToolExecutor.kt` implements the current tool layer. + +### get_current_time + +Returns local datetime, weekday, timezone, and epoch milliseconds. + +### web_search + +Uses Bing HTML search: + +```text +https://www.bing.com/search?q=...&form=QBRE&pq=...&qs=n&sp=-1&lq=0 +``` + +The parser extracts title, URL, and snippet from `b_algo` blocks. Bing redirect URLs are decoded when possible. + +### browser_url + +Reads an HTTP/HTTPS page and extracts readable text from the HTML body. This is a lightweight browser-readable fallback, not the full GeckoView implementation used by ActMe. + +### python_exec + +Runs bounded Python 3.11 code through Chaquopy and the app's `agent_python.py` sandbox. The sandbox supports deterministic computation, JSON/text processing, reusable scripts, py_compile-style checks through `compile_script(name)`, Excel helpers through `read_excel(path)` and `write_excel(filename, sheets)`, and table/data packages including `numpy`, `pandas`, and `openpyxl`. + +Generated file handoff is collected from: + +- files created or modified inside the Python workspace +- declared `output_files`, `generated_files`, `expected_outputs`, or `files` in tool calls +- workspace-looking file references in the final answer + +Only files resolved inside the agent workspace are returned as chat attachments. + +### Memory and Skill + +Agent JSON may include `memory_updates` and `skill_updates`. MnnLlmChat stores them in SQLite tables managed by `ChatDatabaseHelper` / `ChatDataManager`, injects them into future Agent prompts, and appends matching local skill hints when a user message contains a stored trigger keyword. + +## Budgets and Stop Behavior + +Agent mode uses conservative mobile budgets: + +- max passes +- max total tool calls +- max browser calls +- max Python calls + +Repeated search queries, repeated URLs, and repeated Python snippets are skipped. When the budget is exhausted, the model is asked to produce the best final answer from available observations and not request more tools. + +Stop behavior: + +- User stop sets `stopGenerating`. +- The loop checks stop state before model/tool continuation points. +- User stop returns a stopped result. +- Coroutine cancellation from lifecycle destruction is rethrown instead of being treated as a user stop. + +## Implementation Points + +Key files: + +```text +app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticPrompts.kt +app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticProtocol.kt +app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticOutputParser.kt +app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticToolExecutor.kt +app/src/main/java/com/alibaba/mnnllm/android/agent/AgenticPythonEngine.kt +app/src/main/python/agent_python.py +app/src/main/java/com/alibaba/mnnllm/android/chat/ChatPresenter.kt +app/src/main/java/com/alibaba/mnnllm/android/chat/ChatActivity.kt +app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDatabaseHelper.kt +app/src/main/java/com/alibaba/mnnllm/android/chat/model/ChatDataManager.kt +``` + +Important paths: + +- `ChatActivity` chooses and restores the session mode. +- `ChatPresenter` applies the matching system prompt and chooses normal vs Agent execution. +- `ChatDatabaseHelper` and `ChatDataManager` persist `sessionMode`, Agent Memory, and Agent Skill. +- `AgenticToolExecutor` executes current tools. +- `AgenticPythonEngine` owns Chaquopy startup and bounded Python execution. + +## Relationship to ActMe + +ActMe is the upstream/reference project for the mobile agent design: + +```text +https://github.com/huangzhengxiang/ActMe.git +``` + +ActMe contains the fuller implementation: + +- multi-backend search +- GeckoView-rendered browsing +- Python sandbox and Excel processing +- ADB pairing/execution +- memory, schedule, and skill updates +- richer tool execution UI + +MnnLlmChat currently ports the local-model agent loop, browser/time tools, Python execution, and lightweight Skill/Memory persistence. Future work can incrementally add the remaining ActMe capabilities while keeping normal chat stable. + +## Known Limits + +- Search and page reading are lightweight and may fail on CAPTCHA, login, heavy JavaScript, or anti-bot pages. +- Python execution is enabled through Chaquopy and the ActMe-derived `agent_python.py` sandbox. +- No ADB integration in this port. +- No cross-process task resume yet. +- Local models may still emit imperfect JSON; parser fallbacks reduce but do not eliminate this risk. diff --git a/apps/Android/MnnLlmChat/docs/builtin_browser_design.md b/apps/Android/MnnLlmChat/docs/builtin_browser_design.md new file mode 100644 index 0000000000..4d07d8221a --- /dev/null +++ b/apps/Android/MnnLlmChat/docs/builtin_browser_design.md @@ -0,0 +1,67 @@ +# Built-in Browser Design + +This document describes the browser capability to port into `MnnLlmChat`. + +Core technical source: `https://github.com/huangzhengxiang/ActMe.git` + +Status: the current MnnLlmChat Agent mode includes a lightweight Bing/HTTP implementation. The full GeckoView-rendered browser flow remains an ActMe reference capability for future porting. + +## Purpose + +Search snippets are often unstable or incomplete. The agent should be able to open a concrete URL, wait for the page to load, read the visible content, and feed that observation back to the model. + +## Capabilities + +The browser layer should support: + +- `web_search(query)`: returns result titles, snippets, and URLs. +- `browser_url(url, goal)`: opens or reads a specific page. +- page title, final URL, load status, and readable text extraction. +- visible step records in chat: started, loaded, failed, skipped. + +## Recommended Runtime + +For China-accessible web behavior, a real browser-backed path is more reliable than plain HTTP scraping for dynamic pages and anti-bot behavior. The preferred long-term options are: + +- GeckoView-backed hidden or debug-visible browser. +- Android WebView fallback for simple pages. +- OkHttp fallback only for static text pages. + +The tool should expose text observations, not screenshots only. If the browser can render a page but text extraction fails, the observation should say so explicitly. + +## Agent Contract + +`browser_url` input: + +```json +{ + "type": "browser_url", + "url": "https://example.com/page", + "goal": "verify the current price table" +} +``` + +Observation: + +```json +{ + "type": "browser_url", + "status": "ok", + "title": "Page title", + "final_url": "https://example.com/page", + "text": "readable page text", + "error": null +} +``` + +## UI Requirements + +Each browser action should be visible: + +- why it was opened +- URL or host +- loading/failure status +- extracted title and short preview +- optional button to open the full browser view + +The debug browser can be hidden by default, but there should be a developer-visible route for diagnosing search and page extraction problems. diff --git a/apps/Android/MnnLlmChat/docs/builtin_python_design.md b/apps/Android/MnnLlmChat/docs/builtin_python_design.md new file mode 100644 index 0000000000..c0c55c209d --- /dev/null +++ b/apps/Android/MnnLlmChat/docs/builtin_python_design.md @@ -0,0 +1,94 @@ +# Built-in Python Design + +This document describes the Python capability in `MnnLlmChat`. + +Status: implemented for Agent mode through `python_exec`, Chaquopy, and the ActMe-derived `agent_python.py` sandbox. + +Core technical source: `https://github.com/huangzhengxiang/ActMe.git` + +## Purpose + +LLMs are weak at deterministic computation and file transformations. A local Python runtime gives the agent a reliable path for: + +- arithmetic and date calculations +- CSV/Excel-style table processing +- JSON cleanup +- HTML/text parsing +- generating files for the user +- checking Python syntax before execution + +## Tool Set + +The current MnnLlmChat port exposes one Agent tool: + +- `python_exec`: run bounded Python code with declared input files and expected outputs. + +Syntax validation is available inside `python_exec` through `compile_script(name)` after the agent stores code with `save_script(name, source)`. + +Bundled packages: + +- `openpyxl` +- `numpy` +- `pandas` + +`numpy` and `pandas` are native/data Python packages. They are installed at build time through Chaquopy's Android-compatible package support, which increases APK size and may increase build time. + +The sandbox import allowlist must stay aligned with installed packages. Current user-visible import roots include: + +- `openpyxl` +- `numpy` +- `pandas` +- `dateutil` +- `et_xmlfile` +- `pytz` +- `tzdata` + +## Execution Boundaries + +The runtime should be local, temporary, and bounded: + +- per-run timeout +- stdout/stderr size limit +- working directory scoped to the current session +- generated files returned through chat attachments +- no background daemon in the first version + +## Agent Contract + +Execute input: + +```json +{ + "type": "python_exec", + "code": "write_excel('sample.xlsx', {'Sheet1': [['Name'], ['Alice']]})", + "input_files": [], + "output_files": ["sample.xlsx"] +} +``` + +Observation: + +```json +{ + "type": "python_exec", + "status": "ok", + "stdout": "{\"ok\": true}", + "stderr": "", + "output_files": [] +} +``` + +## Excel and File Work + +Excel support should be implemented as Python packages plus Android file handoff: + +- Android registers Excel MIME types and imports the file into a chat session. +- The agent receives the local file path as an attachment observation. +- Python reads the workbook, summarizes sheets, performs analysis, and can generate a new workbook. +- Generated files are returned as chat attachments. + +This keeps spreadsheet logic in Python instead of duplicating it in Kotlin. + +Agents may use `pandas` for larger table operations, but should still prefer the built-in helpers `read_excel(path)` and `write_excel(filename, sheets)` for Android file handoff. For small tables, standard-library modules such as `csv`, `json`, `statistics`, and `collections` are often faster to start and easier to sandbox. + +Generated files can be detected from workspace changes or declared by the model through `output_files`, `generated_files`, `expected_outputs`, or `files`. Path normalization is intentionally workspace-scoped: fake absolute paths or `file://` links are only returned when they can be matched back to files inside the agent workspace. diff --git a/apps/Android/MnnLlmChat/docs/development_notes.md b/apps/Android/MnnLlmChat/docs/development_notes.md new file mode 100644 index 0000000000..a564f586c8 --- /dev/null +++ b/apps/Android/MnnLlmChat/docs/development_notes.md @@ -0,0 +1,161 @@ +# Development Notes + +This document records engineering rules and recurring pitfalls for the MnnLlmChat Android Agent work. + +Core technical source for the Agent port: `https://github.com/huangzhengxiang/ActMe.git` + +## Non-Negotiable Development Rules + +- Keep normal chat and Agent chat as separate session modes. A conversation's mode is chosen when the session is created and is persisted as `Session.sessionMode`. +- Do not route normal chat through the Agent loop. Normal chat must keep the original single-pass path. +- Do not rebuild the prompt from database history on every local ChatActivity turn. The live native `LlmSession` owns active context and KV/prompt-cache state. +- Do not run Gradle builds from automation unless explicitly requested. Many dependency and native build steps are slow or environment-sensitive. +- Prefer small, isolated changes. Agent behavior touches prompt design, parser fallback, tool execution, UI status, file handoff, and native session behavior. + +## Session and History + +There are two different concepts of history: + +- UI/database history: `ChatDataManager.getChatDataBySession(...)`, used for visible conversation history and history list restore. +- Native runtime context: `LlmSession.history_`, held by the active native MNN session and used for generation. + +Important implications: + +- For an active live session, each new user turn should submit only the new input. The native session already has previous context. +- When reopening a historical session after native release, visible database history must be loaded into `LlmSession.savedHistory`. +- Cold restore may pass a bounded, alternating user/assistant history list into native initialization so the model has conversational context again, but this is not the same as persisted KV cache. +- Prompt cache is currently in-memory. Persisting prompt/KV cache by `sessionId` requires explicit native save/load support and should be treated as a separate feature. + +## Agent Prompt Contract + +`AgenticPrompts.kt` is a runtime API contract, not only prompt text. Any change here must be checked against: + +- `AgenticProtocol.kt` +- `AgenticOutputParser.kt` +- `AgenticToolExecutor.kt` +- existing logcat examples from real local models + +Current tool calls should use: + +```json +{ + "reply": "", + "memory_updates": [], + "skill_updates": [], + "system_calls": [ + { + "type": "python_exec", + "code": "write_excel('sample.xlsx', {'Sheet1': [['Name'], ['Alice']]})", + "input": "", + "timeout_ms": 15000, + "output_files": ["sample.xlsx"] + } + ] +} +``` + +Do not add decorative fields unless the executor uses them. The `reason` field was intentionally removed from the advertised protocol because local models tended to overfit to it and it did not improve execution. + +## Parser Robustness + +Local models often produce imperfect JSON. `AgenticOutputParser.kt` should keep accepting: + +- strict JSON object +- fenced JSON blocks +- JSON embedded in normal text +- single tool object, such as `{"type":"python_exec","code":"..."}` +- `system_calls` as either an array or a single object +- string arrays for `memory_updates` or `skill_updates` +- truncated or malformed `python_exec` output when `type` and `code` can still be extracted safely + +The parser should prefer recovering tool calls over failing the whole response because a secondary field is malformed. + +## Python Sandbox + +Python execution is implemented through Chaquopy and `app/src/main/python/agent_python.py`. + +When adding or removing packages in `app/build.gradle`: + +- update `ALLOWED_IMPORT_ROOTS` in `agent_python.py` +- update `AgenticPrompts.kt` +- update `docs/builtin_python_design.md` +- verify common transitive import roots if users are likely to import them directly + +Current installed packages: + +- `openpyxl` +- `numpy` +- `pandas` + +Current user-visible allowlist should include those package roots and common runtime dependencies such as: + +- `dateutil` +- `et_xmlfile` +- `pytz` +- `tzdata` + +Security boundaries: + +- Keep `os`, `sys`, `subprocess`, `socket`, `shutil`, `pathlib`, `multiprocessing`, and `ctypes` denied for user code. +- File access must stay inside `AgenticPythonEngine.workspaceDir(...)`. +- Relative file paths should resolve inside the agent workspace. +- Prefer `write_excel(filename, sheets)` for simple workbook creation because it handles Android file handoff cleanly and starts faster than pandas. + +## Generated Files + +Generated files are returned through `ChatFileAttachment`. + +The attachment pipeline has three sources: + +- Python runner snapshot diff: files created or modified inside the workspace. +- Declared output references in the tool call: `output_files`, `generated_files`, `expected_outputs`, or `files`. +- Final answer references: workspace file names or `file://...` links found in the final text. + +Path normalization must never expose arbitrary filesystem paths. A file reference should only become an attachment if it resolves inside the agent workspace, or if a matching basename is found inside that workspace. + +## Browser and Search + +The current MnnLlmChat port uses lightweight Bing HTML search and HTTP page text extraction. + +Development caveats: + +- Browser/search behavior is network- and region-sensitive. +- Search result parsing must be debugged with the actual request URL and returned HTML shape. +- Dynamic pages, CAPTCHA, login walls, and heavy JavaScript pages may need a future GeckoView-backed implementation. +- Do not claim full browser parity with ActMe until rendered browsing and robust page text extraction are ported. + +## UI and Visibility + +Agent mode must remain visible, controllable, resumable where possible, and interruptible: + +- show tool step progress in the assistant message/process area +- preserve user stop behavior +- avoid exposing raw JSON dictionaries as final answers +- return generated files as clickable attachments +- keep workspace browsing accessible from the history drawer + +For orientation, main chat flows should remain portrait unless a feature explicitly needs another mode. Video/debug/scanner flows may keep their own behavior. + +## Common Failure Patterns + +- Model emits JSON in a fenced code block: parser should strip fences. +- Model emits a single tool object instead of full `system_calls`: parser should wrap it. +- Model emits memory/skill string arrays: parser should not reject the tool call. +- Model says it created a file but did not run code: prompt should tell Python code to directly perform the action. +- Model defines a Python function but does not call it: this is a model behavior issue; prompt and examples should avoid function-only snippets for simple tasks. +- Python package is installed but blocked by sandbox: update both Chaquopy dependencies and `ALLOWED_IMPORT_ROOTS`. +- File link uses a fake path like `file:///samples/a.xlsx`: normalize to workspace path and fallback to basename search. +- Reopened Agent session has no visible history: ensure DB history is passed into `LlmSession.savedHistory`. +- Reopened Agent session has no context: ensure cold native restore receives bounded alternating user/assistant history. + +## Testing Without Full Builds + +When avoiding Gradle builds, use static checks: + +- `rg` for field names and protocol drift. +- inspect manifest changes directly. +- inspect Kotlin call sites after changing data classes. +- inspect Python sandbox allowlist after changing Chaquopy packages. +- use logcat examples to validate parser fallbacks conceptually. + +Full validation still requires a device install and real local-model trials because local model formatting behavior is model-specific. diff --git a/apps/Android/MnnLlmChat/docs/iteration_roadmap.md b/apps/Android/MnnLlmChat/docs/iteration_roadmap.md new file mode 100644 index 0000000000..5ebf9431e1 --- /dev/null +++ b/apps/Android/MnnLlmChat/docs/iteration_roadmap.md @@ -0,0 +1,151 @@ +# Iteration Roadmap + +This document describes how to continue evolving MnnLlmChat's Agent capabilities without destabilizing normal local chat. + +Core technical source for the current Agent direction: `https://github.com/huangzhengxiang/ActMe.git` + +## Product Direction + +MnnLlmChat should remain a local on-device model application first. + +Agent mode should add host-executed capabilities around the model: + +- web search and page reading +- local Python computation and file generation +- lightweight memory and reusable skills +- visible multi-step execution +- safe file handoff back to the user + +Normal chat should remain simple, fast, and compatible with older conversations. + +## Iteration Principles + +- Keep the normal path stable. Every Agent feature must be optional at the session level. +- Prefer host-side deterministic tools over asking the model to simulate exact work. +- Make every tool action visible to the user. +- Keep parser compatibility broad because local models do not reliably follow a single JSON shape. +- Treat prompt text, protocol classes, parser fallback, executor behavior, and docs as one contract. +- Add capability in vertical slices: prompt, protocol, parser, executor, UI, persistence, docs. + +## Suggested Milestones + +### 1. Stabilize Current Agent Loop + +Goals: + +- final answers should not show raw JSON +- generated files should always appear as attachments when present +- stop/cancel should not corrupt native history +- reopened Agent sessions should restore visible history and useful context + +Key work: + +- keep expanding parser fallbacks from real logcat failures +- add focused unit-style parser tests if the project test setup becomes practical +- keep tool budgets conservative on mobile + +### 2. Improve Python and File Workflows + +Goals: + +- make Excel/CSV/JSON workflows reliable +- make generated files easy to inspect and open +- reduce cold-start friction for common data tasks + +Key work: + +- keep `ALLOWED_IMPORT_ROOTS` aligned with Chaquopy packages +- add examples that prefer `write_excel(...)` and `read_excel(...)` +- improve workspace file browser UX +- consider per-session workspace directories if global workspace clutter becomes a problem +- add explicit cleanup/export controls for workspace files + +### 3. Upgrade Browser Capability + +Goals: + +- handle dynamic pages and pages that plain HTTP extraction cannot read +- make search/browse behavior easier to debug + +Possible paths: + +- port a GeckoView-backed browser layer from ActMe +- keep a debug-visible browser activity for inspecting actual rendered pages +- expose page title, final URL, selected text, and readable body text to the agent +- keep Bing as one search source, but design executor interfaces so other sources can be added later + +### 4. Mature Skill and Memory + +Goals: + +- keep useful user preferences and workflows +- avoid saving noisy or hallucinated memory +- let users inspect and control stored information + +Key work: + +- add Memory/Skill management UI +- allow disabling or deleting skills +- require better structure for executable skills before automatic execution +- store source/session metadata for memory updates +- add confidence or review state for uncertain memory + +### 5. Add Durable Task State + +Goals: + +- make long multi-step tasks resumable after app interruption +- make tool observations inspectable after completion + +Key work: + +- persist agent step logs +- persist tool observations +- persist generated file metadata +- design resume semantics separately from native KV cache +- avoid assuming a released native session can continue with the same hidden context + +### 6. Consider Advanced Device Control + +ADB/device control is intentionally not part of the current MnnLlmChat port. + +If ported later, it should be treated as a high-risk capability: + +- explicit user opt-in +- clear pairing UI +- visible command log +- stop button +- strict command allowlist or policy layer +- no hidden background control without user awareness + +## Technical Debt to Watch + +- Parser fallbacks can become hard to reason about. Keep examples in docs and add tests when possible. +- Prompt contract drift is easy. Update prompt, protocol, parser, executor, and docs together. +- Python dependencies increase APK size and build time. +- Native session reuse and history restore are subtle. Do not mix API stateless history submission with ChatActivity live-session semantics. +- Workspace files can accumulate. Add cleanup/export flows before the workspace becomes user-visible clutter. +- Browser scraping is brittle. Treat search/page failures as expected states, not exceptional app failures. + +## Release Checklist for Agent Changes + +- Normal chat still works without Agent mode. +- New conversations can choose normal or Agent mode. +- Reopened history restores the correct mode. +- Agent status steps are visible. +- Stop button interrupts the loop. +- Tool JSON is not shown as final answer. +- Python package allowlist matches installed packages. +- Generated files are attached and openable. +- Workspace file browser shows folders and files. +- Prompt cache behavior is documented if changed. +- README/docs are updated for any new tool, field, or user-visible workflow. + +## Documentation Map + +- `agentic_design.md`: current Agent architecture and loop. +- `builtin_browser_design.md`: web search and browser capability. +- `builtin_python_design.md`: Python runtime, sandbox, Excel/file handling. +- `skill_memory_design.md`: Memory and Skill persistence. +- `development_notes.md`: development rules and pitfalls. +- `iteration_roadmap.md`: future direction and staged work. diff --git a/apps/Android/MnnLlmChat/docs/skill_memory_design.md b/apps/Android/MnnLlmChat/docs/skill_memory_design.md new file mode 100644 index 0000000000..4149cdfebe --- /dev/null +++ b/apps/Android/MnnLlmChat/docs/skill_memory_design.md @@ -0,0 +1,65 @@ +# Skill and Memory Design + +This document adapts ActMe's skill and memory concepts to `MnnLlmChat`. + +Core technical source: `https://github.com/huangzhengxiang/ActMe.git` + +Status: implemented as a lightweight MnnLlmChat SQLite port. Agent mode persists `memory_updates` and `skill_updates`, injects them into future Agent system prompts, and applies local skill hints when trigger keywords match. + +## Memory + +Memory records durable facts about the user or project: + +- goals +- preferences +- recurring context +- recent active projects +- constraints the assistant should remember + +Memory should be injected into the system prompt selectively. It should not blindly store every chat turn. + +Minimal schema: + +```json +{ + "category": "goal", + "content": "User is preparing for an exam.", + "source": "chat", + "updated_at": 1780502400000 +} +``` + +## Skill + +Skill records reusable ways of doing work: + +- study planning workflow +- web research workflow +- spreadsheet analysis workflow +- PDF/report generation workflow + +Minimal schema: + +```json +{ + "name": "spreadsheet_analysis", + "description": "Inspect workbook sheets, summarize columns, run Python analysis, and return a result file.", + "trigger_keywords": ["excel", "xlsx", "spreadsheet"], + "action_template": "Inspect sheets first, then run Python, then explain and attach outputs.", + "enabled": true +} +``` + +## Relation to Agentic Loop + +Memory and skill are prompt context. The loop decides what to do next: + +- memory tells the agent who the user is and what matters. +- skill tells the agent which workflow to prefer. +- tools execute the concrete browser or Python step. + +## Safety + +The user should be able to inspect, disable, or delete memory and skills. Sensitive or uncertain information should not be saved automatically. + +In this MnnLlmChat port, Memory and Skill are persisted in the existing SQLite chat database. A future UI can expose inspection, disable, and deletion controls. diff --git a/apps/Android/MnnLlmChat/gradle/wrapper/gradle-wrapper.properties b/apps/Android/MnnLlmChat/gradle/wrapper/gradle-wrapper.properties index 09523c0e54..e2847c8200 100644 --- a/apps/Android/MnnLlmChat/gradle/wrapper/gradle-wrapper.properties +++ b/apps/Android/MnnLlmChat/gradle/wrapper/gradle-wrapper.properties @@ -1,6 +1,6 @@ distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists -distributionUrl=https\://services.gradle.org/distributions/gradle-8.9-bin.zip +distributionUrl=https\://services.gradle.org/distributions/gradle-8.11.1-bin.zip networkTimeout=10000 validateDistributionUrl=true zipStoreBase=GRADLE_USER_HOME diff --git a/express/Expr.cpp b/express/Expr.cpp index 12f404d213..8632a1e70c 100644 --- a/express/Expr.cpp +++ b/express/Expr.cpp @@ -1063,7 +1063,8 @@ std::vector Variable::load(const uint8_t* buffer, size_t length) { // are metadata-only for these; CONST/TRAIN nodes don't enter this branch, // their weight-bearing host is preserved by Expr::create), so restoring // MEMORY_HOST cannot cause a double-free on destruction. - TensorUtils::getDescribe(expr->inside()->mOutputTensors[index])->memoryType = Tensor::InsideDescribe::MEMORY_HOST; + TensorUtils::getDescribe(expr->inside()->mOutputTensors[index])->memoryType = + Tensor::InsideDescribe::MEMORY_HOST; Utils::copyTensorToInfo(expr->inside()->mOutputInfos.data() + index, expr->inside()->mOutputTensors[index]); } } else if (expr->inputType() == VARP::INPUT) { diff --git a/test/expr/LoadMapInputTest.cpp b/test/expr/LoadMapInputTest.cpp index 7e07339cdf..e702618251 100644 --- a/test/expr/LoadMapInputTest.cpp +++ b/test/expr/LoadMapInputTest.cpp @@ -82,8 +82,8 @@ class LoadMapOutputReadMapTest : public MNNTestCase { auto varMap = Variable::loadMap("tmp/regression_4750.mnn"); auto io = Variable::getInputAndOutput(varMap); if (io.first.size() != 1 || io.second.size() != 1) { - MNN_PRINT("LoadMapOutputTest: expected single input/output, got %zu/%zu\n", - io.first.size(), io.second.size()); + MNN_PRINT("LoadMapOutputTest: expected single input/output, got %zu/%zu\n", io.first.size(), + io.second.size()); return false; } auto input = io.first.begin()->second;