Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/mkdocs/en/model.md
Original file line number Diff line number Diff line change
Expand Up @@ -1843,6 +1843,14 @@ when the same estimates are needed.

Token Tailoring is an intelligent message management technique designed to automatically trim messages when they exceed the model's context window limits, ensuring requests can be successfully sent to the LLM API. This feature is particularly useful for long conversation scenarios, allowing you to keep the message list within the model's token limits while preserving key context.

Token Tailoring is **disabled by default**. Explicitly set
`openai.WithEnableTokenTailoring(true)` to enable automatic message trimming
before provider requests. Setting `openai.WithMaxInputTokens(...)` or
`openai.WithContextWindow(...)` alone only configures the budget or window; it
does not enable trimming. These values remain available to other framework
consumers through `Model.InputTokenBudget` and `Model.Info` even when trimming is
disabled. Session summaries and context compaction are configured separately.

**Automatic Mode (Recommended)**:

```go
Expand Down
6 changes: 6 additions & 0 deletions docs/mkdocs/zh/model.md
Original file line number Diff line number Diff line change
Expand Up @@ -1806,6 +1806,12 @@ summarizer := summary.NewSummarizer(

Token Tailoring 是一种智能的消息管理技术,用于在消息超出模型上下文窗口限制时自动裁剪消息,确保请求能够成功发送到 LLM API。该功能特别适用于长对话场景,能够在保留关键上下文的同时,将消息列表控制在模型的 token 限制内。

Token Tailoring **默认关闭**。如需在请求发送给模型服务前自动裁剪消息,必须显式设置
`openai.WithEnableTokenTailoring(true)`。仅设置 `openai.WithMaxInputTokens(...)`
或 `openai.WithContextWindow(...)` 只会配置预算或窗口,不会开启裁剪。即使关闭裁剪,
其他框架流程仍可通过 `Model.InputTokenBudget` 和 `Model.Info` 读取这些配置。
会话摘要和上下文压缩需要单独配置。

**自动模式(推荐)**:

```go
Expand Down
23 changes: 17 additions & 6 deletions model/openai/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -369,18 +369,25 @@ func WithBatchBaseURL(url string) Option {
}
}

// WithEnableTokenTailoring enables automatic token tailoring based on model context window.
// When enabled, the system will automatically calculate max input tokens using the model's
// context window minus reserved tokens and protocol overhead.
// WithEnableTokenTailoring controls automatic message trimming before sending a
// request to the model provider. It is disabled by default.
// Setting WithMaxInputTokens or WithContextWindow alone does not enable trimming.
// When enabled, trimming uses Model.InputTokenBudget, which derives the budget
// from the context window unless a positive WithMaxInputTokens limit is set.
func WithEnableTokenTailoring(enabled bool) Option {
return func(opts *options) {
opts.EnableTokenTailoring = enabled
}
}

// WithMaxInputTokens sets only the input token limit for token tailoring.
// The counter/strategy will be lazily initialized if not provided.
// Defaults to SimpleTokenCounter and MiddleOutStrategy.
// WithMaxInputTokens sets the messages input budget used by Model.InputTokenBudget
// and token tailoring. A non-positive limit derives the budget from the context
// window instead.
// This option does not enable automatic trimming; use
// WithEnableTokenTailoring(true) to apply trimming before provider requests.
// The input budget remains available to other framework consumers when trimming
// is disabled. The default counter and strategy are SimpleTokenCounter and
// MiddleOutStrategy.
func WithMaxInputTokens(limit int) Option {
return func(opts *options) {
opts.MaxInputTokens = limit
Expand All @@ -389,6 +396,10 @@ func WithMaxInputTokens(limit int) Option {

// WithContextWindow sets the model context window size in tokens for this
// model instance.
// This option does not enable automatic trimming; use
// WithEnableTokenTailoring(true) to apply trimming before provider requests.
// The window is also exposed through Model.Info for other framework consumers,
// such as context-threshold summaries, regardless of the trimming setting.
func WithContextWindow(tokens int) Option {
return func(opts *options) {
if tokens > 0 {
Expand Down
Loading