Skip to content

PERFORMANCE_LOGGING.md

Latest commit

 

History

History
119 lines (94 loc) · 3.54 KB

PERFORMANCE_LOGGING.md

File metadata and controls

119 lines (94 loc) · 3.54 KB
Raw

LLM Response Time Logging Implementation

✅ Features Added

1. Comprehensive Response Time Logging

LLM Handler Level (llm_handler.py)

  • Connection Test Timing: Logs initial LLM connection test time
  • Individual Query Timing: Times each LLM API call precisely
  • Error Timing: Tracks response times even for failed requests
  • Performance Statistics: Tracks aggregate performance metrics

Example Log Output:

2025-06-27 17:15:16,569 - llm_handler - INFO - LLM connection test successful in 1.01s
2025-06-27 17:15:26,342 - llm_handler - INFO - LLM query: 'what are three colors?...' -> response length: 45 chars, time: 9.76s

Bot Level (bot.py)

  • Total Processing Time: Measures end-to-end request processing
  • Includes Context Gathering: Times the complete flow from question to response
  • Error Handling: Logs timing information for failed requests

Example Log Output:

2025-06-27 17:15:26,343 - bot - INFO - Total request processing time for 'what are three colors?...': 9.77s

2. Performance Statistics Tracking

Metrics Collected:

  • Total successful requests
  • Total failed requests
  • Success rate percentage
  • Average response time
  • Min/max response times
  • Recent request sample size (last 100 requests)

New Bot Command: !performance or !perf

Shows comprehensive performance statistics in IRC:

📊 LLM Performance Stats:
• Total requests: 3
• Failed requests: 0
• Success rate: 100.0%
• Average response time: 7.19s
• Response time range: 5.09s - 8.99s
• Recent sample size: 3 requests

3. Memory-Efficient Design

  • Keeps only the last 100 response times to prevent memory growth
  • Lightweight performance tracking with minimal overhead
  • Automatic cleanup of old performance data

4. Updated Help System

  • Added !performance command to help text
  • Updated bot command documentation

Technical Implementation

Timing Precision

  • Uses time.time() for microsecond-level precision
  • Measures pure API call time vs. total processing time
  • Distinguishes between LLM response time and bot overhead

Error Resilience

  • Tracks timing even when LLM calls fail
  • Maintains performance statistics during errors
  • Logs helpful error information with timing context

Low Overhead

  • Minimal performance impact (< 0.01s overhead shown in tests)
  • Efficient data structures for statistics
  • Non-blocking performance tracking

Benefits for Operations

Performance Monitoring

  • Track LLM response time trends
  • Identify slow queries
  • Monitor success rates
  • Debug performance issues

User Experience

  • Users can check bot performance with !performance
  • Transparent response time information
  • Clear error messaging with timing context

Development & Debugging

  • Detailed timing logs for troubleshooting
  • Performance regression detection
  • API response time monitoring

Example Usage

For Operators

# Check logs for performance issues
grep "LLM query" bot.log | tail -10

# Monitor average response times
grep "Total request processing" bot.log

For Users

<user> !performance
<bot> 📊 LLM Performance Stats:
<bot> • Total requests: 25
<bot> • Failed requests: 1
<bot> • Success rate: 96.0%
<bot> • Average response time: 8.34s
<bot> • Response time range: 3.21s - 15.67s
<bot> • Recent sample size: 25 requests

The implementation provides comprehensive performance visibility while maintaining the bot's responsiveness and reliability.