Managing Concurrency in LLM-powered Ambient Agents

Introduction

• Brief intro to Revo.pm: “We’re building an ambient AI Copilot powered by LLMs”
• Core challenge: “LLMs need uninterrupted context to maintain coherent reasoning, but our agent handles multiple concurrent events”

The Problem: LLM Context Management

Real-world Example

• “Let me show you how LLM context gets corrupted…”
• Demo scenario: Website monitoring + reminder conflict
• Highlight the LLM-specific failures:
  • Context window pollution from parallel requests
  • Chain-of-thought reasoning breaks
  • Token waste from repeated prompting
  • Example prompt corruption:

    Initial prompt: "Monitor website hourly..."  
    Corrupted context: [Previous reminder data mixed in]  
    Result: LLM loses track of original task  
    

Our Solution: LLM-aware Concurrency

1. Intelligent Lock System with Context Preservation

# Show the lock acquisition with LLM context management  
Log.Information("acquiring lock for {@llmContext}", ctxLogObject);  
await using var @lock = await context.AcquireLock(copilot, cancellationToken);  
// Preserve LLM conversation history and embeddings  
await context.PreserveLLMState(copilot.CurrentConversation);  

• Key point: “Maintains clean LLM context for each interaction”

2. LLM-aware State Machine

public interface IRevoStateMachine<TState> {  
    // State transitions consider LLM context  
    Task<TState> Reduce(State currentState, RevoChainOfThought chainOfThought);  
    // Handle different types of LLM interactions  
    Task<TState> Reduce(State currentState, LLMResponse response);  
    Task<TState> Reduce(State currentState, UserMessage message);  
}  

• Demo: Show how state transitions preserve LLM reasoning

3. Chain of Thought Management

public interface IRevoChainOfThoughtProvider {  
    // Manages LLM reasoning across events  
    Task<RevoChainOfThought> GetChainOfThought(  
        IRevoState state,  
        LLMContext context,   
        CancellationToken cancellationToken = default  
    );  
}  

• Demo the LLM reasoning flow:
  • Context window management
  • Embedding preservation
  • Multi-step reasoning across events

Results & LLM Performance Monitoring

• Show monitoring dashboard with LLM metrics:
  • Token usage optimization:
    • Before: 2.3M tokens/day wasted on context reloading
    • After: 850K tokens/day saved
  • Reasoning consistency:
    • 97% reduction in context-related reasoning failures
    • 99.9% task completion rate
  • Response latency:
    • 300ms average lock acquisition
    • No impact on LLM response time
  • Cost implications:
    • 40% reduction in token costs
    • Eliminated duplicate LLM calls

Wrap-up & Questions

• Core benefits for LLM systems:
  • Preserved reasoning chains
  • Optimized token usage
  • Reliable concurrent operations
• Technical innovations:
  • LLM-aware state management
  • Context preservation system
  • Efficient token utilization

Speaker Notes

• Focus on LLM-specific challenges and solutions
• Show real monitoring data when discussing metrics
• Prepare for questions about:
  • LLM context window management
  • Token optimization strategies
  • Scaling with multiple LLM providers
  • Handling different types of LLM models
  • Cost optimization techniques
• Have specific examples ready for:
  • Before/after prompts
  • Token usage graphs
  • Reasoning chain preservation