Home
Back to Overview

Streaming & Response Control

Real-time responses and controlled output for better UX

Implement streaming responses and token limits for better user experience

The Streaming Problem

❌ Traditional (Non-Streaming)

User sends prompt
↓
[Wait 10 seconds...]
[Wait...]
[Wait...]
Complete response appears all at once
  • ❌ Poor user experience
  • ❌ No progress indication
  • ❌ Can't interrupt bad responses

✅ Streaming Approach

User sends prompt
↓
"Hoisting" (0.1s) → User sees first word!
"is a" (0.2s) → More text appears
"JavaScript" (0.3s) → Continuous feedback
[Continues token by token...]
  • ✅ Immediate feedback
  • ✅ Progress visible
  • ✅ Can interrupt early

How Streaming Works

Token-by-Token Generation

Internal LLM Process:
Token 1: "Hoisting"
Token 2: "is"
Token 3: "a"
Token 4: "JavaScript"
Token 5: "mechanism"
...
Without Streaming
1. Wait for all tokens
2. Buffer complete response
3. Return all at once
User waits until complete
With Streaming
1. Emit each token immediately
2. Trigger callback per token
3. Display progressively
User sees response in real-time

The onTextChunk Callback

1
Model Generation

LLM generates tokens one at a time

↓
2
Each New Token

Token is emitted as soon as it's generated

↓
3
onTextChunk(text) Callback

Your callback function is invoked with the new token

onTextChunk: (text) => { ... }
↓
4
Your Code Processes It
Display to User
Update UI in real-time
Send Over Network
Stream to client
Log to File
Save incrementally

Token Limits: maxTokens

Without Limits

User: "Explain hoisting"
Model: [Generates 10,000 words including:
- Complete JavaScript history
- Every edge case
- Never stops...]

With Limits

User: "Explain hoisting"
Model: [Generates ~1500 words
- Core concept
- Key examples
- Stops at 2000 tokens]

Token Budgeting

Context Window: 4096 tokens
├─ System Prompt: 200 tokens
├─ User Message: 100 tokens
├─ Response (maxTokens): 2000 tokens
└─ Remaining for history: 1796 tokens

Real-Time Applications

Pattern 1: Interactive CLI

Terminal-based applications with real-time output

User: "Explain closures"
↓
Terminal: |
"A closure is a function..."
(Appears word by word, like typing)
Use Case: Command-line tools, REPLs, interactive debugging assistants

Pattern 2: Web Application

Browser-based streaming with server communication

Browser
Send prompt →
← Chunk 1: "Closures"
← Chunk 2: "are"
← Chunk 3: "functions"
← Chunk 4: "that"
← Chunk 5: "..."
Server
← Send prompt
→ Chunk 1: "Closures"
→ Chunk 2: "are"
→ Chunk 3: "functions"
→ Chunk 4: "that"
→ Chunk 5: "..."
Implementation Methods:
SSE
WebSockets
HTTP Stream
Use Case: Chat interfaces, AI assistants, real-time content generation

Key Benefits of Streaming

⚡ Perceived Performance
Users see results immediately, not after full generation
🔄 Progressive Loading
Content appears incrementally, improving UX
⏱️ Lower Latency
Time to first token is much faster than full response

Performance Characteristics

Time to First Token (TTFT)

Small model (1.7B): ~100ms
Medium model (8B): ~200ms
Large model (20B): ~500ms

TTFT < 500ms → Feels instant

Tokens Per Second

Small model: 50-80 tok/s
Medium model: 20-35 tok/s
Large model: 10-15 tok/s

Tok/s > 20 → Reads naturally

💻 Code Example: Implementing Streaming

Basic Streaming Pattern

// Local LLM (node-llama-cpp)
let fullResponse = '';
await session.prompt(query, {
maxTokens: 2000,
onTextChunk: (chunk) => {
fullResponse += chunk;
process.stdout.write(chunk); // Display immediately
}
});
// OpenAI API
const stream = await client.chat.completions.create({
model: 'gpt-4o',
messages: [{ role: 'user', content: query }],
stream: true,
});
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta?.content || '';
process.stdout.write(content);
}

⚠️ Handling Errors During Streaming

Common Issues

Problem: Stream interrupted

Solution: Wrap in try-catch, save partial response

Problem: User closes connection

Solution: Handle connection errors gracefully

Problem: Token limit reached mid-stream

Solution: Check if response is complete, handle truncation

Best Practices

1. Always Set maxTokens

✓ Good: session.prompt(query, { maxTokens: 2000 })
✗ Risky: session.prompt(query)
May use entire context!

2. Handle Partial Updates

let fullResponse = '';
onTextChunk: (chunk) => {
fullResponse += chunk;
display(chunk); // Show immediately
}

3. Provide Feedback

Show loading state until first chunk arrives, then update in real-time

Key Takeaways

⚡

Streaming improves UX: Users see progress immediately

🎯

maxTokens controls cost: Prevents runaway generation

🔤

Token-by-token generation: LLMs produce one token at a time

🔌

onTextChunk callback: Your hook into the generation process

Next: Function Calling