Run Full Benchmark Suite

Overview

Flow ID: run-full-benchmark
Category: Performance Benchmarking
Estimated Duration: 15-60 minutes (depends on models and hardware)
User Role: All Users (though technical users benefit most)
Complexity: Moderate

Purpose: This flow enables users to run comprehensive performance tests on their AI models to understand how well they perform on their specific hardware. Benchmarking measures speed, throughput, latency, and resource usage across various test scenarios, helping users make informed decisions about model selection and configuration.

Related Flows

• View Benchmark Results - Analyze completed benchmark data
• Export Benchmark Results - Save results to file
• Cancel Running Benchmark - Stop benchmark mid-run
• Resume Incomplete Benchmark - Continue interrupted benchmark
• Upload Large Language Model - Add models to test
• Upload Embedding Model - Add embedding models
• Configure Context Window Size - Optimize based on benchmark

Prerequisites

Before starting, users must have:

• Application installed and running
• At least one chat model uploaded
• At least one embedding model uploaded (for full suite)
• Optional: Dataset uploaded for embedding tests
• Time to let tests run uninterrupted (15-60 minutes)
• Application window should remain open and in focus (recommended)

Step-by-Step Flow

Main Path (Happy Path)

Alternative Paths & Strategies

Strategy A: Continue Partial Benchmark

When to use: Benchmark was interrupted previously

Steps:

1. Navigate to Benchmarking tab
2. See incomplete test results with some tests marked "Cancelled" or not started
3. Click "Continue Benchmark" button (if available)
4. System resumes from where it stopped
5. Only incomplete tests re-run
6. Total time reduced compared to full re-run

Strategy B: Quick Single-Test Benchmark

When to use: Just want to test one specific scenario

Steps:

1. Navigate to Benchmarking
2. Expand test category
3. Click "Re-test" on specific test row
4. Single test runs
5. Results update for just that test
6. Much faster than full suite (30 seconds - 2 minutes)

Strategy C: Model-Specific Benchmark

When to use: Only care about one model (LLM or embedding), not both

Steps:

1. Start benchmark normally
2. When first model completes, click "Stop Benchmark"
3. Accept partial results
4. Complete results for first model
5. Second model tests remain as "Not Started"
6. Saves time if only testing one model

Strategy D: Background Benchmark

When to use: Want to work on other tasks while benchmark runs

Steps:

1. Start benchmark
2. Navigate to other pages or minimize application
3. Benchmark continues in background
4. Periodically check progress via status badge or returning to benchmarking page
5. Return when complete to view results

QA Note: Background operation may be slower or less reliable. Keeping application in focus recommended.

Error States & Recovery

Version History

Notes

Important Considerations:

• Benchmark runs CPU and GPU at high utilization; computer may become hot or loud
• Results are saved and used by system for optimizing future performance
• First benchmark provides baseline; can re-run to track performance over time
• Different models will have vastly different results (expected)
• Tests are designed to stress different aspects of model performance
• Keep application in focus and avoid interruptions for best results

Test Categories Explained:

LLM Tests:

• Context Size: Tests handling different amounts of conversation history
• Non-Streaming: Tests batch processing mode (whole response at once)
• Chat History: Tests multi-turn conversation performance
• VRAM Override: Tests different memory allocations to find optimal

Embedding Tests:

• Embedding Sizes: Tests processing different text lengths
• Bulk Embedding: Tests parallel processing with multiple workers (NPU and CPU)

Metrics Explained:

• Tokens per second: How fast the model generates text (higher is better)
• First token time: Latency before response starts (lower is better)
• Load time: How long model takes to initialize (lower is better)
• CPU load: Processor usage during test (lower is better if adequate performance)
• Embeddings per second: Throughput for embedding generation (higher is better)

Best Practices:

• Run benchmarks when not using computer for other tasks
• Close other applications before benchmarking for accurate results
• Run benchmark after uploading new models to understand their performance
• Keep benchmark results for documentation and comparison
• Re-benchmark periodically to track performance changes
• Use results to guide context window, VRAM, and worker configuration

Common User Questions:

• "How long does benchmark take?" - Typically 15-45 minutes depending on models and hardware
• "Can I stop and resume?" - Yes, use "Continue Benchmark" after stopping
• "Do I need to benchmark every model?" - Recommended, but not required; helps optimize settings
• "What's a good tokens-per-second rate?" - Varies by hardware; 10-50 tokens/sec is typical for consumer hardware
• "Why did some tests fail?" - Tests may exceed hardware capabilities or timeout; this helps identify limits
• "Can I use the application while benchmarking?" - Technically yes, but will slow benchmark and may affect results

Trigger

What initiates this flow:

• User manually initiates

Specific trigger: User wants to test model performance, typically because:

• They just uploaded a new model and want to know how it performs
• They want to compare different models to choose the best one
• They're experiencing performance issues and want diagnostic data
• They want to optimize settings (context window, memory allocation)
• They're documenting system capabilities for deployment planning

User Intent Analysis

Primary Intent

Measure and document how well AI models perform on the current hardware to make informed decisions about model selection, configuration, and deployment.

Secondary Intents

• Compare multiple models to identify the best one
• Identify performance bottlenecks
• Validate hardware capabilities before production use
• Generate performance reports for documentation
• Optimize settings based on measured data
• Troubleshoot slow performance issues

Subintents

• Understand tokens per second (speed) for different scenarios
• Measure memory usage and resource consumption
• Test context window handling
• Validate stability under load
• Collect data for future reference