Strata: Run 125B LLM on Your PC
Run a 125-billion-parameter AI model on your gaming PC with Strata - free, private, and fast.
Have you ever wanted to run a powerful AI model like GPT-4 on your own computer? Until recently, this was impossible - 125-billion-parameter models required expensive servers and specialized GPUs. But with Strata, you can now run Qwen3.8-Flash-Next, a 125B AI model, right on your gaming PC.
Why This Matters#
Imagine this: You’re coding and need AI assistance. Every time you send a prompt to OpenAI, Anthropic, or other cloud APIs, you’re dealing with:
- Cost: API bills (thousands of dollars per month with heavy usage)
- Privacy: Your code and data sent to third-party servers
- Latency: Network delays, sometimes server overload
- Dependency: Reliance on internet and external services
Strata solves all these problems by bringing the power of large AI models directly to your machine.
What is Strata?#
Strata is a C++ inference engine that lets you run Qwen3.8-Flash-Next - a 125-billion-parameter AI model - on ordinary hardware:
- GPU: NVIDIA GeForce RTX 20/30/40/50 series or AMD Radeon RX 7900/7800/7700 series (requires 12GB+ VRAM)
- RAM: 32GB or more (64GB for all model sizes)
- OS: Windows 10/11 or Linux
- Setup: One-click install - no Docker, no complex config
[!TIP] Strata is free and open-source (MIT license). Nothing leaves your PC - everything runs locally.
How Strata Works#
To understand Strata, think of a pizza restaurant:
- Full AI model (125B): The restaurant has 125 chefs. But each order only needs 6 chefs working (6B activated parameters) - this is the Mixture-of-Experts (MoE) architecture.
- Strata: Like a smart manager who knows which chefs to call for each task.
- Quantization: Like simplified cooking - instead of using full ingredients, Strata uses compression techniques (Q2_0, IQ2_XS, IQ3_S) to reduce model size while maintaining quality.
Qwen3.8-Flash-Next Architecture#
Qwen3.8-Flash-Next uses a special architecture:
- 125B parameters total: Total number of parameters
- 6B activated per token: Only 6 billion parameters active per generation (very efficient)
- 51B n-gram embeddings: N-gram embedding tables stored outside GPU
- 48 layers: 48 neural network layers
- Gated DeltaNet + MoE: Combines DeltaNet (for sequence processing) and Mixture-of-Experts (for specialization)
This makes the model both powerful and efficient - using only 1/3 activated parameters compared to previous models.
Real-World Speed#
Strata is fast - on ordinary gaming PCs:
| GPU | Write speed (tokens/s) | Read speed (tokens/s) |
|---|---|---|
| RTX 5070 (12GB) - Q2_0 | 94 | 2,650 |
| RTX 5070 (12GB) - IQ3_S | 53 | 1,620 |
| RX 9070 XT (16GB) - Q2_0 | 60 | 1,160 |
[!NOTE]
- Writes answers: How fast the reply appears in short chat. 60 tokens/s is faster than you can read.
- Reads your prompt: How fast it processes your long prompt (32K tokens = ~24,000 words).
One token ≈ ¾ of a word. So 60 tokens/s ≈ 45 words/second - fast enough for comfortable chatting.
Installation and Usage#
Quick Install#
Windows:
# Download and extract Strata
# Double-click START-HERE.batbashLinux:
git clone https://github.com/Niko1221/Strata.git
cd Strata
./setup.shbashThe installer will automatically:
- Detect your GPU
- Choose the right engine (NVIDIA or AMD)
- Ask you a few questions (model, size, context, image support)
- Download the model (~70GB)
- Start the server at
http://127.0.0.1:8080
[!WARNING]
- First startup: Your PC may be slow or unresponsive for 1-3 minutes - Strata is loading 35-55GB into RAM. This is normal.
- Don’t close the window - wait for it to complete.
Using with Your Apps#
Strata exposes an OpenAI-compatible API:
Base URL: http://127.0.0.1:8080/v1
Example with Claude Code:
ANTHROPIC_BASE_URL=http://127.0.0.1:8080bashAny app that supports OpenAI API works - just change the base URL.
Which Model to Pick?#
The installer will recommend based on your RAM:
| RAM | Recommended model | Why |
|---|---|---|
| 32GB | Coder | Fits 32GB, specialized for code |
| 48GB | IQ2_XS (or Q2_0) | Larger sizes don’t fit |
| 64GB | IQ2_XS (recommended) or IQ3_XXS/IQ3_S | All sizes fit |
| 96GB+ | IQ3_S or UD-IQ4_XS | Room for largest sizes |
Special models:
- Coder: Coding-specialized version, achieves 91% of full model’s SWE-bench Verified score. Weaker at other tasks (especially Chinese and other CJK text).
- Swift 1.5: Fine-tune that thinks less before answering - get answers faster, similar quality.
- Unsloth UD-IQ4_XS: Unsloth’s ~4-bit version, quality between IQ3_S and UD-Q4_K_XL.
Why Choose Strata?#
1. Absolute Privacy#
With Strata, nothing leaves your PC. Code, documents, chat history - everything stays local. Especially important for:
- Companies requiring data security
- Healthcare, legal, finance
- Sensitive projects
2. Cost Savings#
Cloud APIs are expensive:
- OpenAI GPT-4: ~$30/1M tokens
- Claude 3.5 Sonnet: ~$15/1M tokens
- Heavy usage = thousands per month
Strata:
- Cost: $0 (electricity only)
- Volume: unlimited
- Break-even: a few months at moderate volume
3. Network Independence#
No internet needed after downloading the model. Work offline, anywhere.
4. Full Customization#
- Choose model size
- Adjust context length (how much text it remembers)
- Toggle thinking (reasoning effort)
- Multi-GPU (layer split)
- Batching for multiple concurrent requests
Practical Examples#
Local Chat#
Open browser at http://127.0.0.1:8080 and chat like with any AI:
User: Explain Mixture-of-Experts architecture simply, like I'm 10.
Strata: Imagine a school with 125 teachers.
When a student asks a question, only the 6 teachers
who are best at that subject are called to answer.
The other teachers rest. This helps the school save
energy and handle more students at once.textCoding with Claude Code#
# Install Claude Code
# Set environment variable
export ANTHROPIC_BASE_URL=http://127.0.0.1:8080
# Use as normal
claude-code "Add authentication to this API"bashClaude Code will call Strata locally instead of Anthropic API.
Reading Images#
If you enable “Images?” in setup, Strata can understand images:
User: [send screenshot image]
Describe this UI and write equivalent HTML.
Strata: This is a login form with email, password,
and "Sign In" button. Here's the HTML:
<form>
<input type="email" placeholder="Email" />
<input type="password" placeholder="Password" />
<button>Sign In</button>
</form>text[!NOTE]
- AMD on Linux: reads images via CPU
- AMD on Windows: image reading not yet supported
Limitations and Considerations#
Hardware Requirements#
Strata isn’t for every machine:
- GPU: Needs 12GB+ VRAM. Older cards (GTX 10 series) work but slowly.
- RAM: Needs 32GB+. 16GB is insufficient.
- Disk: Needs ~80GB SSD. HDD slows down first startup.
Speed#
- Token generation is fast (50-100 tokens/s)
- Prefill (reading long prompts) is slower: ~1 minute for 30K tokens
- Follow-up messages are faster due to caching
Code-Specialized Model#
The “Coder” model is strong at code but weaker at other tasks, especially Chinese and other CJK languages. If you need multilingual support, choose Q2_0, IQ2_XS, or IQ3_S.
Comparison with Other Solutions#
| Feature | Strata | Ollama | LM Studio | Cloud API |
|---|---|---|---|---|
| 125B Model | ✅ | ❌ (usually 7-70B) | ❌ (usually 7-70B) | ✅ |
| Setup | 1-click | Easy | Easy | Not needed |
| API | OpenAI/Anthropic | OpenAI | OpenAI | Native |
| Privacy | ✅ | ✅ | ✅ | ❌ |
| Cost | $0 | $0 | $0 | $$ |
| Multi-GPU | ✅ | ❌ | ❌ | N/A |
| Image input | ✅ | ⚠️ | ⚠️ | ✅ |
Conclusion#
Strata is a major step forward in local LLM inference:
- Powerful: Run 125B models (server-grade) on consumer hardware
- Fast: 50-100 tokens/s generation
- Easy: 1-click install, OpenAI/Anthropic-compatible API
- Cheap: Free, no API costs
- Private: Everything local
If you have a gaming PC with 12GB+ VRAM and 32GB+ RAM, Strata is an excellent way to run powerful AI without depending on the cloud.