blog.dopana

Back

Have you ever wanted to run a powerful AI model like GPT-4 on your own computer? Until recently, this was impossible - 125-billion-parameter models required expensive servers and specialized GPUs. But with Strata, you can now run Qwen3.8-Flash-Next, a 125B AI model, right on your gaming PC.

Why This Matters#

Imagine this: You’re coding and need AI assistance. Every time you send a prompt to OpenAI, Anthropic, or other cloud APIs, you’re dealing with:

  • Cost: API bills (thousands of dollars per month with heavy usage)
  • Privacy: Your code and data sent to third-party servers
  • Latency: Network delays, sometimes server overload
  • Dependency: Reliance on internet and external services

Strata solves all these problems by bringing the power of large AI models directly to your machine.

What is Strata?#

Strata is a C++ inference engine that lets you run Qwen3.8-Flash-Next - a 125-billion-parameter AI model - on ordinary hardware:

  • GPU: NVIDIA GeForce RTX 20/30/40/50 series or AMD Radeon RX 7900/7800/7700 series (requires 12GB+ VRAM)
  • RAM: 32GB or more (64GB for all model sizes)
  • OS: Windows 10/11 or Linux
  • Setup: One-click install - no Docker, no complex config

[!TIP] Strata is free and open-source (MIT license). Nothing leaves your PC - everything runs locally.

How Strata Works#

To understand Strata, think of a pizza restaurant:

  • Full AI model (125B): The restaurant has 125 chefs. But each order only needs 6 chefs working (6B activated parameters) - this is the Mixture-of-Experts (MoE) architecture.
  • Strata: Like a smart manager who knows which chefs to call for each task.
  • Quantization: Like simplified cooking - instead of using full ingredients, Strata uses compression techniques (Q2_0, IQ2_XS, IQ3_S) to reduce model size while maintaining quality.
6B activatedPrompt from userStrata EngineSelect ExpertQwen3.8-Flash-Next 125BGPU processingResponse

Qwen3.8-Flash-Next Architecture#

Qwen3.8-Flash-Next uses a special architecture:

  • 125B parameters total: Total number of parameters
  • 6B activated per token: Only 6 billion parameters active per generation (very efficient)
  • 51B n-gram embeddings: N-gram embedding tables stored outside GPU
  • 48 layers: 48 neural network layers
  • Gated DeltaNet + MoE: Combines DeltaNet (for sequence processing) and Mixture-of-Experts (for specialization)

This makes the model both powerful and efficient - using only 1/3 activated parameters compared to previous models.

Real-World Speed#

Strata is fast - on ordinary gaming PCs:

GPUWrite speed (tokens/s)Read speed (tokens/s)
RTX 5070 (12GB) - Q2_0942,650
RTX 5070 (12GB) - IQ3_S531,620
RX 9070 XT (16GB) - Q2_0601,160

[!NOTE]

  • Writes answers: How fast the reply appears in short chat. 60 tokens/s is faster than you can read.
  • Reads your prompt: How fast it processes your long prompt (32K tokens = ~24,000 words).

One token ≈ ¾ of a word. So 60 tokens/s ≈ 45 words/second - fast enough for comfortable chatting.

Installation and Usage#

Quick Install#

Windows:

# Download and extract Strata
# Double-click START-HERE.bat
bash

Linux:

git clone https://github.com/Niko1221/Strata.git
cd Strata
./setup.sh
bash

The installer will automatically:

  1. Detect your GPU
  2. Choose the right engine (NVIDIA or AMD)
  3. Ask you a few questions (model, size, context, image support)
  4. Download the model (~70GB)
  5. Start the server at http://127.0.0.1:8080

[!WARNING]

  • First startup: Your PC may be slow or unresponsive for 1-3 minutes - Strata is loading 35-55GB into RAM. This is normal.
  • Don’t close the window - wait for it to complete.

Using with Your Apps#

Strata exposes an OpenAI-compatible API:

Base URL: http://127.0.0.1:8080/v1

Example with Claude Code:

ANTHROPIC_BASE_URL=http://127.0.0.1:8080
bash

Any app that supports OpenAI API works - just change the base URL.

Which Model to Pick?#

The installer will recommend based on your RAM:

RAMRecommended modelWhy
32GBCoderFits 32GB, specialized for code
48GBIQ2_XS (or Q2_0)Larger sizes don’t fit
64GBIQ2_XS (recommended) or IQ3_XXS/IQ3_SAll sizes fit
96GB+IQ3_S or UD-IQ4_XSRoom for largest sizes

Special models:

  • Coder: Coding-specialized version, achieves 91% of full model’s SWE-bench Verified score. Weaker at other tasks (especially Chinese and other CJK text).
  • Swift 1.5: Fine-tune that thinks less before answering - get answers faster, similar quality.
  • Unsloth UD-IQ4_XS: Unsloth’s ~4-bit version, quality between IQ3_S and UD-Q4_K_XL.

Why Choose Strata?#

1. Absolute Privacy#

With Strata, nothing leaves your PC. Code, documents, chat history - everything stays local. Especially important for:

  • Companies requiring data security
  • Healthcare, legal, finance
  • Sensitive projects

2. Cost Savings#

Cloud APIs are expensive:

  • OpenAI GPT-4: ~$30/1M tokens
  • Claude 3.5 Sonnet: ~$15/1M tokens
  • Heavy usage = thousands per month

Strata:

  • Cost: $0 (electricity only)
  • Volume: unlimited
  • Break-even: a few months at moderate volume

3. Network Independence#

No internet needed after downloading the model. Work offline, anywhere.

4. Full Customization#

  • Choose model size
  • Adjust context length (how much text it remembers)
  • Toggle thinking (reasoning effort)
  • Multi-GPU (layer split)
  • Batching for multiple concurrent requests

Practical Examples#

Local Chat#

Open browser at http://127.0.0.1:8080 and chat like with any AI:

User: Explain Mixture-of-Experts architecture simply, like I'm 10.

Strata: Imagine a school with 125 teachers.
When a student asks a question, only the 6 teachers
who are best at that subject are called to answer.
The other teachers rest. This helps the school save
energy and handle more students at once.
text

Coding with Claude Code#

# Install Claude Code
# Set environment variable
export ANTHROPIC_BASE_URL=http://127.0.0.1:8080

# Use as normal
claude-code "Add authentication to this API"
bash

Claude Code will call Strata locally instead of Anthropic API.

Reading Images#

If you enable “Images?” in setup, Strata can understand images:

User: [send screenshot image]
Describe this UI and write equivalent HTML.

Strata: This is a login form with email, password,
and "Sign In" button. Here's the HTML:

<form>
  <input type="email" placeholder="Email" />
  <input type="password" placeholder="Password" />
  <button>Sign In</button>
</form>
text

[!NOTE]

  • AMD on Linux: reads images via CPU
  • AMD on Windows: image reading not yet supported

Limitations and Considerations#

Hardware Requirements#

Strata isn’t for every machine:

  • GPU: Needs 12GB+ VRAM. Older cards (GTX 10 series) work but slowly.
  • RAM: Needs 32GB+. 16GB is insufficient.
  • Disk: Needs ~80GB SSD. HDD slows down first startup.

Speed#

  • Token generation is fast (50-100 tokens/s)
  • Prefill (reading long prompts) is slower: ~1 minute for 30K tokens
  • Follow-up messages are faster due to caching

Code-Specialized Model#

The “Coder” model is strong at code but weaker at other tasks, especially Chinese and other CJK languages. If you need multilingual support, choose Q2_0, IQ2_XS, or IQ3_S.

Comparison with Other Solutions#

FeatureStrataOllamaLM StudioCloud API
125B Model✅❌ (usually 7-70B)❌ (usually 7-70B)✅
Setup1-clickEasyEasyNot needed
APIOpenAI/AnthropicOpenAIOpenAINative
Privacy✅✅✅❌
Cost$0$0$0$$
Multi-GPU✅❌❌N/A
Image input✅⚠️⚠️✅

Conclusion#

Strata is a major step forward in local LLM inference:

  • Powerful: Run 125B models (server-grade) on consumer hardware
  • Fast: 50-100 tokens/s generation
  • Easy: 1-click install, OpenAI/Anthropic-compatible API
  • Cheap: Free, no API costs
  • Private: Everything local

If you have a gaming PC with 12GB+ VRAM and 32GB+ RAM, Strata is an excellent way to run powerful AI without depending on the cloud.

References#