How to get started with Qwen3.8-Flash-Next on Runpod Serverless

Qwen3.8-Flash-Next is Alibaba's preview of the Qwen4 serving stack. Four aspects of the model shape how you deploy it:

• It is a 125B-parameter ultra-sparse mixture-of-experts model with only 6B parameters active per token.
• Its native context is 262,144 tokens, extensible to 1M with YaRN.
• It ships with multi-token prediction (MTP) and a vision encoder.
• It pairs Gated DeltaNet (GDN) linear-attention layers with Qwen Sparse Attention (QSA), which is what makes very long context cheap to serve.

Everything below was verified end-to-end on Runpod using 4× H200, via the OpenAI Python SDK.

Choosing the deployment

Runpod Hub's vLLM worker template (runpod-workers/worker-vllm, currently v2.26.0) ships vLLM 0.28.0, and Qwen3.8-Flash-Next requires vLLM 0.29 or newer.

The architecture (GDN, QSA, and n-gram embeddings) exists only in that build and later, so the Hub's one-click path does not serve this model today. vLLM publishes a dedicated image for it, vllm/vllm-openai:qwen38-flash-next, and we validated that image live on 2026-09-03 in two configurations.

You can run it as a load-balancing Serverless endpoint or as a GPU pod. The load-balancing endpoint is direct HTTPS to vLLM's OpenAI server, with streaming, autoscaling, per-second billing and no queue, which suits API-style traffic. A pod runs the same image and flags behind Runpod's HTTP proxy, and it suits interactive work and debugging because there is no health watchdog, so a slow first boot is not fatal.

Hardware

We used the official FP8 checkpoint Qwen/Qwen3.8-Flash-Next-FP8, roughly 173 GiB of weights across 131 shards, on NVIDIA H200s with 141 GB of HBM3e each. The official vLLM recipe supports TEP8, tensor-plus-expert parallelization across 8 GPUs. Eight-wide H200 nodes were capacity-constrained during our test, so we ran the other supported configuration, TP4 with expert parallelism on 4× H200. That gives 564 GB of HBM, which holds the weights, the KV cache, the vision tower and the MTP module with a large KV budget left over: vLLM reported 37.28 GiB of KV cache per GPU, about 3.02M tokens, enough for 11.5 concurrent sequences at the full 262K context.

On Hopper you must pass --enable-expert-parallel --moe-backend triton; plain tensor parallelism is incompatible with this checkpoint's 128-wide quantization blocks.

Cost (measured config)

Serverless bills only while a worker is running, so bursty API traffic is usually cheaper there. Steady or interactive workloads favor a pod.

Creating the load-balancing endpoint

Select Serverless, then New Endpoint, then choose a custom image. These are the settings we validated:

Container start args (the image entrypoint is vllm serve):

Qwen/Qwen3.8-Flash-Next-FP8 \
  --tensor-parallel-size 4 --enable-expert-parallel --moe-backend triton \
  --gpu-memory-utilization 0.85 --max-num-seqs 256 --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --enforce-eager

--enforce-eager skips torch.compile and CUDA-graph capture. Our pod boot spent 46.6 s compiling and 36.2 s capturing 86 CUDA graphs; eager moves that time off the boot path so the endpoint passes the platform health window. Add --gdn-prefill-backend triton to also skip the FlashInfer GDN JIT that otherwise hits the first request (see the performance section).

Environment variables:

The Runpod gateway authenticates requests with your Runpod API key, so do not add --api-key to vLLM on this path. The gateway cannot know the value, and every request would 401 inside the worker. On a pod there is no gateway, so you can set --api-key there.

Measured on September 3, 2026, on a four-GPU H200 worker, startup took about 11 minutes from scale-up to the first successful health check: roughly 4.5 minutes to pull and extract the image, under a minute for NCCL initialization (with NCCL_NVLS_ENABLE=0), 167.7 seconds to download roughly 173 GiB of model weights to a warm host, 38 seconds to load the shards, 35.3 seconds to initialize the engine in eager mode and 19.2 seconds for multimodal warmup.

On a fully cold host, allow 20–25 minutes for the Hugging Face download. Monitor progress in the Workers tab or with stream-worker-logs; the worker is ready when the logs report that application startup is complete.

Making a request

Once the worker is healthy, call the endpoint with the standard OpenAI Python client. Set RUNPOD_API_KEY in your environment, replace <ENDPOINT_ID> with your Runpod endpoint ID and make sure the model value matches the model your deployment serves.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["RUNPOD_API_KEY"],
    base_url="https://.api.runpod.ai/v1",
    timeout=600.0,
)

response = client.chat.completions.create(
    model="Qwen/Qwen3.8-Flash-Next-FP8",
    messages=[{"role": "user",
               "content": "Explain how a mixture-of-experts model works in a couple of sentences."}],
    max_tokens=2048,
    temperature=1.0, top_p=0.95,       # model's recommended defaults
    extra_body={"top_k": 20},
)
print(response.choices[0].message.content)
print(response.usage)

Response:

A mixture-of-experts model combines multiple specialized submodels, or
"experts," along with a gating mechanism that decides which experts to use
for each input. This allows the model to allocate different parts of the
problem space to the experts best suited for each example.

usage: prompt=66 completion=101 (reasoning_tokens=45), total=167

Thinking is on by default. usage.completion_tokens_details.reasoning_tokens counts the reasoning trace, and with --reasoning-parser qwen3 the visible content excludes it. Streaming works with the standard stream=True.

Non-thinking mode, tool calling and vision

All six checks in scripts/test_endpoint.py passed against the exact configuration above on 2026-09-03.

Instruct mode works by passing extra_body={"chat_template_kwargs": {"enable_thinking": false}} with the model's recommended sampling for that mode (temperature=0.7, top_p=0.8, presence_penalty=1.5, top_k=20). It answered a mini-math question with just 391 and no reasoning preamble.

Tool calling works through the qwen3_xml parser. With a get_weather tool defined, response.choices[0].message.tool_calls[0] came back as get_weather({"city": "Reykjavik"}).

Vision needs no extra flags because the tower loads by default. We sent a data-URI PNG of a red, white and blue striped field, and the model described it as a tricolor flag with three equal vertical stripes of red, white and blue from left to right.

Measured performance

Eager mode gets the server reachable 5–7 minutes sooner and serves its early requests an order of magnitude slower than the compiled engine, because every Triton kernel JITs cold on first use; even at steady state, eager decode runs well below the compiled pod's 137 tok/s. You have three ways out: accept eager for the first worker and let REQUEST_COUNT scaling bring the others up the same way, run compiled and persist VLLM_CACHE_ROOT on a network volume so compile artifacts carry across workers, or use pods for anything latency-sensitive.

Cold-start levers, in order of effort:

1. --enforce-eager removes compile and graph capture from the boot path.
2. Keep workersMin: 1 during business hours so the second boot on a host reuses the HF cache. Our warm-host download was 167.7 s for 173 GiB; budget up to 20–25 minutes on a cold host.
3. Attach a network volume with VLLM_CACHE_ROOT (and HF_HOME) so the compile cache and weights survive worker recycling. The full playbook is in Cut vLLM Cold Starts on Runpod Serverless.
4. On the request side, --gdn-prefill-backend triton avoids the FlashInfer GDN JIT that made our first-request TTFT 3.60 s against 3.12 s warm.

Disable NVLS before your first boot

Every H200 node we landed on failed NVLS (NVLink SHARP fabric) initialization: all four NCCL ranks died at ncclCommInitRank with unhandled cuda error before a single byte of weights downloaded. The fix is a single environment variable: NCCL_NVLS_ENABLE=0. With it set, all ranks initialize in about 0.5 s via P2P/CUMEM and serving proceeds. You will also see SymmMemCommunicator: symmetric memory multicast operations are not supported warnings; those are the expected consequence of disabling NVLS and are safe to ignore. Save yourself four crash-looped pods and set the variable on day one.

Conclusion

Until the Hub worker ships vLLM 0.29 or newer, both paths run the same vllm/vllm-openai:qwen38-flash-next image with the same flags: the load-balancing Serverless endpoint for API traffic and a pod for interactive work and maximum single-stream throughput. One environment variable, NCCL_NVLS_ENABLE=0, made the difference between a crash loop and a clean boot. With the configuration tables above, you can go from zero to an OpenAI-compatible API serving 262K-context Qwen3.8-Flash-Next in about eleven minutes.

Next steps

Runpod Serverless runs your container as an autoscaling endpoint that scales to zero and bills by the second.

• Try Serverless: Runpod console
• Python SDK: github.com/runpod/runpod-python
• Examples: github.com/runpod-workers
• Docs: docs.runpod.io/serverless/overview
• Discord: join us here

Resources

• Qwen/Qwen3.8-Flash-Next on Hugging Face
• FP8 checkpoint
• vLLM recipes
• Runpod docs