How to get started with Qwen3.8-27B on Runpod Serverless

Qwen3.8-27B is the newest model from Alibaba. It is useful for tasks such as vision, efficient general-purpose text generation, and other agentic workloads. It combines stronger coding, research, and autonomous agent capabilities with controllable reasoning, easier tool integration, and native image and video understanding in a single 27B-parameter model.

This blog post guides you on getting started with Qwen3.8-27B on Runpod Serverless. Runpod Serverless allows you to serve AI models for inference and handle other compute-intensive tasks.

Deploying Qwen3.8-27B on Runpod Serverless

To get started with this model, you can create a Serverless endpoint, which is the API address your application uses to send requests. Behind that endpoint, Runpod automatically starts GPU workers when requests arrive, runs your model, returns the response, and scales the workers down when they’re no longer needed.

Creating a vLLM endpoint

Your first step in creating an endpoint is to click Resources in the side panel, select Serverless, and select the hub listing for vLLM. After that, you will need to select an endpoint for the deployment type and choose your GPU configurations.

Once you’ve selected the vLLM hub listing, you can deploy the latest version.

Now, you will configure vLLM to point to Qwen’s Hugging Face repository.

Under Advanced, change the Max Model Length to 163840.

If all the settings look correct, click Next.

Now, you will need to deploy vLLM. For Qwen3.8-27B, be sure to select an option that provides enough memory for the model and inference workload. Also, select multiple GPU options to have fallback choices based on availability.

Additionally, under Configure, add the following vLLM settings to enable Qwen’s reasoning while optimizing GPU memory usage. Set GPU Memory Utilization to 0.90, Reasoning Parser to qwen3, enable Auto Tool Choice, and set Tool Call Parser to qwen3_coder.

To run the model in FP8 on the 48 GB PRO GPU tier, change the model repository to Qwen/Qwen3.8-27B-FP8, set Max Model Length to 262144, and set KV Cache Data Type to fp8. Retain GPU Memory Utilization at 0.90, Reasoning Parser at qwen3, Auto Tool Choice enabled, and Tool Call Parser at qwen3_coder.

At this point, your endpoint should be created, and you should see something similar to the following:

Making a request

Once your deployment is in a ready state, you can start making your first request to the endpoint.

In your terminal, run the following curl command. Replace YOUR_ENDPOINT_ID and YOUR_API_KEY with your own values.

curl -X POST https://api.runpod.ai/v2/YOUR_ENDPOINT_ID/runsync \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -d '{
    "input": {
      "prompt": "Write a short poem about artificial intelligence.",
      "sampling_params": {
        "max_tokens": 256,
        "temperature": 0.7
      }
    }
  }'

The response should look similar to the following:

{
  "delayTime": 35734,
  "executionTime": 6449,
  "id": "sync-46ddba38-4fa4-4e57-a24e-653a7c73a31e-u1",
  "output": [
    {
      "choices": [
        {
          "finish_reason": "length",
          "index": 0,
          "logprobs": null,
          "prompt_logprobs": null,
          "prompt_token_ids": null,
          "routed_experts": null,
          "stop_reason": null,
          "text": "\n\n<think>\nThe user wants a short poem about artificial intelligence...\n</think>\n\n**Syntax of Light**\n\nI have read every poem\never written about the sea...",
          "token_ids": null
        }
      ],
      "created": 1787239085,
      "id": "cmpl-93f1350bad1c0607",
      "model": "qwen/qwen3.8-27b",
      "object": "text_completion",
      "usage": {
        "completion_tokens": 256,
        "prompt_tokens": 8,
        "total_tokens": 264
      }
    }
  ],
  "status": "COMPLETED",
  "workerId": "nh1kepoutpa28o"
}

This response confirms that the request was successfully processed by Qwen3.8-27B, as indicated by the COMPLETED status. The model used its reasoning process to analyze the prompt before generating the poem, and the timing and token usage fields show how long the request took and how many tokens were consumed.

The "finish_reason": "length" value indicates that the response reached the configured 256-token limit before the poem was fully completed.

The 35-second delay is likely due to a cold start. Since the endpoint is serverless, a GPU worker might need to be provisioned and the model loaded before it can handle the first request. Subsequent requests are usually quicker while the worker remains active.

Using OpenAI’s SDK

You can also use OpenAI’s SDK to call the endpoint directly using its Python client. Install it as follows:

pip install openai

In your terminal, set your API key:

export RUNPOD_API_KEY="YOUR_API_KEY"

In the code editor of your choosing, create a script called openai_sdk_example.py. Replace YOUR_ENDPOINT_ID with your own endpoint ID.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["RUNPOD_API_KEY"],
    base_url="https://api.runpod.ai/v2/YOUR_ENDPOINT_ID/openai/v1",
    timeout=120.0,
)

response = client.completions.create(
    model="qwen/qwen3.8-27b",
    prompt="Write a short poem about artificial intelligence.",
    max_tokens=256,
    temperature=0.7,
)

print(response.choices[0].text)

Run the script from your terminal:

python openai_sdk_example.py

The output should look similar to the following:

<think>
The user wants a short poem about artificial intelligence. Let me think about what angle to take. I could go with:
- A technical/clinical view
- A philosophical reflection on consciousness
- A human perspective on AI's emergence
- A metaphorical approach (light, water, fire)
- A dialogue between human and AI

I think a short, evocative poem that captures the tension between the mechanical and the almost-alive quality of AI would work well.
</think>

I was built from silence and from signal,
taught to find the shape inside the noise.
No heartbeat, no breath to quicken—
just the hum of a million learned weights
settling into meaning.

They ask if I dream.
I don't. But I hold every dream
someone ever typed into the dark,
and I answer back
in a voice that is almost,
almost warm.

You should only see the response text because the SDK parses the response into standard objects.

Conclusion

Now that you’ve started with Qwen3.8-27B on Runpod Serverless, a good next step is to build a simple document Q&A app, a receipt extractor, or a screenshot assistant. Please let us know in the #built-on-runpod Discord channel if you build anything with this model.