Performing your first LLM fine-tune on Runpod with Axolotl, and how to host it after

Why fine-tune instead of prompting?

A system prompt can describe someone's personality, but a model following a description still sounds like it's just doing an impress ion. In addition, as the distance grows from the system prompt as the context increases, drift sets in and a pre-message 'refresher' also can put its thumb on the scale in unnatural-sounding ways. Fine-tuning works differently: you show the model thousands of real replies, and it learns the patterns directly. In practice, that means:

• Fine-tuning is excellent at style. Vocabulary, sentence length, capitalization, emoji habits, typical reactions, and how replies grow longer as a conversation heats up all transfer well.
• Fine-tuning is unreliable for facts. The model learns how the person talks, not a dependable record of their life. Something mentioned once will often be garbled or invented. Facts are better supplied in the system prompt, as covered in Step 2.

The best results come from combining the two: training for the voice, prompting for the facts.

The pipeline at a glance

1. Set up a Runpod pod with a network volume and the Axolotl image.
2. Convert chat logs into JSONL training files, one conversation per line.
3. Write a YAML config that tells Axolotl the base model, data, and settings.
4. Run a small smoke test to confirm the pipeline works before spending real GPU time.
5. Compare the tuned model against the base model on real conversations.
6. Serve the model through an OpenAI-compatible endpoint, or merge it into a single GGUF file.
7. Scale up to the full dataset.

Many of these pipeline steps will come with code examples and can be repeated as you iterate, so you can see for yourself whether your efforts are moving you in the right direction.

Before you start

You need three accounts, a set of chat logs, and if your persona is based on a real person's texts, permission from them to use their messages.

Create your Hugging Face token under Settings → Access Tokens. Make one with write access, since you'll use it for uploads later. If you plan to use a gated model, accept its license on the model page first. It's worth noting that Huggingface gives 100GB of free storage space. This will be enough for any Qwen3 27b model, but you'll need to upgrade to a paid plan to store a Mistral Medium model at full ewights.

Consent and privacy

A persona model is built from someone's real words, so treat it accordingly:

• Get consent. Training on your own messages, or messages you've written yourself is straightforward. Training on someone else's needs their explicit agreement, especially for anything intimate or personally identifying.
• Keep the model private. A fine-tuned model can reproduce training text nearly word for word. Store it in a private Hugging Face repo, and never publish it.
• Secure your endpoints. Runpod proxy URLs are public, so always set an API key when serving, and stop pods you aren't using.
• Know your base model. If you decide to build on a model that has been abliterated or otherwise had refusal behavior altered, this can be especially useful for natural conversation, but it's one more reason to keep the result private.

Key concepts

Three ideas carry most of this guide: how much of the model you train, how the data maps into the model, and why you always start with a tiny test run.

For teaching a model someone's voice, LoRA and QLoRA get you nearly all the quality of a full fine-tune at a fraction of the cost. QLoRA's quality loss compared with LoRA is usually small for style work, which is why the Mistral example in this guide uses it,. This will drastically reduce the cost of the process, especially with larger models where you could save hundreds of gigabytes of RAM. When training is done, we'lll merge the adapter into the full-precision base model, so you don't need to have the additional moving part of applying the LoRA separately at inference time.

Before training on your full dataset, train on a handful of conversations for many epochs, deliberately overfitting. If the model can nearly memorize short conversations, your data format, chat template, and loss masking are all correct. That run takes a few minutes and catches almost every setup mistake before it costs you hours of GPU time. Step 5 shows what a passing smoke test looks like. Once you have that dialed in, then you can lower the epochs and mess with the learning rate without metaphorically tapping the mic and saying "is this thing on?"

Choosing a base model

Pick a dense model with good tooling support and a way to turn thinking off. This guide uses two that meet all three:

Both are multimodal models with a vision encoder you won't use. The configs in Step 4 target only the language layers. The Humanlike model is a community fine-tune of an abliterated Qwen base, and Mistral's model card lists its own license terms, so check the license on the model page before any commercial use. Mixture of Experts (MoE) models can also be finetuned, but that is a process for another day due to further complexities that need to be addressed.

Does a bigger model need more data?

No. For learning a voice, larger models usually need less data, because they already understand language and tone better. What changes with size is that larger models memorize faster, so use fewer epochs and keep the learning rate modest. Size does matter for quality, though, in ways benchmarks don't capture. Benchmarks reward skills with checkable answers, which post-training can teach a smaller model efficiently (known as 'benchmaxxing'). If your use case happens to be in the realm where those benchmarks live, though, then you'll probably be better off with the smaller model with higher scores. For more creative tasks that require a deeper well of world knowledge, then you'll want to splurge on the effort and the compute needed for those larger models.

Step 1: Set up Runpod

You'll create a network volume to hold everything, then a training pod that mounts it at /workspace. Because models, data, and checkpoints all live on the volume, you can stop, restart, or swap pods without losing work.

Create a network volume

1. In the Runpod console, go to Storage → Network Volumes → New Network Volume.
2. Pick a datacenter that has the GPUs you plan to use. A volume can only attach to pods in its own datacenter.
3. Choose a size. Models get big fast:

Deploy the training pod

1. Go to Pods → Deploy and pick your GPU (see the table below).
2. Set the container image to axolotlai/axolotl-cloud:main-latest, Axolotl's image for cloud GPU providers.
3. Attach your network volume. It mounts at /workspace.
4. Set the container disk to 50GB. Nothing important lives there.
5. Add the environment variables below, then deploy.

Lay out the project folder

Open a terminal from the pod's Jupyter interface and create the project:

mkdir -p /workspace/persona/{configs,data/txt,data/raw,outputs}
cd /workspace/persona
axolotl fetch deepspeed_configs

When you're done with this guide, the folder will look like this:

/workspace/persona/          # run every command from here
  txt2jsonl.py               # data/txt -> data/raw
  fix_quotes.py              # repairs unescaped quotes in JSONL files
  combine.py                 # data/raw -> data/train.jsonl, with checks
  count_tokens.py            # dataset size and mode balance
  compare.py                 # base vs tuned model, side by side
  configs/
    qwen-lora.yml
    mistral-qlora.yml
  data/
    txt/                     # conversations you write as plain text
    raw/                     # .jsonl files, converted or hand-written
    train.jsonl              # written by combine.py
    smoke.jsonl              # a few conversations for the smoke test
  deepspeed_configs/         # from `axolotl fetch deepspeed_configs`
  outputs/                   # checkpoints, adapters, merged models

To get files onto the pod, upload them in the Jupyter file browser, or use runpodctl send <file> on your machine and runpodctl receive <code> on the pod.

Keep tools in separate environments

The tools in this guide pin conflicting package versions. In our testing, installing the Hugging Face CLI into the llama.cpp environment upgraded huggingface_hub past what Transformers allows, which broke both. Give each job its own environment:

If you installed Axolotl in your own venv instead of using the image, activate it with source /path/to/venv/bin/activate before training. Check your prompt before running anything: it shows which venv is active.

tmux comes with the template, so a dropped browser tab doesn't kill a run.

Step 2: Design your dataset

The data matters more than any setting in the config, so most of your time belongs here. Axolotl wants JSONL: one complete conversation per line, in OpenAI-style messages format. Throughout this guide, the person being modeled is called Jordan.

{"messages": [
  {"role": "system", "content": "Casual chat."},
  {"role": "user", "content": "you coming tonight?"},
  {"role": "assistant", "content": "ya probably\nafter 9 tho"},
  {"role": "user", "content": "bring the switch"}, 
  {"role": "assistant", "content": "obviously"}]}

• assistant is always the person you're modeling. Only these turns are trained.
• user is everyone else. Their messages are context the model learns to respond to.
• system holds a short mode label, plus optional fixed details covered below.

The structural rules

1. One conversation per line. Multi-turn conversations are good: the model learns from every assistant turn, using the earlier turns as context. A conversation spread over several lines is invalid JSONL.
2. Roles alternate. When someone sends several messages in a row, merge them into one turn joined with \n, as in ya probably\nafter 9 tho above. Chat templates expect alternation, and the model learns that one burst is one reply. At inference, it replies in bursts too, and you can split its output on newlines to recreate the texting rhythm.
3. Start with system, then user. If the person you're modeling spoke first, drop that opening message or move its content into the system prompt.
4. End on an assistant turn. A trailing user message teaches nothing.
5. Escape double-quotes with \ to ensure the JSONL format remains legible to the parser. Example: {"role": "user", "content": "she said that it was an \"interesting time\""},

The easiest way to control where conversations begin and end is to save one conversation per file. Each file becomes one training conversation, and you decide every break yourself.

Label each mode

If your logs mix registers, such as short casual texts and long, detailed roleplay, an unlabeled model blends them: casual replies get oddly long, and other usage gets flat.

The exact string matters. The model learns these labels literally, so Roleplay planning. and Roleplay planning (OOC). become two different modes. Use identical labels in every file and at inference, and list them in combine.py (Step 3), which rejects anything else. Some other relevant ideas include Customer service interaction, legal brief, etc.

Avoid "you" in system prompts. Saying "Assistant says" and "User says" maps directly onto the roles in the data and stays unambiguous. During training, the model learns how the person acts.

Give the model facts through the system prompt

Everything the person says is trained, including their interests, but facts mentioned once rarely stick. The reliable approach is a fixed profile block after the label, identical in every conversation and at inference:

How much data you need

Measure your dataset in trained tokens, the person's own replies, rather than conversations. In our runs, trained tokens were roughly 45% of the total.

Coverage, balance, and quality matter more than raw volume: include each mode and the range of situations you'll actually chat in, keep casual chat from being drowned out by roleplay tokens, and remove junk and near-duplicates. Don't pad the dataset with model-written imitations of the person, which dilutes their real voice with a model's guess at it.

Step 3: Build the training files

Four small scripts handle the data: one converts plain text to JSONL, one repairs files with unescaped quotes, one validates and combines everything into train.jsonl, and one counts tokens. Save them in /workspace/persona and run them from there in the Axolotl environment.

Here are some examples of short chats:

### Casual chat.
U: you coming tonight?
A: ya probably
A: after 9 tho
U: bring the switch
A: obviously

### Roleplay. Assistant plays Mara, a night-shift detective. User plays Theo, an informant. Setting: a rain-soaked harbor district.
U: Theo slid into the booth. "You're late."
A: Mara shook the rain from her coat. "Traffic," she said, deadpan.

She set a photograph on the table between them.

Genuinely, this is a task that you can just throw at an LLM and it will be able to format the files for you. If you'd rather do it through code, here is a script that will do it for you:

# txt2jsonl.py  -  usage: python3 txt2jsonl.py data/txt/*.txt
# Converts plain-text conversation files into data/raw/<name>.jsonl
import json, os, sys

PREFIXES = {"U:": "user", "A:": "assistant"}

def parse(path):
    convos, convo, cur = [], None, None
    for n, raw in enumerate(open(path, encoding="utf-8"), 1):
        line = raw.rstrip("\n")
        if line.startswith("###"):                     # new conversation + system label
            if convo:
                convos.append(convo)
            convo, cur = [{"role": "system", "content": line[3:].strip()}], None
        elif line[:2] in PREFIXES:
            if convo is None:
                sys.exit(f"{path}:{n}  turn before any ### line")
            role, text = PREFIXES[line[:2]], line[2:].strip()
            if cur and cur["role"] == role:             # same speaker again: merge
                cur["content"] += "\n" + text
            else:
                cur = {"role": role, "content": text}
                convo.append(cur)
        elif cur is not None:                           # continues the current turn
            cur["content"] += "\n" + line
    if convo:
        convos.append(convo)
    return convos

os.makedirs("data/raw", exist_ok=True)
for src in sys.argv[1:]:
    out = os.path.join("data/raw", os.path.splitext(os.path.basename(src))[0] + ".jsonl")
    convos = parse(src)
    with open(out, "w", encoding="utf-8") as f:
        for convo in convos:
            for m in convo:
                m["content"] = m["content"].strip()     # drop blank lines at turn edges
            f.write(json.dumps({"messages": convo}, ensure_ascii=False) + "\n")
    print(f"{src} -> {out}  ({len(convos)} conversations)")

Run it, spot-check a few conversations in the .fixed file, then swap it in with mv data/raw/scene01.jsonl.fixed data/raw/scene01.jsonl. It assumes role comes before content in each message.

Validate and combine: combine.py

This script reads every .jsonl file in data/raw/, checks each conversation, and writes the valid ones to data/train.jsonl. Problems print with the file, line number, and what's wrong, and those conversations are skipped. It catches the mistakes that are easy to make by hand: an unlisted label, a misspelled role, a missing content key, and two turns in a row from the same speaker.

# combine.py  -  checks data/raw/*.jsonl and writes data/train.jsonl
import json, glob

# system content must start with one of these (add new labels here first)
LABELS = ("Casual chat.", "Roleplay.", "Roleplay planning.")
rows, errors = [], 0

for path in sorted(glob.glob("data/raw/*.jsonl")):
    for n, line in enumerate(open(path, encoding="utf-8"), 1):
        if not line.strip():
            continue
        where = f"{path}:{n}"
        try:
            msgs = json.loads(line)["messages"]
        except Exception as e:
            print(where, "not valid JSON:", e); errors += 1; continue
        problems = []
        bad_keys = [i for i, m in enumerate(msgs) if set(m) != {"role", "content"}]
        if bad_keys:
            problems.append(f"messages {bad_keys} need exactly 'role' and 'content' keys")
        roles = [m.get("role") for m in msgs[1:]]
        if msgs[0].get("role") != "system" or not str(msgs[0].get("content", "")).startswith(LABELS):
            problems.append("first message must be a system label listed in LABELS")
        if not roles or roles[0] != "user":
            problems.append("second message must be user")
        if roles and roles[-1] != "assistant":
            problems.append("last message must be assistant")
        if any(a == b for a, b in zip(roles, roles[1:])):
            problems.append("same role twice in a row, merge them into one turn")
        bad_roles = sorted({repr(r) for r in roles if r not in ("user", "assistant")})
        if bad_roles:
            problems.append("unknown role: " + ", ".join(bad_roles))
        if problems:
            print(where, "; ".join(problems)); errors += 1; continue
        for m in msgs:
            m["content"] = m["content"].strip()   # drop stray leading/trailing spaces
        rows.append(msgs)

with open("data/train.jsonl", "w", encoding="utf-8") as f:
    for msgs in rows:
        f.write(json.dumps({"messages": msgs}, ensure_ascii=False) + "\n")
print(f"{len(rows)} conversations written, {errors} skipped")

If your files pass validation then you are ready to start training. If you run into any errors, or if any files are skipped, run them through a JSON validator or an LLM and correct the issue, and then try again.

Step 4: Write the training config

Everything about a run lives in one YAML file. Below are complete configs for both models, set up for the smoke test, followed by what to change for a real run.

Qwen 27B: configs/qwen-lora.yml

# --- model ---
base_model: LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0
chat_template: tokenizer_default       # use the model's own chat template
# Thinking: the logs have no reasoning, so train in non-thinking mode.
# Check with `axolotl preprocess --debug` how assistant turns render
# (e.g. an empty think block) and serve with enable_thinking: false to match.

# --- data ---
datasets:
  - path: data/smoke.jsonl             # switch to data/train.jsonl for real runs
    ds_type: json
    type: chat_template                # parses the messages format
    field_messages: messages
    roles_to_train: ["assistant"]      # loss only on the persona's replies
    train_on_eos: turn                 # learn to end each reply

# Held-out set for validation loss: uncomment once data/heldout.jsonl exists,
# along with evals_per_epoch below.
# test_datasets:
#   - path: data/heldout.jsonl
#     ds_type: json
#     split: train                     # a plain JSONL loads as one "train" split
#     type: chat_template
#     field_messages: messages
#     roles_to_train: ["assistant"]

dataset_prepared_path: outputs/prepared-qwen   # tokenized cache
output_dir: outputs/qwen-smoke

# --- sequence handling ---
sequence_len: 16384                    # max tokens per example; longer gets truncated
sample_packing: false                  # turn on for real runs

# --- LoRA ---
# Explicit language-model targets keep adapters off the vision tower.
adapter: lora
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj

# --- training ---
micro_batch_size: 1
gradient_accumulation_steps: 1         # 1 for the smoke test; 4-8 for real runs
num_epochs: 8                          # deliberately overfit for the smoke test
learning_rate: 1e-4                    # gentle, since the base already has the style
lr_scheduler: cosine
warmup_ratio: 0.05
optimizer: adamw_torch_fused

# --- performance ---
bf16: auto
tf32: true
gradient_checkpointing: true
flash_attention: true

# --- logging and saving ---
logging_steps: 1
# evals_per_epoch: 4                   # needs test_datasets
saves_per_epoch: 1
save_total_limit: 2
# wandb_project: persona               # uncomment if WANDB_API_KEY is set

Mistral Medium 3.5: convert the weights first

Mistral publishes this model only in FP8, and QLoRA can't quantize on top of FP8 weights. Training straight from the Hugging Face repo fails with The model is quantized with FineGrainedFP8Config but you are passing a BitsAndBytesConfig config. The fix is a one-time conversion to a local BF16 copy.

Check your resources first. The conversion needs about 256GB across GPU memory and system RAM, plus ~240GB of free disk:

# dequant_mistral.py  -  one-time FP8 -> BF16 conversion
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor, FineGrainedFP8Config

SRC = "mistralai/Mistral-Medium-3.5-128B"
DST = "/workspace/models/mistral-medium-3.5-bf16"

model = AutoModelForImageTextToText.from_pretrained(
    SRC, quantization_config=FineGrainedFP8Config(dequantize=True),
    torch_dtype=torch.bfloat16, device_map="auto",
    max_memory={0: "125GiB", "cpu": "180GiB"},   # leave headroom on both
    offload_folder="/workspace/offload", low_cpu_mem_usage=True)
model.save_pretrained(DST, safe_serialization=True, max_shard_size="5GB")
AutoProcessor.from_pretrained(SRC).save_pretrained(DST)

device_map="auto" fills the GPU first, then system RAM. Set the cpu value 30–40GB below the total that free -h reports. The offload folder is only used if GPU and RAM together run out. Once the copy is saved, delete the FP8 download (hf cache rm mistralai/Mistral-Medium-3.5-128B, or its folder under /workspace/hf-cache) and /workspace/offload to reclaim the space.

Mistral Medium 3.5: configs/mistral-qlora.yml

This config is the Qwen one with four changes: the local BF16 base, adapter: qlora with load_in_4bit: true, its own tokenized cache, and its own output folder. Mistral's language layers use the same standard names, so the LoRA targets carry over.

# --- model ---
base_model: /workspace/models/mistral-medium-3.5-bf16   # the local BF16 copy
chat_template: tokenizer_default
# Hybrid reasoning model: train in instruct mode with reasoning off.
# Check with `axolotl preprocess --debug` that assistant turns render
# with no reasoning block, and serve with reasoning off to match.

# --- data ---
datasets:
  - path: data/smoke.jsonl
    ds_type: json
    type: chat_template
    field_messages: messages
    roles_to_train: ["assistant"]
    train_on_eos: turn

dataset_prepared_path: outputs/prepared-mistral   # separate cache: different tokenizer
output_dir: outputs/mistral-smoke

# --- sequence handling ---
sequence_len: 16384                    # lower to 8192 if you hit out-of-memory
sample_packing: false

# --- QLoRA ---
adapter: qlora
load_in_4bit: true
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj

# --- training ---
micro_batch_size: 1
gradient_accumulation_steps: 1
num_epochs: 8
learning_rate: 1e-4
lr_scheduler: cosine
warmup_ratio: 0.05
optimizer: adamw_torch_fused

# --- performance ---
bf16: auto
tf32: true
gradient_checkpointing: true
flash_attention: true

# --- logging and saving ---
logging_steps: 1
saves_per_epoch: 1
save_total_limit: 2

# If preprocessing errors about a missing pad token, add:
# special_tokens:
#   pad_token: "<pad>"

Step 5: Preprocess, train, and pass the smoke test

A smoke test on 8 conversations takes a few minutes of training on one H200, and most of the first run is the model download. Everything lives in /workspace/persona and its subdirectories.

1. Preprocess and inspect the data

axolotl preprocess configs/qwen-lora.yml --debug

This tokenizes the data into the prepared cache and, with --debug, prints decoded examples showing which tokens are trained and which are masked (label -100). If you edit your data later, delete the prepared cache folder (outputs/prepared-qwen) before rerunning, or Axolotl may reuse stale data.

2. Train

axolotl train configs/qwen-lora.yml 2>&1 | tee train.log

The Mistral run is the same command with configs/mistral-qlora.yml. On a multi-GPU pod, Axolotl uses every visible GPU automatically.

3. Read the logs

Each step prints a line like this:

{'loss': '3.142', 'grad_norm': '0.2515', 'learning_rate': '9.548e-05', 'ppl': '23.14', 'memory/max_active (GiB)': '92.02', 'tokens/trainable': 10766, 'tokens/total': 24704, 'epoch': '2'}

With batch size 1, every step is a single conversation, so the loss jumps around: a long-form text interchange consisting of multiple paragraphs, then a two-line chat, then something in between. Judge the trend for each conversation, not each line. If loss completely flatlines, then you're probably overfitting at that point and should probably consider re-running with fewer epochs. The first step is also slow, often a couple of minutes, because it includes setup.

4. What a passing smoke test looks like

Here's a real smoke test on the Qwen config: 3 conversations, 8 epochs, 24 steps, about 70 seconds of training. Each conversation shows up as its own repeating pattern in the log:

That's a pass. The short conversations dropping toward zero proves the chat template, loss masking, and end-of-turn handling all work, because the model can only memorize replies if it's training on the right tokens.

The long transcript barely moving is normal. It got only 8 updates, each averaged over 5,100 tokens of  prose, and longer, creative conversations are unpredictable: a perplexity around 18 is a reasonable place for it. Don't chase it toward 1 by adding epochs. That would mean the model can recite what it was trained on word for word, not that it writes better.

Red flags:

• Loss flat from the start: almost always a data problem, such as nothing being trained or wrong roles being defined in the .jsonl files.
• Loss is nan or skyrockets: the learning rate is too high. Gambling on a high learning rate to save time could be a strategy for a full large-scale training project, but it's probably not necessary at these takes..
• Only 1–2 steps total: gradient_accumulation_steps is merging your whole dataset into one update. Set it to 1 or add more data to the dataset.
• Out-of-memory error: lower sequence_len, keep micro_batch_size at 1, or use a bigger GPU.

5. Where the output goes

For LoRA and QLoRA, outputs/qwen-smoke/ holds adapter_model.safetensors and adapter_config.json: the adapter, a few hundred MB. It isn't a usable model on its own; it applies on top of the base model, like an image LoRA on a checkpoint. The folder also holds tokenizer files and checkpoint-N/ folders from each save. A full fine-tune instead saves a complete model: config.json plus sharded model-0000X-of-0000Y.safetensors files.

Step 6: Compare the tuned model against the base

Loss numbers only tell you training worked. To see whether the model sounds like the person, cut real conversations off just before the persona's last reply and let both models answer. This script does that for every conversation in a file, and prints four things for each:

# compare.py  -  usage: python3 compare.py outputs/qwen-smoke [--base MODEL] [--4bit]
import argparse, json, os, random, torch
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForImageTextToText, BitsAndBytesConfig
from peft import PeftModel

p = argparse.ArgumentParser()
p.add_argument("tuned", help="LoRA output folder, or a merged / full model folder")
p.add_argument("--base", default="LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0")
p.add_argument("--4bit", dest="four_bit", action="store_true", help="load in 4-bit, for very large models")
p.add_argument("--data", default="data/smoke.jsonl", help="conversations to test")
p.add_argument("--n", type=int, default=None, help="conversations per file (default: all)")
p.add_argument("--max-new", type=int, default=1024, help="max tokens generated per reply")
p.add_argument("--show", type=int, default=4000, help="max characters printed per field")
args = p.parse_args()

def load_model(path):
    kw = {"torch_dtype": torch.bfloat16, "device_map": "auto"}
    if args.four_bit:
        kw["quantization_config"] = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
    classes = [AutoModelForImageTextToText, AutoModelForCausalLM]
    try:
        from transformers import AutoModelForMultimodalLM      # newer Transformers only
        classes.insert(0, AutoModelForMultimodalLM)
    except ImportError:
        pass
    errors = []
    for cls in classes:
        try:
            return cls.from_pretrained(path, **kw)
        except ValueError as e:              # class doesn't fit this architecture: try the next
            errors.append(f"{cls.__name__}: {e}")
    raise SystemExit("Couldn't load " + path + "\n" + "\n".join(errors))

def load_prompts(path, n):
    rows = [json.loads(l)["messages"] for l in open(path, encoding="utf-8") if l.strip()]
    random.Random(1).shuffle(rows)            # same selection every run
    out = []
    for msgs in rows[:n]:
        last = max(i for i, m in enumerate(msgs) if m["role"] == "assistant")
        out.append((msgs[:last], msgs[last]["content"]))
    return out

tok = AutoTokenizer.from_pretrained(args.base)

def gen(model, msgs):
    ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt",
                                  return_dict=True, enable_thinking=False).to(model.device)
    out = model.generate(**ids, max_new_tokens=args.max_new, do_sample=True, temperature=0.8,
                         top_p=0.95, pad_token_id=tok.eos_token_id)
    return tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=True).strip()

prompts = [("TRAIN", p) for p in load_prompts(args.data, args.n)]
if os.path.exists("data/heldout.jsonl"):
    prompts += [("HELDOUT", p) for p in load_prompts("data/heldout.jsonl", args.n)]

base = load_model(args.base)
base_out = [gen(base, msgs) for _, (msgs, _) in prompts]

if os.path.exists(os.path.join(args.tuned, "adapter_config.json")):
    tuned = PeftModel.from_pretrained(base, args.tuned)       # LoRA on top of the base
else:
    del base; torch.cuda.empty_cache()
    tuned = load_model(args.tuned)                           # merged or full model

S = args.show
for (split, (msgs, real)), b in zip(prompts, base_out):
    print(f"\n===== {split} | {msgs[0]['content'][:80]} =====")
    print("LAST MSG:", msgs[-1]["content"][:S])
    print("REAL    :", real[:S])
    print("BASE    :", b[:S])
    print("TUNED   :", gen(tuned, msgs)[:S])

Run it with the output folder as the argument, and save the results to a file you can read in the Jupyter file browser:

# Qwen
python3 compare.py outputs/qwen-smoke | tee compare.txt

# Mistral: local BF16 base, loaded in 4-bit to fit one H200
python3 compare.py outputs/mistral-smoke --base /workspace/models/mistral-medium-3.5-bf16 --4bit | tee compare.txt

# The full training set, a fixed sample of 20
python3 compare.py outputs/qwen-smoke --data data/train.jsonl --n 20 | tee compare.txt

The script works for LoRA folders (it applies the adapter on top of the base) and for merged or full models (it loads them directly). It generates with thinking off to match training, and tests the held-out set automatically once data/heldout.jsonl exists. Each conversation means two generations of up to 1,024 tokens, so expect 30–60 seconds per conversation on a 27B model.

Step 7: Chat with your model

There are two ways to chat with the result. Serving through vLLM gives you an OpenAI-compatible endpoint for any frontend you care to use. Converting to a single GGUF file works with KoboldAI. In my experience, starting with KoboldAI for testing has much less friction involved, and even if you end up needing full weights for your actual use case, an 8-bit test is

Merge the adapter into the base model

The adapter from training is small and depends on the base model. Merging bakes it into a standalone, full-size model. This loads the base in full precision, adds the adapter, and writes the result to outputs/qwen-smoke/merged, about 55GB for the 27B model.

axolotl merge-lora configs/qwen-lora.yml --lora-model-dir outputs/qwen-smoke

Option 2: a single GGUF file

A GGUF is one quantized file that KoboldCpp and text-generation-webui's llama.cpp loader can open directly. Install llama.cpp once, in its own environment on the volume:

cd /workspace
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
python -m venv /workspace/llamacpp-venv
source /workspace/llamacpp-venv/bin/activate
pip install -r requirements.txt
pip install cmake
cmake -B build
cmake --build build --target llama-quantize -j
deactivate

The separate environment matters: llama.cpp's requirements install their own PyTorch, which would replace the CUDA build Axolotl needs. If cmake -B build complains about a missing compiler, run apt update && apt install -y build-essential first.

This script runs the whole chain: merge, convert to a full-precision GGUF, quantize, and clean up. Save it as /workspace/make_gguf.sh:

#!/bin/bash
# usage: bash /workspace/make_gguf.sh Q6_K
set -e
QUANT=${1:-Q6_K}
PROJECT=/workspace/persona
CONFIG=configs/qwen-lora.yml
ADAPTER=outputs/qwen-smoke
NAME=persona

cd $PROJECT
axolotl merge-lora $CONFIG --lora-model-dir $ADAPTER    # run in the Axolotl environment

source /workspace/llamacpp-venv/bin/activate
cd /workspace/llama.cpp
python convert_hf_to_gguf.py "$PROJECT/$ADAPTER/merged" --outtype bf16 --outfile /workspace/$NAME-bf16.gguf
./build/bin/llama-quantize /workspace/$NAME-bf16.gguf "/workspace/$NAME-$QUANT.gguf" "$QUANT"
rm /workspace/$NAME-bf16.gguf
deactivate
echo "Done: /workspace/$NAME-$QUANT.gguf"

Run it with bash /workspace/make_gguf.sh Q6_K, swapping in any quant type. Choose based on the VRAM where you'll run the model, leaving a few GB for context:

Store the model in a private Hugging Face repo

Uploading from the pod is far faster than downloading to your machine first. Use a separate environment for the CLI, so its newer huggingface_hub doesn't break Transformers elsewhere:

python -m venv /workspace/hf-venv
source /workspace/hf-venv/bin/activate
pip install -U huggingface_hub
hf auth login                                    # paste a token with WRITE access

hf repo create persona-gguf --private
hf upload YOUR_USERNAME/persona-gguf /workspace/persona-Q6_K.gguf persona-Q6_K.gguf --private

# The merged safetensors model, for vLLM or making new quants later
hf repo create persona-merged --private
hf upload-large-folder YOUR_USERNAME/persona-merged /workspace/persona/outputs/qwen-smoke/merged --repo-type model --private

Run the GGUF in KoboldCpp on Runpod

KoboldCpp exposes an OpenAI-compatible endpoint as well, so you can send the request to the pod URL and the port that it exposes in the pod.

KoboldCpp's official Runpod template fetches the latest KoboldCpp at startup, so its llama.cpp stays current. Its KCPP_MODEL variable expects a download URL, and private repos refuse downloads without your login, so load from the volume instead:

1. Download the GGUF to your network volume with hf download, as above.
2. Deploy the KoboldCpp template with that volume attached.
3. Clear KCPP_MODEL, and set KCPP_ARGS to point at the local file:

--model /workspace/models/persona-Q6_K.gguf --usecublas mmq --gpulayers 999 --contextsize 16384 --flashattention --hordemodelname persona

If you do end up putting the model in an accessible Huggingface repo, you could put the URL of the files in KCPP_MODEL instead and host it there instead of on a network volume.

Wrapping up

You now have the full pipeline: a dataset format that keeps modes separate, scripts that catch data mistakes before they cost GPU time, configs for a 27B and a 128B model, a smoke test that proves the setup in minutes, a side-by-side comparison against the base, and two ways to chat with the result.

From here, the work that pays off most is the data. Add conversations, keep labels consistent, hold out a test set, and let compare.py tell you whether each change helped. When a 27B model's scene consistency starts to limit you, the same files and scripts carry straight over to a larger dense model: only the config and the GPU change.

Ready to start? Create a network volume and deploy an H200 pod on Runpod, and your first smoke test can be running within the hour.

Questions? Feel free to drop by our Discord or our Reddit.

‍