Common Terms of Reference
Before I start, some common terms of reference would be helpful so that we are on the same page:
- Quantisation (Quant, Q) -
- Mixture of Experts vs Dense (MoE, A3B) -
- Number of Parameters, Billions (27B, 31B)
I previously used to document detailed instructions on how to get things going, but with AI these days, I think the basics are well covered, so I’ll focus more on the nuances between the decisions that I needed to make.
Quickstart: Runpod
Disclosure: the above link is a referral link
The above link will take you to runpod and launch a template that I’ve pre-configured to run Qwen 3.6 27B (Q8_XL) with an RTX 6000. The api key is in the launch template, so please make sure to change it.
On launching, a sidebar like the above will open up. The inference server will be exposed via a HTTP endpoint that follows this format: https://[POD-ID]-[PORT].proxy.runpod.net/. You can also click the llama link to obtain the URL.
It will take a few minutes for the container to initialise as it is downloading a big model from huggingface. So you can wait until the container indicates it is ready. You can view the logs under Logs > Container.
This usually indicates that the container is still downloading.
Meanwhile, the coding harness needs to be set up. For ease of use, I’ll be using Opencode as an example. After installing opencode, depending on your install location, you’ll need to find opencode.json. On my Windows machine it is at [User Directory]/.config/opencode/opencode.json. Add the following configuration. Change the api key if you have changed it from the default (you have changed it from the default, right?). Please note that you need to add a v1 behind the base URL.
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"vllm": {
"npm": "@ai-sdk/openai-compatible",
"name": "RunPod vLLM",
"options": {
"baseURL": "https://[POD-ID]-[POD].proxy.runpod.net/v1",
"apiKey": "my-secret-key"
},
"models": {
"qwen3.6": {
"name": "Qwen 3.6 27B"
}
}
}
}
}
And when runpod is ready, you’ll be able to access your very own Qwen instance!
Some additional items:
- It is set up with two concurrent request streams, so you can have two agents working simultaneously
- It is set up for coding.
Choosing an LLM
Lets get started with the fun part! There are a ton of models to choose from and many different opinions so here is mine (June 2026). The models here are selected
Qwen 3.6
Qwen comes in two variants 27B (Dense) and 35B-A3B (MoE). Dense models activate all parameters for every token, MoE (Mixture of Experts) model only activate part of the model per token depending on the context. Dense is smarter but more memory intensive, MoE is faster and is less demanding on hardware. Qwen is really good at coding, and I’m using the UD-IQ3_XXS variant from unsloth locally. On the cloud, I use the the Q6 variants as a good balance between speed and intelligence.
I generally find the 35B model to be faster, but it tends to make more mistakes, and get stuck more often. The 27B model is slightly better.
Gemma 4
Gemma 4 comes from Google and also comes in the same two variants, a 31B dense and a 24B MoE model. I used the Q3_K_M variant from unsloth, but in my experience, it got stuck and started to loop a few times, so my impression wasn’t as good, but the smaller E4B variants are quicker so YMMV.
Running your chosen LLM
Smartest - Cloud
I’ve provided my own template that I use to spin up Qwen 3.6 26B UD-Q8_K_XL, which allows it to perform at near native accuracy while keeping the inference speed reasonably fast. Since there is plenty of headroom on a RTX 6000 PRO, I’ve also provisioned two slots to maximise usage. This allows it to perform inference on two parallel threads concurrently.
At the time of writing, this costs $1.89/hr on Runpod’s Secure Cloud.
Cost efficient - Cloud
For when the money seems to tick down a bit too quickly on Runpod
Local
While Ollama still wins out on being the easiest to use, I don’t recommend them.
Lmstudio is a very easy, all-in-one solution to run local LLMs. It also includes a model selector of curated models.
![[Pasted image 20260506165310.png]]
After downloading the model, remember to set the context window to the size that the model and your GPU can support
![[Pasted image 20260506170214.png]]
You can already start using it through lmstudio.
My personal fav
There are a few ways you can run this. I personally use wsl2 in Windows to run, but if you have Docker installed, it may be quicker to use a prebuilt docker image and just have it download the LLM in the container.
Expose ports
Interacting with your LLM - Harness
I lied, this is the best part. You have chosen your LLM, got it running, and finally you get to play with it!
From prompt engineering, to context engineering, and now to harness engineering.
Chat
Openwebui
Agentic
Claude Code is optimised for Anthropic models, so I recommend choosing something else. For an out of the box experience, I recommend going with opencode. Once you get more experience and want a lighter harness, I recommend pi.dev for its sheer extensibility.
Opencode can be a bit prompt heavy - about 10k tokens before the first word of your prompt is read in. This diluted my desired system prompt, and affected prompt adherence, especially for edge cases.
pi goes in the opposite philosophy of a lot of harnesses these days, which are trying to build in more and more checks to prevent any unintended agentic mishaps. However, it is my belief that overly restricting an agent’s harness prevents it from being fully agentic, so I prefer to deploy agents in a container, which isolates any harmful actions from the host.
General advice
-
Development: I like having a spec-coder-reviewer loop. I will first start planning with the heaviest/smartest model I can get my hands on, and once I am happy with the plan, I’ll tell it to generate a handoff document so that I can pass to another LLM for development. For the coder agent, I like having a reviewer subagent that judges the outcome of the code and measure if it meets the spec. This allows me to get 95% of the way to my vision and I don’t need to intervene.
-
Grounding: LLMs are hampered because their training data only reaches up to a certain date. For it to be useful and accurate, having a
websearchtool to look up recent changes, new API, new terms, is very useful and one of the first things I would try to build or adopt into the harness. -
Advisor
WARNING - HERE BE DRAGONS
There are some model variant types (yes I know - more??), and this section specifically deals with the spicy variants.
- Abliteration - Directional ablation. A model’s refusal behaviour is guided by a specific direction in the model’s stream (think of this as a path that goes through a model’s layers). If we prevent the model from moving
- Heretic - Heretic is a tool that removes censorship
Local Considerations
Some thoughts I had against using a Local LLM*:
- Cost does not make sense. The cost of the GPU has to be amortized over multiple years to come close to the price you are paying to access the latest frontier models via a subscription.
- The performance does not make sense. Running a local model is inherently less capable than a cloud model like Opus.
- The middle way does not make sense. Even renting a VPS with a beefy GPU attached to run the best local LLM works out to be higher than the monthly subscription.
However, recent events have made me wonder if my usage was being subsidised. Anthropic, the ‘good guys’, throttled the usage of models and removed Claude Code from the Pro tier.
If we also consider using the API vs the subscription, it works out to be almost 10x difference in price, which is strange because it is still the same model being served. To me, the true cost is via the API (when the price settles).
However, the recent events from Anthropic: usage throttling and removal of Claude Code from the Pro tier thrusts running a Local LLM into a new light for me.
The trap of dependence
I’ve never been a cloud guy. A few years ago I argued that if you were not paying for the true cost of maintaining a service in perpetuity, you will one day be deprived of it.
And I think this is the same with these AI companies. The recent changes from Anthropic show that the service is subsidised.
There is nothing wrong with taking advantage of a good deal with it comes along.
But for something like IoT infrastructure, where I wouldn’t want to change it once it is installed, or having an LLM that turns my thoughts into code, the danger is dependence.
There is a formal phrase for this: vendor lock-in. Terrible if you are the client, great if you are the vendor. In this scenario, I’m the client.
The restraint of guardrails
AI companies are in a constant battle to ensure that their products do not produce any harmful content because they are businesses. And no one wants to be known as the business that produces harmful content.
Therefore, ever since the dawn of GPT-1, LLMs had guardrails to prevent the generation of harmful content, because if you think about it, LLMs are stochastic machines and do not have any inherent morality, so there needs to be defences against people who would use it for evil.
I think this is necessary.
But I also work in cybersecurity, which means some questions I ask of an LLM may have dual use.
Therefore, ever since the dawn of GPT-1, LLMs had guardrails to prevent the generation of harmful content, because if you think about it, LLMs are stochastic machines and do not have any inherent morality, therefore, they need external guides to tell them what not to do.
I think this is necessary.
But I also work in cybersecurity, specifically offensive security. This is one of the things that an LLM is told explicitly NOT to do.
For all other purposes, I think taking advantage of the cloud models is fine, especially when the prices are so low. However, for my specific use-case, I would need to run a Local LLM.
Public companies cannot be seen abetting crimes, so the natural course of action is to restrict by default.
While there are ways to bypass this restriction by certifying your use with the company, it is quite a cumbersome process, and it may not succeed.
Therefore, being able to ensure that I can get work done, as needed is also another reason why I started to look into running local models.