Bringing up local inference on the Intel Arc Pro B70

The first lesson from bringing up local inference on Intel’s Arc Pro B70 was simple: a model producing tokens is not the same thing as a working GPU inference stack.

I wanted to answer three practical questions:

1. Can the card run the 20B–30B open-weight models I use for coding-agent work?
2. Which runtime gives me a trustworthy GPU path today?
3. What happens when I push beyond a short demo and toward 128K or 262K context?

The short answer is that llama.cpp with SYCL is the path I trust so far. A patched vLLM build can load and decode, but I have not yet shown that it is using the B70 correctly. The long answer is mostly about measurement discipline.

The two stacks

I brought up two separate paths rather than treating “Intel GPU support” as one thing.

Path	Environment	Why I tried it	Current state
llama.cpp + SYCL	Native build from source	Direct control, GGUF support, simple GPU-offload checks	Working
Patched vLLM + XPU	Ubuntu 24.04 under WSL	OpenAI-compatible serving and paged attention	Loads and decodes; GPU execution is not yet verified

The distinction matters. The llama.cpp path exposes enough information to check layer placement and run focused CLI benchmarks. vLLM gives me the serving interface I ultimately want, but a healthy API response can conceal a fallback or incomplete backend.

The model set

I used three model families that stress different parts of the stack:

• Qwen3-Coder-30B-A3B-Instruct, IQ4_XS GGUF, about 15.6 GiB. This was the main coding workload.
• Qwen3-30B-A3B-Thinking-2507, UD-Q5_K_XL GGUF, about 20.2 GiB. This gives the runtime a larger quantized load and a reasoning-oriented prompt format.
• GPT-OSS 20B, MXFP4, about 12.8 GiB. This was useful for exposing the difference between throughput and valid output.

The Qwen models advertise context windows in the 128K–262K range. That does not mean a single-device configuration can materialize every advertised token with an ordinary full-size KV cache. I return to that below.

Bringing up the SYCL path

The native route was straightforward in shape, if not in every build detail:

1. Install Intel’s oneAPI runtime and compiler components.
2. Build llama.cpp from source with the SYCL backend enabled.
3. Run a short prompt with all eligible layers requested on the GPU.
4. Inspect startup logs and device activity before collecting a number.

The benchmark and interactive invocations looked like this:

./llama-batched-bench \
  -m /path/to/model.gguf \
  --n-gpu-layers 999 \
  --ctx-size 32768

./llama-cli \
  -m /path/to/model.gguf \
  --n-gpu-layers 999 \
  --ctx-size 32768 \
  --prompt "Write a small Python LRU cache and explain the eviction rule."

The important flag is not the exact value 999; it is the intent to place every supported layer on the accelerator, followed by checking whether that actually happened. A launch command is a request, not evidence.

A result worth keeping

For Qwen3-Coder-30B-A3B-Instruct IQ4_XS, the SYCL build produced:

• 38.07 generated tokens/second
• a coherent coding response
• observed GPU activity during the run

This is an exploratory result, not a cross-platform benchmark. It came from one model, one quantization, one prompt shape, and an evolving software stack. I am keeping it because it proves that this particular path can do useful work and gives me a baseline for regressions.

A result worth rejecting

GPT-OSS 20B initially looked measurable:

• three-run average wall time: 409.249 seconds
• average output length: 1,205 characters

The output was repetitive and malformed. That makes the timing unsuitable as a model-performance result. It still told me something about runtime behavior, but it did not answer the question a user cares about: how quickly can the system produce a valid answer?

That failure changed the harness. Every run now needs a basic quality gate alongside timing: required structure, repetition checks, and a saved sample for inspection. Throughput without output validation is an easy way to benchmark a broken path.

The vLLM path: success at the wrong layer

The second stack used a patched vLLM build under Ubuntu 24.04 in WSL. The target was an OpenAI-compatible server with a large configured context:

vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --device xpu \
  --max-model-len 262144 \
  --enforce-eager

After patching the XPU compatibility points, vLLM could:

• discover and load the model,
• initialize the server,
• accept a short request,
• and return decoded text.

That sounds like success until the telemetry is included. During the test, Arc GPU utilization did not rise in a way I could trust, while CPU load was substantial. Intel’s GPU process reporting also did not show the kind of per-process evidence I expected.

So the accurate status is:

The vLLM request path works, but the intended B70 execution path is unproven.

I am not counting those tokens as GPU inference until I can correlate a request with device utilization, memory allocation, and backend logs. A working endpoint is a functional milestone; it is not a hardware benchmark.

Why the advertised context does not fit by default

Long context turns KV-cache arithmetic into a first-order design constraint. A rough upper bound is:

KV bytes ≈ token count × KV bytes per token

Using the model metadata I recorded during bring-up:

Model	Approx. KV bytes/token	KV at 131,072 tokens	KV at 262,144 tokens
Qwen3-Coder 30B-A3B	274,944 B	33.6 GiB	67.1 GiB
Qwen3 30B-A3B Thinking	174,080 B	21.3 GiB	42.5 GiB
GPT-OSS 20B	163,840 B	20.0 GiB	40.0 GiB

These are planning estimates, not measured peak allocations. They omit runtime overhead and sit alongside the model weights themselves. The practical conclusion is still clear: setting --max-model-len 262144 does not reserve a magical 262K-token capability on one card.

Paged KV allocation helps avoid reserving the entire cache up front. It does not erase the eventual byte cost. Reaching the top of the advertised window may require some combination of KV quantization, host offload, sliding-window behavior, cache compression, or multiple devices.

What I would measure before publishing a comparison

My initial runs were useful for bring-up, but they were not controlled enough for a vendor or runtime comparison. A defensible pass needs:

• pinned runtime commits, driver versions, and model hashes;
• separate prompt-processing and token-generation rates;
• fixed prompt and generation lengths;
• warm-up runs separated from measured runs;
• saved outputs with an explicit validity check;
• device utilization and memory telemetry captured over the same interval;
• a context sweep rather than a single max_model_len setting;
• and failure results reported beside successful ones.

The next harness will run context checkpoints at 32K, 64K, 128K, and then higher only if the memory trace supports it. That is slower than copying a tokens-per-second number into a chart, but it answers the question I actually have: can this stack support a coding agent over a long session without silently changing execution paths or collapsing output quality?

What I know now

Three conclusions survived the bring-up work:

1. SYCL llama.cpp is currently the credible B70 path for my workloads. It runs the quantized coding model, exposes enough diagnostics to verify placement, and produced a valid 38.07 tok/s baseline.
2. vLLM is not verified just because it serves a response. The patched XPU path needs better telemetry and backend confirmation before its results belong in a benchmark.
3. Maximum context is a memory architecture problem, not a command-line number. Model support for 262K tokens and a single-device deployment capable of using 262K tokens are different claims.

That is less exciting than declaring the card solved on day one. It is also much more useful. The stack now has one path I can trust, one path with a clearly defined blocker, and a benchmark plan designed to catch the exact mistakes the first pass exposed.