Как я тестировал локально новый Qwen 3.6 и Gemma 4

Запускаю Qwen3.6-35B-A3B-UD-Q4_K_M.gguf+mmproj от Unsloth на MI50 32GB. Собирал llama.cpp по инструкции.

Параметры запуска (llama-server запущен в режиме router через --models-presets):

[*]
host = 0.0.0.0
batch-size = 4096
n-gpu-layers = all
flash-attn = on
jinja = true
threads = -1
no-context-shift = false

[qwen3.6-35b-a3b]
model = /var/lib/llama/models/unsloth/Qwen3.6-35B-A3B-GGUF/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
mmproj = /var/lib/llama/models/unsloth/Qwen3.6-35B-A3B-GGUF/mmproj-F16.gguf
ctx-size = 262144
batch-size = 8192
ubatch-size = 2048
cache-ram = 24576
ctx-checkpoints = 8
parallel = 2
load-on-startup = false
n-predict = 32768
temperature = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
presence-penalty = 0.0
repeat-penalty = 1.0
chat-template-kwargs = {"enable_thinking":true}

Скорость ~56 ток/сек: