Open-weight 4B models approach o3-level medical question answering in Swedish [P]

Background

I have been running some experiments with smaller open-weight LLMs on multiple-choice questions of Swedish medical licensing exams. On a dataset called MedQA-SWE, GPT-4 scored 84% accuracy in 2024 and o3 scored 88% in 2025 on a smaller, overlapping dataset.

Post-training Results

With post-training (SFT) on data from earlier years, I got MedGemma-1.5-4B to a passing score of 60% on the final year’s exam. Find the implementation here: https://github.com/tarolangner/medqaswe_medgemma_sft.

But even though they were released just three months later, Gemma4-E4B and Qwen3.5-4B are flat out superior already, at 77% with no post-training at all.

Newer Models and Reasoning

With reasoning enabled, Qwen3.5-4B can get to 87% accuracy. It can even push a bit further if no length cap is put on the reasoning traces, but some of them spiral into repetitive loops about formatting that fill the entire context length without giving any answer. Here, I found it helpful to use an “early exit” thinking intervention proposed in the S-GRPO paper that simply injects a phrase and closes the thinking trace at a predetermined sequence length.

I also tried their proposed reinforcement learning method to get shorter reasoning traces, but with only minor gains (probably a somewhat underdimensioned training setup).

Observations

Curiously, Qwen3.5-4B does all reasoning in English despite the Swedish prompt, questions and answer options. But it really seems like the language is no obstacle, even though it’s often estimated to be just 1% of LLM training data.

Further Reading

I also have a more detailed write-up on the details and experiments here for anyone interested: https://tensorlabbet.com/2026/07/19/medqaswe_post_training/.