How I Engineered the World's First Khattak Pashto LLM (Qwen2 + LoRA)

Introduction

Standard Large Language Models (LLMs) are incredible, but they have a massive blind spot: rural and regional dialects. While standard Pashto is supported by some models, they completely fail to capture the rich, localized grammar of the Khattak dialect spoken in Karak, Nowshera, and Kohat (Pakistan). To solve this, I built Khatta-ka-LLM - the world's first AI language model fine-tuned specifically for the Khattak (Khatak) dialect. (Note: You can read the original architectural breakdown on my Software Engineering Portfolio or test the model live below). Watch the Live Testing Video: 🛠️

Technical Architecture

As a Software Engineer specializing in AI Infrastructure, my focus was on creating a clean, scalable, and reproducible pipeline for low-resource languages. Here is how I built it:

The Custom Dataset

Because no dataset existed for this dialect, I curated a custom dataset of 2,000+ English-to-Khattak pairs. This represents the first digitized collection of Khattak linguistic markers, focusing heavily on the unique grammar used in the Karak and Kohat regions.

The Fine-Tuning Pipeline

To achieve high performance on consumer-grade hardware, I utilized the following stack:

• Base Model: Qwen2 Architecture
• Framework: Unsloth (for highly optimized, faster training)
• Technique: LoRA (Low-Rank Adaptation)
• Performance: Through careful hyperparameter tuning, the training loss dropped significantly from 3.44 to 0.22.

The Linguistic Engineering (Rulebook)

To ensure the AI didn't just speak "standard Pashto with a Khattak accent," I had to engineer specific linguistic rules into the model's training weights. Here are a few of the core rules the model learned:

1. Pronouns & Possession

• My / Mine: The model uses مو والا (Mo wala) instead of the standard Zama.
• To me: Uses موته (Mota) instead of standard Mala.
• We: Uses موخ (Moxh) instead of standard Mung.

2. The "Noon Ghunna" (ں) Rule

In the Khattak dialect, first-person verbs end in a nasal Noon Ghunna (ں) rather than the standard "M".

• Standard: Za Kar Kawom (I do work)
• Khattak AI: زه چار کاوں (Za char kaon)

3. Phonetic "O" Shifts

The model successfully converts standard "A" sounds to the deep Khattak "O":

• Asmaan ➡️ Asmon (اسمون)
• Baraan ➡️ Baron (بارون)
• Bazaar ➡️ Bazor (بازور)

Why This Matters

By focusing on high-quality, localized data, Khatta-ka-LLM successfully translates English into pure Khattak Pashto. As a Software Engineer from Pakistan, my goal is to bridge the gap between modern AI infrastructure and regional linguistic heritage. We shouldn't have to lose our dialects to use modern technology. I would love to hear your thoughts on fine-tuning models for low-resource languages! Have you worked with Unsloth or LoRA recently? Let me know in the comments. 👇

About the Developer

I am Muhammad Zubair, a Software, DevOps, and Platform Engineer from Pakistan. I specialize in architecting scalable AI infrastructure, cloud platforms (AWS/Kubernetes), and full-stack systems. My passion lies in solving real-world problems through code, whether that's building real-time mobility radars for the visually impaired or preserving regional languages through AI.

Let's Connect

• Portfolio & Case Studies: mzubair.online
• GitHub (76+ Repos): Muhammad-Zubair796
• LinkedIn: Muhammad Zubair

Top comments (0)