Discriminative Fine-Tuning: Why Your Backbone and Your Head Shouldn't Learn at the Same Speed

I was recently fine-tuning DEIMv2 - a DETR-style detector with a DINOv3 Vision Transformer backbone - starting from a COCO-pretrained checkpoint, to build a single-class face detector. While wiring up the optimizer, I gave the backbone a learning rate of 5e-6 and the decoder head 1e-4 - a 20x gap. That wasn't an arbitrary choice. It's a named, well-established technique called discriminative fine-tuning, or layer-wise learning rate decay (LLRD). This post walks through the reasoning in the order I actually arrived at it - not as an abstract lecture, but as a practical decision framework, illustrated with two real fine-tuning runs from the same week of work: one where LLRD was the right call, and one where it wasn't. The hidden assumption in "just lower the learning rate" The most common fine-tuning recipe looks like this: load a pretrained checkpoint, drop the learning rate relative to what you'd use training from scratch, and continue training - with one single learning rate for the entire network. It's simple, and it's the default in almost every framework's train() call. That recipe carries a hidden assumption: every layer in the model needs to be updated at the same rate. This assumption holds in exactly one situation - when the pretrained checkpoint already solves your task, and you're simply continuing training on more or different data. It breaks down the moment you're doing something closer to transfer learning across tasks or domains - for example, taking a general-purpose 80-class COCO detector and repurposing it to detect exactly one thing it was never trained to recognize as a distinct category: faces. The backbone has already learned what an edge, a texture, a spatial gradient looks like - knowledge that's broadly reusable across almost any image. The classification head, on the other hand, has never seen "face" as its own class. It needs to learn that concept close to from scratch. Force both to update at the same learning rate, and you land in one of two failure modes: - LR high enough for the head to converge quickly → the backbone gets dragged along at the same pace, and its general-purpose features start degrading before the head even has time to stabilize. This is the classic symptom of catastrophic forgetting: a network's earlier, more general representations get overwritten by gradient updates optimized for a narrower, newer objective. - LR low enough to keep the backbone safe → the head, which genuinely needs to move fast, converges painfully slowly, burning epochs and compute that would otherwise be productive. Discriminative fine-tuning resolves this tension directly: give every part of the network its own learning rate, scaled inversely to how "general" that part's learned representations already are. Not a new idea - just an underused one This isn't a novel trick. It traces back to ULMFiT (Howard & Ruder, 2018), where it was introduced for fine-tuning language models in NLP under the name "discriminative fine-tuning." The core idea generalized cleanly and is now standard practice across transformer fine-tuning - layer-wise LR decay shows up in BERT-family fine-tuning recipes, and in vision transformer fine-tuning work (e.g. MAE, BEiT-style recipes commonly apply a per-layer decay factor from the output layers back toward the input). The rule of thumb, in one sentence: layers closer to the input (more general) get a lower learning rate; layers closer to the output (more task-specific) get a higher one. A real example - and a real counterexample In the same week, I was fine-tuning two different detectors for the exact same downstream task (face detection), and I only applied LLRD to one of them. That contrast is the clearest way to explain when this technique actually matters. | YOLO (this project) | DEIMv2-S | | |---|---|---| | Starting checkpoint | Already a face detector (previously fine-tuned on WIDER FACE) | General-purpose COCO detector, 80 classes - had never seen "face" as a category | | What this training run actually is | Continued training - same task, more/different data | Cross-task transfer - a genuine task change | | Learning rate setup | One flat LR (1e-3 ) for the entire network | 5e-6 (backbone) vs 1e-4 (head/decoder) - a 20x split | | Does it need discriminative LR? | No - the real risk here is a different one (an aggressive warmup schedule disturbing an already-converged optimum) | Yes - the head has to learn a new concept from near-scratch while the backbone should mostly stay put | The insight worth internalizing here: discriminative fine-tuning is not a "just add it, can't hurt" default. It's a targeted answer to one specific problem - cross-task or cross-domain transfer, where different parts of the network genuinely have different amounts of "relearning" to do. When the starting checkpoint already solves your task and you're just extending training, a single global LR is often the right (and simpler) choice - the real risks in that case tend to be elsewhere, like warmup schedules that perturb an optimum that didn't need perturbing. A quick self-test: has your pretrained checkpoint already solved this exact task before (just on less data), or is it solving a different task and you're borrowing its learned features? The answer determines whether you need discriminative LR - not the architecture, not the framework. Implementing it: it's just parameter groups No special framework support is strictly required - it's a matter of partitioning the model's parameters into named groups (by module path, typically via regex or explicit submodule references) and assigning each group its own learning rate at optimizer construction time: # Group by module -- no changes needed inside the model itself, # this all happens at optimizer construction time. optimizer = torch.optim.AdamW([ {"params": backbone.parameters(), "lr": 5e-6}, # pretrained, keep it stable {"params": encoder.parameters(), "lr": 3e-5}, # intermediate {"params": decoder_head.parameters(),"lr": 1e-4}, # relearning from near-scratch, move faster ]) Some modern detection frameworks (DEIMv2, D-FINE, RT-DETR and relatives) expose this directly through config - you declare regex patterns that match parameter names, and each pattern gets its own LR override. Ultralytics YOLO, notably, does not support this out of the box. Its build_optimizer only splits parameters into groups by parameter type - decayed weights, non-decayed weights/BatchNorm, and biases - every group still shares the same lr value: # ultralytics/engine/trainer.py (abbreviated) optim_args = {"lr": lr, "betas": (momentum, 0.999), "weight_decay": 0.0} ... g[2] = {"params": g[2], **optim_args, "param_group": "bias"} g[0] = {"params": g[0], **optim_args, "weight_decay": decay, "param_group": "weight"} If you want backbone/head LLRD in ultralytics, you have to build the optimizer yourself and hand it to a custom trainer - there's no backbone_lr= argument waiting for you in the CLI. A decision checklist Reach for discriminative fine-tuning when: - The pretrained checkpoint comes from a different domain or task than the one you're fine-tuning for. - The backbone is large/capable and you specifically want to avoid disturbing what it already learned. - The head/decoder has a substantially different architecture or role than the backbone (e.g. a DETR-style attention decoder sitting on top of a CNN or ViT backbone). Skip it, use a single global LR, when: - The starting checkpoint already solves your exact task - you're extending training, not transferring across tasks. - The model is small/simple enough that per-group tuning adds complexity without a clear payoff. - You're training from scratch - there's no "already-learned" representation to protect in the first place. Going a bit further: how to actually pick the ratio A flat two-group split (backbone vs. head) is often enough, but if you want a smoother gradient across many layers, the common recipe from BERT/ViT fine-tuning literature applies a multiplicative decay per layer, working backward from the output: lr_l = lr_top * (decay_factor) ** (num_layers - l) with decay_factor typically somewhere in the 0.9 -0.95 range per layer. In practice, a simpler two- or three-tier split (backbone / encoder / head, as shown above) captures most of the benefit for far less config complexity - reserve the full per-layer decay for cases where you have a very deep backbone and empirical evidence that a coarser split isn't cutting it. The takeaway Discriminative fine-tuning isn't a score-boosting trick to bolt onto every training run "just in case." It's a precise answer to a specific question: when different parts of a network need different amounts of relearning, why force them through the same optimizer step size? Recognizing which situation you're actually in - cross-task transfer versus continued training on the same task - matters more than knowing the technique exists at all. Written up from a real fine-tuning session comparing DEIMv2 (DINOv3 backbone) and YOLO for face detection, on the differences in how each checkpoint's starting point shaped the right optimizer strategy. Top comments (0)