DEV Community

Silent nginx Config Bugs That Pass nginx -t: if, location, add_header, alias

nginx -t returns syntax is ok and test is successful. nginx -s reload finishes without a complaint. Nothing shows up in the error log. And yet the behavior is wrong.

I've done this to myself more times than I'd like to admit. The one that hurt most was a CSP header I shipped to production that never reached a single browser for a full day - I only noticed because I got suspicious that no violation reports were coming in. There have been others: a file I put under /static/ getting swallowed by a different location, old behavior lingering after a reload. In every case nginx -t passed, and nothing was grammatically wrong anywhere.

The nature of this gap is clear: nginx -t validates only syntactic correctness, and never looks at semantic correctness.

What nginx -t actually guarantees for you

What nginx -t does, roughly speaking, is check "is this config file in a shape that nginx can load?" Does the directive name exist? Is the argument count right? Are the {} blocks closed? Are the value types valid? In compiler terms, it goes about as far as syntax errors and maybe type checking. Everything past that point - "will this config process requests the way you intended?" - is outside the scope of nginx -t.

This is where the blind spot is. When the test passes, you quietly reread it as "the config is correct." But all that passed was the syntax test.

Outside nginx -t, there are three layers of breakage:

  • Semantic level - the config isn't read the way you intended. The grammar is perfect.
  • Security level - it works exactly as you intended. It just also works exactly as the attacker intended.
  • Runtime level - the config text is irrelevant. It's the behavior of the reload operation itself.

The higher layers are the ones you step on daily, and their cause is easy to look for inside the config file. The lower you go, the more it becomes the kind of incident where reading the config file gives you no answer at all. We'll go top to bottom.

Reading this with your own nginx.conf open beside you should help. Note that this article assumes the behavior of the nginx 1.31 (mainline) series. The stable series is 1.30, and the behavior covered here - if, location selection, add_header inheritance, alias, and reload - is the same across both. Where a version makes a difference, I'll call it out explicitly.

Semantic level: the grammar is correct, but it's read differently

The contents of if aren't evaluated in the order you think

When you want to "branch on a condition" in an nginx config, if is the first thing you reach for. But an if inside a location has long been called "if is evil" in the nginx community. It sounds like a religious argument if you only hear the name, but it's a statement about behavior.

First, why it breaks. if is a directive of the rewrite module, evaluated in an early phase of request processing (the rewrite phase). And the nastier part: when you write an if inside a location, that if block is treated as an implicit nested location. When the condition is true, nginx switches the request's entire configuration context into that inner location. Not every directive gets dropped (many settings like root or proxy_set_header carry over). But array-style directives in particular - like add_header - and content-processing directives may not carry over the way you expect. That's the trap.

Let's look at a concrete example. This is completely valid grammatically and passes nginx -t:

location /images/ {
    add_header X-Served-By images;
    if ($arg_size = large) {
        add_header X-Size large; # I just want to add one header conditionally...
    }
}

The author's intent is "add an extra X-Size header only when ?size=large." But if creates an implicit nested location, so when the condition is true, the config context switches to the inside of it. And add_header "does not inherit from the parent if there's even one at the current level" (we'll look at this behavior in detail in the add_header section below). The result: on a size=large request, only the inner X-Size rides along, and the outer X-Served-By disappears. What you meant to add has replaced instead.

For the same reason, putting try_files or proxy_pass inside an if can get them ignored or make the URI silently rewrite, all because of this switch. And nginx -t says nothing. You ship it without noticing.

Detection and avoidance: The things that are genuinely safe inside an if in a location are, in practice, only return ..., rewrite ... last, set ... (and break). For anything else - header manipulation, try_files, proxy_pass - the moment you want to do it inside an if, that's a sign the design is wrong. Push the condition out into a map block (think of it as a lookup table that maps an input value to an output value), make a variable, and use that variable - or split the location. Not "if is evil, so don't use it," but "if creates an implicit location, so the outer config isn't inherited." Remember it by the reason, and next time your hand reaches for it, you can stop yourself.

Rewriting the earlier example with map looks like this:

# map goes in the http context (outside the server block)
map $arg_size $x_size {
    default "";       # normally an empty string
    large   "large";  # gets a value only on ?size=large
}

server {
    location /images/ {
        add_header X-Served-By images;
        add_header X-Size $x_size; # if the value is empty, this line isn't emitted
    }
}

When the value of add_header is an empty string, it doesn't emit that header. So X-Size is attached only on ?size=large and not otherwise - exactly what you wanted originally, expressed without an if. And since both add_header directives are at the same level in location /images/, the problem of X-Served-By disappearing never happens. Replacing conditional branching with a "variable lookup" rather than a "control statement" is the basic form of avoiding if.

Location priority is not the order you wrote them in

When you write multiple location blocks, don't you assume "the one written higher up takes priority"? I did. And it bit me. nginx's location selection runs on a rule that differs from the order you wrote.

The procedure is this:

  1. If an = (exact match) location matches the URI exactly, it's decided right there and nothing else is examined.
  2. Otherwise, among the prefix locations (no modifier, or ^~), pick and remember the one with the longest match. Order within the config file is irrelevant here. The longest match wins.
  3. If the selected prefix has ^~, it's decided there and regex is not checked.
  4. Otherwise, check the regex locations (~, ~*) in written order, and the first that matches wins. Here, written order matters.
  5. If no regex matches, use the prefix remembered in step 2.

So: "prefixes are longest-match (order-independent), regexes are first-match by written order, and regexes take priority over prefixes." That last point is what gets people.

A reproduction:

location /static/ {
    root /var/www;
}

location ~ \.(png|jpe?g|gif)$ {
    expires 30d;
    root /var/www/cache; # <- a different root
}

Let's trace a request to /static/logo.png. As a prefix, /static/ is the longest match and gets "remembered." But /static/ has no ^~. So nginx proceeds to the regex check, \.(png|jpe?g|gif)$ matches, and that one wins. The result: /static/logo.png goes looking in /var/www/cache. The /static/ block is skipped past. The grammar is perfect, nginx -t passes. Only your assumption - "I'm handling all static files under /static/" - is off.

Detection: Don't guess which location you're hitting - confirm it. The easy way is to temporarily add a marker header to the suspect locations:

location /static/ {
    add_header X-Loc "static" always;
    ...
}
location ~ \.(png|jpe?g|gif)$ {
    add_header X-Loc "regex" always;
    ...
}

Then hit curl -sI http://localhost/static/logo.png | grep X-Loc and you'll know instantly which one answered. Nailing down "where am I hitting right now" as a fact comes first.

Once you have that, if you want to lock it down, put ^~ on the prefix to stop the regex check itself:

location ^~ /static/ { # decided the moment this prefix is the longest match; regex is not examined
    root /var/www;
}

With add_header, adding one in the child makes all the parents disappear

This is the trap that most symbolizes the "semantic level" of the three layers, and the one I was stuck on the longest.

As a premise, many people write security headers like HSTS or CSP once in the server block, intending them to apply to every location.

server {
    add_header Strict-Transport-Security "max-age=63072000" always;
    add_header Content-Security-Policy "default-src 'self'" always;

    location /api/ {
        add_header Cache-Control "no-store" always; # <- the moment you add this
        proxy_pass http://backend;
    }
}

You just added one Cache-Control header to /api/. But nginx's add_header is specified so that if there's even one add_header at the current level, it does not inherit any add_header from the parent level at all. It's replacement, not merging. The result: HSTS and CSP disappear from the /api/ response, and all that's left is Cache-Control. Only the security headers go silently missing.

This "array-style directives don't inherit (don't merge with) the parent once defined in a child" behavior is not unique to add_header. proxy_set_header and fastcgi_param have the same trap. Write this, for example, and Host and X-Forwarded-For stop being passed to the backend:

server {
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    location /api/ {
        proxy_set_header X-Request-Id $request_id; # <- just added one
        proxy_pass http://backend; # neither Host nor X-Forwarded-For is inherited here
    }
}

Detection: Missing headers - the only sure method is to actually look at the response. At each endpoint where you "think" you set a header, compare against the real thing.

curl -sI https://example.com/ | grep -i -E 'strict-transport|content-security'
curl -sI https://example.com/api/foo | grep -i -E 'strict-transport|content-security'

Present at the top but gone under /api/ is the classic signature of this trap.

How to deal with it: The most portable fix is to re-list every header you need in the child block. Split the common part into a separate file and include it, so you gather the duplication into one place while re-declaring it at each level:

# security-headers.conf
add_header Strict-Transport-Security "max-age=63072000" always;
add_header Content-Security-Policy "default-src 'self'" always;

server {
    include security-headers.conf;

    location /api/ {
        include security-headers.conf; # re-read and re-declare in the child too
        add_header Cache-Control "no-store" always;
        proxy_pass http://backend;
    }
}

The add_header_inherit merge; added in nginx 1.29.3 lets you change the behavior to "inherit the parent, then add the child" (it's in both the current stable 1.30 series and mainline 1.31 series). But it's not in the 1.28-and-earlier stable series, so it may not be available on your version - getting the premise "it's replacement" into your bones is more effective first.

Forget one trailing slash on alias, and you can climb outside the directory

Everything so far has been about "behavior that differs from intent." This alias trap is the same kind of mistake - a single-slash difference - but the quality of the outcome is different. The config looks like it works as intended and does, yet you can read files outside the published directory. It's not just a behavior bug; it's a security hole as-is.

A reproduction. It's the commonplace setup of "I want to serve static files from a different directory":

location /assets {   # <- no trailing slash
    alias /var/www/static/; # <- this one has a trailing slash
}

/assets/logo.png is served normally. nginx -t passes, and as far as the browser is concerned there's no problem. It looks like it works as intended.

Why it's dangerous. alias builds the real file path by replacing the portion of the URI that matched the location (here, /assets) with the value of alias. When the location is /assets (no slash), the string following /assets gets stuck on directly at the end. So a request for /assets../ resolves to /var/www/static/ + ../ = /var/www/static/../ = /var/www/. That means with a little trick like /assets../../etc/passwd, you can climb outside the intended published directory. It's a classic path traversal (an attack that walks back up directories to read non-public files on the server). Normal access never notices it at all.

Detection and fix: The way to spot it is simple - line up the trailing slashes on the location and the alias. Either both have one, or neither does. In the example above, make it location /assets/. Then /assets../ no longer matches this location in the first place, and the climb is gone. To audit an existing config, use nginx -T (below) to dump all locations, and eyeball just the blocks containing alias for the trailing-slash correspondence. To reproduce it at hand, poke one level up with curl: if curl -sI 'http://localhost/assets../' returns not a 200 or 403 but "something that shouldn't be visible," you've got a hit.

Security level: it works exactly as intended, which is why it's dangerous

The semantic-level traps were "behavior that differs from intent." From here, the quality changes. The config works exactly as you intended. It just also works exactly as the attacker intended. So it passes not only nginx -t but even a visual review, on a "well, it's working" basis.

Passing a client-touchable variable straight into proxy_pass

Using a variable for the proxy_pass destination lets you decide the destination dynamically per request. Handy. But it's a different story if the source of that variable is client-derived.

location /fetch/ {
    proxy_pass http://$arg_target; # ?target=... decides the destination
}

With a request like /fetch/?target=internal-admin:8080/, nginx can relay the request to any host inside your network. This is a textbook entry point for SSRF (server-side request forgery, an attack that uses

Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.