The Insecure Default Was the Configuration

I was moving a pile of automation scripts into version control - years of accumulated cron jobs that had only ever existed on one machine - and ran a secret scan before the first commit. It found this, in two files: SERP_API_KEY = os.environ.get("SERP_API_KEY", "a1b2c3d4…") A live API key, sitting in source, as the default argument. The fix is so obvious it barely counts as a decision. Delete the literal, put the value in an env file, read it from there. Ten minutes. I did the ten minutes. Then, before committing, I checked one more thing: what actually sets SERP_API_KEY when these run? Nothing does. Not the crontab, which has no env line for it. Not the wrapper the jobs run through, which passes the environment along untouched. Not a shell profile - cron does not read one. So that hardcoded literal was not a fallback. It was the configuration. Every scheduled run of those jobs for months had used the second argument, because the first was never populated. What I had just deleted as a bad habit was the only reason the jobs worked. Two halves that look alike os.environ.get(KEY, default) reads as a preference and a safety net. Use the environment; if it is not there, fall back to this. The shape is honest about the mechanism and silent about the reality. It does not tell you which half runs. Both branches are valid code, both produce a working value, and the expression looks identical whether the variable is set on every machine or on none of them. process.env.KEY || 'literal' has the same problem. So does every config layer with a default: the fallback is invisible precisely when it is doing all the work. The asymmetry that makes this dangerous is that the two failure modes arrive at different times. If the environment variable is set and I delete the default, nothing happens - the code was already taking the first branch. If it is not set and I delete the default, the job dies. Both edits look the same in a diff. Both feel like the same small act of hygiene. And the death is quiet. These were scheduled jobs. They fail at 09:10 on a Monday into a log file, and the next thing that reads that log is me, whenever I next think to look. The question that separates them Not "is there a hardcoded secret here." I could see that. The scanner saw it. The question is: which branch is running in production right now? For an environment variable that is one command, and it has to be run in the same context the job runs in - not in my shell, where my profile has been sourced and everything looks populated. My interactive shell is the least representative environment on the machine. Cron gets a nearly empty one. Once I asked that, the fix changed shape. It was no longer "delete the literal." It was "make the environment actually carry the value, then delete the literal, then prove the job still resolves it." The last step is the one I would have skipped. It is also the only one that distinguishes a security fix from an outage. The scanner was too clever There is a second half to this, and it is worse. My scan looked for high-entropy strings - long runs of hex and base64, the shape of an API key. It caught both hardcoded keys immediately. It also caught a third one in a different file, and I felt good about the coverage. Then, reading around an unrelated line in a config block, I found this a few lines below: EMAIL = "m*@example.com" PASSWORD = "…" A real account login, in plaintext, used to authenticate against a hosted database. Fifteen characters. My pattern required thirty-two. That is not a tuning problem I can fix by lowering the threshold - at fifteen characters and lower, everything matches. It is a category error. Entropy-based secret detection is built for machine-generated credentials, and a password is human-generated by definition. It is short because a person types it. It contains a word because a person remembers it. Every property that makes it a bad password makes it invisible to a scanner looking for randomness. The credential with the widest blast radius on the machine - an account login, plausibly reused elsewhere - was the one my tooling was structurally incapable of finding. I found it by reading. So the scan needs two passes with different logic: one for shape, matching entropy and known key prefixes, and one for name, matching PASSWORD , SECRET , TOKEN , PASSWD as assignment targets regardless of what is on the right-hand side. The second pass is dumber and catches what the first cannot. Not everything that looks like a secret is one The mirror of that failure showed up in the same commit, and it is worth naming because over-correcting here has its own cost. Two more long random-looking strings survived my scan, and I left both in source deliberately. One was a verification key for a search-engine ping protocol. That protocol works by having you host the key as a text file at your own domain root, so a crawler can confirm you control the site. The key is published by design. I confirmed it in one request: fetching the key file at the domain returned 200. Stripping it would have been theatre. The other was a JSON Web Token for a database client. JWTs are three base64 segments joined by dots, and the middle one is the payload - you can decode it without any credential at all. Its role field said anon : the public client key, meant to ship to browsers, protected by row-level policies rather than by being secret. Also fine to commit. Both of those look exactly like the thing I was hunting. The difference is not visible in the string. You get it by asking what the value is for - and for a JWT you can simply read the answer out of the token. Blindly stripping every high-entropy string would have broken two working systems in the name of security. That is the same failure as leaving the real password in, just pointed the other way: acting on the shape of a value instead of its function. What I actually changed The refactor that shipped is unremarkable, which is the point: - Each script reads its credential from a single env file, using the same loader the rest of the codebase already used. Consistency here matters more than elegance; a second loader is a second thing to get wrong. - Every one raises a clear error naming the missing variable, instead of falling back. If it cannot find the value, I want the job dead and loud, not running on a stale literal. - I verified each one by executing it and watching it resolve a real value. Not a syntax check. A syntax check would have passed on all three, including the one where I had used os.environ without importingos . That last one is the small joke at the end. Having reasoned carefully about which branch runs in production, I introduced a NameError and nearly shipped it, because ast.parse validates grammar and knows nothing about names. The check that caught it was running the thing. The short version A default value is a claim about what happens when the real source is missing. It is not a claim that the real source exists. Before you delete one as a bad habit, find out which half has been load-bearing. If the answer is "the literal," you are not tightening security - you are removing the configuration, and you will find out at 09:10 on a Monday, in a log file, if you find out at all. And run two scans. One for what secrets look like, one for what they are called. The scanner that finds your API keys will walk straight past your password. Top comments (0)