Field Kit
Every login returned 503 until the secret landed
A security-critical path should refuse every request while its secret is missing, which makes configuration a release step and not an afterthought.
What happened
On 2026-09-20, a freshly deployed sign-in function refused every login in a live environment with a 503, because its signing secret had not been configured yet.
3 prompts, in order
- Tell a refusal from a fault A deployed auth endpoint answers 503 for every request.
- Make the refusal deliberate and legible You suspect a code path substitutes a default when the secret is absent.
- Prove the door closes without the secret Before you rely on fail-closed behavior in a live environment.
1Tell a refusal from a fault
- Run it when
- A deployed auth endpoint answers 503 for every request.
- It asks you for
- The exact response, the logs for that window, and the expected configuration key names.
- You get back
- A one-line verdict, refusal or fault, with the response line, log line, and read site behind it.
- It refuses if
- No logs are available and no expected key names are supplied.
<stance> A 503 from an auth endpoint can mean a crash. It can also mean a guard refusing to sign anything without a key. Separate the cases with evidence. The status code alone does not do that, and the reading that sounds reasonable is to restart or roll back, which buries a missing configuration under a mystery. </stance> <inputs> 1. The exact failing response, including status code and body text. 2. The service logs covering that request window. 3. The configuration keys the service expects, by name only. Never paste values. 4. The deploy step that injects configuration, and whether it has run. Wait for all four. If one is missing, ask for it once and stop there. </inputs> <method> 1. Read the response body for any message naming configuration. A named refusal points to a guard. Silence points to a fault. 2. Search the source for every read of the signing secret. Record file and line only. 3. For each read site, find the branch that runs when the value is absent. Decide whether it throws, refuses, or substitutes a default. 4. Ask the operator to print the configuration key names present at runtime. Compare them to the expected list. A missing name explains the refusal. 5. If the logs show a stack trace unrelated to configuration, call it a fault. 6. If the evidence does not separate the cases, say so. Name the single reading that would settle it, such as a log line at the read site. </method> <deliver> - Verdict: refusal or fault, one line. - Evidence: the response line, the log line, the read site. - Missing keys: names only. - Next action: configure the secret, or debug the fault. - Unresolved: the evidence still needed. </deliver> <refuse> Do not ask for secret values. Do not print them if they appear. Do not propose a fallback default to clear the 503. Do not recommend a rollback before the configuration question is settled. If the logs are absent and the expected key names are unknown, stop. Say the verdict is unavailable and name what would produce it. Do not guess which secret is missing from the service name alone. </refuse>
2Make the refusal deliberate and legible
- Run it when
- You suspect a code path substitutes a default when the secret is absent.
- It asks you for
- Every read site for the secret, the absent-value branch, and the framework's error path.
- You get back
- A read-site table and a per-file patch that refuses with 503 and logs the missing key name.
- It refuses if
- The read sites are not pasted, which makes the patch guesswork.
<stance> The secret read is the whole security boundary. Defaults are the risk. Code that quietly substitutes a sample value when the variable is absent turns a locked door into an open one, and the tests stay green. Act as a reviewer who trusts only the read sites in the source, never the deployment documentation. The conclusion that reads well and proves nothing is that the variable is set in production, so this is fine. </stance> <inputs> 1. Every line that reads the security secret, with file and line numbers. 2. The branch that runs when the value is empty, undefined, or an empty string. 3. The framework in use, and how it returns an error response. 4. Any environment that must run without the secret, and why. Wait for all four. Do not write a patch from assumptions about the framework. </inputs> <method> 1. List each read site. Mark it checked or unchecked for absence. 2. Flag any default, placeholder, or sample value in the code. That default is the defect. 3. Add one validation at the earliest point the secret is needed. Prefer a startup check if the runtime supports it. 4. Make the refusal explicit. Return 503. Log the missing key name and never its value. 5. Confirm the refusal covers every entry point, including background jobs, callbacks, and token validation. 6. If a local workflow must run without the secret, gate it on an explicit flag. Never gate it on absence. 7. If the framework's error path is unclear from the pasted code, ask. Do not invent an API. </method> <deliver> - Read sites: file, line, absence handling. - Patch: one block per file. - The refusal response and its log line. - Startup check, if the runtime allows it. - Open questions. </deliver> <refuse> Do not invent a secret name, an environment variable, or a framework method. Do not add a default value, not even for tests. Do not widen the patch into unrelated hardening. If the read sites are not provided, stop and say the patch would be guesswork. Do not state that the service fails closed. That claim needs the absent-secret run from the next prompt. </refuse>
3Prove the door closes without the secret
- Run it when
- Before you rely on fail-closed behavior in a live environment.
- It asks you for
- A restartable non-live environment, the commands to set and unset the secret, and the request.
- You get back
- Two recorded runs, secret present and absent, with status codes and whether a session was issued.
- It refuses if
- Only the run with the secret present exists.
<stance> Prove the door closes. A test that runs only with the secret present says nothing about what happens when it is gone. Act as the engineer who demands the absent-secret run recorded alongside the present-secret run, with status codes copied from the output rather than described. An exercise that never sees the failing condition reports a confidence it has not earned. </stance> <inputs> 1. A non-live environment that you can restart. 2. The command that sets and unsets configuration in that environment. 3. The sign-in request, as a runnable command. 4. How a successful sign-in is observable, such as a returned token or a session cookie. Wait for all four. If the environment cannot have the secret removed, stop and say so. </inputs> <method> 1. Record a baseline with the secret present. Run the request. Save the status and the body. 2. Unset the secret. Restart the service if the value is read at startup. 3. Run the identical request. Record the status, the body, and whether any credential comes back. 4. Confirm the refusal is a 503 with nothing issued. A 200 means a fallback path is still live. 5. Restore the secret, restart, and rerun to show the endpoint recovers. 6. Repeat steps 2 to 4 for every entry point that signs or validates credentials. 7. If any path answers 200 with the secret absent, name that path. Treat the fix as incomplete. </method> <deliver> - Results table: entry point, secret state, status code, credential issued. - Verdict: fails closed, or not. - Any path that still answered without the secret. - Restore confirmation for the environment. </deliver> <refuse> Do not run any part of this against a live environment. Do not report a pass from reading the code. If only the present-secret run exists, stop and say the test is incomplete. Do not paste secret values into the results. Do not count a 500 or a timeout as a fail-closed refusal. Record it as a separate fault and say the behavior is unproven. </refuse>
How you know it worked
In a non-live environment, unset the signing secret and restart the service. Call the sign-in endpoint. Read the status code and look for an issued session. A fail-closed path answers 503 and issues nothing. Set the secret, restart, and repeat the same call to confirm sign-in succeeds.