API Key Rotation After the Railway Incident: My Apps Still Had the Old Keys

I build and operate my own software as a solo founder. The recovery work described here involved a web application and background workers: programs that run jobs without someone clicking through a page. Those programs depend on credentials to access other services.

An API key is one kind of credential. Rather than a person entering a password, the application sends a secret value to identify itself to a service. If that value may have been exposed, replacing it means making the old value unusable and arranging for the application to use a new one.

When Railway advised users to rotate credentials in its 21 October security disclosure, I started doing that. But I began revoking or replacing keys before I had a complete list of the software using each one. I also missed some replacement values that their providers displayed only once.

The result was a second problem to work through: some applications still held keys I had already replaced. I needed to find where those values were stored, change them and check what the running software was actually using. Creating the new keys was only the beginning.

This article follows those decisions through my notes dated 23 October 2026. It is a personal account of getting applications working with replacement credentials, rather than a forensic investigation into whose secrets the attacker accessed.

Railway is a platform developers use to host applications. Applications commonly receive configuration through environment variables: named settings available to a running program. Some settings contain secrets, including API keys and database passwords.

In its 21 October disclosure, Railway reported possible access to encrypted environment variables and decrypted contents for some values. It recommended rotating credentials and reviewing logs. It also described a pause on new deployments during the investigation.

Railway's preliminary report, reviewed for this article on 25 October, said the investigation remained ongoing. It reported restored builds and deployments on 22 October, so the earlier pause should not be read as a claim that deployments were still unavailable.

The report also explained that some providers had revoked credentials as a precaution beyond those established as affected. A reset notice therefore did not prove that an attacker had read my application's secrets. I still needed to respond to the possible exposure, and then establish whether my applications could use their replacements.

I had treated credential rotation as though changing the value at the service provider would complete the job. But the provider and my application each held a value. Changing what the provider accepted did not change what my software sent.

There were more copies to consider than the one I had just replaced. A web application and a background worker could each need the same credential, and development and production could have separate settings. I use “consumer” to mean any piece of software that needs a particular credential.

Even finding and editing a saved setting was not enough. A program can load its environment variables when it starts and continue using those values while it runs. In that arrangement, saving a replacement elsewhere does not update the already running program.

For the worker I focused on, I needed a clear startup sequence: retrieve the current values, check them and then launch the worker with those values. I also needed a record of every other consumer that would require its own update or restart.

During an incident, disabling a potentially exposed key may be more urgent than keeping every application available. The lesson I took was to prepare that list before an emergency, so I could see which services would need attention after a revocation.

To improve secret rotation, I started small: one background worker and Infisical EU Cloud, a managed service for storing application secrets. A central secret store gives a program a place to request the values it needs. It still needs a controlled way to connect to that store and permission to read the right values.

I kept Keeper for recovery material I would need as a person. For the worker's application secrets, I used a dedicated Infisical project. That separation mattered because Infisical's role documentation describes its Viewer role as read-only access to all resources in the assigned project.

Choosing one folder when requesting secrets would not, by itself, prevent that identity from reading elsewhere in the project. Infisical also documents custom roles and additional permission mechanisms. For this initial setup, I chose a separate project as the boundary around the worker's access.

The worker also needed a credential to authenticate to Infisical itself. With Universal Auth, a client ID and client secret are exchanged for an access token. That token lets the program make authorized requests; retiring it does not retire the reusable client secret that obtained it.

I wrote a startup script, also called a wrapper, to do the preparation before launching the worker. It retrieved the required secrets and checked their names and the expected project, environment and path. It then revoked the temporary token and launched the worker with only the selected values, excluding the credentials used to log in to Infisical.

This made the handoff easier to inspect. It did not remove the need to protect the computer running the script, or make a running worker automatically reload later changes.

I needed to answer several different questions: did the startup script handle secrets correctly, could it retrieve them in the deployed environment, did the external service accept a replacement, and could the application complete the operation I cared about?

I started with offline tests. These supplied invented values and simulated responses to check behavior such as refusing to start when required data was missing. They helped check the script's logic without contacting real services, but could not tell me whether the deployed application had the right permissions or configuration.

Checks in the hosted environment addressed the next question: could the script retrieve the selected values and pass them to the worker? A limited read from an external service checked whether that service accepted a credential for that request. I recorded whether the running program used the changed configuration separately.

A successful login checked a particular authentication flow. It did not establish that every background job worked. Similarly, my notes at the 23 October checkpoint reported checks of changes to database access permissions and encryption, while exercises to restore the application and its data remained unfinished.

One worker run reached an existing spending limit and stopped before doing fresh paid work. That result did not show that it could successfully generate AI output, scrape pages or publish anything. I kept those outcomes separate rather than calling the whole workflow recovered.

There was also a later editorial check on 25 October: an adapted example of the startup script passed 36 offline tests using Node.js 24.19.0. Those tests used synthetic values and mocked HTTP responses. They checked the example code, not the original production script or my live applications.

env -i PATH=/usr/bin:/bin /opt/codex/runtimes/codex-primary-runtime/dependencies/node/bin/node --test research/offline-example/managed-worker.test.mjs

ℹ tests 36
ℹ suites 0
ℹ pass 36
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0

A central store changed where the worker obtained its secrets. The worker still needed those values in memory to use them, and the startup script still needed permission to retrieve them. The setup depended on trusting the machine and the script running on it.

That matters when coding agents also have access to a machine. Keeping the Infisical login credentials out of the worker's environment did not prove that another program running under the same operating-system user could not access sensitive information. I had not established enforced isolation for a coding agent in that situation.

My recovery notes also left several practical tasks open on 23 October. I had not completed restore exercises, reviewed all historical copies of secrets, demonstrated sustained worker stability or verified the broader application workflows. The full list of services and programs using each credential was still unfinished.

Each gap needed its own check. Proving that a new key works does not remove an old copy from a file or backup, and proving that a program starts does not establish that it stays healthy. The pilot gave me a clearer way to supply secrets and investigate failures; it was not a declaration that recovery was complete.

The most useful preparation would have been a list connecting each credential to the software that needs it. For each entry, I wanted to know which service issued it, whether it belonged to development or production, where it was configured and which applications or jobs used it.

I also wanted to record how each program picks up a replacement. Does it read the value on startup, need a redeployment or use some other reload mechanism? Beside that, I would write the specific operation to check afterward, such as a login or a limited service request.

The list would contain descriptive names, not the secret values themselves. Its purpose would be to show me what to update and how to tell whether the update worked.

If I need to rotate API keys again, that is the map I want beside me. My next work is to finish it and test the recovery tasks still open, so the next replacement does not leave me searching for applications that are sending an old key.