API Key Cleanup Post-Mortem: 1.2M Requests/Day from a Stale Build

Posted by Michael S. on June 9, 2026

TL;DR

We set out to find four market-data API keys (60c6a7dd…, 7e862fac…, 5zR1SWz…, bd0944b0… for four different providers), determine where they're actually used, and stop one of them from being burned. We discovered a desktop trading app on a headless Linux box was hitting the paid data provider at ~1.2M requests/day from a month-old build that should have been replaced weeks ago. We stopped it, deleted the key, fixed the broken auto-updater, redeployed a Schwab/E*TRADE-only build from HEAD (eff519d), and verified zero calls to the old provider.

An AI assistant did the investigation. It gave several wrong conclusions along the way that later evidence overturned. Those are documented in full below, because that's the useful part.

What we did (timeline)

  1. Found the keys. All four were in plaintext in a Java constants file in the codebase.
  2. Swept every machine (local Mac, headless Linux build host, older Mac workstation, Hetzner VPS, DigitalOcean droplets) for the values and aliases in shell configs, .env files, scripts, systemd/launchd/cron. Real shell exports only on the Mac workstation (TIINGO_API_KEY, TWELVE_DATA_API_KEY) plus a .secrets/ env file; everything else was code.
  3. Mapped call sites. Two real consumers: a Python trade-review script (granular_trade_review.py) and a Tauri desktop trading app.
  4. Runtime + network audit overturned the "not used" conclusion and proved the Linux box was live-polling the provider.
  5. Remediation: stopped the app, deleted the API key env var from all 7 .env.local files (backed up), diagnosed and patched the auto-updater, rebuilt a clean Schwab/E*TRADE-only build from HEAD, reinstalled, restarted.
  6. Verified (adversarially): zero network calls to the provider; keyless binary; app running on the broker's own sidecar feed.

Where the AI gave wrong information

The point of this doc. Each item: what was claimed, what was actually true, and what corrected it.

  1. "The data-provider path is wired but not called. It's vestigial." Wrong; biggest error. The installed, running build (dated May 3) still had the old provider as its default adapter and was polling it live at ~1.2M req/day. The AI analyzed the source at HEAD (which had migrated to Schwab on 2026-05-27) instead of the deployed binary. Caught by the runtime/network audit: 109 TIME-WAIT sockets to the provider's domain, and the provider's own dashboard showing 1.2M calls.

  2. "E*TRADE auth is dead. Run npm run etrade:auth; live data is down." Wrong conclusion and wrong fix. E*TRADE was live: a separate trade-placer app owned the session and refreshed its token daily (etrade-morning.sh, cron 6:25 AM weekdays). The trading app borrows that token (sync-etrade-tokens.sh → borrowed-tokens.etrade.json, re-read by the sidecar). The 401s came from a stale 4-day-old sidecar that never re-synced. The AI parroted the sidecar's generic log hint instead of reading the token-borrow architecture. Real fix: restart the sidecar to re-borrow, not a re-auth. Caught when I pushed back ("E*TRADE data IS live").

  3. "On 5/28 the app was up and polling." Wrong. The service was down from 5/27 to 6/07 (~11.5 days). The AI inferred continuity from "first start 5/25, last entry today" without reading the start/stop events in the journal. Caught by the forensic pass.

  4. "granular_trade_review.py last ran May 18." Wrong by ~7 weeks. True last run was March 30; the May 18 artifacts were from a separate reconciliation job.

  5. "The auto-updater has been silently failing every 5 minutes since May 3." Imprecise. For most of that span origin/main hadn't advanced past the box's HEAD, so the updater correctly skipped. The latent nvm/PATH bug only triggered once new commits arrived. Self-corrected after diagnosis.

  6. "App.tsx says 'No polling'" (implying disabled code). Imprecise. It's a descriptive comment; the provider integration was actually deleted in a later commit (31c3644).

The pattern: every error clustered around inferring runtime behavior from static/surface signals (HEAD source, a log string, a single timestamp) instead of tracing the actual deployed artifact, event log, and data flow. Every correction came from runtime/network/journal evidence or from reading the architecture end-to-end.

Final result (verified)

  • App: running, NRestarts=0, HEAD eff519d (Schwab/E*TRADE-only source).
  • Old provider: zero network calls across 10 samples in two time windows. Installed binary, dist/, and all .env* files are keyless. The only residual code path is gated behind an env var that exists nowhere on the machine; the settings panel can accept a user-typed key, but nothing is baked in.
  • E*TRADE: live via trade-placer (token refreshed daily). Trading app borrows it.
  • Key: deleted from all 7 .env.local (backups in ~/eodhd-key-backups/). Not yet revoked provider-side.
  • Auto-updater: both build bugs fixed and proven via the real systemd trigger: nvm/cargo now load correctly, and the deb path points at the real artifact.

Open items

  1. Restart the sidecar service so it re-borrows trade-placer's fresh token (clears the stale 401s). Then reconcile the duplicate-sidecar/port conflict so future deploys auto-borrow.
  2. End-to-end smoke test fails 5 steps, so the updater rolls back every deploy. Some failures trace to the stale-sidecar 401; others look like display/threshold flakiness on the headless GPU-Xorg session. Needs a pass once data flows.
  3. Revoke the key provider-side (paid account, needs dashboard access).
← Claude 5 Fable Just Dropped