2026-09-30 · 9 min
Q3 Tech Debt Audit: Production Review Checklist Before Q4 Planning
Your Q3 just ended. Before you let the retrospective slide deck become the only artifact of the last three months, run an audit on what your production systems actually look like — not what the architecture diagram says they look like.
This is the checklist I run every September. Not a theoretical framework — specific things to check, specific tools to use, and the failure modes I keep finding at Indonesian startups where I’ve consulted, from fintech teams in BSD City to e-commerce backends running out of colocated racks near Serpong.
Why Q3 specifically
Q4 is when planning happens and budgets move. If you surface tech debt in October, you’re competing against feature work that already has stakeholder buy-in. Surface it in late September with data and you can actually influence what gets resourced.
The other reason: six months of production data is enough to see patterns. A dependency that was updated in March and introduced a subtle memory leak shows up in the Q3 p99 latency graphs. A database index that was “good enough” in January starts showing up in slow query logs by August once the data volume triples.
1. Dependency hygiene
Run this first. It takes twenty minutes and consistently surfaces the worst surprises.
# Node.js projects
npx npm-check-updates --format group
# Check for known vulnerabilities separately
npm audit --audit-level=moderate
# Python
pip list --outdated
pip-audit
# Go
go list -m -u all | grep '\[v'
govulncheck ./...
What you’re looking for:
- Major version jumps that were silently skipped. If you’re on
[email protected]and[email protected]has been stable for months, that’s not just a missing upgrade — it’s a sign your dependency update process has stalled. - Direct dependencies pinned to patch versions that haven’t moved in over a year. Sometimes this is fine (stable library). Often it means nobody is watching.
- Transitive dependencies with known CVEs.
npm auditwill find these even if your direct dependencies are clean. Fix or document a risk acceptance decision — undocumented CVEs in production are a compliance problem waiting to surface.
The number I look for: more than 30% of direct dependencies with available minor/patch updates means your update cadence is broken. Fix the cadence, not just the packages.
2. Database schema drift and migration hygiene
Schema drift is the tech debt that can actually kill you during an incident.
-- PostgreSQL: find tables with no primary key
SELECT schemaname, tablename
FROM pg_tables
WHERE schemaname = 'public'
AND tablename NOT IN (
SELECT c.table_name
FROM information_schema.table_constraints c
WHERE c.constraint_type = 'PRIMARY KEY'
);
-- Find indexes that exist in code but not in DB (or vice versa)
-- Compare your ORM migration history against actual pg_indexes
SELECT indexname, tablename, indexdef
FROM pg_indexes
WHERE schemaname = 'public'
ORDER BY tablename, indexname;
-- Bloated tables: dead tuple ratio above 20% needs VACUUM
SELECT relname, n_dead_tup, n_live_tup,
round(n_dead_tup::numeric / NULLIF(n_live_tup + n_dead_tup, 0) * 100, 2) AS dead_ratio
FROM pg_stat_user_tables
WHERE n_live_tup > 10000
ORDER BY dead_ratio DESC;
Three things to audit specifically:
Migration files versus actual schema. pg_dump --schema-only your production database and diff it against what your ORM thinks the schema looks like after running all migrations in sequence. In my experience, one in three production databases has at least one column or index that exists in production but is absent from the migration history, or vice versa. This happens when someone ran a manual ALTER TABLE during an incident and forgot to backport it.
Autovacuum configuration. The default PostgreSQL autovacuum settings are conservative. Tables that receive heavy UPDATE or DELETE traffic need tuned autovacuum_vacuum_scale_factor. Check if any table with high write volume has dead tuple ratios above 20% — that’s a sign autovacuum is losing the race.
Backup restore verification. Do you have a record of the last time you actually restored from backup and verified the data? Not just “we have backups” — actually restored. If not, this goes to the top of the Q4 list regardless of everything else.
3. Observability gaps
Monitoring that doesn’t get acted on is theater. This audit is about finding the gaps in what you can actually see.
# Checklist — answer honestly for each production service
observability_audit:
- "Does the service emit request duration histograms, not just counters?"
- "Are p95 and p99 latency tracked separately from average?"
- "Is there an alert on error rate, not just absolute error count?"
- "Does the alert have a documented runbook — or does it just page someone?"
- "Are database query durations visible per query, not just total DB time?"
- "Is structured logging in place, or still printf-style strings?"
- "Can you trace a single user request across all services end-to-end?"
- "Are background jobs tracked for duration and failure rate?"
The gap I find most often in Q3 audits: services that have metrics but no alert thresholds that mean anything. Someone set up Prometheus, added a Grafana dashboard, and then set the alert threshold to > 100 errors/minute because it seemed like a big number. By the time 100 errors per minute is firing, you’ve had several thousand 5xx responses served to real users.
Check your alert thresholds against your actual baseline error rates from the last 90 days. An alert that never fires because the threshold is set too high is the same as no alert.
4. CI/CD rot
Your pipeline from January and your pipeline from September probably have different characteristics, and not in a good way.
# Measure your actual CI duration over the last 30 days
# GitHub Actions: use the API or gh CLI
gh run list --limit 100 --json durationMs,status,conclusion \
| jq '[.[] | select(.conclusion == "success")] | [.[].durationMs] | add / length / 60000'
# That gives you average successful run duration in minutes
Things that rot quietly in CI:
Test suite duration creep. If your test suite took 4 minutes in January and takes 11 minutes now, you’ve accumulated enough test debt that every engineer is losing at least 30 minutes a day waiting. Calculate the actual cost: if 8 engineers each run CI 4 times a day and wait 7 extra minutes each time, that’s 224 minutes of engineering time daily — before you account for context switching.
Flaky tests that everyone ignores. Pull the last 200 CI runs and count how many failed on the first attempt but passed on retry. Any test that requires retrying more than 2% of the time is masking real failures. The team has trained itself to re-run instead of investigate.
Docker layer cache invalidation. COPY . . as the first meaningful layer in your Dockerfile means every code change rebuilds from scratch. Check your Dockerfiles. If package.json and the lockfile aren’t copied and installed before the rest of the source, you’re wasting minutes per build.
# Before (bad — cache busts on every code change)
COPY . .
RUN npm install
# After (cache-friendly)
COPY package.json package-lock.json ./
RUN npm ci --only=production
COPY . .
5. Configuration and secrets hygiene
This one gets skipped because it feels like security theater until the moment it isn’t.
# Find hardcoded secrets in git history (run from repo root)
# trufflehog is thorough but slow; gitleaks is faster for regular audits
gitleaks detect --source . --verbose
# List all environment variables actually read by your application
# (for Node.js, trace process.env references)
grep -r "process\.env\." ./src --include="*.ts" \
| grep -oP '(?<=process\.env\.)\w+' \
| sort -u > env-vars-in-code.txt
# Compare against your .env.example or documented vars
The specific failure mode I’ve seen in three separate production systems: a .env file committed to a private repo that was later made public, or an engineer who committed their local .env to a feature branch that was never cleaned up. Private repositories becoming public is more common than people think — acquisitions, migrations, accidental settings changes.
Document every secret: what it is, where it’s rotated, who has access, when it was last rotated. If the answer to “when was this secret last rotated” is “I don’t know,” that’s the answer you needed.
Triaging what you found
After running this audit you’ll have a list. Here’s how to sort it:
| Priority | Criteria | Action |
|---|---|---|
| Immediate | Backup not verified, known CVE in critical path, secret rotation unknown | Fix in current sprint |
| Q4 roadmap | CI duration >10min, observable gaps, migration drift | Scope and resource in planning |
| Parking lot | Outdated minor versions, cosmetic debt | Revisit next audit |
The goal is not to fix everything — it’s to make the invisible visible before Q4 roadmap locks. Tech debt you haven’t measured can’t be prioritized. Tech debt you’ve measured but not communicated won’t get resources.
One more thing: write down what you found even for items you’re deferring. A three-sentence note in your engineering wiki about “we know autovacuum is losing the race on the transactions table, accepted risk until Q1” is worth more than institutional silence about it.
The audit only pays off if someone reads the output.
Ditulis oleh Reza Pradipta