149 lines
4.9 KiB
Markdown
149 lines
4.9 KiB
Markdown
# Deployment Runbook
|
|
|
|
## Scope
|
|
|
|
- Runtime: `console.svc.plus`
|
|
- Frontend host: Vercel
|
|
- Edge: Cloudflare
|
|
- Auth backend: `https://accounts.svc.plus`
|
|
|
|
This runbook is the minimum checklist for production incidents where login or MFA stops working and browser devtools show `/api/auth/login` or `/api/auth/mfa/*` failures.
|
|
|
|
## Expected Request Flow
|
|
|
|
1. Browser loads `https://console.svc.plus/login`
|
|
2. Browser calls same-origin Next routes on `console.svc.plus`
|
|
3. Next route proxies server-side to `https://accounts.svc.plus/api/auth/*`
|
|
4. `accounts.svc.plus` returns either a session token or an MFA challenge
|
|
|
|
The browser should not call `accounts.svc.plus` directly for login.
|
|
|
|
## Fast Triage
|
|
|
|
Run these checks first:
|
|
|
|
```bash
|
|
curl -si https://console.svc.plus/login | sed -n '1,20p'
|
|
curl -si https://console.svc.plus/api/auth/login | sed -n '1,20p'
|
|
curl -si https://accounts.svc.plus/healthz | sed -n '1,20p'
|
|
curl -si https://accounts.svc.plus/api/auth/login | sed -n '1,20p'
|
|
```
|
|
|
|
Interpretation:
|
|
|
|
- `console.svc.plus` returns `403` with `cf-mitigated: challenge`
|
|
Cloudflare is blocking the page or auth API before Vercel sees it.
|
|
- `console.svc.plus/api/auth/login` returns `404`
|
|
Vercel production is not serving the expected Next route, or Cloudflare is pointing at the wrong origin/deployment behavior.
|
|
- `accounts.svc.plus/healthz` fails
|
|
Back-end outage. Fix backend first.
|
|
- `accounts.svc.plus/api/auth/login` returns `200` with `mfaRequired`
|
|
Backend is healthy; continue on console/Vercel/Cloudflare.
|
|
|
|
## Application Checks
|
|
|
|
Verify the current build still contains the auth routes:
|
|
|
|
```bash
|
|
cd /Users/shenlan/workspaces/cloud-neutral-toolkit/console.svc.plus
|
|
yarn build
|
|
cat .next/app-path-routes-manifest.json | jq 'with_entries(select(.key|test("/api/auth/")))'
|
|
```
|
|
|
|
Verify the login page still uses same-origin routes:
|
|
|
|
```bash
|
|
nl -ba 'src/app/(auth)/login/LoginForm.tsx' | sed -n '64,180p'
|
|
nl -ba 'src/app/api/auth/login/route.ts' | sed -n '1,180p'
|
|
nl -ba 'src/app/api/auth/mfa/verify/route.ts' | sed -n '1,180p'
|
|
```
|
|
|
|
Expected behavior:
|
|
|
|
- `LoginForm` posts to `/api/auth/login`
|
|
- login proxy accepts backend `mfaRequired` / `mfaTicket`
|
|
- MFA verify proxy calls `/api/auth/mfa/verify`
|
|
|
|
## Vercel Checks
|
|
|
|
In the Vercel project for `console-svc-plus`, verify:
|
|
|
|
1. The production deployment corresponds to the intended git commit.
|
|
2. Framework preset is `Next.js`.
|
|
3. Build command is `yarn build` or the project default, not a static export command.
|
|
4. Output is not being overridden to static export.
|
|
5. Production Functions include `app/api/auth/login` and the other `app/api/auth/*` handlers.
|
|
6. Required runtime env vars are present for the auth proxy path if they are managed in Vercel.
|
|
|
|
If the route exists locally but Vercel returns `404`, suspect:
|
|
|
|
- wrong production deployment selected
|
|
- wrong root directory/project link
|
|
- stale alias or domain assignment
|
|
- build output mismatch between local and Vercel
|
|
|
|
## Cloudflare Checks
|
|
|
|
If `curl` shows `cf-mitigated: challenge`, check Cloudflare first.
|
|
|
|
Look for:
|
|
|
|
1. Managed Challenge or WAF custom rules affecting `/login`
|
|
2. Managed Challenge or WAF custom rules affecting `/api/auth/*`
|
|
3. Bot Fight Mode or Super Bot Fight Mode interactions
|
|
4. Transform/redirect/cache rules that alter `/api/auth/*`
|
|
5. Page Rules or Ruleset Engine policies applied only to the production hostname
|
|
|
|
Recommended policy for auth API:
|
|
|
|
- Do not cache `/api/auth/*`
|
|
- Do not apply JS challenge to `/api/auth/*`
|
|
- Keep standard security headers, but let requests reach Vercel
|
|
|
|
## Backend Verification
|
|
|
|
Use the backend directly to prove whether auth is healthy:
|
|
|
|
```bash
|
|
cd /Users/shenlan/workspaces/cloud-neutral-toolkit/accounts.svc.plus
|
|
set -a; source .env; set +a
|
|
payload=$(printf '{"identifier":"admin@svc.plus","password":"%s"}' "$SUPERADMIN_PASSWORD")
|
|
curl -sS -X POST https://accounts.svc.plus/api/auth/login \
|
|
-H 'Content-Type: application/json' \
|
|
-d "$payload"
|
|
```
|
|
|
|
Expected for an MFA-enabled admin:
|
|
|
|
- HTTP `200`
|
|
- response contains `mfaRequired`
|
|
- response contains `mfaTicket` or `mfaToken`
|
|
|
|
## Known Failure Signatures
|
|
|
|
- `POST https://console.svc.plus/api/auth/login 404`
|
|
Likely Vercel deployment mismatch or route not published.
|
|
- `403` with `cf-mitigated: challenge`
|
|
Cloudflare blocked request before Vercel.
|
|
- login returns generic failure even though backend returns MFA challenge
|
|
Console auth proxy is not parsing MFA fields correctly.
|
|
- MFA code accepted by authenticator but web login still fails
|
|
Console proxy may be calling the setup endpoint instead of the login MFA endpoint.
|
|
|
|
## Rollback Strategy
|
|
|
|
When a release breaks auth:
|
|
|
|
1. Remove or relax Cloudflare rules affecting `/login` and `/api/auth/*`
|
|
2. Re-point domain to last known-good Vercel production deployment
|
|
3. Roll back `console.svc.plus`
|
|
4. Only then consider `accounts.svc.plus` rollback
|
|
|
|
## Related Files
|
|
|
|
- `src/app/(auth)/login/LoginForm.tsx`
|
|
- `src/app/api/auth/login/route.ts`
|
|
- `src/app/api/auth/mfa/status/route.ts`
|
|
- `src/app/api/auth/mfa/verify/route.ts`
|
|
- `src/server/serviceConfig.ts`
|