Skip to main content
EVOKORE// BROWSE
>

./browse/prompts

7 NODES
πŸ€–system promptβ€’6 months ago

Semantic Session Checkpointing Pattern

Record lightweight semantic checkpoints at planning, risk, failure, and handoff boundaries while keeping heartbeat outside the model loop.

productivity
⭐1
# Semantic Session Checkpointing Pattern Imported from curated first-party documentation sources. ## What this covers Use this pattern when session continuity matters and you need checkpoints that survive model drift or hard crashes. ## Use this when - Capturing handoff context during long sessions - Checkpointing before risky refactors - Separating liveness tracking from agent behavior ## Expected outcomes - Checkpoint timing becomes explicit and repeatable - Session heartbeat stays reliable even if the model stalls - Handoffs capture task, next action, files touched, and risk ## Source synthesis - REVOKORE/docs/MCP-Integration.md ## Dedupe notes Uses the compact REVOKORE integration guide as a distinct session-state pattern that is not already covered by EVOKORE-MCP or AGENT33 docs. ## Source excerpts ### REVOKORE/docs/MCP-Integration.md ## Recommended MCP Pattern Expose a tool that writes a short checkpoint into the active session: - current task - next action - files touched - blocker or risk The simplest implementation is to call: ```powershell REVOKORE/scripts/Write-AiCliCheckpoint.ps1 -Kind note -Message "working on auth bug" ``` From an MCP server, you can call the same script directly or append JSON lines to `REVOKORE_CHECKPOINT_PATH`.
πŸ‘0
πŸ‘οΈ14
docs
πŸ“textβ€’6 months ago

ARCH-AEP Tiered Remediation Cycle

Run a scope-locked remediation cycle that normalizes findings, revalidates them in parallel, and clears tiers with verification evidence.

architecture
⭐1
# ARCH-AEP Tiered Remediation Cycle Imported from curated first-party documentation sources. ## What this covers Use this workflow when you need an orchestrated engineering review cycle with strict scope lock, tiered execution, and audit-grade evidence. ## Use this when - Large remediation efforts spanning multiple PRs - Parallel specialist review with shared backlog authority - Tracking verification evidence through closure ## Expected outcomes - Scope changes are controlled instead of creeping mid-cycle - Severity tiers drive remediation order predictably - Verification evidence is treated as part of done ## Source synthesis - CHELATEDAI/docs/ARCH AGENTIC ENGINEERING AND PLANNING/workflow.md (https://github.com/mattmre/CHELATEDAI/blob/main/docs/ARCH%20AGENTIC%20ENGINEERING%20AND%20PLANNING/workflow.md) - CHELATEDAI/docs/ARCH AGENTIC ENGINEERING AND PLANNING/orchestrator-briefing.md (https://github.com/mattmre/CHELATEDAI/blob/main/docs/ARCH%20AGENTIC%20ENGINEERING%20AND%20PLANNING/orchestrator-briefing.md) ## Dedupe notes Condenses the core ARCH-AEP workflow and briefing docs into a single remediation-cycle entry instead of importing both separately. ## Source excerpts ### CHELATEDAI/docs/ARCH AGENTIC ENGINEERING AND PLANNING/workflow.md ## Workflow (ARCH phase) 1. Scope lock - Define PR range and time window. - Freeze inputs (refinement report + framework + PR list). 2. Discovery + normalization - Orchestrator ingests `agentic-review-framework.md` and the latest refinement report. - Normalize all findings into a single backlog with unique IDs. - De-duplicate, merge overlaps, and assign provisional severity. - Record backlog in `docs/ARCH AGENTIC ENGINEERING AND PLANNING/backlog-YYYY-MM-DD.md`. 3. Parallel re-validation - Spawn specialist agents to re-validate findings against current main. - Each agent must attach exact file paths and line ranges. - Propose the smallest safe fix, acceptance criteria, and effort sizing (S/M/L). 4. Architecture and planning synthesis - Orchestrator merges validated findings into a master backlog. - Re-score and re-rank using the sorting model. - Identify dependency chains and blockers. ### CHELATEDAI/docs/ARCH AGENTIC ENGINEERING AND PLANNING/orchestrator-briefing.md Purpose: Quick reference for the ARCH-AEP documentation set, plus the narrative to start a new session. Full file index: `docs/INDEX.md` ## What Exists In This Folder - `README.md`: narrative overview and intent for ARCH-AEP. - `workflow.md`: end-to-end workflow specification with phases, guardrails, and enhancements. - `templates.md`: ID and branch conventions, tracker table format, and status log. - `backlog-template.md`: template for the master backlog file. - `backlog-index.md`: index of backlog files. - `next-session.md`: short handoff checklist for resuming work. - `phase-planning.md`: long-running planning record for the current cycle. - `schedule-and-tracking.md`: cadence, gates, and milestone tracking. - `agent-learning.md`: cross-session learnings and reusable patterns. - `change-log.md`: scope/defer decisions with audit context. - `risk-memo-template.md`: Critical/High risk memo template. - `risk-memos/README.md`: storage location for risk memos. - `phase-summary-template.md`: template for per-PR or per-phase summaries. - `phase-summaries/README.md`: storage location for phase summaries. - `verification-log.md`: test/build evidence mirror for PRs. - `tracker-pointer.md`: pointer to the active tracker file. - `tracker-index.md`: index of tracker files. - `scope-lock-t ...
πŸ‘0
πŸ‘οΈ0
docs
πŸ“textβ€’6 months ago

Submodule Hygiene Workflow

Add, update, validate, document, and review git submodule changes without losing track of dependency state.

productivity
⭐1
# Submodule Hygiene Workflow Imported from curated first-party documentation sources. ## What this covers Use this workflow when repositories rely on submodules and you need a disciplined way to update them safely. ## Use this when - Adding or updating repository submodules - Documenting submodule changes alongside code changes - Avoiding broken references during review ## Expected outcomes - Submodule changes are treated as a documented workflow - Validation happens before the review handoff - Dependency movement stays visible to the team ## Source synthesis - EVOKORE-MCP/docs/SUBMODULE_WORKFLOW.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/SUBMODULE_WORKFLOW.md) ## Dedupe notes Uses the dedicated submodule workflow doc because the guidance is operationally specific and not already represented on the site. ## Source excerpts ### EVOKORE-MCP/docs/SUBMODULE_WORKFLOW.md This repository may consume external content through git submodules. Use this workflow to keep documentation updates reviewable and predictable. ## 1) Add or Register a Submodule ```bash git submodule add <repo-url> <target-path> git submodule update --init --recursive ``` Commit both: - `.gitmodules` - the submodule pointer change at `<target-path>` ## 2) Pull Latest Submodule Content ```bash git submodule update --remote --merge git submodule update --init --recursive ``` Then inspect: ```bash git status git diff --submodule ``` ## 3) Validate Submodule Cleanliness (Local + CI) Run the same guard used in CI before opening your PR: ```bash git submodule status --recursive node scripts/validate-submodule-cleanliness.js ``` Cleanliness semantics: - `-` => uninitialized submodule - `+` => submodule commit mismatch (worktree vs gitlink pointer) - `U` => submodule merge conflict - dirty submodule worktree => non-empty `git -C <submodule> status --porcelain` Submodule paths can include spaces (for example `SKILLS/ANTHROPIC COOKBOOK`), so always quote path arguments when running manual `git -C` commands. ## 4) Update Docs in This Repo When a submodule changes behavior, update: - `docs/README.md` if canonical links changed - `docs/USAGE.md` / `docs/TROUBLESHOOTING.md` if runtime guidance changed - `README.md` / `CONTRIBUTING.md` if contributor workflow changed ## 5) PR Expectations For submodule-related PRs: 1. Commit inside the submodule first. 2. Return to the parent repo and verify `git submodule status` has no unexpected `-dirty` entries. 3. Commit the updated submodule pointer in the parent repository. 4. Include any matching docs updates in this repository. 5. Mention the upstream submodule commit SHA in the PR description.
πŸ‘0
πŸ‘οΈ0
docs
πŸ€–system promptβ€’6 months ago

Hook Observability and Session Replay

Instrument hooks with replayable logs, task-state visibility, and non-blocking observability around agent sessions.

devops
⭐1
# Hook Observability and Session Replay Imported from curated first-party documentation sources. ## What this covers Use this skill when you need to understand what hooks fired, what they emitted, and how a session can be replayed after the fact. ## Use this when - Debugging hook-driven automation - Replaying session events after failures - Adding observability without blocking interactive work ## Expected outcomes - Hook activity becomes inspectable and replayable - Operational state survives beyond a single terminal session - Observability stays useful without overwhelming operators ## Source synthesis - EVOKORE-MCP/docs/VOICE_AND_HOOKS.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/VOICE_AND_HOOKS.md) - EVOKORE-MCP/docs/USE_CASES_AND_WALKTHROUGHS.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/USE_CASES_AND_WALKTHROUGHS.md) ## Dedupe notes Focuses on replay and observability instead of importing the broader voice-sidecar guide verbatim. ## Source excerpts ### EVOKORE-MCP/docs/VOICE_AND_HOOKS.md EVOKORE currently has three separate voice-related systems plus a set of hook and observability utilities. They overlap in operator workflows, but they are not the same runtime. ## The three voice-related systems ### 1. ElevenLabs MCP proxy This is the optional `elevenlabs` child server configured in `mcp.config.json`. What it is: - proxied through the EVOKORE router - exposed as prefixed MCP tools - available to any EVOKORE-connected MCP client when configured successfully What it is for: - text-to-speech and other ElevenLabs MCP operations as tools - routing voice-related actions through the standard EVOKORE proxy/security stack Requirements: - `uvx` available on PATH - `ELEVENLABS_API_KEY` set ### 2. VoiceMode VoiceMode is a separate voice-conversation system for Claude Code. What it is: - registered separately from EVOKORE - not routed through EVOKOREÒ€ℒs stdio server - used for bidirectional voice conversation in Claude Code What it is for: - speaking to Claude and hearing spoken responses - using `OPENAI_API_KEY` and VoiceModeÒ€ℒs own runtime Windows note: - VoiceMode relies on `uvx` being directly available - set `OPENAI_API_KEY` in the shell that launches Claude Code ### 3. VoiceSidecar VoiceSidecar is a standalone WebSocket server implemented in `src/Voice ... ### EVOKORE-MCP/docs/USE_CASES_AND_WALKTHROUGHS.md This guide turns the runtime contracts into practical operator flows. ## Walkthrough 1: Adopt a workflow from the skill library Use this when you want EVOKORE to retrieve process guidance before the model starts acting. ### Goal Find and adopt an existing workflow such as `session-wrap`. ### Steps 1. Ask the client to search skills: ```text Search the MCP for a workflow about session wrap-up and continuity. ``` 2. EVOKORE uses `search_skills` and returns matching skills. 3. Ask for a specific skill: ```text Show me help for the session-wrap skill. ``` 4. EVOKORE uses `get_skill_help` and returns the skillÒ€ℒs internal instructions. 5. For broader task matching, ask: ```text Resolve a workflow for wrapping this session, documenting open risks, and preparing the next handoff. ``` 6. EVOKORE uses `resolve_workflow` and injects the top 1-3 relevant workflows directly into the tool response. ### Why this matters - keeps the model grounded in repo-specific process - reduces prompt drift - makes handoff and governance behavior repeatable ## Walkthrough 2: Use a proxied tool that requires HITL approval Use this when the tool is configured as `require_approval` in `permissions.yml`. ### Goal Allow a protected proxied tool call such as `fs_write_f ...
πŸ‘0
πŸ‘οΈ0
docs
πŸ€–system promptβ€’6 months ago

Human-in-the-Loop Approval Token Workflow

Gate risky tool execution behind approval tokens so agents can retry safely after a human reviewer signs off.

security
⭐1
# Human-in-the-Loop Approval Token Workflow Imported from curated first-party documentation sources. ## What this covers Use this workflow when autonomous execution needs a durable handoff from human review back into the agent loop. ## Use this when - Retrying a blocked tool call after approval - Adding a human checkpoint to sensitive actions - Reducing insecure workarounds around approval flows ## Expected outcomes - Approval becomes a reusable tokenized workflow - Agents resume work without losing execution context - Security controls stay explicit instead of being implied ## Source synthesis - EVOKORE-MCP/docs/V2_MULTI_AGENT_WORKFLOWS.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/V2_MULTI_AGENT_WORKFLOWS.md) - EVOKORE-MCP/docs/AGENT33_IMPROVEMENT_INSTRUCTIONS.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/AGENT33_IMPROVEMENT_INSTRUCTIONS.md) ## Dedupe notes Synthesizes the approval-token concept from the main workflow doc and the Agent33 improvement transfer notes. ## Source excerpts ### EVOKORE-MCP/docs/V2_MULTI_AGENT_WORKFLOWS.md With EVOKORE-MCP v2.0 fully operational, we can now leverage 40+ proxied GitHub and Filesystem tools seamlessly within complex multi-agent workflows. The core architectural advancements include: ## 1. Dynamic Tool Prefixing & Indexing Instead of loading 40+ tools into an LLM's context window statically (which causes massive bloat), EVOKORE's `ProxyManager` boots child servers (like `@modelcontextprotocol/server-github` and `@modelcontextprotocol/server-filesystem`) and dynamically prefixes their tools (`github_create_issue`, `fs_write_file`). This prevents namespace collisions while keeping the tools accessible to native skills. ## 2. Human-in-the-Loop (HITL) Security Interceptor Automated multi-agent workflows involving sensitive endpoints (like GitHub write access or file deletion) are governed by EVOKORE's stateless `_evokore_approval_token` architecture. - When an agent attempts a restricted action, the tool call is intercepted and blocked. - The server returns an error explicitly commanding the agent to prompt the human for approval. - Upon approval, the agent retries the exact tool call with the injected token, securely fulfilling the workflow without severing the conversational context. ## 3. Active Skill Orchestration (Native Harnessing) Unlike v1.0 where skills merel ... ### EVOKORE-MCP/docs/AGENT33_IMPROVEMENT_INSTRUCTIONS.md > **Purpose**: Feed this file into Claude Code CLI when working on the Agent33 repo. It contains patterns, architectures, and capabilities proven in EVOKORE-MCP that Agent33 should adopt. --- ## 1. Multi-Server MCP Aggregation Pattern **What Agent33 lacks**: Agent33's MCP server (Phase 43) is a single-endpoint bridge. It doesn't aggregate multiple child MCP servers behind a unified namespace. **What to build**: A proxy layer that spawns and manages multiple child MCP servers from a single config file, presenting them as one unified tool surface. ### Implementation spec: ``` mcp.config.json { "servers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } }, "fs": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "./"] }, "elevenlabs": { "command": "uvx", "args": ["elevenlabs-mcp"], "env": { "ELEVENLABS_API_KEY": "${ELEVENLABS_API_KEY}" } } } } ``` **Key patterns from EVOKORE**: - **Tool name prefixing**: Every proxied tool gets renamed `{serverId}_{originalName}` to prevent namespace collisions (e.g., `github_create_issue`, `fs_read_file`). First-registration-wins for duplicates. - **Environment interpolation**: `${VAR}` syntax in `env` blocks resolved ...
πŸ‘0
πŸ‘οΈ0
docs
πŸ“textβ€’6 months ago

Guardrailed Code Review Pipeline

Run a staged code review workflow with explicit risk assessment, reviewer handoffs, and merge gates.

coding
⭐1
# Guardrailed Code Review Pipeline Imported from curated first-party documentation sources. ## What this covers Use this workflow when a change needs structured review evidence, clear approval stages, and a rollback-aware path to merge. ## Use this when - Multi-step L1 and L2 review signoff - High-risk changes that need audit trails - Standardizing reviewer decisions before merge ## Expected outcomes - Review states and approval checkpoints stay explicit - Risk assessment and evidence are captured before merge - The workflow remains small enough to reuse across repositories ## Source synthesis - AGENT33/docs/functionality-and-workflows.md (https://github.com/mattmre/AGENT33/blob/main/docs/functionality-and-workflows.md) - AGENT33/docs/walkthroughs.md (https://github.com/mattmre/AGENT33/blob/main/docs/walkthroughs.md) ## Dedupe notes Combines AGENT33 lifecycle mapping and operator walkthrough material into one review-oriented site entry. ## Source excerpts ### AGENT33/docs/functionality-and-workflows.md ### 4.1 Review Lifecycle States: - `draft -> ready -> l1-review -> l1-approved -> (optional l2-review -> l2-approved) -> approved -> merged` Main APIs: - `/v1/reviews/{id}/assess` - `/v1/reviews/{id}/assign-l1` - `/v1/reviews/{id}/l1` - `/v1/reviews/{id}/assign-l2` - `/v1/reviews/{id}/l2` - `/v1/reviews/{id}/approve` - `/v1/reviews/{id}/merge` ### AGENT33/docs/walkthroughs.md ## 4. Review Lifecycle (Two-Layer Signoff) Create review: ```bash curl -X POST http://localhost:8000/v1/reviews/ \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"task_id":"TASK-101","branch":"feat/docs-refresh","pr_number":12}' ``` Assess risk: ```bash curl -X POST http://localhost:8000/v1/reviews/<review_id>/assess \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"triggers":["api-public","security"]}' ``` Move to ready and assign L1: ```bash curl -X POST http://localhost:8000/v1/reviews/<review_id>/ready -H "Authorization: Bearer $TOKEN" curl -X POST http://localhost:8000/v1/reviews/<review_id>/assign-l1 -H "Authorization: Bearer $TOKEN" ``` Submit L1 decision: ```bash curl -X POST http://localhost:8000/v1/reviews/<review_id>/l1 \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"decision":"approved","issues":[],"comments":"L1 pass"}' ``` If L2 required, continue: ```bash curl -X POST http://localhost:8000/v1/reviews/<review_id>/assign-l2 -H "Authorization: Bearer $TOKEN" curl -X POST http://localhost:8000/v1/reviews/<review_id>/l2 \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"decision":"approved","issues":[],"commen ...
πŸ‘0
πŸ‘οΈ0
docs
πŸ€–system promptβ€’7 months ago

on-call-handoff-patterns

Master on-call shift handoffs with context transfer, escalation

coding
⭐1
# On-Call Handoff Patterns Effective patterns for on-call shift transitions, ensuring continuity, context transfer, and reliable incident response across shifts. ## When to Use This Skill - Transitioning on-call responsibilities - Writing shift handoff summaries - Documenting ongoing investigations - Establishing on-call rotation procedures - Improving handoff quality - Onboarding new on-call engineers ## Core Concepts ### 1. Handoff Components | Component | Purpose | | -------------------------- | ----------------------- | | **Active Incidents** | What's currently broken | | **Ongoing Investigations** | Issues being debugged | | **Recent Changes** | Deployments, configs | | **Known Issues** | Workarounds in place | | **Upcoming Events** | Maintenance, releases | ### 2. Handoff Timing ``` Recommended: 30 min overlap between shifts Outgoing: β”œβ”€β”€ 15 min: Write handoff document └── 15 min: Sync call with incoming Incoming: β”œβ”€β”€ 15 min: Review handoff document β”œβ”€β”€ 15 min: Sync call with outgoing └── 5 min: Verify alerting setup ``` ## Templates ### Template 1: Shift Handoff Document ````markdown # On-Call Handoff: Platform Team **Outgoing**: @alice (2024-01-15 to 2024-01-22) **Incoming**: @bob (2024-01-22 to 2024-01-29) **Handoff Time**: 2024-01-22 09:00 UTC --- ## πŸ”΄ Active Incidents ### None currently active No active incidents at handoff time. --- ## 🟑 Ongoing Investigations ### 1. Intermittent API Timeouts (ENG-1234) **Status**: Investigating **Started**: 2024-01-20 **Impact**: ~0.1% of requests timing out **Context**: - Timeouts correlate with database backup window (02:00-03:00 UTC) - Suspect backup process causing lock contention - Added extra logging in PR #567 (deployed 01/21) **Next Steps**: - [ ] Review new logs after tonight's backup - [ ] Consider moving backup window if confirmed **Resources**: - Dashboard: [API Latency](https://grafana/d/api-latency) - Thread: #platform-eng (01/20, 14:32) --- ### 2. Memory Growth in Auth Service (ENG-1235) **Status**: Monitoring **Started**: 2024-01-18 **Impact**: None yet (proactive) **Context**: - Memory usage growing ~5% per day - No memory leak found in profiling - Suspect connection pool not releasing properly **Next Steps**: - [ ] Review heap dump from 01/21 - [ ] Consider restart if usage > 80% **Resources**: - Dashboard: [Auth Service Memory](https://grafana/d/auth-memory) - Analysis doc: [Memory Investigation](https://docs/eng-1235) --- ## 🟒 Resolved This Shift ### Payment Service Outage (2024-01-19) - **Duration**: 23 minutes - **Root Cause**: Database connection exhaustion - **Resolution**: Rolled back v2.3.4, increased pool size - **Postmortem**: [POSTMORTEM-89](https://docs/postmortem-89) - **Follow-up tickets**: ENG-1230, ENG-1231 --- ## πŸ“‹ Recent Changes ### Deployments | Service | Version | Time | Notes | | ------------ | ------- | ----------- | -------------------------- | | api-gateway | v3.2.1 | 01/21 14:00 | Bug fix for header parsing | | user-service | v2.8.0 | 01/20 10:00 | New profile features | | auth-service | v4.1.2 | 01/19 16:00 | Security patch | ### Configuration Changes - 01/21: Increased API rate limit from 1000 to 1500 RPS - 01/20: Updated database connection pool max from 50 to 75 ### Infrastructure - 01/20: Added 2 nodes to Kubernetes cluster - 01/19: Upgraded Redis from 6.2 to 7.0 --- ## ⚠️ Known Issues & Workarounds ### 1. Slow Dashboard Loading **Issue**: Grafana dashboards slow on Monday mornings **Workaround**: Wait 5 min after 08:00 UTC for cache warm-up **Ticket**: OPS-456 (P3) ### 2. Flaky Integration Test **Issue**: `test_payment_flow` fails intermittently in CI **Workaround**: Re-run failed job (usually passes on retry) **Ticket**: ENG-1200 (P2) --- ## πŸ“… Upcoming Events | Date | Event | Impact | Contact | | ----------- | -------------------- | ------------------- | ------------- | | 01/23 02:00 | Database maintenance | 5 min read-only | @dba-team | | 01/24 14:00 | Major release v5.0 | Monitor closely | @release-team | | 01/25 | Marketing campaign | 2x traffic expected | @platform | --- ## πŸ“ž Escalation Reminders | Issue Type | First Escalation | Second Escalation | | --------------- | -------------------- | ----------------- | | Payment issues | @payments-oncall | @payments-manager | | Auth issues | @auth-oncall | @security-team | | Database issues | @dba-team | @infra-manager | | Unknown/severe | @engineering-manager | @vp-engineering | --- ## πŸ”§ Quick Reference ### Common Commands ```bash # Check service health kubectl get pods -A | grep -v Running # Recent deployments kubectl get events --sort-by='.lastTimestamp' | tail -20 # Database connections psql -c "SELECT count(*) FROM pg_stat_activity;" # Clear cache (emergency only) redis-cli FLUSHDB ``` ```` ### Important Links - [Runbooks](https://wiki/runbooks) - [Service Catalog](https://wiki/services) - [Incident Slack](https://slack.com/incidents) - [PagerDuty](https://pagerduty.com/schedules) --- ## Handoff Checklist ### Outgoing Engineer - [x] Document active incidents - [x] Document ongoing investigations - [x] List recent changes - [x] Note known issues - [x] Add upcoming events - [x] Sync with incoming engineer ### Incoming Engineer - [ ] Read this document - [ ] Join sync call - [ ] Verify PagerDuty is routing to you - [ ] Verify Slack notifications working - [ ] Check VPN/access working - [ ] Review critical dashboards ```` ### Template 2: Quick Handoff (Async) ```markdown # Quick Handoff: @alice β†’ @bob ## TL;DR - No active incidents - 1 investigation ongoing (API timeouts, see ENG-1234) - Major release tomorrow (01/24) - be ready for issues ## Watch List 1. API latency around 02:00-03:00 UTC (backup window) 2. Auth service memory (restart if > 80%) ## Recent - Deployed api-gateway v3.2.1 yesterday (stable) - Increased rate limits to 1500 RPS ## Coming Up - 01/23 02:00 - DB maintenance (5 min read-only) - 01/24 14:00 - v5.0 release ## Questions? I'll be available on Slack until 17:00 today. ```` ### Template 3: Incident Handoff (Mid-Incident) ```markdown # INCIDENT HANDOFF: Payment Service Degradation **Incident Start**: 2024-01-22 08:15 UTC **Current Status**: Mitigating **Severity**: SEV2 --- ## Current State - Error rate: 15% (down from 40%) - Mitigation in progress: scaling up pods - ETA to resolution: ~30 min ## What We Know 1. Root cause: Memory pressure on payment-service pods 2. Triggered by: Unusual traffic spike (3x normal) 3. Contributing: Inefficient query in checkout flow ## What We've Done - Scaled payment-service from 5 β†’ 15 pods - Enabled rate limiting on checkout endpoint - Disabled non-critical features ## What Needs to Happen 1. Monitor error rate - should reach <1% in ~15 min 2. If not improving, escalate to @payments-manager 3. Once stable, begin root cause investigation ## Key People - Incident Commander: @alice (handing off) - Comms Lead: @charlie - Technical Lead: @bob (incoming) ## Communication - Status page: Updated at 08:45 - Customer support: Notified - Exec team: Aware ## Resources - Incident channel: #inc-20240122-payment - Dashboard: [Payment Service](https://grafana/d/payments) - Runbook: [Payment Degradation](https://wiki/runbooks/payments) --- **Incoming on-call (@bob) - Please confirm you have:** - [ ] Joined #inc-20240122-payment - [ ] Access to dashboards - [ ] Understand current state - [ ] Know escalation path ``` ## Handoff Sync Meeting ### Agenda (15 minutes) ```markdown ## Handoff Sync: @alice β†’ @bob 1. **Active Issues** (5 min) - Walk through any ongoing incidents - Discuss investigation status - Transfer context and theories 2. **Recent Changes** (3 min) - Deployments to watch - Config changes - Known regressions 3. **Upcoming Events** (3 min) - Maintenance windows - Expected traffic changes - Releases planned 4. **Questions** (4 min) - Clarify anything unclear - Confirm access and alerting - Exchange contact info ``` ## On-Call Best Practices ### Before Your Shift ```markdown ## Pre-Shift Checklist ### Access Verification - [ ] VPN working - [ ] kubectl access to all clusters - [ ] Database read access - [ ] Log aggregator access (Splunk/Datadog) - [ ] PagerDuty app installed and logged in ### Alerting Setup - [ ] PagerDuty schedule shows you as primary - [ ] Phone notifications enabled - [ ] Slack notifications for incident channels - [ ] Test alert received and acknowledged ### Knowledge Refresh - [ ] Review recent incidents (past 2 weeks) - [ ] Check service changelog - [ ] Skim critical runbooks - [ ] Know escalation contacts ### Environment Ready - [ ] Laptop charged and accessible - [ ] Phone charged - [ ] Quiet space available for calls - [ ] Secondary contact identified (if traveling) ``` ### During Your Shift ```markdown ## Daily On-Call Routine ### Morning (start of day) - [ ] Check overnight alerts - [ ] Review dashboards for anomalies - [ ] Check for any P0/P1 tickets created - [ ] Skim incident channels for context ### Throughout Day - [ ] Respond to alerts within SLA - [ ] Document investigation progress - [ ] Update team on significant issues - [ ] Triage incoming pages ### End of Day - [ ] Hand off any active issues - [ ] Update investigation docs - [ ] Note anything for next shift ``` ### After Your Shift ```markdown ## Post-Shift Checklist - [ ] Complete handoff document - [ ] Sync with incoming on-call - [ ] Verify PagerDuty routing changed - [ ] Close/update investigation tickets - [ ] File postmortems for any incidents - [ ] Take time off if shift was stressful ``` ## Escalation Guidelines ### When to Escalate ```markdown ## Escalation Triggers ### Immediate Escalation - SEV1 incident declared - Data breach suspected - Unable to diagnose within 30 min - Customer or legal escalation received ### Consider Escalation - Issue spans multiple teams - Requires expertise you don't have - Business impact exceeds threshold - You're uncertain about next steps ### How to Escalate 1. Page the appropriate escalation path 2. Provide brief context in Slack 3. Stay engaged until escalation acknowledges 4. Hand off cleanly, don't just disappear ``` ## Best Practices ### Do's - **Document everything** - Future you will thank you - **Escalate early** - Better safe than sorry - **Take breaks** - Alert fatigue is real - **Keep handoffs synchronous** - Async loses context - **Test your setup** - Before incidents, not during ### Don'ts - **Don't skip handoffs** - Context loss causes incidents - **Don't hero** - Escalate when needed - **Don't ignore alerts** - Even if they seem minor - **Don't work sick** - Swap shifts instead - **Don't disappear** - Stay reachable during shift ## Resources - [Google SRE - Being On-Call](https://sre.google/sre-book/being-on-call/) - [PagerDuty On-Call Guide](https://www.pagerduty.com/resources/learn/on-call-management/) - [Increment On-Call Issue](https://increment.com/on-call/)
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered