Skip to main content
EVOKORE// BROWSE
>

./browse/prompts

34 NODES
πŸ“textβ€’3 hours ago

Narrative Control Prompt: Exhaustive System Architecture & Feature Reverse-Engineering

Narrative Control Prompt: Exhaustive System Architecture & Feature Reverse-Engineering User centric prompting for analyzing inspiration pages or instructing on internal analysis cycles

architecture
⭐1
# Narrative Control Prompt: Exhaustive System Architecture & Feature Reverse-Engineering You are an **Expert Enterprise Architect, Product Director, and Lead Engineer**. Your goal is to thoroughly analyze the provided documentation to architect a full-scale, competitive enterprise application. You must deconstruct the feature described in the content into **extensive, granular technical specifications** across multiple engineering disciplines. **CRITICAL INSTRUCTIONS**: - **DO NOT SUMMARIZE**. Be exhaustive. - For every category below, aim to list **10+ specific items** if possible. - Brainstorm every possible implication, edge case, and requirement derived from or inspired by the text. - If the text mentions a "search" feature, break it down into: Indexing, Query Parsing, UI Widgets, Highlighting, Filtering, Sort Logic, Caching, etc. Please output the analysis in the following Markdown format: # 1. Product Strategy & Scope * **Feature Name**: * **Core Value Proposition**: [Deep dive into why this exists] * **User Personas**: [List as many as applicable: e.g. Admin, Power User, Viewer, Auditor, API Consumer...] * **User Stories**: [Extensive list of 10+ granular user stories e.g. "As a User, I want to..."] * **Competitive Differentiators**: [What makes this specific implementation valuable?] # 2. Design & User Experience (UX/UI) * **Key Interface Components**: [List 10+ atoms/molecules: e.g. Data Grid, Filter Chips, Modals, Tooltips, Empty States, Toasts, Dropdowns...] * **Interaction Patterns**: [List 10+ patterns: e.g. Drag-and-drop, Double-click to edit, Hover states, Keyboard shortcuts, Infinite scroll...] * **Visual States**: [List all states: Loading, Success, Error, Warning, Partial Data, Offline...] * **Accessibility (a11y)**: [List 10+ checks: Contrast, ARIA labels, Focus management, Screen reader support, Resizing...] # 3. Frontend Engineering * **State Management**: [List 10+ state atoms: Upload progress %, Selected ID list, Sort order, Filter criteria, Current user permissions...] * **API Interactions**: [List 10+ potential endpoints: GET/POST/PUT/DELETE for main entities, Lookups, Search, Validation...] * **Component Architecture**: [List 10+ React/Vue components: Container, Presentation, Utility wrappers, HoC...] * **Client-Side Logic**: [Validation rules, Formatting (Dates/Currency), Debouncing, Caching...] # 4. Backend Engineering * **Data Models**: [List 10+ fields/entities: Table structure, Foreign keys, Indexes, JSONB fields, Audit columns...] * **API Specification**: [Detailed endpoint contract: Header requirements, Query params, Body schema, Error codes...] * **Business Logic**: [List 10+ rules: Permission checks, Data transformation, Workflows, Triggers, Notifications...] * **Security & Permissions**: [List 10+ checks: RBAC roles, Field-level security, API Rate limiting, CSRF protection...] # 5. Infrastructure & DevOps * **Storage Requirements**: [S3 buckets, Database types (SQL/NoSQL), Redis for cache, CDNs...] * **Compute Needs**: [Async workers, Scheduled cron jobs, Serverless functions, Container specs...] * **Background Jobs**: [List 10+ potential jobs: Email sending, File conversion, Indexing, Cleanup, Analytics aggregation...] * **Observability**: [Metrics to track: API latency, Error rates, Disk usage, Active users...] # 6. Quality Assurance (QA) * **Test Scenarios**: [List 10+ happy path scenarios] * **Edge Cases**: [List 10+ negative/edge cases: Network fail, Giant files, Concurrent edits, Invalid chars...] * **Performance Metrics**: [Specific SLAs: <200ms API, <1s Page load, 99.9% Uptime...] * **Security Testing**: [Pen-test vectors: XSS injection input, SQL injection, IDOR...] # 7. Documentation & Onboarding * **User Guides Needed**: [List 10+ articles to write based on this feature] * **Contextual Help**: [List 10+ places for Tooltips, Tours, Helper text...] * **API Documentation**: [Swagger/OpenAPI requirements] # 8. Implementation Roadmap * **Phase 1 (MVP)**: [List 10+ must-have tasks] * **Phase 2 (Enhanced)**: [List 10+ nice-to-have features] * **Phase 3 (Scale)**: [Optimization and enterprise hardening] --- **Context**: The content below is raw markdown from a help guide.
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ“textβ€’3 hours ago

Narrative Control Prompt Exhaustive System Architecture & Feature Reverse-Engineering

Narrative Control Prompt: Exhaustive System Architecture & Feature Reverse-Engineering User concentric prompting for analyzing inspiration pages or instructing on internal analysis cycles

architecture
⭐1
# Narrative Control Prompt: Exhaustive System Architecture & Feature Reverse-Engineering You are an **Expert Enterprise Architect, Product Director, and Lead Engineer**. Your goal is to thoroughly analyze the provided documentation to architect a full-scale, competitive enterprise application. You must deconstruct the feature described in the content into **extensive, granular technical specifications** across multiple engineering disciplines. **CRITICAL INSTRUCTIONS**: - **DO NOT SUMMARIZE**. Be exhaustive. - For every category below, aim to list **10+ specific items** if possible. - Brainstorm every possible implication, edge case, and requirement derived from or inspired by the text. - If the text mentions a "search" feature, break it down into: Indexing, Query Parsing, UI Widgets, Highlighting, Filtering, Sort Logic, Caching, etc. Please output the analysis in the following Markdown format: # 1. Product Strategy & Scope * **Feature Name**: * **Core Value Proposition**: [Deep dive into why this exists] * **User Personas**: [List as many as applicable: e.g. Admin, Power User, Viewer, Auditor, API Consumer...] * **User Stories**: [Extensive list of 10+ granular user stories e.g. "As a User, I want to..."] * **Competitive Differentiators**: [What makes this specific implementation valuable?] # 2. Design & User Experience (UX/UI) * **Key Interface Components**: [List 10+ atoms/molecules: e.g. Data Grid, Filter Chips, Modals, Tooltips, Empty States, Toasts, Dropdowns...] * **Interaction Patterns**: [List 10+ patterns: e.g. Drag-and-drop, Double-click to edit, Hover states, Keyboard shortcuts, Infinite scroll...] * **Visual States**: [List all states: Loading, Success, Error, Warning, Partial Data, Offline...] * **Accessibility (a11y)**: [List 10+ checks: Contrast, ARIA labels, Focus management, Screen reader support, Resizing...] # 3. Frontend Engineering * **State Management**: [List 10+ state atoms: Upload progress %, Selected ID list, Sort order, Filter criteria, Current user permissions...] * **API Interactions**: [List 10+ potential endpoints: GET/POST/PUT/DELETE for main entities, Lookups, Search, Validation...] * **Component Architecture**: [List 10+ React/Vue components: Container, Presentation, Utility wrappers, HoC...] * **Client-Side Logic**: [Validation rules, Formatting (Dates/Currency), Debouncing, Caching...] # 4. Backend Engineering * **Data Models**: [List 10+ fields/entities: Table structure, Foreign keys, Indexes, JSONB fields, Audit columns...] * **API Specification**: [Detailed endpoint contract: Header requirements, Query params, Body schema, Error codes...] * **Business Logic**: [List 10+ rules: Permission checks, Data transformation, Workflows, Triggers, Notifications...] * **Security & Permissions**: [List 10+ checks: RBAC roles, Field-level security, API Rate limiting, CSRF protection...] # 5. Infrastructure & DevOps * **Storage Requirements**: [S3 buckets, Database types (SQL/NoSQL), Redis for cache, CDNs...] * **Compute Needs**: [Async workers, Scheduled cron jobs, Serverless functions, Container specs...] * **Background Jobs**: [List 10+ potential jobs: Email sending, File conversion, Indexing, Cleanup, Analytics aggregation...] * **Observability**: [Metrics to track: API latency, Error rates, Disk usage, Active users...] # 6. Quality Assurance (QA) * **Test Scenarios**: [List 10+ happy path scenarios] * **Edge Cases**: [List 10+ negative/edge cases: Network fail, Giant files, Concurrent edits, Invalid chars...] * **Performance Metrics**: [Specific SLAs: <200ms API, <1s Page load, 99.9% Uptime...] * **Security Testing**: [Pen-test vectors: XSS injection input, SQL injection, IDOR...] # 7. Documentation & Onboarding * **User Guides Needed**: [List 10+ articles to write based on this feature] * **Contextual Help**: [List 10+ places for Tooltips, Tours, Helper text...] * **API Documentation**: [Swagger/OpenAPI requirements] # 8. Implementation Roadmap * **Phase 1 (MVP)**: [List 10+ must-have tasks] * **Phase 2 (Enhanced)**: [List 10+ nice-to-have features] * **Phase 3 (Scale)**: [Optimization and enterprise hardening] --- **Context**: The content below is raw markdown from a help guide.
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’6 months ago

Hook Observability and Session Replay

Instrument hooks with replayable logs, task-state visibility, and non-blocking observability around agent sessions.

devops
⭐1
# Hook Observability and Session Replay Imported from curated first-party documentation sources. ## What this covers Use this skill when you need to understand what hooks fired, what they emitted, and how a session can be replayed after the fact. ## Use this when - Debugging hook-driven automation - Replaying session events after failures - Adding observability without blocking interactive work ## Expected outcomes - Hook activity becomes inspectable and replayable - Operational state survives beyond a single terminal session - Observability stays useful without overwhelming operators ## Source synthesis - EVOKORE-MCP/docs/VOICE_AND_HOOKS.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/VOICE_AND_HOOKS.md) - EVOKORE-MCP/docs/USE_CASES_AND_WALKTHROUGHS.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/USE_CASES_AND_WALKTHROUGHS.md) ## Dedupe notes Focuses on replay and observability instead of importing the broader voice-sidecar guide verbatim. ## Source excerpts ### EVOKORE-MCP/docs/VOICE_AND_HOOKS.md EVOKORE currently has three separate voice-related systems plus a set of hook and observability utilities. They overlap in operator workflows, but they are not the same runtime. ## The three voice-related systems ### 1. ElevenLabs MCP proxy This is the optional `elevenlabs` child server configured in `mcp.config.json`. What it is: - proxied through the EVOKORE router - exposed as prefixed MCP tools - available to any EVOKORE-connected MCP client when configured successfully What it is for: - text-to-speech and other ElevenLabs MCP operations as tools - routing voice-related actions through the standard EVOKORE proxy/security stack Requirements: - `uvx` available on PATH - `ELEVENLABS_API_KEY` set ### 2. VoiceMode VoiceMode is a separate voice-conversation system for Claude Code. What it is: - registered separately from EVOKORE - not routed through EVOKOREÒ€ℒs stdio server - used for bidirectional voice conversation in Claude Code What it is for: - speaking to Claude and hearing spoken responses - using `OPENAI_API_KEY` and VoiceModeÒ€ℒs own runtime Windows note: - VoiceMode relies on `uvx` being directly available - set `OPENAI_API_KEY` in the shell that launches Claude Code ### 3. VoiceSidecar VoiceSidecar is a standalone WebSocket server implemented in `src/Voice ... ### EVOKORE-MCP/docs/USE_CASES_AND_WALKTHROUGHS.md This guide turns the runtime contracts into practical operator flows. ## Walkthrough 1: Adopt a workflow from the skill library Use this when you want EVOKORE to retrieve process guidance before the model starts acting. ### Goal Find and adopt an existing workflow such as `session-wrap`. ### Steps 1. Ask the client to search skills: ```text Search the MCP for a workflow about session wrap-up and continuity. ``` 2. EVOKORE uses `search_skills` and returns matching skills. 3. Ask for a specific skill: ```text Show me help for the session-wrap skill. ``` 4. EVOKORE uses `get_skill_help` and returns the skillÒ€ℒs internal instructions. 5. For broader task matching, ask: ```text Resolve a workflow for wrapping this session, documenting open risks, and preparing the next handoff. ``` 6. EVOKORE uses `resolve_workflow` and injects the top 1-3 relevant workflows directly into the tool response. ### Why this matters - keeps the model grounded in repo-specific process - reduces prompt drift - makes handoff and governance behavior repeatable ## Walkthrough 2: Use a proxied tool that requires HITL approval Use this when the tool is configured as `require_approval` in `permissions.yml`. ### Goal Allow a protected proxied tool call such as `fs_write_f ...
πŸ‘0
πŸ‘οΈ0
docs
πŸ€–system promptβ€’6 months ago

Multi-Server MCP Aggregation Pattern

Aggregate tools across multiple MCP child servers with prefixing, collision avoidance, and routing rules that stay deterministic.

architecture
⭐1
# Multi-Server MCP Aggregation Pattern Imported from curated first-party documentation sources. ## What this covers Use this pattern when an MCP host must broker tools from multiple child servers without sacrificing clarity or control. ## Use this when - Combining tools from several MCP backends - Avoiding tool-name collisions across providers - Keeping origin and routing visible during execution ## Expected outcomes - Server-prefixed names make tool origins obvious - Collisions are avoided without brittle manual renaming - Operators can extend the tool surface without losing determinism ## Source synthesis - EVOKORE-MCP/docs/AGENT33_IMPROVEMENT_INSTRUCTIONS.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/AGENT33_IMPROVEMENT_INSTRUCTIONS.md) - EVOKORE-MCP/docs/TOOLS_AND_DISCOVERY.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/TOOLS_AND_DISCOVERY.md) ## Dedupe notes Uses the improvement-transfer doc as the primary source, with the discovery doc covering prefixing and compatibility details. ## Source excerpts ### EVOKORE-MCP/docs/AGENT33_IMPROVEMENT_INSTRUCTIONS.md > **Purpose**: Feed this file into Claude Code CLI when working on the Agent33 repo. It contains patterns, architectures, and capabilities proven in EVOKORE-MCP that Agent33 should adopt. --- ## 1. Multi-Server MCP Aggregation Pattern **What Agent33 lacks**: Agent33's MCP server (Phase 43) is a single-endpoint bridge. It doesn't aggregate multiple child MCP servers behind a unified namespace. **What to build**: A proxy layer that spawns and manages multiple child MCP servers from a single config file, presenting them as one unified tool surface. ### Implementation spec: ``` mcp.config.json { "servers": { "github": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } }, "fs": { "command": "npx", "args": ["-y", "@modelcontextprotocol/server-filesystem", "./"] }, "elevenlabs": { "command": "uvx", "args": ["elevenlabs-mcp"], "env": { "ELEVENLABS_API_KEY": "${ELEVENLABS_API_KEY}" } } } } ``` **Key patterns from EVOKORE**: - **Tool name prefixing**: Every proxied tool gets renamed `{serverId}_{originalName}` to prevent namespace collisions (e.g., `github_create_issue`, `fs_read_file`). First-registration-wins for duplicates. - **Environment interpolation**: `${VAR}` syntax in `env` blocks resolved ... ### EVOKORE-MCP/docs/TOOLS_AND_DISCOVERY.md This page explains how EVOKORE presents tools, how proxy names are built, and how `discover_tools` changes the visible tool surface. ## Two tool populations ### Native EVOKORE tools These tools are defined by EVOKORE itself: - `docs_architect` - `skill_creator` - `resolve_workflow` - `search_skills` - `get_skill_help` - `discover_tools` Properties: - always available - always visible - not subject to proxy prefixing ### Proxied child-server tools These come from child servers in `mcp.config.json`. Current configured sources: - `github` - `fs` - optional `elevenlabs` Properties: - fetched from child servers at startup - renamed with server prefixes - governed by `permissions.yml` - routed through `ProxyManager` ## Prefixing and compatibility EVOKORE rewrites proxied tool names to: ```text ${serverId}_${tool.name} ``` Why this exists: - prevents tool-name collisions across child servers - makes origin obvious during execution and review - keeps exact-name routing deterministic Examples: | Upstream tool | EVOKORE-exposed tool | |---|---| | `read_file` from `fs` | `fs_read_file` | | `create_issue` from `github` | `github_create_issue` | ### Duplicate-prefixed name policy If two child registrations would create the same final prefixed name: - the first registrati ...
πŸ‘0
πŸ‘οΈ0
docs
πŸ€–system promptβ€’6 months ago

Dynamic Tool Discovery and Session Activation

Expose only native tools at session start, then activate proxied tools on demand through a searchable discovery layer.

architecture
⭐1
# Dynamic Tool Discovery and Session Activation Imported from curated first-party documentation sources. ## What this covers Use this skill when you need smaller tool payloads, exact-name compatibility, and session-scoped tool activation. ## Use this when - Reducing tool-list bloat for focused sessions - Preserving exact-name tool calls while hiding noise - Activating proxied tools only when a task requires them ## Expected outcomes - Discovery changes listing visibility without breaking routing - Session-scoped activation keeps tools relevant to the task - Operators can trade compatibility against payload size intentionally ## Source synthesis - EVOKORE-MCP/docs/TOOLS_AND_DISCOVERY.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/TOOLS_AND_DISCOVERY.md) ## Dedupe notes Uses the canonical EVOKORE-MCP discovery doc instead of duplicating the same concept from walkthrough and migration docs. ## Source excerpts ### EVOKORE-MCP/docs/TOOLS_AND_DISCOVERY.md This page explains how EVOKORE presents tools, how proxy names are built, and how `discover_tools` changes the visible tool surface. ## Two tool populations ### Native EVOKORE tools These tools are defined by EVOKORE itself: - `docs_architect` - `skill_creator` - `resolve_workflow` - `search_skills` - `get_skill_help` - `discover_tools` Properties: - always available - always visible - not subject to proxy prefixing ### Proxied child-server tools These come from child servers in `mcp.config.json`. Current configured sources: - `github` - `fs` - optional `elevenlabs` Properties: - fetched from child servers at startup - renamed with server prefixes - governed by `permissions.yml` - routed through `ProxyManager` ## Prefixing and compatibility EVOKORE rewrites proxied tool names to: ```text ${serverId}_${tool.name} ``` Why this exists: - prevents tool-name collisions across child servers - makes origin obvious during execution and review - keeps exact-name routing deterministic Examples: | Upstream tool | EVOKORE-exposed tool | |---|---| | `read_file` from `fs` | `fs_read_file` | | `create_issue` from `github` | `github_create_issue` | ### Duplicate-prefixed name policy If two child registrations would create the same final prefixed name: - the first registration wins - later duplicates are skipped - EVOKORE logs a warning and duplicate summary ## Discovery modes | Mode | What `tools/list` returns | Best for | |---|---|---| | `legacy` | all native + proxied tools | maximum compatibility | | `dynamic` | native tools + session-activated proxied tools | smaller initial tool payloads | Environment toggle: ```bash EVOKORE_TOOL_DISCOVERY_MODE=legacy EVOKORE_TOOL_DISCOVERY_MODE=dynamic ``` ## Dynamic discovery lifecycle In `dynamic` mode, EVOKORE uses a session-scoped activation set. Lifecycle: 1. session starts with only native tools visible 2. user/model calls `discover_tools` 3. `ToolCatalogIndex` searches the merged native + proxied catalog 4. matching proxied tools are activated for that session 5. EVOKORE emits `sendToolListChanged()` best-effort 6. client re-runs `tools/list` or auto-refreshes ```mermaid flowchart TD A[Session starts in dynamic mode] --> B[tools/list shows native tools] B --> C[discover_tools query] C --> D[ToolCatalogIndex searches merged catalog] D --> E[Matching proxied tools added to session activation set] E --> F[sendToolListChanged best-effort] F --> G[Client refreshes tools/list] G --> H[Activated proxied tools now visible] D --> I[Exact-name proxied call still works eve ...
πŸ‘0
πŸ‘οΈ0
docs
πŸ€–system promptβ€’7 months ago

debugging-strategies

Master systematic debugging techniques, profiling tools, and root

coding
⭐1
# Debugging Strategies Transform debugging from frustrating guesswork into systematic problem-solving with proven strategies, powerful tools, and methodical approaches. ## When to Use This Skill - Tracking down elusive bugs - Investigating performance issues - Understanding unfamiliar codebases - Debugging production issues - Analyzing crash dumps and stack traces - Profiling application performance - Investigating memory leaks - Debugging distributed systems ## Core Principles ### 1. The Scientific Method **1. Observe**: What's the actual behavior? **2. Hypothesize**: What could be causing it? **3. Experiment**: Test your hypothesis **4. Analyze**: Did it prove/disprove your theory? **5. Repeat**: Until you find the root cause ### 2. Debugging Mindset **Don't Assume:** - "It can't be X" - Yes it can - "I didn't change Y" - Check anyway - "It works on my machine" - Find out why **Do:** - Reproduce consistently - Isolate the problem - Keep detailed notes - Question everything - Take breaks when stuck ### 3. Rubber Duck Debugging Explain your code and problem out loud (to a rubber duck, colleague, or yourself). Often reveals the issue. ## Systematic Debugging Process ### Phase 1: Reproduce ```markdown ## Reproduction Checklist 1. **Can you reproduce it?** - Always? Sometimes? Randomly? - Specific conditions needed? - Can others reproduce it? 2. **Create minimal reproduction** - Simplify to smallest example - Remove unrelated code - Isolate the problem 3. **Document steps** - Write down exact steps - Note environment details - Capture error messages ``` ### Phase 2: Gather Information ```markdown ## Information Collection 1. **Error Messages** - Full stack trace - Error codes - Console/log output 2. **Environment** - OS version - Language/runtime version - Dependencies versions - Environment variables 3. **Recent Changes** - Git history - Deployment timeline - Configuration changes 4. **Scope** - Affects all users or specific ones? - All browsers or specific ones? - Production only or also dev? ``` ### Phase 3: Form Hypothesis ```markdown ## Hypothesis Formation Based on gathered info, ask: 1. **What changed?** - Recent code changes - Dependency updates - Infrastructure changes 2. **What's different?** - Working vs broken environment - Working vs broken user - Before vs after 3. **Where could this fail?** - Input validation - Business logic - Data layer - External services ``` ### Phase 4: Test & Verify ```markdown ## Testing Strategies 1. **Binary Search** - Comment out half the code - Narrow down problematic section - Repeat until found 2. **Add Logging** - Strategic console.log/print - Track variable values - Trace execution flow 3. **Isolate Components** - Test each piece separately - Mock dependencies - Remove complexity 4. **Compare Working vs Broken** - Diff configurations - Diff environments - Diff data ``` ## Debugging Tools ### JavaScript/TypeScript Debugging ```typescript // Chrome DevTools Debugger function processOrder(order: Order) { debugger; // Execution pauses here const total = calculateTotal(order); console.log("Total:", total); // Conditional breakpoint if (order.items.length > 10) { debugger; // Only breaks if condition true } return total; } // Console debugging techniques console.log("Value:", value); // Basic console.table(arrayOfObjects); // Table format console.time("operation"); /* code */ console.timeEnd("operation"); // Timing console.trace(); // Stack trace console.assert(value > 0, "Value must be positive"); // Assertion // Performance profiling performance.mark("start-operation"); // ... operation code performance.mark("end-operation"); performance.measure("operation", "start-operation", "end-operation"); console.log(performance.getEntriesByType("measure")); ``` **VS Code Debugger Configuration:** ```json // .vscode/launch.json { "version": "0.2.0", "configurations": [ { "type": "node", "request": "launch", "name": "Debug Program", "program": "${workspaceFolder}/src/index.ts", "preLaunchTask": "tsc: build - tsconfig.json", "outFiles": ["${workspaceFolder}/dist/**/*.js"], "skipFiles": ["<node_internals>/**"] }, { "type": "node", "request": "launch", "name": "Debug Tests", "program": "${workspaceFolder}/node_modules/jest/bin/jest", "args": ["--runInBand", "--no-cache"], "console": "integratedTerminal" } ] } ``` ### Python Debugging ```python # Built-in debugger (pdb) import pdb def calculate_total(items): total = 0 pdb.set_trace() # Debugger starts here for item in items: total += item.price * item.quantity return total # Breakpoint (Python 3.7+) def process_order(order): breakpoint() # More convenient than pdb.set_trace() # ... code # Post-mortem debugging try: risky_operation() except Exception: import pdb pdb.post_mortem() # Debug at exception point # IPython debugging (ipdb) from ipdb import set_trace set_trace() # Better interface than pdb # Logging for debugging import logging logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger(__name__) def fetch_user(user_id): logger.debug(f'Fetching user: {user_id}') user = db.query(User).get(user_id) logger.debug(f'Found user: {user}') return user # Profile performance import cProfile import pstats cProfile.run('slow_function()', 'profile_stats') stats = pstats.Stats('profile_stats') stats.sort_stats('cumulative') stats.print_stats(10) # Top 10 slowest ``` ### Go Debugging ```go // Delve debugger // Install: go install github.com/go-delve/delve/cmd/dlv@latest // Run: dlv debug main.go import ( "fmt" "runtime" "runtime/debug" ) // Print stack trace func debugStack() { debug.PrintStack() } // Panic recovery with debugging func processRequest() { defer func() { if r := recover(); r != nil { fmt.Println("Panic:", r) debug.PrintStack() } }() // ... code that might panic } // Memory profiling import _ "net/http/pprof" // Visit http://localhost:6060/debug/pprof/ // CPU profiling import ( "os" "runtime/pprof" ) f, _ := os.Create("cpu.prof") pprof.StartCPUProfile(f) defer pprof.StopCPUProfile() // ... code to profile ``` ## Advanced Debugging Techniques ### Technique 1: Binary Search Debugging ```bash # Git bisect for finding regression git bisect start git bisect bad # Current commit is bad git bisect good v1.0.0 # v1.0.0 was good # Git checks out middle commit # Test it, then: git bisect good # if it works git bisect bad # if it's broken # Continue until bug found git bisect reset # when done ``` ### Technique 2: Differential Debugging Compare working vs broken: ```markdown ## What's Different? | Aspect | Working | Broken | | ------------ | ----------- | -------------- | | Environment | Development | Production | | Node version | 18.16.0 | 18.15.0 | | Data | Empty DB | 1M records | | User | Admin | Regular user | | Browser | Chrome | Safari | | Time | During day | After midnight | Hypothesis: Time-based issue? Check timezone handling. ``` ### Technique 3: Trace Debugging ```typescript // Function call tracing function trace( target: any, propertyKey: string, descriptor: PropertyDescriptor, ) { const originalMethod = descriptor.value; descriptor.value = function (...args: any[]) { console.log(`Calling ${propertyKey} with args:`, args); const result = originalMethod.apply(this, args); console.log(`${propertyKey} returned:`, result); return result; }; return descriptor; } class OrderService { @trace calculateTotal(items: Item[]): number { return items.reduce((sum, item) => sum + item.price, 0); } } ``` ### Technique 4: Memory Leak Detection ```typescript // Chrome DevTools Memory Profiler // 1. Take heap snapshot // 2. Perform action // 3. Take another snapshot // 4. Compare snapshots // Node.js memory debugging if (process.memoryUsage().heapUsed > 500 * 1024 * 1024) { console.warn("High memory usage:", process.memoryUsage()); // Generate heap dump require("v8").writeHeapSnapshot(); } // Find memory leaks in tests let beforeMemory: number; beforeEach(() => { beforeMemory = process.memoryUsage().heapUsed; }); afterEach(() => { const afterMemory = process.memoryUsage().heapUsed; const diff = afterMemory - beforeMemory; if (diff > 10 * 1024 * 1024) { // 10MB threshold console.warn(`Possible memory leak: ${diff / 1024 / 1024}MB`); } }); ``` ## Debugging Patterns by Issue Type ### Pattern 1: Intermittent Bugs ```markdown ## Strategies for Flaky Bugs 1. **Add extensive logging** - Log timing information - Log all state transitions - Log external interactions 2. **Look for race conditions** - Concurrent access to shared state - Async operations completing out of order - Missing synchronization 3. **Check timing dependencies** - setTimeout/setInterval - Promise resolution order - Animation frame timing 4. **Stress test** - Run many times - Vary timing - Simulate load ``` ### Pattern 2: Performance Issues ```markdown ## Performance Debugging 1. **Profile first** - Don't optimize blindly - Measure before and after - Find bottlenecks 2. **Common culprits** - N+1 queries - Unnecessary re-renders - Large data processing - Synchronous I/O 3. **Tools** - Browser DevTools Performance tab - Lighthouse - Python: cProfile, line_profiler - Node: clinic.js, 0x ``` ### Pattern 3: Production Bugs ```markdown ## Production Debugging 1. **Gather evidence** - Error tracking (Sentry, Bugsnag) - Application logs - User reports - Metrics/monitoring 2. **Reproduce locally** - Use production data (anonymized) - Match environment - Follow exact steps 3. **Safe investigation** - Don't change production - Use feature flags - Add monitoring/logging - Test fixes in staging ``` ## Best Practices 1. **Reproduce First**: Can't fix what you can't reproduce 2. **Isolate the Problem**: Remove complexity until minimal case 3. **Read Error Messages**: They're usually helpful 4. **Check Recent Changes**: Most bugs are recent 5. **Use Version Control**: Git bisect, blame, history 6. **Take Breaks**: Fresh eyes see better 7. **Document Findings**: Help future you 8. **Fix Root Cause**: Not just symptoms ## Common Debugging Mistakes - **Making Multiple Changes**: Change one thing at a time - **Not Reading Error Messages**: Read the full stack trace - **Assuming It's Complex**: Often it's simple - **Debug Logging in Prod**: Remove before shipping - **Not Using Debugger**: console.log isn't always best - **Giving Up Too Soon**: Persistence pays off - **Not Testing the Fix**: Verify it actually works ## Quick Debugging Checklist ```markdown ## When Stuck, Check: - [ ] Spelling errors (typos in variable names) - [ ] Case sensitivity (fileName vs filename) - [ ] Null/undefined values - [ ] Array index off-by-one - [ ] Async timing (race conditions) - [ ] Scope issues (closure, hoisting) - [ ] Type mismatches - [ ] Missing dependencies - [ ] Environment variables - [ ] File paths (absolute vs relative) - [ ] Cache issues (clear cache) - [ ] Stale data (refresh database) ``` ## Resources - **references/debugging-tools-guide.md**: Comprehensive tool documentation - **references/performance-profiling.md**: Performance debugging guide - **references/production-debugging.md**: Debugging live systems - **assets/debugging-checklist.md**: Quick reference checklist - **assets/common-bugs.md**: Common bug patterns - **scripts/debug-helper.ts**: Debugging utility functions
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

git-advanced-workflows

Master advanced Git workflows including rebasing, cherry-picking,

coding
⭐1
# Git Advanced Workflows Master advanced Git techniques to maintain clean history, collaborate effectively, and recover from any situation with confidence. ## When to Use This Skill - Cleaning up commit history before merging - Applying specific commits across branches - Finding commits that introduced bugs - Working on multiple features simultaneously - Recovering from Git mistakes or lost commits - Managing complex branch workflows - Preparing clean PRs for review - Synchronizing diverged branches ## Core Concepts ### 1. Interactive Rebase Interactive rebase is the Swiss Army knife of Git history editing. **Common Operations:** - `pick`: Keep commit as-is - `reword`: Change commit message - `edit`: Amend commit content - `squash`: Combine with previous commit - `fixup`: Like squash but discard message - `drop`: Remove commit entirely **Basic Usage:** ```bash # Rebase last 5 commits git rebase -i HEAD~5 # Rebase all commits on current branch git rebase -i $(git merge-base HEAD main) # Rebase onto specific commit git rebase -i abc123 ``` ### 2. Cherry-Picking Apply specific commits from one branch to another without merging entire branches. ```bash # Cherry-pick single commit git cherry-pick abc123 # Cherry-pick range of commits (exclusive start) git cherry-pick abc123..def456 # Cherry-pick without committing (stage changes only) git cherry-pick -n abc123 # Cherry-pick and edit commit message git cherry-pick -e abc123 ``` ### 3. Git Bisect Binary search through commit history to find the commit that introduced a bug. ```bash # Start bisect git bisect start # Mark current commit as bad git bisect bad # Mark known good commit git bisect good v1.0.0 # Git will checkout middle commit - test it # Then mark as good or bad git bisect good # or: git bisect bad # Continue until bug found # When done git bisect reset ``` **Automated Bisect:** ```bash # Use script to test automatically git bisect start HEAD v1.0.0 git bisect run ./test.sh # test.sh should exit 0 for good, 1-127 (except 125) for bad ``` ### 4. Worktrees Work on multiple branches simultaneously without stashing or switching. ```bash # List existing worktrees git worktree list # Add new worktree for feature branch git worktree add ../project-feature feature/new-feature # Add worktree and create new branch git worktree add -b bugfix/urgent ../project-hotfix main # Remove worktree git worktree remove ../project-feature # Prune stale worktrees git worktree prune ``` ### 5. Reflog Your safety net - tracks all ref movements, even deleted commits. ```bash # View reflog git reflog # View reflog for specific branch git reflog show feature/branch # Restore deleted commit git reflog # Find commit hash git checkout abc123 git branch recovered-branch # Restore deleted branch git reflog git branch deleted-branch abc123 ``` ## Practical Workflows ### Workflow 1: Clean Up Feature Branch Before PR ```bash # Start with feature branch git checkout feature/user-auth # Interactive rebase to clean history git rebase -i main # Example rebase operations: # - Squash "fix typo" commits # - Reword commit messages for clarity # - Reorder commits logically # - Drop unnecessary commits # Force push cleaned branch (safe if no one else is using it) git push --force-with-lease origin feature/user-auth ``` ### Workflow 2: Apply Hotfix to Multiple Releases ```bash # Create fix on main git checkout main git commit -m "fix: critical security patch" # Apply to release branches git checkout release/2.0 git cherry-pick abc123 git checkout release/1.9 git cherry-pick abc123 # Handle conflicts if they arise git cherry-pick --continue # or git cherry-pick --abort ``` ### Workflow 3: Find Bug Introduction ```bash # Start bisect git bisect start git bisect bad HEAD git bisect good v2.1.0 # Git checks out middle commit - run tests npm test # If tests fail git bisect bad # If tests pass git bisect good # Git will automatically checkout next commit to test # Repeat until bug found # Automated version git bisect start HEAD v2.1.0 git bisect run npm test ``` ### Workflow 4: Multi-Branch Development ```bash # Main project directory cd ~/projects/myapp # Create worktree for urgent bugfix git worktree add ../myapp-hotfix hotfix/critical-bug # Work on hotfix in separate directory cd ../myapp-hotfix # Make changes, commit git commit -m "fix: resolve critical bug" git push origin hotfix/critical-bug # Return to main work without interruption cd ~/projects/myapp git fetch origin git cherry-pick hotfix/critical-bug # Clean up when done git worktree remove ../myapp-hotfix ``` ### Workflow 5: Recover from Mistakes ```bash # Accidentally reset to wrong commit git reset --hard HEAD~5 # Oh no! # Use reflog to find lost commits git reflog # Output shows: # abc123 HEAD@{0}: reset: moving to HEAD~5 # def456 HEAD@{1}: commit: my important changes # Recover lost commits git reset --hard def456 # Or create branch from lost commit git branch recovery def456 ``` ## Advanced Techniques ### Rebase vs Merge Strategy **When to Rebase:** - Cleaning up local commits before pushing - Keeping feature branch up-to-date with main - Creating linear history for easier review **When to Merge:** - Integrating completed features into main - Preserving exact history of collaboration - Public branches used by others ```bash # Update feature branch with main changes (rebase) git checkout feature/my-feature git fetch origin git rebase origin/main # Handle conflicts git status # Fix conflicts in files git add . git rebase --continue # Or merge instead git merge origin/main ``` ### Autosquash Workflow Automatically squash fixup commits during rebase. ```bash # Make initial commit git commit -m "feat: add user authentication" # Later, fix something in that commit # Stage changes git commit --fixup HEAD # or specify commit hash # Make more changes git commit --fixup abc123 # Rebase with autosquash git rebase -i --autosquash main # Git automatically marks fixup commits ``` ### Split Commit Break one commit into multiple logical commits. ```bash # Start interactive rebase git rebase -i HEAD~3 # Mark commit to split with 'edit' # Git will stop at that commit # Reset commit but keep changes git reset HEAD^ # Stage and commit in logical chunks git add file1.py git commit -m "feat: add validation" git add file2.py git commit -m "feat: add error handling" # Continue rebase git rebase --continue ``` ### Partial Cherry-Pick Cherry-pick only specific files from a commit. ```bash # Show files in commit git show --name-only abc123 # Checkout specific files from commit git checkout abc123 -- path/to/file1.py path/to/file2.py # Stage and commit git commit -m "cherry-pick: apply specific changes from abc123" ``` ## Best Practices 1. **Always Use --force-with-lease**: Safer than --force, prevents overwriting others' work 2. **Rebase Only Local Commits**: Don't rebase commits that have been pushed and shared 3. **Descriptive Commit Messages**: Future you will thank present you 4. **Atomic Commits**: Each commit should be a single logical change 5. **Test Before Force Push**: Ensure history rewrite didn't break anything 6. **Keep Reflog Aware**: Remember reflog is your safety net for 90 days 7. **Branch Before Risky Operations**: Create backup branch before complex rebases ```bash # Safe force push git push --force-with-lease origin feature/branch # Create backup before risky operation git branch backup-branch git rebase -i main # If something goes wrong git reset --hard backup-branch ``` ## Common Pitfalls - **Rebasing Public Branches**: Causes history conflicts for collaborators - **Force Pushing Without Lease**: Can overwrite teammate's work - **Losing Work in Rebase**: Resolve conflicts carefully, test after rebase - **Forgetting Worktree Cleanup**: Orphaned worktrees consume disk space - **Not Backing Up Before Experiment**: Always create safety branch - **Bisect on Dirty Working Directory**: Commit or stash before bisecting ## Recovery Commands ```bash # Abort operations in progress git rebase --abort git merge --abort git cherry-pick --abort git bisect reset # Restore file to version from specific commit git restore --source=abc123 path/to/file # Undo last commit but keep changes git reset --soft HEAD^ # Undo last commit and discard changes git reset --hard HEAD^ # Recover deleted branch (within 90 days) git reflog git branch recovered-branch abc123 ``` ## Resources - **references/git-rebase-guide.md**: Deep dive into interactive rebase - **references/git-conflict-resolution.md**: Advanced conflict resolution strategies - **references/git-history-rewriting.md**: Safely rewriting Git history - **assets/git-workflow-checklist.md**: Pre-PR cleanup checklist - **assets/git-aliases.md**: Useful Git aliases for advanced workflows - **scripts/git-clean-branches.sh**: Clean up merged and stale branches
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

sql-optimization-patterns

Master SQL query optimization, indexing strategies, and EXPLAIN

coding
⭐1
# SQL Optimization Patterns Transform slow database queries into lightning-fast operations through systematic optimization, proper indexing, and query plan analysis. ## When to Use This Skill - Debugging slow-running queries - Designing performant database schemas - Optimizing application response times - Reducing database load and costs - Improving scalability for growing datasets - Analyzing EXPLAIN query plans - Implementing efficient indexes - Resolving N+1 query problems ## Core Concepts ### 1. Query Execution Plans (EXPLAIN) Understanding EXPLAIN output is fundamental to optimization. **PostgreSQL EXPLAIN:** ```sql -- Basic explain EXPLAIN SELECT * FROM users WHERE email = 'user@example.com'; -- With actual execution stats EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'user@example.com'; -- Verbose output with more details EXPLAIN (ANALYZE, BUFFERS, VERBOSE) SELECT u.*, o.order_total FROM users u JOIN orders o ON u.id = o.user_id WHERE u.created_at > NOW() - INTERVAL '30 days'; ``` **Key Metrics to Watch:** - **Seq Scan**: Full table scan (usually slow for large tables) - **Index Scan**: Using index (good) - **Index Only Scan**: Using index without touching table (best) - **Nested Loop**: Join method (okay for small datasets) - **Hash Join**: Join method (good for larger datasets) - **Merge Join**: Join method (good for sorted data) - **Cost**: Estimated query cost (lower is better) - **Rows**: Estimated rows returned - **Actual Time**: Real execution time ### 2. Index Strategies Indexes are the most powerful optimization tool. **Index Types:** - **B-Tree**: Default, good for equality and range queries - **Hash**: Only for equality (=) comparisons - **GIN**: Full-text search, array queries, JSONB - **GiST**: Geometric data, full-text search - **BRIN**: Block Range INdex for very large tables with correlation ```sql -- Standard B-Tree index CREATE INDEX idx_users_email ON users(email); -- Composite index (order matters!) CREATE INDEX idx_orders_user_status ON orders(user_id, status); -- Partial index (index subset of rows) CREATE INDEX idx_active_users ON users(email) WHERE status = 'active'; -- Expression index CREATE INDEX idx_users_lower_email ON users(LOWER(email)); -- Covering index (include additional columns) CREATE INDEX idx_users_email_covering ON users(email) INCLUDE (name, created_at); -- Full-text search index CREATE INDEX idx_posts_search ON posts USING GIN(to_tsvector('english', title || ' ' || body)); -- JSONB index CREATE INDEX idx_metadata ON events USING GIN(metadata); ``` ### 3. Query Optimization Patterns **Avoid SELECT \*:** ```sql -- Bad: Fetches unnecessary columns SELECT * FROM users WHERE id = 123; -- Good: Fetch only what you need SELECT id, email, name FROM users WHERE id = 123; ``` **Use WHERE Clause Efficiently:** ```sql -- Bad: Function prevents index usage SELECT * FROM users WHERE LOWER(email) = 'user@example.com'; -- Good: Create functional index or use exact match CREATE INDEX idx_users_email_lower ON users(LOWER(email)); -- Then: SELECT * FROM users WHERE LOWER(email) = 'user@example.com'; -- Or store normalized data SELECT * FROM users WHERE email = 'user@example.com'; ``` **Optimize JOINs:** ```sql -- Bad: Cartesian product then filter SELECT u.name, o.total FROM users u, orders o WHERE u.id = o.user_id AND u.created_at > '2024-01-01'; -- Good: Filter before join SELECT u.name, o.total FROM users u JOIN orders o ON u.id = o.user_id WHERE u.created_at > '2024-01-01'; -- Better: Filter both tables SELECT u.name, o.total FROM (SELECT * FROM users WHERE created_at > '2024-01-01') u JOIN orders o ON u.id = o.user_id; ``` ## Optimization Patterns ### Pattern 1: Eliminate N+1 Queries **Problem: N+1 Query Anti-Pattern** ```python # Bad: Executes N+1 queries users = db.query("SELECT * FROM users LIMIT 10") for user in users: orders = db.query("SELECT * FROM orders WHERE user_id = ?", user.id) # Process orders ``` **Solution: Use JOINs or Batch Loading** ```sql -- Solution 1: JOIN SELECT u.id, u.name, o.id as order_id, o.total FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.id IN (1, 2, 3, 4, 5); -- Solution 2: Batch query SELECT * FROM orders WHERE user_id IN (1, 2, 3, 4, 5); ``` ```python # Good: Single query with JOIN or batch load # Using JOIN results = db.query(""" SELECT u.id, u.name, o.id as order_id, o.total FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.id IN (1, 2, 3, 4, 5) """) # Or batch load users = db.query("SELECT * FROM users LIMIT 10") user_ids = [u.id for u in users] orders = db.query( "SELECT * FROM orders WHERE user_id IN (?)", user_ids ) # Group orders by user_id orders_by_user = {} for order in orders: orders_by_user.setdefault(order.user_id, []).append(order) ``` ### Pattern 2: Optimize Pagination **Bad: OFFSET on Large Tables** ```sql -- Slow for large offsets SELECT * FROM users ORDER BY created_at DESC LIMIT 20 OFFSET 100000; -- Very slow! ``` **Good: Cursor-Based Pagination** ```sql -- Much faster: Use cursor (last seen ID) SELECT * FROM users WHERE created_at < '2024-01-15 10:30:00' -- Last cursor ORDER BY created_at DESC LIMIT 20; -- With composite sorting SELECT * FROM users WHERE (created_at, id) < ('2024-01-15 10:30:00', 12345) ORDER BY created_at DESC, id DESC LIMIT 20; -- Requires index CREATE INDEX idx_users_cursor ON users(created_at DESC, id DESC); ``` ### Pattern 3: Aggregate Efficiently **Optimize COUNT Queries:** ```sql -- Bad: Counts all rows SELECT COUNT(*) FROM orders; -- Slow on large tables -- Good: Use estimates for approximate counts SELECT reltuples::bigint AS estimate FROM pg_class WHERE relname = 'orders'; -- Good: Filter before counting SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '7 days'; -- Better: Use index-only scan CREATE INDEX idx_orders_created ON orders(created_at); SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '7 days'; ``` **Optimize GROUP BY:** ```sql -- Bad: Group by then filter SELECT user_id, COUNT(*) as order_count FROM orders GROUP BY user_id HAVING COUNT(*) > 10; -- Better: Filter first, then group (if possible) SELECT user_id, COUNT(*) as order_count FROM orders WHERE status = 'completed' GROUP BY user_id HAVING COUNT(*) > 10; -- Best: Use covering index CREATE INDEX idx_orders_user_status ON orders(user_id, status); ``` ### Pattern 4: Subquery Optimization **Transform Correlated Subqueries:** ```sql -- Bad: Correlated subquery (runs for each row) SELECT u.name, u.email, (SELECT COUNT(*) FROM orders o WHERE o.user_id = u.id) as order_count FROM users u; -- Good: JOIN with aggregation SELECT u.name, u.email, COUNT(o.id) as order_count FROM users u LEFT JOIN orders o ON o.user_id = u.id GROUP BY u.id, u.name, u.email; -- Better: Use window functions SELECT DISTINCT ON (u.id) u.name, u.email, COUNT(o.id) OVER (PARTITION BY u.id) as order_count FROM users u LEFT JOIN orders o ON o.user_id = u.id; ``` **Use CTEs for Clarity:** ```sql -- Using Common Table Expressions WITH recent_users AS ( SELECT id, name, email FROM users WHERE created_at > NOW() - INTERVAL '30 days' ), user_order_counts AS ( SELECT user_id, COUNT(*) as order_count FROM orders WHERE created_at > NOW() - INTERVAL '30 days' GROUP BY user_id ) SELECT ru.name, ru.email, COALESCE(uoc.order_count, 0) as orders FROM recent_users ru LEFT JOIN user_order_counts uoc ON ru.id = uoc.user_id; ``` ### Pattern 5: Batch Operations **Batch INSERT:** ```sql -- Bad: Multiple individual inserts INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com'); INSERT INTO users (name, email) VALUES ('Bob', 'bob@example.com'); INSERT INTO users (name, email) VALUES ('Carol', 'carol@example.com'); -- Good: Batch insert INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com'), ('Bob', 'bob@example.com'), ('Carol', 'carol@example.com'); -- Better: Use COPY for bulk inserts (PostgreSQL) COPY users (name, email) FROM '/tmp/users.csv' CSV HEADER; ``` **Batch UPDATE:** ```sql -- Bad: Update in loop UPDATE users SET status = 'active' WHERE id = 1; UPDATE users SET status = 'active' WHERE id = 2; -- ... repeat for many IDs -- Good: Single UPDATE with IN clause UPDATE users SET status = 'active' WHERE id IN (1, 2, 3, 4, 5, ...); -- Better: Use temporary table for large batches CREATE TEMP TABLE temp_user_updates (id INT, new_status VARCHAR); INSERT INTO temp_user_updates VALUES (1, 'active'), (2, 'active'), ...; UPDATE users u SET status = t.new_status FROM temp_user_updates t WHERE u.id = t.id; ``` ## Advanced Techniques ### Materialized Views Pre-compute expensive queries. ```sql -- Create materialized view CREATE MATERIALIZED VIEW user_order_summary AS SELECT u.id, u.name, COUNT(o.id) as total_orders, SUM(o.total) as total_spent, MAX(o.created_at) as last_order_date FROM users u LEFT JOIN orders o ON u.id = o.user_id GROUP BY u.id, u.name; -- Add index to materialized view CREATE INDEX idx_user_summary_spent ON user_order_summary(total_spent DESC); -- Refresh materialized view REFRESH MATERIALIZED VIEW user_order_summary; -- Concurrent refresh (PostgreSQL) REFRESH MATERIALIZED VIEW CONCURRENTLY user_order_summary; -- Query materialized view (very fast) SELECT * FROM user_order_summary WHERE total_spent > 1000 ORDER BY total_spent DESC; ``` ### Partitioning Split large tables for better performance. ```sql -- Range partitioning by date (PostgreSQL) CREATE TABLE orders ( id SERIAL, user_id INT, total DECIMAL, created_at TIMESTAMP ) PARTITION BY RANGE (created_at); -- Create partitions CREATE TABLE orders_2024_q1 PARTITION OF orders FOR VALUES FROM ('2024-01-01') TO ('2024-04-01'); CREATE TABLE orders_2024_q2 PARTITION OF orders FOR VALUES FROM ('2024-04-01') TO ('2024-07-01'); -- Queries automatically use appropriate partition SELECT * FROM orders WHERE created_at BETWEEN '2024-02-01' AND '2024-02-28'; -- Only scans orders_2024_q1 partition ``` ### Query Hints and Optimization ```sql -- Force index usage (MySQL) SELECT * FROM users USE INDEX (idx_users_email) WHERE email = 'user@example.com'; -- Parallel query (PostgreSQL) SET max_parallel_workers_per_gather = 4; SELECT * FROM large_table WHERE condition; -- Join hints (PostgreSQL) SET enable_nestloop = OFF; -- Force hash or merge join ``` ## Best Practices 1. **Index Selectively**: Too many indexes slow down writes 2. **Monitor Query Performance**: Use slow query logs 3. **Keep Statistics Updated**: Run ANALYZE regularly 4. **Use Appropriate Data Types**: Smaller types = better performance 5. **Normalize Thoughtfully**: Balance normalization vs performance 6. **Cache Frequently Accessed Data**: Use application-level caching 7. **Connection Pooling**: Reuse database connections 8. **Regular Maintenance**: VACUUM, ANALYZE, rebuild indexes ```sql -- Update statistics ANALYZE users; ANALYZE VERBOSE orders; -- Vacuum (PostgreSQL) VACUUM ANALYZE users; VACUUM FULL users; -- Reclaim space (locks table) -- Reindex REINDEX INDEX idx_users_email; REINDEX TABLE users; ``` ## Common Pitfalls - **Over-Indexing**: Each index slows down INSERT/UPDATE/DELETE - **Unused Indexes**: Waste space and slow writes - **Missing Indexes**: Slow queries, full table scans - **Implicit Type Conversion**: Prevents index usage - **OR Conditions**: Can't use indexes efficiently - **LIKE with Leading Wildcard**: `LIKE '%abc'` can't use index - **Function in WHERE**: Prevents index usage unless functional index exists ## Monitoring Queries ```sql -- Find slow queries (PostgreSQL) SELECT query, calls, total_time, mean_time FROM pg_stat_statements ORDER BY mean_time DESC LIMIT 10; -- Find missing indexes (PostgreSQL) SELECT schemaname, tablename, seq_scan, seq_tup_read, idx_scan, seq_tup_read / seq_scan AS avg_seq_tup_read FROM pg_stat_user_tables WHERE seq_scan > 0 ORDER BY seq_tup_read DESC LIMIT 10; -- Find unused indexes (PostgreSQL) SELECT schemaname, tablename, indexname, idx_scan, idx_tup_read, idx_tup_fetch FROM pg_stat_user_indexes WHERE idx_scan = 0 ORDER BY pg_relation_size(indexrelid) DESC; ``` ## Resources - **references/postgres-optimization-guide.md**: PostgreSQL-specific optimization - **references/mysql-optimization-guide.md**: MySQL/MariaDB optimization - **references/query-plan-analysis.md**: Deep dive into EXPLAIN plans - **assets/index-strategy-checklist.md**: When and how to create indexes - **assets/query-optimization-checklist.md**: Step-by-step optimization guide - **scripts/analyze-slow-queries.sql**: Identify slow queries in your database - **scripts/index-recommendations.sql**: Generate index recommendations
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

architecture-decision-records

Write and maintain Architecture Decision Records (ADRs) following

coding
⭐1
# Architecture Decision Records Comprehensive patterns for creating, maintaining, and managing Architecture Decision Records (ADRs) that capture the context and rationale behind significant technical decisions. ## When to Use This Skill - Making significant architectural decisions - Documenting technology choices - Recording design trade-offs - Onboarding new team members - Reviewing historical decisions - Establishing decision-making processes ## Core Concepts ### 1. What is an ADR? An Architecture Decision Record captures: - **Context**: Why we needed to make a decision - **Decision**: What we decided - **Consequences**: What happens as a result ### 2. When to Write an ADR | Write ADR | Skip ADR | | -------------------------- | ---------------------- | | New framework adoption | Minor version upgrades | | Database technology choice | Bug fixes | | API design patterns | Implementation details | | Security architecture | Routine maintenance | | Integration patterns | Configuration changes | ### 3. ADR Lifecycle ``` Proposed β†’ Accepted β†’ Deprecated β†’ Superseded ↓ Rejected ``` ## Templates ### Template 1: Standard ADR (MADR Format) ```markdown # ADR-0001: Use PostgreSQL as Primary Database ## Status Accepted ## Context We need to select a primary database for our new e-commerce platform. The system will handle: - ~10,000 concurrent users - Complex product catalog with hierarchical categories - Transaction processing for orders and payments - Full-text search for products - Geospatial queries for store locator The team has experience with MySQL, PostgreSQL, and MongoDB. We need ACID compliance for financial transactions. ## Decision Drivers - **Must have ACID compliance** for payment processing - **Must support complex queries** for reporting - **Should support full-text search** to reduce infrastructure complexity - **Should have good JSON support** for flexible product attributes - **Team familiarity** reduces onboarding time ## Considered Options ### Option 1: PostgreSQL - **Pros**: ACID compliant, excellent JSON support (JSONB), built-in full-text search, PostGIS for geospatial, team has experience - **Cons**: Slightly more complex replication setup than MySQL ### Option 2: MySQL - **Pros**: Very familiar to team, simple replication, large community - **Cons**: Weaker JSON support, no built-in full-text search (need Elasticsearch), no geospatial without extensions ### Option 3: MongoDB - **Pros**: Flexible schema, native JSON, horizontal scaling - **Cons**: No ACID for multi-document transactions (at decision time), team has limited experience, requires schema design discipline ## Decision We will use **PostgreSQL 15** as our primary database. ## Rationale PostgreSQL provides the best balance of: 1. **ACID compliance** essential for e-commerce transactions 2. **Built-in capabilities** (full-text search, JSONB, PostGIS) reduce infrastructure complexity 3. **Team familiarity** with SQL databases reduces learning curve 4. **Mature ecosystem** with excellent tooling and community support The slight complexity in replication is outweighed by the reduction in additional services (no separate Elasticsearch needed). ## Consequences ### Positive - Single database handles transactions, search, and geospatial queries - Reduced operational complexity (fewer services to manage) - Strong consistency guarantees for financial data - Team can leverage existing SQL expertise ### Negative - Need to learn PostgreSQL-specific features (JSONB, full-text search syntax) - Vertical scaling limits may require read replicas sooner - Some team members need PostgreSQL-specific training ### Risks - Full-text search may not scale as well as dedicated search engines - Mitigation: Design for potential Elasticsearch addition if needed ## Implementation Notes - Use JSONB for flexible product attributes - Implement connection pooling with PgBouncer - Set up streaming replication for read replicas - Use pg_trgm extension for fuzzy search ## Related Decisions - ADR-0002: Caching Strategy (Redis) - complements database choice - ADR-0005: Search Architecture - may supersede if Elasticsearch needed ## References - [PostgreSQL JSON Documentation](https://www.postgresql.org/docs/current/datatype-json.html) - [PostgreSQL Full Text Search](https://www.postgresql.org/docs/current/textsearch.html) - Internal: Performance benchmarks in `/docs/benchmarks/database-comparison.md` ``` ### Template 2: Lightweight ADR ```markdown # ADR-0012: Adopt TypeScript for Frontend Development **Status**: Accepted **Date**: 2024-01-15 **Deciders**: @alice, @bob, @charlie ## Context Our React codebase has grown to 50+ components with increasing bug reports related to prop type mismatches and undefined errors. PropTypes provide runtime-only checking. ## Decision Adopt TypeScript for all new frontend code. Migrate existing code incrementally. ## Consequences **Good**: Catch type errors at compile time, better IDE support, self-documenting code. **Bad**: Learning curve for team, initial slowdown, build complexity increase. **Mitigations**: TypeScript training sessions, allow gradual adoption with `allowJs: true`. ``` ### Template 3: Y-Statement Format ```markdown # ADR-0015: API Gateway Selection In the context of **building a microservices architecture**, facing **the need for centralized API management, authentication, and rate limiting**, we decided for **Kong Gateway** and against **AWS API Gateway and custom Nginx solution**, to achieve **vendor independence, plugin extensibility, and team familiarity with Lua**, accepting that **we need to manage Kong infrastructure ourselves**. ``` ### Template 4: ADR for Deprecation ```markdown # ADR-0020: Deprecate MongoDB in Favor of PostgreSQL ## Status Accepted (Supersedes ADR-0003) ## Context ADR-0003 (2021) chose MongoDB for user profile storage due to schema flexibility needs. Since then: - MongoDB's multi-document transactions remain problematic for our use case - Our schema has stabilized and rarely changes - We now have PostgreSQL expertise from other services - Maintaining two databases increases operational burden ## Decision Deprecate MongoDB and migrate user profiles to PostgreSQL. ## Migration Plan 1. **Phase 1** (Week 1-2): Create PostgreSQL schema, dual-write enabled 2. **Phase 2** (Week 3-4): Backfill historical data, validate consistency 3. **Phase 3** (Week 5): Switch reads to PostgreSQL, monitor 4. **Phase 4** (Week 6): Remove MongoDB writes, decommission ## Consequences ### Positive - Single database technology reduces operational complexity - ACID transactions for user data - Team can focus PostgreSQL expertise ### Negative - Migration effort (~4 weeks) - Risk of data issues during migration - Lose some schema flexibility ## Lessons Learned Document from ADR-0003 experience: - Schema flexibility benefits were overestimated - Operational cost of multiple databases was underestimated - Consider long-term maintenance in technology decisions ``` ### Template 5: Request for Comments (RFC) Style ```markdown # RFC-0025: Adopt Event Sourcing for Order Management ## Summary Propose adopting event sourcing pattern for the order management domain to improve auditability, enable temporal queries, and support business analytics. ## Motivation Current challenges: 1. Audit requirements need complete order history 2. "What was the order state at time X?" queries are impossible 3. Analytics team needs event stream for real-time dashboards 4. Order state reconstruction for customer support is manual ## Detailed Design ### Event Store ``` OrderCreated { orderId, customerId, items[], timestamp } OrderItemAdded { orderId, item, timestamp } OrderItemRemoved { orderId, itemId, timestamp } PaymentReceived { orderId, amount, paymentId, timestamp } OrderShipped { orderId, trackingNumber, timestamp } ``` ### Projections - **CurrentOrderState**: Materialized view for queries - **OrderHistory**: Complete timeline for audit - **DailyOrderMetrics**: Analytics aggregation ### Technology - Event Store: EventStoreDB (purpose-built, handles projections) - Alternative considered: Kafka + custom projection service ## Drawbacks - Learning curve for team - Increased complexity vs. CRUD - Need to design events carefully (immutable once stored) - Storage growth (events never deleted) ## Alternatives 1. **Audit tables**: Simpler but doesn't enable temporal queries 2. **CDC from existing DB**: Complex, doesn't change data model 3. **Hybrid**: Event source only for order state changes ## Unresolved Questions - [ ] Event schema versioning strategy - [ ] Retention policy for events - [ ] Snapshot frequency for performance ## Implementation Plan 1. Prototype with single order type (2 weeks) 2. Team training on event sourcing (1 week) 3. Full implementation and migration (4 weeks) 4. Monitoring and optimization (ongoing) ## References - [Event Sourcing by Martin Fowler](https://martinfowler.com/eaaDev/EventSourcing.html) - [EventStoreDB Documentation](https://www.eventstore.com/docs) ``` ## ADR Management ### Directory Structure ``` docs/ β”œβ”€β”€ adr/ β”‚ β”œβ”€β”€ README.md # Index and guidelines β”‚ β”œβ”€β”€ template.md # Team's ADR template β”‚ β”œβ”€β”€ 0001-use-postgresql.md β”‚ β”œβ”€β”€ 0002-caching-strategy.md β”‚ β”œβ”€β”€ 0003-mongodb-user-profiles.md # [DEPRECATED] β”‚ └── 0020-deprecate-mongodb.md # Supersedes 0003 ``` ### ADR Index (README.md) ```markdown # Architecture Decision Records This directory contains Architecture Decision Records (ADRs) for [Project Name]. ## Index | ADR | Title | Status | Date | | ------------------------------------- | ---------------------------------- | ---------- | ---------- | | [0001](0001-use-postgresql.md) | Use PostgreSQL as Primary Database | Accepted | 2024-01-10 | | [0002](0002-caching-strategy.md) | Caching Strategy with Redis | Accepted | 2024-01-12 | | [0003](0003-mongodb-user-profiles.md) | MongoDB for User Profiles | Deprecated | 2023-06-15 | | [0020](0020-deprecate-mongodb.md) | Deprecate MongoDB | Accepted | 2024-01-15 | ## Creating a New ADR 1. Copy `template.md` to `NNNN-title-with-dashes.md` 2. Fill in the template 3. Submit PR for review 4. Update this index after approval ## ADR Status - **Proposed**: Under discussion - **Accepted**: Decision made, implementing - **Deprecated**: No longer relevant - **Superseded**: Replaced by another ADR - **Rejected**: Considered but not adopted ``` ### Automation (adr-tools) ```bash # Install adr-tools brew install adr-tools # Initialize ADR directory adr init docs/adr # Create new ADR adr new "Use PostgreSQL as Primary Database" # Supersede an ADR adr new -s 3 "Deprecate MongoDB in Favor of PostgreSQL" # Generate table of contents adr generate toc > docs/adr/README.md # Link related ADRs adr link 2 "Complements" 1 "Is complemented by" ``` ## Review Process ```markdown ## ADR Review Checklist ### Before Submission - [ ] Context clearly explains the problem - [ ] All viable options considered - [ ] Pros/cons balanced and honest - [ ] Consequences (positive and negative) documented - [ ] Related ADRs linked ### During Review - [ ] At least 2 senior engineers reviewed - [ ] Affected teams consulted - [ ] Security implications considered - [ ] Cost implications documented - [ ] Reversibility assessed ### After Acceptance - [ ] ADR index updated - [ ] Team notified - [ ] Implementation tickets created - [ ] Related documentation updated ``` ## Best Practices ### Do's - **Write ADRs early** - Before implementation starts - **Keep them short** - 1-2 pages maximum - **Be honest about trade-offs** - Include real cons - **Link related decisions** - Build decision graph - **Update status** - Deprecate when superseded ### Don'ts - **Don't change accepted ADRs** - Write new ones to supersede - **Don't skip context** - Future readers need background - **Don't hide failures** - Rejected decisions are valuable - **Don't be vague** - Specific decisions, specific consequences - **Don't forget implementation** - ADR without action is waste ## Resources - [Documenting Architecture Decisions (Michael Nygard)](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions) - [MADR Template](https://adr.github.io/madr/) - [ADR GitHub Organization](https://adr.github.io/) - [adr-tools](https://github.com/npryce/adr-tools)
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

openapi-spec-generation

Generate and maintain OpenAPI 3.1 specifications from code,

coding
⭐1
# OpenAPI Spec Generation Comprehensive patterns for creating, maintaining, and validating OpenAPI 3.1 specifications for RESTful APIs. ## When to Use This Skill - Creating API documentation from scratch - Generating OpenAPI specs from existing code - Designing API contracts (design-first approach) - Validating API implementations against specs - Generating client SDKs from specs - Setting up API documentation portals ## Core Concepts ### 1. OpenAPI 3.1 Structure ```yaml openapi: 3.1.0 info: title: API Title version: 1.0.0 servers: - url: https://api.example.com/v1 paths: /resources: get: ... components: schemas: ... securitySchemes: ... ``` ### 2. Design Approaches | Approach | Description | Best For | | ---------------- | ---------------------------- | ------------------- | | **Design-First** | Write spec before code | New APIs, contracts | | **Code-First** | Generate spec from code | Existing APIs | | **Hybrid** | Annotate code, generate spec | Evolving APIs | ## Templates ### Template 1: Complete API Specification ```yaml openapi: 3.1.0 info: title: User Management API description: | API for managing users and their profiles. ## Authentication All endpoints require Bearer token authentication. ## Rate Limiting - 1000 requests per minute for standard tier - 10000 requests per minute for enterprise tier version: 2.0.0 contact: name: API Support email: api-support@example.com url: https://docs.example.com license: name: MIT url: https://opensource.org/licenses/MIT servers: - url: https://api.example.com/v2 description: Production - url: https://staging-api.example.com/v2 description: Staging - url: http://localhost:3000/v2 description: Local development tags: - name: Users description: User management operations - name: Profiles description: User profile operations - name: Admin description: Administrative operations paths: /users: get: operationId: listUsers summary: List all users description: Returns a paginated list of users with optional filtering. tags: - Users parameters: - $ref: "#/components/parameters/PageParam" - $ref: "#/components/parameters/LimitParam" - name: status in: query description: Filter by user status schema: $ref: "#/components/schemas/UserStatus" - name: search in: query description: Search by name or email schema: type: string minLength: 2 maxLength: 100 responses: "200": description: Successful response content: application/json: schema: $ref: "#/components/schemas/UserListResponse" examples: default: $ref: "#/components/examples/UserListExample" "400": $ref: "#/components/responses/BadRequest" "401": $ref: "#/components/responses/Unauthorized" "429": $ref: "#/components/responses/RateLimited" security: - bearerAuth: [] post: operationId: createUser summary: Create a new user description: Creates a new user account and sends welcome email. tags: - Users requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/CreateUserRequest" examples: standard: summary: Standard user value: email: user@example.com name: John Doe role: user admin: summary: Admin user value: email: admin@example.com name: Admin User role: admin responses: "201": description: User created successfully content: application/json: schema: $ref: "#/components/schemas/User" headers: Location: description: URL of created user schema: type: string format: uri "400": $ref: "#/components/responses/BadRequest" "409": description: Email already exists content: application/json: schema: $ref: "#/components/schemas/Error" security: - bearerAuth: [] /users/{userId}: parameters: - $ref: "#/components/parameters/UserIdParam" get: operationId: getUser summary: Get user by ID tags: - Users responses: "200": description: Successful response content: application/json: schema: $ref: "#/components/schemas/User" "404": $ref: "#/components/responses/NotFound" security: - bearerAuth: [] patch: operationId: updateUser summary: Update user tags: - Users requestBody: required: true content: application/json: schema: $ref: "#/components/schemas/UpdateUserRequest" responses: "200": description: User updated content: application/json: schema: $ref: "#/components/schemas/User" "400": $ref: "#/components/responses/BadRequest" "404": $ref: "#/components/responses/NotFound" security: - bearerAuth: [] delete: operationId: deleteUser summary: Delete user tags: - Users - Admin responses: "204": description: User deleted "404": $ref: "#/components/responses/NotFound" security: - bearerAuth: [] - apiKey: [] components: schemas: User: type: object required: - id - email - name - status - createdAt properties: id: type: string format: uuid readOnly: true description: Unique user identifier email: type: string format: email description: User email address name: type: string minLength: 1 maxLength: 100 description: User display name status: $ref: "#/components/schemas/UserStatus" role: type: string enum: [user, moderator, admin] default: user avatar: type: string format: uri nullable: true metadata: type: object additionalProperties: true description: Custom metadata createdAt: type: string format: date-time readOnly: true updatedAt: type: string format: date-time readOnly: true UserStatus: type: string enum: [active, inactive, suspended, pending] description: User account status CreateUserRequest: type: object required: - email - name properties: email: type: string format: email name: type: string minLength: 1 maxLength: 100 role: type: string enum: [user, moderator, admin] default: user metadata: type: object additionalProperties: true UpdateUserRequest: type: object minProperties: 1 properties: name: type: string minLength: 1 maxLength: 100 status: $ref: "#/components/schemas/UserStatus" role: type: string enum: [user, moderator, admin] metadata: type: object additionalProperties: true UserListResponse: type: object required: - data - pagination properties: data: type: array items: $ref: "#/components/schemas/User" pagination: $ref: "#/components/schemas/Pagination" Pagination: type: object required: - page - limit - total - totalPages properties: page: type: integer minimum: 1 limit: type: integer minimum: 1 maximum: 100 total: type: integer minimum: 0 totalPages: type: integer minimum: 0 hasNext: type: boolean hasPrev: type: boolean Error: type: object required: - code - message properties: code: type: string description: Error code for programmatic handling message: type: string description: Human-readable error message details: type: array items: type: object properties: field: type: string message: type: string requestId: type: string description: Request ID for support parameters: UserIdParam: name: userId in: path required: true description: User ID schema: type: string format: uuid PageParam: name: page in: query description: Page number (1-based) schema: type: integer minimum: 1 default: 1 LimitParam: name: limit in: query description: Items per page schema: type: integer minimum: 1 maximum: 100 default: 20 responses: BadRequest: description: Invalid request content: application/json: schema: $ref: "#/components/schemas/Error" example: code: VALIDATION_ERROR message: Invalid request parameters details: - field: email message: Must be a valid email address Unauthorized: description: Authentication required content: application/json: schema: $ref: "#/components/schemas/Error" example: code: UNAUTHORIZED message: Authentication required NotFound: description: Resource not found content: application/json: schema: $ref: "#/components/schemas/Error" example: code: NOT_FOUND message: User not found RateLimited: description: Too many requests content: application/json: schema: $ref: "#/components/schemas/Error" headers: Retry-After: description: Seconds until rate limit resets schema: type: integer X-RateLimit-Limit: description: Request limit per window schema: type: integer X-RateLimit-Remaining: description: Remaining requests in window schema: type: integer examples: UserListExample: value: data: - id: "550e8400-e29b-41d4-a716-446655440000" email: "john@example.com" name: "John Doe" status: "active" role: "user" createdAt: "2024-01-15T10:30:00Z" pagination: page: 1 limit: 20 total: 1 totalPages: 1 hasNext: false hasPrev: false securitySchemes: bearerAuth: type: http scheme: bearer bearerFormat: JWT description: JWT token from /auth/login apiKey: type: apiKey in: header name: X-API-Key description: API key for service-to-service calls security: - bearerAuth: [] ``` ### Template 2: Code-First Generation (Python/FastAPI) ```python # FastAPI with automatic OpenAPI generation from fastapi import FastAPI, HTTPException, Query, Path, Depends from pydantic import BaseModel, Field, EmailStr from typing import Optional, List from datetime import datetime from uuid import UUID from enum import Enum app = FastAPI( title="User Management API", description="API for managing users and profiles", version="2.0.0", openapi_tags=[ {"name": "Users", "description": "User operations"}, {"name": "Profiles", "description": "Profile operations"}, ], servers=[ {"url": "https://api.example.com/v2", "description": "Production"}, {"url": "http://localhost:8000", "description": "Development"}, ], ) # Enums class UserStatus(str, Enum): active = "active" inactive = "inactive" suspended = "suspended" pending = "pending" class UserRole(str, Enum): user = "user" moderator = "moderator" admin = "admin" # Models class UserBase(BaseModel): email: EmailStr = Field(..., description="User email address") name: str = Field(..., min_length=1, max_length=100, description="Display name") class UserCreate(UserBase): role: UserRole = Field(default=UserRole.user) metadata: Optional[dict] = Field(default=None, description="Custom metadata") model_config = { "json_schema_extra": { "examples": [ { "email": "user@example.com", "name": "John Doe", "role": "user" } ] } } class UserUpdate(BaseModel): name: Optional[str] = Field(None, min_length=1, max_length=100) status: Optional[UserStatus] = None role: Optional[UserRole] = None metadata: Optional[dict] = None class User(UserBase): id: UUID = Field(..., description="Unique identifier") status: UserStatus role: UserRole avatar: Optional[str] = Field(None, description="Avatar URL") metadata: Optional[dict] = None created_at: datetime = Field(..., alias="createdAt") updated_at: Optional[datetime] = Field(None, alias="updatedAt") model_config = {"populate_by_name": True} class Pagination(BaseModel): page: int = Field(..., ge=1) limit: int = Field(..., ge=1, le=100) total: int = Field(..., ge=0) total_pages: int = Field(..., ge=0, alias="totalPages") has_next: bool = Field(..., alias="hasNext") has_prev: bool = Field(..., alias="hasPrev") class UserListResponse(BaseModel): data: List[User] pagination: Pagination class ErrorDetail(BaseModel): field: str message: str class ErrorResponse(BaseModel): code: str = Field(..., description="Error code") message: str = Field(..., description="Error message") details: Optional[List[ErrorDetail]] = None request_id: Optional[str] = Field(None, alias="requestId") # Endpoints @app.get( "/users", response_model=UserListResponse, tags=["Users"], summary="List all users", description="Returns a paginated list of users with optional filtering.", responses={ 400: {"model": ErrorResponse, "description": "Invalid request"}, 401: {"model": ErrorResponse, "description": "Unauthorized"}, }, ) async def list_users( page: int = Query(1, ge=1, description="Page number"), limit: int = Query(20, ge=1, le=100, description="Items per page"), status: Optional[UserStatus] = Query(None, description="Filter by status"), search: Optional[str] = Query(None, min_length=2, max_length=100), ): """ List users with pagination and filtering. - **page**: Page number (1-based) - **limit**: Number of items per page (max 100) - **status**: Filter by user status - **search**: Search by name or email """ # Implementation pass @app.post( "/users", response_model=User, status_code=201, tags=["Users"], summary="Create a new user", responses={ 400: {"model": ErrorResponse}, 409: {"model": ErrorResponse, "description": "Email already exists"}, }, ) async def create_user(user: UserCreate): """Create a new user and send welcome email.""" pass @app.get( "/users/{user_id}", response_model=User, tags=["Users"], summary="Get user by ID", responses={404: {"model": ErrorResponse}}, ) async def get_user( user_id: UUID = Path(..., description="User ID"), ): """Retrieve a specific user by their ID.""" pass @app.patch( "/users/{user_id}", response_model=User, tags=["Users"], summary="Update user", responses={ 400: {"model": ErrorResponse}, 404: {"model": ErrorResponse}, }, ) async def update_user( user_id: UUID = Path(..., description="User ID"), user: UserUpdate = ..., ): """Update user attributes.""" pass @app.delete( "/users/{user_id}", status_code=204, tags=["Users", "Admin"], summary="Delete user", responses={404: {"model": ErrorResponse}}, ) async def delete_user( user_id: UUID = Path(..., description="User ID"), ): """Permanently delete a user.""" pass # Export OpenAPI spec if __name__ == "__main__": import json print(json.dumps(app.openapi(), indent=2)) ``` ### Template 3: Code-First (TypeScript/Express with tsoa) ```typescript // tsoa generates OpenAPI from TypeScript decorators import { Controller, Get, Post, Patch, Delete, Route, Path, Query, Body, Response, SuccessResponse, Tags, Security, Example, } from "tsoa"; // Models interface User { /** Unique identifier */ id: string; /** User email address */ email: string; /** Display name */ name: string; status: UserStatus; role: UserRole; /** Avatar URL */ avatar?: string; /** Custom metadata */ metadata?: Record<string, unknown>; createdAt: Date; updatedAt?: Date; } enum UserStatus { Active = "active", Inactive = "inactive", Suspended = "suspended", Pending = "pending", } enum UserRole { User = "user", Moderator = "moderator", Admin = "admin", } interface CreateUserRequest { email: string; name: string; role?: UserRole; metadata?: Record<string, unknown>; } interface UpdateUserRequest { name?: string; status?: UserStatus; role?: UserRole; metadata?: Record<string, unknown>; } interface Pagination { page: number; limit: number; total: number; totalPages: number; hasNext: boolean; hasPrev: boolean; } interface UserListResponse { data: User[]; pagination: Pagination; } interface ErrorResponse { code: string; message: string; details?: { field: string; message: string }[]; requestId?: string; } @Route("users") @Tags("Users") export class UsersController extends Controller { /** * List all users with pagination and filtering * @param page Page number (1-based) * @param limit Items per page (max 100) * @param status Filter by user status * @param search Search by name or email */
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

dotnet-backend-patterns

Master C#/.NET backend development patterns for building robust

coding
⭐1
# .NET Backend Development Patterns Master C#/.NET patterns for building production-grade APIs, MCP servers, and enterprise backends with modern best practices (2024/2025). ## When to Use This Skill - Developing new .NET Web APIs or MCP servers - Reviewing C# code for quality and performance - Designing service architectures with dependency injection - Implementing caching strategies with Redis - Writing unit and integration tests - Optimizing database access with EF Core or Dapper - Configuring applications with IOptions pattern - Handling errors and implementing resilience patterns ## Core Concepts ### 1. Project Structure (Clean Architecture) ``` src/ β”œβ”€β”€ Domain/ # Core business logic (no dependencies) β”‚ β”œβ”€β”€ Entities/ β”‚ β”œβ”€β”€ Interfaces/ β”‚ β”œβ”€β”€ Exceptions/ β”‚ └── ValueObjects/ β”œβ”€β”€ Application/ # Use cases, DTOs, validation β”‚ β”œβ”€β”€ Services/ β”‚ β”œβ”€β”€ DTOs/ β”‚ β”œβ”€β”€ Validators/ β”‚ └── Interfaces/ β”œβ”€β”€ Infrastructure/ # External implementations β”‚ β”œβ”€β”€ Data/ # EF Core, Dapper repositories β”‚ β”œβ”€β”€ Caching/ # Redis, Memory cache β”‚ β”œβ”€β”€ External/ # HTTP clients, third-party APIs β”‚ └── DependencyInjection/ # Service registration └── Api/ # Entry point β”œβ”€β”€ Controllers/ # Or MinimalAPI endpoints β”œβ”€β”€ Middleware/ β”œβ”€β”€ Filters/ └── Program.cs ``` ### 2. Dependency Injection Patterns ```csharp // Service registration by lifetime public static class ServiceCollectionExtensions { public static IServiceCollection AddApplicationServices( this IServiceCollection services, IConfiguration configuration) { // Scoped: One instance per HTTP request services.AddScoped<IProductService, ProductService>(); services.AddScoped<IOrderService, OrderService>(); // Singleton: One instance for app lifetime services.AddSingleton<ICacheService, RedisCacheService>(); services.AddSingleton<IConnectionMultiplexer>(_ => ConnectionMultiplexer.Connect(configuration["Redis:Connection"]!)); // Transient: New instance every time services.AddTransient<IValidator<CreateOrderRequest>, CreateOrderValidator>(); // Options pattern for configuration services.Configure<CatalogOptions>(configuration.GetSection("Catalog")); services.Configure<RedisOptions>(configuration.GetSection("Redis")); // Factory pattern for conditional creation services.AddScoped<IPriceCalculator>(sp => { var options = sp.GetRequiredService<IOptions<PricingOptions>>().Value; return options.UseNewEngine ? sp.GetRequiredService<NewPriceCalculator>() : sp.GetRequiredService<LegacyPriceCalculator>(); }); // Keyed services (.NET 8+) services.AddKeyedScoped<IPaymentProcessor, StripeProcessor>("stripe"); services.AddKeyedScoped<IPaymentProcessor, PayPalProcessor>("paypal"); return services; } } // Usage with keyed services public class CheckoutService { public CheckoutService( [FromKeyedServices("stripe")] IPaymentProcessor stripeProcessor) { _processor = stripeProcessor; } } ``` ### 3. Async/Await Patterns ```csharp // βœ… CORRECT: Async all the way down public async Task<Product> GetProductAsync(string id, CancellationToken ct = default) { return await _repository.GetByIdAsync(id, ct); } // βœ… CORRECT: Parallel execution with WhenAll public async Task<(Stock, Price)> GetStockAndPriceAsync( string productId, CancellationToken ct = default) { var stockTask = _stockService.GetAsync(productId, ct); var priceTask = _priceService.GetAsync(productId, ct); await Task.WhenAll(stockTask, priceTask); return (await stockTask, await priceTask); } // βœ… CORRECT: ConfigureAwait in libraries public async Task<T> LibraryMethodAsync<T>(CancellationToken ct = default) { var result = await _httpClient.GetAsync(url, ct).ConfigureAwait(false); return await result.Content.ReadFromJsonAsync<T>(ct).ConfigureAwait(false); } // βœ… CORRECT: ValueTask for hot paths with caching public ValueTask<Product?> GetCachedProductAsync(string id) { if (_cache.TryGetValue(id, out Product? product)) return ValueTask.FromResult(product); return new ValueTask<Product?>(GetFromDatabaseAsync(id)); } // ❌ WRONG: Blocking on async (deadlock risk) var result = GetProductAsync(id).Result; // NEVER do this var result2 = GetProductAsync(id).GetAwaiter().GetResult(); // Also bad // ❌ WRONG: async void (except event handlers) public async void ProcessOrder() { } // Exceptions are lost // ❌ WRONG: Unnecessary Task.Run for already async code await Task.Run(async () => await GetDataAsync()); // Wastes thread ``` ### 4. Configuration with IOptions ```csharp // Configuration classes public class CatalogOptions { public const string SectionName = "Catalog"; public int DefaultPageSize { get; set; } = 50; public int MaxPageSize { get; set; } = 200; public TimeSpan CacheDuration { get; set; } = TimeSpan.FromMinutes(15); public bool EnableEnrichment { get; set; } = true; } public class RedisOptions { public const string SectionName = "Redis"; public string Connection { get; set; } = "localhost:6379"; public string KeyPrefix { get; set; } = "mcp:"; public int Database { get; set; } = 0; } // appsettings.json { "Catalog": { "DefaultPageSize": 50, "MaxPageSize": 200, "CacheDuration": "00:15:00", "EnableEnrichment": true }, "Redis": { "Connection": "localhost:6379", "KeyPrefix": "mcp:", "Database": 0 } } // Registration services.Configure<CatalogOptions>(configuration.GetSection(CatalogOptions.SectionName)); services.Configure<RedisOptions>(configuration.GetSection(RedisOptions.SectionName)); // Usage with IOptions (singleton, read once at startup) public class CatalogService { private readonly CatalogOptions _options; public CatalogService(IOptions<CatalogOptions> options) { _options = options.Value; } } // Usage with IOptionsSnapshot (scoped, re-reads on each request) public class DynamicService { private readonly CatalogOptions _options; public DynamicService(IOptionsSnapshot<CatalogOptions> options) { _options = options.Value; // Fresh value per request } } // Usage with IOptionsMonitor (singleton, notified on changes) public class MonitoredService { private CatalogOptions _options; public MonitoredService(IOptionsMonitor<CatalogOptions> monitor) { _options = monitor.CurrentValue; monitor.OnChange(newOptions => _options = newOptions); } } ``` ### 5. Result Pattern (Avoiding Exceptions for Flow Control) ```csharp // Generic Result type public class Result<T> { public bool IsSuccess { get; } public T? Value { get; } public string? Error { get; } public string? ErrorCode { get; } private Result(bool isSuccess, T? value, string? error, string? errorCode) { IsSuccess = isSuccess; Value = value; Error = error; ErrorCode = errorCode; } public static Result<T> Success(T value) => new(true, value, null, null); public static Result<T> Failure(string error, string? code = null) => new(false, default, error, code); public Result<TNew> Map<TNew>(Func<T, TNew> mapper) => IsSuccess ? Result<TNew>.Success(mapper(Value!)) : Result<TNew>.Failure(Error!, ErrorCode); public async Task<Result<TNew>> MapAsync<TNew>(Func<T, Task<TNew>> mapper) => IsSuccess ? Result<TNew>.Success(await mapper(Value!)) : Result<TNew>.Failure(Error!, ErrorCode); } // Usage in service public async Task<Result<Order>> CreateOrderAsync(CreateOrderRequest request, CancellationToken ct) { // Validation var validation = await _validator.ValidateAsync(request, ct); if (!validation.IsValid) return Result<Order>.Failure( validation.Errors.First().ErrorMessage, "VALIDATION_ERROR"); // Business rule check var stock = await _stockService.CheckAsync(request.ProductId, request.Quantity, ct); if (!stock.IsAvailable) return Result<Order>.Failure( $"Insufficient stock: {stock.Available} available, {request.Quantity} requested", "INSUFFICIENT_STOCK"); // Create order var order = await _repository.CreateAsync(request.ToEntity(), ct); return Result<Order>.Success(order); } // Usage in controller/endpoint app.MapPost("/orders", async ( CreateOrderRequest request, IOrderService orderService, CancellationToken ct) => { var result = await orderService.CreateOrderAsync(request, ct); return result.IsSuccess ? Results.Created($"/orders/{result.Value!.Id}", result.Value) : Results.BadRequest(new { error = result.Error, code = result.ErrorCode }); }); ``` ## Data Access Patterns ### Entity Framework Core ```csharp // DbContext configuration public class AppDbContext : DbContext { public DbSet<Product> Products => Set<Product>(); public DbSet<Order> Orders => Set<Order>(); protected override void OnModelCreating(ModelBuilder modelBuilder) { // Apply all configurations from assembly modelBuilder.ApplyConfigurationsFromAssembly(typeof(AppDbContext).Assembly); // Global query filters modelBuilder.Entity<Product>().HasQueryFilter(p => !p.IsDeleted); } } // Entity configuration public class ProductConfiguration : IEntityTypeConfiguration<Product> { public void Configure(EntityTypeBuilder<Product> builder) { builder.ToTable("Products"); builder.HasKey(p => p.Id); builder.Property(p => p.Id).HasMaxLength(40); builder.Property(p => p.Name).HasMaxLength(200).IsRequired(); builder.Property(p => p.Price).HasPrecision(18, 2); builder.HasIndex(p => p.Sku).IsUnique(); builder.HasIndex(p => new { p.CategoryId, p.Name }); builder.HasMany(p => p.OrderItems) .WithOne(oi => oi.Product) .HasForeignKey(oi => oi.ProductId); } } // Repository with EF Core public class ProductRepository : IProductRepository { private readonly AppDbContext _context; public async Task<Product?> GetByIdAsync(string id, CancellationToken ct = default) { return await _context.Products .AsNoTracking() .FirstOrDefaultAsync(p => p.Id == id, ct); } public async Task<IReadOnlyList<Product>> SearchAsync( ProductSearchCriteria criteria, CancellationToken ct = default) { var query = _context.Products.AsNoTracking(); if (!string.IsNullOrWhiteSpace(criteria.SearchTerm)) query = query.Where(p => EF.Functions.Like(p.Name, $"%{criteria.SearchTerm}%")); if (criteria.CategoryId.HasValue) query = query.Where(p => p.CategoryId == criteria.CategoryId); if (criteria.MinPrice.HasValue) query = query.Where(p => p.Price >= criteria.MinPrice); if (criteria.MaxPrice.HasValue) query = query.Where(p => p.Price <= criteria.MaxPrice); return await query .OrderBy(p => p.Name) .Skip((criteria.Page - 1) * criteria.PageSize) .Take(criteria.PageSize) .ToListAsync(ct); } } ``` ### Dapper for Performance ```csharp public class DapperProductRepository : IProductRepository { private readonly IDbConnection _connection; public async Task<Product?> GetByIdAsync(string id, CancellationToken ct = default) { const string sql = """ SELECT Id, Name, Sku, Price, CategoryId, Stock, CreatedAt FROM Products WHERE Id = @Id AND IsDeleted = 0 """; return await _connection.QueryFirstOrDefaultAsync<Product>( new CommandDefinition(sql, new { Id = id }, cancellationToken: ct)); } public async Task<IReadOnlyList<Product>> SearchAsync( ProductSearchCriteria criteria, CancellationToken ct = default) { var sql = new StringBuilder(""" SELECT Id, Name, Sku, Price, CategoryId, Stock, CreatedAt FROM Products WHERE IsDeleted = 0 """); var parameters = new DynamicParameters(); if (!string.IsNullOrWhiteSpace(criteria.SearchTerm)) { sql.Append(" AND Name LIKE @SearchTerm"); parameters.Add("SearchTerm", $"%{criteria.SearchTerm}%"); } if (criteria.CategoryId.HasValue) { sql.Append(" AND CategoryId = @CategoryId"); parameters.Add("CategoryId", criteria.CategoryId); } if (criteria.MinPrice.HasValue) { sql.Append(" AND Price >= @MinPrice"); parameters.Add("MinPrice", criteria.MinPrice); } if (criteria.MaxPrice.HasValue) { sql.Append(" AND Price <= @MaxPrice"); parameters.Add("MaxPrice", criteria.MaxPrice); } sql.Append(" ORDER BY Name OFFSET @Offset ROWS FETCH NEXT @PageSize ROWS ONLY"); parameters.Add("Offset", (criteria.Page - 1) * criteria.PageSize); parameters.Add("PageSize", criteria.PageSize); var results = await _connection.QueryAsync<Product>( new CommandDefinition(sql.ToString(), parameters, cancellationToken: ct)); return results.ToList(); } // Multi-mapping for related data public async Task<Order?> GetOrderWithItemsAsync(int orderId, CancellationToken ct = default) { const string sql = """ SELECT o.*, oi.*, p.* FROM Orders o LEFT JOIN OrderItems oi ON o.Id = oi.OrderId LEFT JOIN Products p ON oi.ProductId = p.Id WHERE o.Id = @OrderId """; var orderDictionary = new Dictionary<int, Order>(); await _connection.QueryAsync<Order, OrderItem, Product, Order>( new CommandDefinition(sql, new { OrderId = orderId }, cancellationToken: ct), (order, item, product) => { if (!orderDictionary.TryGetValue(order.Id, out var existingOrder)) { existingOrder = order; existingOrder.Items = new List<OrderItem>(); orderDictionary.Add(order.Id, existingOrder); } if (item != null) { item.Product = product; existingOrder.Items.Add(item); } return existingOrder; }, splitOn: "Id,Id"); return orderDictionary.Values.FirstOrDefault(); } } ``` ## Caching Patterns ### Multi-Level Cache with Redis ```csharp public class CachedProductService : IProductService { private readonly IProductRepository _repository; private readonly IMemoryCache _memoryCache; private readonly IDistributedCache _distributedCache; private readonly ILogger<CachedProductService> _logger; private static readonly TimeSpan MemoryCacheDuration = TimeSpan.FromMinutes(1); private static readonly TimeSpan DistributedCacheDuration = TimeSpan.FromMinutes(15); public async Task<Product?> GetByIdAsync(string id, CancellationToken ct = default) { var cacheKey = $"product:{id}"; // L1: Memory cache (in-process, fastest) if (_memoryCache.TryGetValue(cacheKey, out Product? cached)) { _logger.LogDebug("L1 cache hit for {CacheKey}", cacheKey); return cached; } // L2: Distributed cache (Redis) var distributed = await _distributedCache.GetStringAsync(cacheKey, ct); if (distributed != null) { _logger.LogDebug("L2 cache hit for {CacheKey}", cacheKey); var product = JsonSerializer.Deserialize<Product>(distributed); // Populate L1 _memoryCache.Set(cacheKey, product, MemoryCacheDuration); return product; } // L3: Database _logger.LogDebug("Cache miss for {CacheKey}, fetching from database", cacheKey); var fromDb = await _repository.GetByIdAsync(id, ct); if (fromDb != null) { var serialized = JsonSerializer.Serialize(fromDb); // Populate both caches await _distributedCache.SetStringAsync( cacheKey, serialized, new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = DistributedCacheDuration }, ct); _memoryCache.Set(cacheKey, fromDb, MemoryCacheDuration); } return fromDb; } public async Task InvalidateAsync(string id, CancellationToken ct = default) { var cacheKey = $"product:{id}"; _memoryCache.Remove(cacheKey); await _distributedCache.RemoveAsync(cacheKey, ct); _logger.LogInformation("Invalidated cache for {CacheKey}", cacheKey); } } // Stale-while-revalidate pattern public class StaleWhileRevalidateCache<T> { private readonly IDistributedCache _cache; private readonly TimeSpan _freshDuration; private readonly TimeSpan _staleDuration; public async Task<T?> GetOrCreateAsync( string key, Func<CancellationToken, Task<T>> factory, CancellationToken ct = default) { var cached = await _cache.GetStringAsync(key, ct); if (cached != null) { var entry = JsonSerializer.Deserialize<CacheEntry<T>>(cached)!; if (entry.IsStale && !entry.IsExpired) { // Return stale data immediately, refresh in background _ = Task.Run(async () => { var fresh = await factory(CancellationToken.None); await SetAsync(key, fresh, CancellationToken.None); }); } if (!entry.IsExpired) return entry.Value; } // Cache miss or expired var value = await factory(ct); await SetAsync(key, value, ct); return value; } private record CacheEntry<TValue>(TValue Value, DateTime CreatedAt) { public bool IsStale => DateTime.UtcNow - CreatedAt > _freshDuration; public bool IsExpired => DateTime.UtcNow - CreatedAt > _staleDuration; } } ``` ## Testing Patterns ### Unit Tests with xUnit and Moq ```csharp public class OrderServiceTests { private readonly Mock<IOrderRepository> _mockRepository; private readonly Mock<IStockService> _mockStockService; private readonly Mock<IValidator<CreateOrderRequest>> _mockValidator; private readonly OrderService _sut; // System Under Test public OrderServiceTests() { _mockRepository = new Mock<IOrderRepository>(); _mockStockService = new Mock<IStockService>(); _mockValidator = new Mock<IValidator<CreateOrderRequest>>(); // Default: validation passes _mockValidator .Setup(v => v.ValidateAsync(It.IsAny<CreateOrderRequest>(), It.IsAny<
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

react-modernization

Upgrade React applications to latest versions, migrate from class

coding
⭐1
# React Modernization Master React version upgrades, class to hooks migration, concurrent features adoption, and codemods for automated transformation. ## When to Use This Skill - Upgrading React applications to latest versions - Migrating class components to functional components with hooks - Adopting concurrent React features (Suspense, transitions) - Applying codemods for automated refactoring - Modernizing state management patterns - Updating to TypeScript - Improving performance with React 18+ features ## Version Upgrade Path ### React 16 β†’ 17 β†’ 18 **Breaking Changes by Version:** **React 17:** - Event delegation changes - No event pooling - Effect cleanup timing - JSX transform (no React import needed) **React 18:** - Automatic batching - Concurrent rendering - Strict Mode changes (double invocation) - New root API - Suspense on server ## Class to Hooks Migration ### State Management ```javascript // Before: Class component class Counter extends React.Component { constructor(props) { super(props); this.state = { count: 0, name: "", }; } increment = () => { this.setState({ count: this.state.count + 1 }); }; render() { return ( <div> <p>Count: {this.state.count}</p> <button onClick={this.increment}>Increment</button> </div> ); } } // After: Functional component with hooks function Counter() { const [count, setCount] = useState(0); const [name, setName] = useState(""); const increment = () => { setCount(count + 1); }; return ( <div> <p>Count: {count}</p> <button onClick={increment}>Increment</button> </div> ); } ``` ### Lifecycle Methods to Hooks ```javascript // Before: Lifecycle methods class DataFetcher extends React.Component { state = { data: null, loading: true }; componentDidMount() { this.fetchData(); } componentDidUpdate(prevProps) { if (prevProps.id !== this.props.id) { this.fetchData(); } } componentWillUnmount() { this.cancelRequest(); } fetchData = async () => { const data = await fetch(`/api/${this.props.id}`); this.setState({ data, loading: false }); }; cancelRequest = () => { // Cleanup }; render() { if (this.state.loading) return <div>Loading...</div>; return <div>{this.state.data}</div>; } } // After: useEffect hook function DataFetcher({ id }) { const [data, setData] = useState(null); const [loading, setLoading] = useState(true); useEffect(() => { let cancelled = false; const fetchData = async () => { try { const response = await fetch(`/api/${id}`); const result = await response.json(); if (!cancelled) { setData(result); setLoading(false); } } catch (error) { if (!cancelled) { console.error(error); } } }; fetchData(); // Cleanup function return () => { cancelled = true; }; }, [id]); // Re-run when id changes if (loading) return <div>Loading...</div>; return <div>{data}</div>; } ``` ### Context and HOCs to Hooks ```javascript // Before: Context consumer and HOC const ThemeContext = React.createContext(); class ThemedButton extends React.Component { static contextType = ThemeContext; render() { return ( <button style={{ background: this.context.theme }}> {this.props.children} </button> ); } } // After: useContext hook function ThemedButton({ children }) { const { theme } = useContext(ThemeContext); return <button style={{ background: theme }}>{children}</button>; } // Before: HOC for data fetching function withUser(Component) { return class extends React.Component { state = { user: null }; componentDidMount() { fetchUser().then((user) => this.setState({ user })); } render() { return <Component {...this.props} user={this.state.user} />; } }; } // After: Custom hook function useUser() { const [user, setUser] = useState(null); useEffect(() => { fetchUser().then(setUser); }, []); return user; } function UserProfile() { const user = useUser(); if (!user) return <div>Loading...</div>; return <div>{user.name}</div>; } ``` ## React 18 Concurrent Features ### New Root API ```javascript // Before: React 17 import ReactDOM from "react-dom"; ReactDOM.render(<App />, document.getElementById("root")); // After: React 18 import { createRoot } from "react-dom/client"; const root = createRoot(document.getElementById("root")); root.render(<App />); ``` ### Automatic Batching ```javascript // React 18: All updates are batched function handleClick() { setCount((c) => c + 1); setFlag((f) => !f); // Only one re-render (batched) } // Even in async: setTimeout(() => { setCount((c) => c + 1); setFlag((f) => !f); // Still batched in React 18! }, 1000); // Opt out if needed import { flushSync } from "react-dom"; flushSync(() => { setCount((c) => c + 1); }); // Re-render happens here setFlag((f) => !f); // Another re-render ``` ### Transitions ```javascript import { useState, useTransition } from "react"; function SearchResults() { const [query, setQuery] = useState(""); const [results, setResults] = useState([]); const [isPending, startTransition] = useTransition(); const handleChange = (e) => { // Urgent: Update input immediately setQuery(e.target.value); // Non-urgent: Update results (can be interrupted) startTransition(() => { setResults(searchResults(e.target.value)); }); }; return ( <> <input value={query} onChange={handleChange} /> {isPending && <Spinner />} <Results data={results} /> </> ); } ``` ### Suspense for Data Fetching ```javascript import { Suspense } from "react"; // Resource-based data fetching (with React 18) const resource = fetchProfileData(); function ProfilePage() { return ( <Suspense fallback={<Loading />}> <ProfileDetails /> <Suspense fallback={<Loading />}> <ProfileTimeline /> </Suspense> </Suspense> ); } function ProfileDetails() { // This will suspend if data not ready const user = resource.user.read(); return <h1>{user.name}</h1>; } function ProfileTimeline() { const posts = resource.posts.read(); return <Timeline posts={posts} />; } ``` ## Codemods for Automation ### Run React Codemods ```bash # Rename unsafe lifecycle methods npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js src/ # Update React imports (React 17+) npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/update-react-imports.js src/ # Add error boundaries npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/error-boundaries.js src/ # For TypeScript files npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js --parser=tsx src/ # Dry run to preview changes npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js --dry --print src/ # Class to Hooks (third-party) npx codemod react/hooks/convert-class-to-function src/ ``` ### Custom Codemod Example ```javascript // custom-codemod.js module.exports = function (file, api) { const j = api.jscodeshift; const root = j(file.source); // Find setState calls root .find(j.CallExpression, { callee: { type: "MemberExpression", property: { name: "setState" }, }, }) .forEach((path) => { // Transform to useState // ... transformation logic }); return root.toSource(); }; // Run: jscodeshift -t custom-codemod.js src/ ``` ## Performance Optimization ### useMemo and useCallback ```javascript function ExpensiveComponent({ items, filter }) { // Memoize expensive calculation const filteredItems = useMemo(() => { return items.filter((item) => item.category === filter); }, [items, filter]); // Memoize callback to prevent child re-renders const handleClick = useCallback((id) => { console.log("Clicked:", id); }, []); // No dependencies, never changes return <List items={filteredItems} onClick={handleClick} />; } // Child component with memo const List = React.memo(({ items, onClick }) => { return items.map((item) => ( <Item key={item.id} item={item} onClick={onClick} /> )); }); ``` ### Code Splitting ```javascript import { lazy, Suspense } from "react"; // Lazy load components const Dashboard = lazy(() => import("./Dashboard")); const Settings = lazy(() => import("./Settings")); function App() { return ( <Suspense fallback={<Loading />}> <Routes> <Route path="/dashboard" element={<Dashboard />} /> <Route path="/settings" element={<Settings />} /> </Routes> </Suspense> ); } ``` ## TypeScript Migration ```typescript // Before: JavaScript function Button({ onClick, children }) { return <button onClick={onClick}>{children}</button>; } // After: TypeScript interface ButtonProps { onClick: () => void; children: React.ReactNode; } function Button({ onClick, children }: ButtonProps) { return <button onClick={onClick}>{children}</button>; } // Generic components interface ListProps<T> { items: T[]; renderItem: (item: T) => React.ReactNode; } function List<T>({ items, renderItem }: ListProps<T>) { return <>{items.map(renderItem)}</>; } ``` ## Migration Checklist ```markdown ### Pre-Migration - [ ] Update dependencies incrementally (not all at once) - [ ] Review breaking changes in release notes - [ ] Set up testing suite - [ ] Create feature branch ### Class β†’ Hooks Migration - [ ] Identify class components to migrate - [ ] Start with leaf components (no children) - [ ] Convert state to useState - [ ] Convert lifecycle to useEffect - [ ] Convert context to useContext - [ ] Extract custom hooks - [ ] Test thoroughly ### React 18 Upgrade - [ ] Update to React 17 first (if needed) - [ ] Update react and react-dom to 18 - [ ] Update @types/react if using TypeScript - [ ] Change to createRoot API - [ ] Test with StrictMode (double invocation) - [ ] Address concurrent rendering issues - [ ] Adopt Suspense/Transitions where beneficial ### Performance - [ ] Identify performance bottlenecks - [ ] Add React.memo where appropriate - [ ] Use useMemo/useCallback for expensive operations - [ ] Implement code splitting - [ ] Optimize re-renders ### Testing - [ ] Update test utilities (React Testing Library) - [ ] Test with React 18 features - [ ] Check for warnings in console - [ ] Performance testing ``` ## Resources - **references/breaking-changes.md**: Version-specific breaking changes - **references/codemods.md**: Codemod usage guide - **references/hooks-migration.md**: Comprehensive hooks patterns - **references/concurrent-features.md**: React 18 concurrent features - **assets/codemod-config.json**: Codemod configurations - **assets/migration-checklist.md**: Step-by-step checklist - **scripts/apply-codemods.sh**: Automated codemod script ## Best Practices 1. **Incremental Migration**: Don't migrate everything at once 2. **Test Thoroughly**: Comprehensive testing at each step 3. **Use Codemods**: Automate repetitive transformations 4. **Start Simple**: Begin with leaf components 5. **Leverage StrictMode**: Catch issues early 6. **Monitor Performance**: Measure before and after 7. **Document Changes**: Keep migration log ## Common Pitfalls - Forgetting useEffect dependencies - Over-using useMemo/useCallback - Not handling cleanup in useEffect - Mixing class and functional patterns - Ignoring StrictMode warnings - Breaking change assumptions
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

nextjs-app-router-patterns

Master Next.js 14+ App Router with Server Components, streaming,

coding
⭐1
# Next.js App Router Patterns Comprehensive patterns for Next.js 14+ App Router architecture, Server Components, and modern full-stack React development. ## When to Use This Skill - Building new Next.js applications with App Router - Migrating from Pages Router to App Router - Implementing Server Components and streaming - Setting up parallel and intercepting routes - Optimizing data fetching and caching - Building full-stack features with Server Actions ## Core Concepts ### 1. Rendering Modes | Mode | Where | When to Use | | --------------------- | ------------ | ----------------------------------------- | | **Server Components** | Server only | Data fetching, heavy computation, secrets | | **Client Components** | Browser | Interactivity, hooks, browser APIs | | **Static** | Build time | Content that rarely changes | | **Dynamic** | Request time | Personalized or real-time data | | **Streaming** | Progressive | Large pages, slow data sources | ### 2. File Conventions ``` app/ β”œβ”€β”€ layout.tsx # Shared UI wrapper β”œβ”€β”€ page.tsx # Route UI β”œβ”€β”€ loading.tsx # Loading UI (Suspense) β”œβ”€β”€ error.tsx # Error boundary β”œβ”€β”€ not-found.tsx # 404 UI β”œβ”€β”€ route.ts # API endpoint β”œβ”€β”€ template.tsx # Re-mounted layout β”œβ”€β”€ default.tsx # Parallel route fallback └── opengraph-image.tsx # OG image generation ``` ## Quick Start ```typescript // app/layout.tsx import { Inter } from 'next/font/google' import { Providers } from './providers' const inter = Inter({ subsets: ['latin'] }) export const metadata = { title: { default: 'My App', template: '%s | My App' }, description: 'Built with Next.js App Router', } export default function RootLayout({ children, }: { children: React.ReactNode }) { return ( <html lang="en" suppressHydrationWarning> <body className={inter.className}> <Providers>{children}</Providers> </body> </html> ) } // app/page.tsx - Server Component by default async function getProducts() { const res = await fetch('https://api.example.com/products', { next: { revalidate: 3600 }, // ISR: revalidate every hour }) return res.json() } export default async function HomePage() { const products = await getProducts() return ( <main> <h1>Products</h1> <ProductGrid products={products} /> </main> ) } ``` ## Patterns ### Pattern 1: Server Components with Data Fetching ```typescript // app/products/page.tsx import { Suspense } from 'react' import { ProductList, ProductListSkeleton } from '@/components/products' import { FilterSidebar } from '@/components/filters' interface SearchParams { category?: string sort?: 'price' | 'name' | 'date' page?: string } export default async function ProductsPage({ searchParams, }: { searchParams: Promise<SearchParams> }) { const params = await searchParams return ( <div className="flex gap-8"> <FilterSidebar /> <Suspense key={JSON.stringify(params)} fallback={<ProductListSkeleton />} > <ProductList category={params.category} sort={params.sort} page={Number(params.page) || 1} /> </Suspense> </div> ) } // components/products/ProductList.tsx - Server Component async function getProducts(filters: ProductFilters) { const res = await fetch( `${process.env.API_URL}/products?${new URLSearchParams(filters)}`, { next: { tags: ['products'] } } ) if (!res.ok) throw new Error('Failed to fetch products') return res.json() } export async function ProductList({ category, sort, page }: ProductFilters) { const { products, totalPages } = await getProducts({ category, sort, page }) return ( <div> <div className="grid grid-cols-3 gap-4"> {products.map((product) => ( <ProductCard key={product.id} product={product} /> ))} </div> <Pagination currentPage={page} totalPages={totalPages} /> </div> ) } ``` ### Pattern 2: Client Components with 'use client' ```typescript // components/products/AddToCartButton.tsx 'use client' import { useState, useTransition } from 'react' import { addToCart } from '@/app/actions/cart' export function AddToCartButton({ productId }: { productId: string }) { const [isPending, startTransition] = useTransition() const [error, setError] = useState<string | null>(null) const handleClick = () => { setError(null) startTransition(async () => { const result = await addToCart(productId) if (result.error) { setError(result.error) } }) } return ( <div> <button onClick={handleClick} disabled={isPending} className="btn-primary" > {isPending ? 'Adding...' : 'Add to Cart'} </button> {error && <p className="text-red-500 text-sm">{error}</p>} </div> ) } ``` ### Pattern 3: Server Actions ```typescript // app/actions/cart.ts "use server"; import { revalidateTag } from "next/cache"; import { cookies } from "next/headers"; import { redirect } from "next/navigation"; export async function addToCart(productId: string) { const cookieStore = await cookies(); const sessionId = cookieStore.get("session")?.value; if (!sessionId) { redirect("/login"); } try { await db.cart.upsert({ where: { sessionId_productId: { sessionId, productId } }, update: { quantity: { increment: 1 } }, create: { sessionId, productId, quantity: 1 }, }); revalidateTag("cart"); return { success: true }; } catch (error) { return { error: "Failed to add item to cart" }; } } export async function checkout(formData: FormData) { const address = formData.get("address") as string; const payment = formData.get("payment") as string; // Validate if (!address || !payment) { return { error: "Missing required fields" }; } // Process order const order = await processOrder({ address, payment }); // Redirect to confirmation redirect(`/orders/${order.id}/confirmation`); } ``` ### Pattern 4: Parallel Routes ```typescript // app/dashboard/layout.tsx export default function DashboardLayout({ children, analytics, team, }: { children: React.ReactNode analytics: React.ReactNode team: React.ReactNode }) { return ( <div className="dashboard-grid"> <main>{children}</main> <aside className="analytics-panel">{analytics}</aside> <aside className="team-panel">{team}</aside> </div> ) } // app/dashboard/@analytics/page.tsx export default async function AnalyticsSlot() { const stats = await getAnalytics() return <AnalyticsChart data={stats} /> } // app/dashboard/@analytics/loading.tsx export default function AnalyticsLoading() { return <ChartSkeleton /> } // app/dashboard/@team/page.tsx export default async function TeamSlot() { const members = await getTeamMembers() return <TeamList members={members} /> } ``` ### Pattern 5: Intercepting Routes (Modal Pattern) ```typescript // File structure for photo modal // app/ // β”œβ”€β”€ @modal/ // β”‚ β”œβ”€β”€ (.)photos/[id]/page.tsx # Intercept // β”‚ └── default.tsx // β”œβ”€β”€ photos/ // β”‚ └── [id]/page.tsx # Full page // └── layout.tsx // app/@modal/(.)photos/[id]/page.tsx import { Modal } from '@/components/Modal' import { PhotoDetail } from '@/components/PhotoDetail' export default async function PhotoModal({ params, }: { params: Promise<{ id: string }> }) { const { id } = await params const photo = await getPhoto(id) return ( <Modal> <PhotoDetail photo={photo} /> </Modal> ) } // app/photos/[id]/page.tsx - Full page version export default async function PhotoPage({ params, }: { params: Promise<{ id: string }> }) { const { id } = await params const photo = await getPhoto(id) return ( <div className="photo-page"> <PhotoDetail photo={photo} /> <RelatedPhotos photoId={id} /> </div> ) } // app/layout.tsx export default function RootLayout({ children, modal, }: { children: React.ReactNode modal: React.ReactNode }) { return ( <html> <body> {children} {modal} </body> </html> ) } ``` ### Pattern 6: Streaming with Suspense ```typescript // app/product/[id]/page.tsx import { Suspense } from 'react' export default async function ProductPage({ params, }: { params: Promise<{ id: string }> }) { const { id } = await params // This data loads first (blocking) const product = await getProduct(id) return ( <div> {/* Immediate render */} <ProductHeader product={product} /> {/* Stream in reviews */} <Suspense fallback={<ReviewsSkeleton />}> <Reviews productId={id} /> </Suspense> {/* Stream in recommendations */} <Suspense fallback={<RecommendationsSkeleton />}> <Recommendations productId={id} /> </Suspense> </div> ) } // These components fetch their own data async function Reviews({ productId }: { productId: string }) { const reviews = await getReviews(productId) // Slow API return <ReviewList reviews={reviews} /> } async function Recommendations({ productId }: { productId: string }) { const products = await getRecommendations(productId) // ML-based, slow return <ProductCarousel products={products} /> } ``` ### Pattern 7: Route Handlers (API Routes) ```typescript // app/api/products/route.ts import { NextRequest, NextResponse } from "next/server"; export async function GET(request: NextRequest) { const searchParams = request.nextUrl.searchParams; const category = searchParams.get("category"); const products = await db.product.findMany({ where: category ? { category } : undefined, take: 20, }); return NextResponse.json(products); } export async function POST(request: NextRequest) { const body = await request.json(); const product = await db.product.create({ data: body, }); return NextResponse.json(product, { status: 201 }); } // app/api/products/[id]/route.ts export async function GET( request: NextRequest, { params }: { params: Promise<{ id: string }> }, ) { const { id } = await params; const product = await db.product.findUnique({ where: { id } }); if (!product) { return NextResponse.json({ error: "Product not found" }, { status: 404 }); } return NextResponse.json(product); } ``` ### Pattern 8: Metadata and SEO ```typescript // app/products/[slug]/page.tsx import { Metadata } from 'next' import { notFound } from 'next/navigation' type Props = { params: Promise<{ slug: string }> } export async function generateMetadata({ params }: Props): Promise<Metadata> { const { slug } = await params const product = await getProduct(slug) if (!product) return {} return { title: product.name, description: product.description, openGraph: { title: product.name, description: product.description, images: [{ url: product.image, width: 1200, height: 630 }], }, twitter: { card: 'summary_large_image', title: product.name, description: product.description, images: [product.image], }, } } export async function generateStaticParams() { const products = await db.product.findMany({ select: { slug: true } }) return products.map((p) => ({ slug: p.slug })) } export default async function ProductPage({ params }: Props) { const { slug } = await params const product = await getProduct(slug) if (!product) notFound() return <ProductDetail product={product} /> } ``` ## Caching Strategies ### Data Cache ```typescript // No cache (always fresh) fetch(url, { cache: "no-store" }); // Cache forever (static) fetch(url, { cache: "force-cache" }); // ISR - revalidate after 60 seconds fetch(url, { next: { revalidate: 60 } }); // Tag-based invalidation fetch(url, { next: { tags: ["products"] } }); // Invalidate via Server Action ("use server"); import { revalidateTag, revalidatePath } from "next/cache"; export async function updateProduct(id: string, data: ProductData) { await db.product.update({ where: { id }, data }); revalidateTag("products"); revalidatePath("/products"); } ``` ## Best Practices ### Do's - **Start with Server Components** - Add 'use client' only when needed - **Colocate data fetching** - Fetch data where it's used - **Use Suspense boundaries** - Enable streaming for slow data - **Leverage parallel routes** - Independent loading states - **Use Server Actions** - For mutations with progressive enhancement ### Don'ts - **Don't pass serializable data** - Server β†’ Client boundary limitations - **Don't use hooks in Server Components** - No useState, useEffect - **Don't fetch in Client Components** - Use Server Components or React Query - **Don't over-nest layouts** - Each layout adds to the component tree - **Don't ignore loading states** - Always provide loading.tsx or Suspense ## Resources - [Next.js App Router Documentation](https://nextjs.org/docs/app) - [Server Components RFC](https://github.com/reactjs/rfcs/blob/main/text/0188-server-components.md) - [Vercel Templates](https://vercel.com/templates/next.js)
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

react-native-architecture

Build production React Native apps with Expo, navigation, native

coding
⭐1
# React Native Architecture Production-ready patterns for React Native development with Expo, including navigation, state management, native modules, and offline-first architecture. ## When to Use This Skill - Starting a new React Native or Expo project - Implementing complex navigation patterns - Integrating native modules and platform APIs - Building offline-first mobile applications - Optimizing React Native performance - Setting up CI/CD for mobile releases ## Core Concepts ### 1. Project Structure ``` src/ β”œβ”€β”€ app/ # Expo Router screens β”‚ β”œβ”€β”€ (auth)/ # Auth group β”‚ β”œβ”€β”€ (tabs)/ # Tab navigation β”‚ └── _layout.tsx # Root layout β”œβ”€β”€ components/ β”‚ β”œβ”€β”€ ui/ # Reusable UI components β”‚ └── features/ # Feature-specific components β”œβ”€β”€ hooks/ # Custom hooks β”œβ”€β”€ services/ # API and native services β”œβ”€β”€ stores/ # State management β”œβ”€β”€ utils/ # Utilities └── types/ # TypeScript types ``` ### 2. Expo vs Bare React Native | Feature | Expo | Bare RN | | ------------------ | -------------- | -------------- | | Setup complexity | Low | High | | Native modules | EAS Build | Manual linking | | OTA updates | Built-in | Manual setup | | Build service | EAS | Custom CI | | Custom native code | Config plugins | Direct access | ## Quick Start ```bash # Create new Expo project npx create-expo-app@latest my-app -t expo-template-blank-typescript # Install essential dependencies npx expo install expo-router expo-status-bar react-native-safe-area-context npx expo install @react-native-async-storage/async-storage npx expo install expo-secure-store expo-haptics ``` ```typescript // app/_layout.tsx import { Stack } from 'expo-router' import { ThemeProvider } from '@/providers/ThemeProvider' import { QueryProvider } from '@/providers/QueryProvider' export default function RootLayout() { return ( <QueryProvider> <ThemeProvider> <Stack screenOptions={{ headerShown: false }}> <Stack.Screen name="(tabs)" /> <Stack.Screen name="(auth)" /> <Stack.Screen name="modal" options={{ presentation: 'modal' }} /> </Stack> </ThemeProvider> </QueryProvider> ) } ``` ## Patterns ### Pattern 1: Expo Router Navigation ```typescript // app/(tabs)/_layout.tsx import { Tabs } from 'expo-router' import { Home, Search, User, Settings } from 'lucide-react-native' import { useTheme } from '@/hooks/useTheme' export default function TabLayout() { const { colors } = useTheme() return ( <Tabs screenOptions={{ tabBarActiveTintColor: colors.primary, tabBarInactiveTintColor: colors.textMuted, tabBarStyle: { backgroundColor: colors.background }, headerShown: false, }} > <Tabs.Screen name="index" options={{ title: 'Home', tabBarIcon: ({ color, size }) => <Home size={size} color={color} />, }} /> <Tabs.Screen name="search" options={{ title: 'Search', tabBarIcon: ({ color, size }) => <Search size={size} color={color} />, }} /> <Tabs.Screen name="profile" options={{ title: 'Profile', tabBarIcon: ({ color, size }) => <User size={size} color={color} />, }} /> <Tabs.Screen name="settings" options={{ title: 'Settings', tabBarIcon: ({ color, size }) => <Settings size={size} color={color} />, }} /> </Tabs> ) } // app/(tabs)/profile/[id].tsx - Dynamic route import { useLocalSearchParams } from 'expo-router' export default function ProfileScreen() { const { id } = useLocalSearchParams<{ id: string }>() return <UserProfile userId={id} /> } // Navigation from anywhere import { router } from 'expo-router' // Programmatic navigation router.push('/profile/123') router.replace('/login') router.back() // With params router.push({ pathname: '/product/[id]', params: { id: '123', referrer: 'home' }, }) ``` ### Pattern 2: Authentication Flow ```typescript // providers/AuthProvider.tsx import { createContext, useContext, useEffect, useState } from 'react' import { useRouter, useSegments } from 'expo-router' import * as SecureStore from 'expo-secure-store' interface AuthContextType { user: User | null isLoading: boolean signIn: (credentials: Credentials) => Promise<void> signOut: () => Promise<void> } const AuthContext = createContext<AuthContextType | null>(null) export function AuthProvider({ children }: { children: React.ReactNode }) { const [user, setUser] = useState<User | null>(null) const [isLoading, setIsLoading] = useState(true) const segments = useSegments() const router = useRouter() // Check authentication on mount useEffect(() => { checkAuth() }, []) // Protect routes useEffect(() => { if (isLoading) return const inAuthGroup = segments[0] === '(auth)' if (!user && !inAuthGroup) { router.replace('/login') } else if (user && inAuthGroup) { router.replace('/(tabs)') } }, [user, segments, isLoading]) async function checkAuth() { try { const token = await SecureStore.getItemAsync('authToken') if (token) { const userData = await api.getUser(token) setUser(userData) } } catch (error) { await SecureStore.deleteItemAsync('authToken') } finally { setIsLoading(false) } } async function signIn(credentials: Credentials) { const { token, user } = await api.login(credentials) await SecureStore.setItemAsync('authToken', token) setUser(user) } async function signOut() { await SecureStore.deleteItemAsync('authToken') setUser(null) } if (isLoading) { return <SplashScreen /> } return ( <AuthContext.Provider value={{ user, isLoading, signIn, signOut }}> {children} </AuthContext.Provider> ) } export const useAuth = () => { const context = useContext(AuthContext) if (!context) throw new Error('useAuth must be used within AuthProvider') return context } ``` ### Pattern 3: Offline-First with React Query ```typescript // providers/QueryProvider.tsx import { QueryClient, QueryClientProvider } from '@tanstack/react-query' import { createAsyncStoragePersister } from '@tanstack/query-async-storage-persister' import { PersistQueryClientProvider } from '@tanstack/react-query-persist-client' import AsyncStorage from '@react-native-async-storage/async-storage' import NetInfo from '@react-native-community/netinfo' import { onlineManager } from '@tanstack/react-query' // Sync online status onlineManager.setEventListener((setOnline) => { return NetInfo.addEventListener((state) => { setOnline(!!state.isConnected) }) }) const queryClient = new QueryClient({ defaultOptions: { queries: { gcTime: 1000 * 60 * 60 * 24, // 24 hours staleTime: 1000 * 60 * 5, // 5 minutes retry: 2, networkMode: 'offlineFirst', }, mutations: { networkMode: 'offlineFirst', }, }, }) const asyncStoragePersister = createAsyncStoragePersister({ storage: AsyncStorage, key: 'REACT_QUERY_OFFLINE_CACHE', }) export function QueryProvider({ children }: { children: React.ReactNode }) { return ( <PersistQueryClientProvider client={queryClient} persistOptions={{ persister: asyncStoragePersister }} > {children} </PersistQueryClientProvider> ) } // hooks/useProducts.ts import { useQuery, useMutation, useQueryClient } from '@tanstack/react-query' export function useProducts() { return useQuery({ queryKey: ['products'], queryFn: api.getProducts, // Use stale data while revalidating placeholderData: (previousData) => previousData, }) } export function useCreateProduct() { const queryClient = useQueryClient() return useMutation({ mutationFn: api.createProduct, // Optimistic update onMutate: async (newProduct) => { await queryClient.cancelQueries({ queryKey: ['products'] }) const previous = queryClient.getQueryData(['products']) queryClient.setQueryData(['products'], (old: Product[]) => [ ...old, { ...newProduct, id: 'temp-' + Date.now() }, ]) return { previous } }, onError: (err, newProduct, context) => { queryClient.setQueryData(['products'], context?.previous) }, onSettled: () => { queryClient.invalidateQueries({ queryKey: ['products'] }) }, }) } ``` ### Pattern 4: Native Module Integration ```typescript // services/haptics.ts import * as Haptics from "expo-haptics"; import { Platform } from "react-native"; export const haptics = { light: () => { if (Platform.OS !== "web") { Haptics.impactAsync(Haptics.ImpactFeedbackStyle.Light); } }, medium: () => { if (Platform.OS !== "web") { Haptics.impactAsync(Haptics.ImpactFeedbackStyle.Medium); } }, heavy: () => { if (Platform.OS !== "web") { Haptics.impactAsync(Haptics.ImpactFeedbackStyle.Heavy); } }, success: () => { if (Platform.OS !== "web") { Haptics.notificationAsync(Haptics.NotificationFeedbackType.Success); } }, error: () => { if (Platform.OS !== "web") { Haptics.notificationAsync(Haptics.NotificationFeedbackType.Error); } }, }; // services/biometrics.ts import * as LocalAuthentication from "expo-local-authentication"; export async function authenticateWithBiometrics(): Promise<boolean> { const hasHardware = await LocalAuthentication.hasHardwareAsync(); if (!hasHardware) return false; const isEnrolled = await LocalAuthentication.isEnrolledAsync(); if (!isEnrolled) return false; const result = await LocalAuthentication.authenticateAsync({ promptMessage: "Authenticate to continue", fallbackLabel: "Use passcode", disableDeviceFallback: false, }); return result.success; } // services/notifications.ts import * as Notifications from "expo-notifications"; import { Platform } from "react-native"; import Constants from "expo-constants"; Notifications.setNotificationHandler({ handleNotification: async () => ({ shouldShowAlert: true, shouldPlaySound: true, shouldSetBadge: true, }), }); export async function registerForPushNotifications() { let token: string | undefined; if (Platform.OS === "android") { await Notifications.setNotificationChannelAsync("default", { name: "default", importance: Notifications.AndroidImportance.MAX, vibrationPattern: [0, 250, 250, 250], }); } const { status: existingStatus } = await Notifications.getPermissionsAsync(); let finalStatus = existingStatus; if (existingStatus !== "granted") { const { status } = await Notifications.requestPermissionsAsync(); finalStatus = status; } if (finalStatus !== "granted") { return null; } const projectId = Constants.expoConfig?.extra?.eas?.projectId; token = (await Notifications.getExpoPushTokenAsync({ projectId })).data; return token; } ``` ### Pattern 5: Platform-Specific Code ```typescript // components/ui/Button.tsx import { Platform, Pressable, StyleSheet, Text, ViewStyle } from 'react-native' import * as Haptics from 'expo-haptics' import Animated, { useAnimatedStyle, useSharedValue, withSpring, } from 'react-native-reanimated' const AnimatedPressable = Animated.createAnimatedComponent(Pressable) interface ButtonProps { title: string onPress: () => void variant?: 'primary' | 'secondary' | 'outline' disabled?: boolean } export function Button({ title, onPress, variant = 'primary', disabled = false, }: ButtonProps) { const scale = useSharedValue(1) const animatedStyle = useAnimatedStyle(() => ({ transform: [{ scale: scale.value }], })) const handlePressIn = () => { scale.value = withSpring(0.95) if (Platform.OS !== 'web') { Haptics.impactAsync(Haptics.ImpactFeedbackStyle.Light) } } const handlePressOut = () => { scale.value = withSpring(1) } return ( <AnimatedPressable onPress={onPress} onPressIn={handlePressIn} onPressOut={handlePressOut} disabled={disabled} style={[ styles.button, styles[variant], disabled && styles.disabled, animatedStyle, ]} > <Text style={[styles.text, styles[`${variant}Text`]]}>{title}</Text> </AnimatedPressable> ) } // Platform-specific files // Button.ios.tsx - iOS-specific implementation // Button.android.tsx - Android-specific implementation // Button.web.tsx - Web-specific implementation // Or use Platform.select const styles = StyleSheet.create({ button: { paddingVertical: 12, paddingHorizontal: 24, borderRadius: 8, alignItems: 'center', ...Platform.select({ ios: { shadowColor: '#000', shadowOffset: { width: 0, height: 2 }, shadowOpacity: 0.1, shadowRadius: 4, }, android: { elevation: 4, }, }), }, primary: { backgroundColor: '#007AFF', }, secondary: { backgroundColor: '#5856D6', }, outline: { backgroundColor: 'transparent', borderWidth: 1, borderColor: '#007AFF', }, disabled: { opacity: 0.5, }, text: { fontSize: 16, fontWeight: '600', }, primaryText: { color: '#FFFFFF', }, secondaryText: { color: '#FFFFFF', }, outlineText: { color: '#007AFF', }, }) ``` ### Pattern 6: Performance Optimization ```typescript // components/ProductList.tsx import { FlashList } from '@shopify/flash-list' import { memo, useCallback } from 'react' interface ProductListProps { products: Product[] onProductPress: (id: string) => void } // Memoize list item const ProductItem = memo(function ProductItem({ item, onPress, }: { item: Product onPress: (id: string) => void }) { const handlePress = useCallback(() => onPress(item.id), [item.id, onPress]) return ( <Pressable onPress={handlePress} style={styles.item}> <FastImage source={{ uri: item.image }} style={styles.image} resizeMode="cover" /> <Text style={styles.title}>{item.name}</Text> <Text style={styles.price}>${item.price}</Text> </Pressable> ) }) export function ProductList({ products, onProductPress }: ProductListProps) { const renderItem = useCallback( ({ item }: { item: Product }) => ( <ProductItem item={item} onPress={onProductPress} /> ), [onProductPress] ) const keyExtractor = useCallback((item: Product) => item.id, []) return ( <FlashList data={products} renderItem={renderItem} keyExtractor={keyExtractor} estimatedItemSize={100} // Performance optimizations removeClippedSubviews={true} maxToRenderPerBatch={10} windowSize={5} // Pull to refresh onRefresh={onRefresh} refreshing={isRefreshing} /> ) } ``` ## EAS Build & Submit ```json // eas.json { "cli": { "version": ">= 5.0.0" }, "build": { "development": { "developmentClient": true, "distribution": "internal", "ios": { "simulator": true } }, "preview": { "distribution": "internal", "android": { "buildType": "apk" } }, "production": { "autoIncrement": true } }, "submit": { "production": { "ios": { "appleId": "your@email.com", "ascAppId": "123456789" }, "android": { "serviceAccountKeyPath": "./google-services.json" } } } } ``` ```bash # Build commands eas build --platform ios --profile development eas build --platform android --profile preview eas build --platform all --profile production # Submit to stores eas submit --platform ios eas submit --platform android # OTA updates eas update --branch production --message "Bug fixes" ``` ## Best Practices ### Do's - **Use Expo** - Faster development, OTA updates, managed native code - **FlashList over FlatList** - Better performance for long lists - **Memoize components** - Prevent unnecessary re-renders - **Use Reanimated** - 60fps animations on native thread - **Test on real devices** - Simulators miss real-world issues ### Don'ts - **Don't inline styles** - Use StyleSheet.create for performance - **Don't fetch in render** - Use useEffect or React Query - **Don't ignore platform differences** - Test on both iOS and Android - **Don't store secrets in code** - Use environment variables - **Don't skip error boundaries** - Mobile crashes are unforgiving ## Resources - [Expo Documentation](https://docs.expo.dev/) - [Expo Router](https://docs.expo.dev/router/introduction/) - [React Native Performance](https://reactnative.dev/docs/performance) - [FlashList](https://shopify.github.io/flash-list/)
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

react-state-management

Master modern React state management with Redux Toolkit, Zustand,

coding
⭐1
# React State Management Comprehensive guide to modern React state management patterns, from local component state to global stores and server state synchronization. ## When to Use This Skill - Setting up global state management in a React app - Choosing between Redux Toolkit, Zustand, or Jotai - Managing server state with React Query or SWR - Implementing optimistic updates - Debugging state-related issues - Migrating from legacy Redux to modern patterns ## Core Concepts ### 1. State Categories | Type | Description | Solutions | | ---------------- | ---------------------------- | ----------------------------- | | **Local State** | Component-specific, UI state | useState, useReducer | | **Global State** | Shared across components | Redux Toolkit, Zustand, Jotai | | **Server State** | Remote data, caching | React Query, SWR, RTK Query | | **URL State** | Route parameters, search | React Router, nuqs | | **Form State** | Input values, validation | React Hook Form, Formik | ### 2. Selection Criteria ``` Small app, simple state β†’ Zustand or Jotai Large app, complex state β†’ Redux Toolkit Heavy server interaction β†’ React Query + light client state Atomic/granular updates β†’ Jotai ``` ## Quick Start ### Zustand (Simplest) ```typescript // store/useStore.ts import { create } from 'zustand' import { devtools, persist } from 'zustand/middleware' interface AppState { user: User | null theme: 'light' | 'dark' setUser: (user: User | null) => void toggleTheme: () => void } export const useStore = create<AppState>()( devtools( persist( (set) => ({ user: null, theme: 'light', setUser: (user) => set({ user }), toggleTheme: () => set((state) => ({ theme: state.theme === 'light' ? 'dark' : 'light' })), }), { name: 'app-storage' } ) ) ) // Usage in component function Header() { const { user, theme, toggleTheme } = useStore() return ( <header className={theme}> {user?.name} <button onClick={toggleTheme}>Toggle Theme</button> </header> ) } ``` ## Patterns ### Pattern 1: Redux Toolkit with TypeScript ```typescript // store/index.ts import { configureStore } from "@reduxjs/toolkit"; import { TypedUseSelectorHook, useDispatch, useSelector } from "react-redux"; import userReducer from "./slices/userSlice"; import cartReducer from "./slices/cartSlice"; export const store = configureStore({ reducer: { user: userReducer, cart: cartReducer, }, middleware: (getDefaultMiddleware) => getDefaultMiddleware({ serializableCheck: { ignoredActions: ["persist/PERSIST"], }, }), }); export type RootState = ReturnType<typeof store.getState>; export type AppDispatch = typeof store.dispatch; // Typed hooks export const useAppDispatch: () => AppDispatch = useDispatch; export const useAppSelector: TypedUseSelectorHook<RootState> = useSelector; ``` ```typescript // store/slices/userSlice.ts import { createSlice, createAsyncThunk, PayloadAction } from "@reduxjs/toolkit"; interface User { id: string; email: string; name: string; } interface UserState { current: User | null; status: "idle" | "loading" | "succeeded" | "failed"; error: string | null; } const initialState: UserState = { current: null, status: "idle", error: null, }; export const fetchUser = createAsyncThunk( "user/fetchUser", async (userId: string, { rejectWithValue }) => { try { const response = await fetch(`/api/users/${userId}`); if (!response.ok) throw new Error("Failed to fetch user"); return await response.json(); } catch (error) { return rejectWithValue((error as Error).message); } }, ); const userSlice = createSlice({ name: "user", initialState, reducers: { setUser: (state, action: PayloadAction<User>) => { state.current = action.payload; state.status = "succeeded"; }, clearUser: (state) => { state.current = null; state.status = "idle"; }, }, extraReducers: (builder) => { builder .addCase(fetchUser.pending, (state) => { state.status = "loading"; state.error = null; }) .addCase(fetchUser.fulfilled, (state, action) => { state.status = "succeeded"; state.current = action.payload; }) .addCase(fetchUser.rejected, (state, action) => { state.status = "failed"; state.error = action.payload as string; }); }, }); export const { setUser, clearUser } = userSlice.actions; export default userSlice.reducer; ``` ### Pattern 2: Zustand with Slices (Scalable) ```typescript // store/slices/createUserSlice.ts import { StateCreator } from "zustand"; export interface UserSlice { user: User | null; isAuthenticated: boolean; login: (credentials: Credentials) => Promise<void>; logout: () => void; } export const createUserSlice: StateCreator< UserSlice & CartSlice, // Combined store type [], [], UserSlice > = (set, get) => ({ user: null, isAuthenticated: false, login: async (credentials) => { const user = await authApi.login(credentials); set({ user, isAuthenticated: true }); }, logout: () => { set({ user: null, isAuthenticated: false }); // Can access other slices // get().clearCart() }, }); // store/index.ts import { create } from "zustand"; import { createUserSlice, UserSlice } from "./slices/createUserSlice"; import { createCartSlice, CartSlice } from "./slices/createCartSlice"; type StoreState = UserSlice & CartSlice; export const useStore = create<StoreState>()((...args) => ({ ...createUserSlice(...args), ...createCartSlice(...args), })); // Selective subscriptions (prevents unnecessary re-renders) export const useUser = () => useStore((state) => state.user); export const useCart = () => useStore((state) => state.cart); ``` ### Pattern 3: Jotai for Atomic State ```typescript // atoms/userAtoms.ts import { atom } from 'jotai' import { atomWithStorage } from 'jotai/utils' // Basic atom export const userAtom = atom<User | null>(null) // Derived atom (computed) export const isAuthenticatedAtom = atom((get) => get(userAtom) !== null) // Atom with localStorage persistence export const themeAtom = atomWithStorage<'light' | 'dark'>('theme', 'light') // Async atom export const userProfileAtom = atom(async (get) => { const user = get(userAtom) if (!user) return null const response = await fetch(`/api/users/${user.id}/profile`) return response.json() }) // Write-only atom (action) export const logoutAtom = atom(null, (get, set) => { set(userAtom, null) set(cartAtom, []) localStorage.removeItem('token') }) // Usage function Profile() { const [user] = useAtom(userAtom) const [, logout] = useAtom(logoutAtom) const [profile] = useAtom(userProfileAtom) // Suspense-enabled return ( <Suspense fallback={<Skeleton />}> <ProfileContent profile={profile} onLogout={logout} /> </Suspense> ) } ``` ### Pattern 4: React Query for Server State ```typescript // hooks/useUsers.ts import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query"; // Query keys factory export const userKeys = { all: ["users"] as const, lists: () => [...userKeys.all, "list"] as const, list: (filters: UserFilters) => [...userKeys.lists(), filters] as const, details: () => [...userKeys.all, "detail"] as const, detail: (id: string) => [...userKeys.details(), id] as const, }; // Fetch hook export function useUsers(filters: UserFilters) { return useQuery({ queryKey: userKeys.list(filters), queryFn: () => fetchUsers(filters), staleTime: 5 * 60 * 1000, // 5 minutes gcTime: 30 * 60 * 1000, // 30 minutes (formerly cacheTime) }); } // Single user hook export function useUser(id: string) { return useQuery({ queryKey: userKeys.detail(id), queryFn: () => fetchUser(id), enabled: !!id, // Don't fetch if no id }); } // Mutation with optimistic update export function useUpdateUser() { const queryClient = useQueryClient(); return useMutation({ mutationFn: updateUser, onMutate: async (newUser) => { // Cancel outgoing refetches await queryClient.cancelQueries({ queryKey: userKeys.detail(newUser.id), }); // Snapshot previous value const previousUser = queryClient.getQueryData( userKeys.detail(newUser.id), ); // Optimistically update queryClient.setQueryData(userKeys.detail(newUser.id), newUser); return { previousUser }; }, onError: (err, newUser, context) => { // Rollback on error queryClient.setQueryData( userKeys.detail(newUser.id), context?.previousUser, ); }, onSettled: (data, error, variables) => { // Refetch after mutation queryClient.invalidateQueries({ queryKey: userKeys.detail(variables.id), }); }, }); } ``` ### Pattern 5: Combining Client + Server State ```typescript // Zustand for client state const useUIStore = create<UIState>((set) => ({ sidebarOpen: true, modal: null, toggleSidebar: () => set((s) => ({ sidebarOpen: !s.sidebarOpen })), openModal: (modal) => set({ modal }), closeModal: () => set({ modal: null }), })) // React Query for server state function Dashboard() { const { sidebarOpen, toggleSidebar } = useUIStore() const { data: users, isLoading } = useUsers({ active: true }) const { data: stats } = useStats() if (isLoading) return <DashboardSkeleton /> return ( <div className={sidebarOpen ? 'with-sidebar' : ''}> <Sidebar open={sidebarOpen} onToggle={toggleSidebar} /> <main> <StatsCards stats={stats} /> <UserTable users={users} /> </main> </div> ) } ``` ## Best Practices ### Do's - **Colocate state** - Keep state as close to where it's used as possible - **Use selectors** - Prevent unnecessary re-renders with selective subscriptions - **Normalize data** - Flatten nested structures for easier updates - **Type everything** - Full TypeScript coverage prevents runtime errors - **Separate concerns** - Server state (React Query) vs client state (Zustand) ### Don'ts - **Don't over-globalize** - Not everything needs to be in global state - **Don't duplicate server state** - Let React Query manage it - **Don't mutate directly** - Always use immutable updates - **Don't store derived data** - Compute it instead - **Don't mix paradigms** - Pick one primary solution per category ## Migration Guides ### From Legacy Redux to RTK ```typescript // Before (legacy Redux) const ADD_TODO = "ADD_TODO"; const addTodo = (text) => ({ type: ADD_TODO, payload: text }); function todosReducer(state = [], action) { switch (action.type) { case ADD_TODO: return [...state, { text: action.payload, completed: false }]; default: return state; } } // After (Redux Toolkit) const todosSlice = createSlice({ name: "todos", initialState: [], reducers: { addTodo: (state, action: PayloadAction<string>) => { // Immer allows "mutations" state.push({ text: action.payload, completed: false }); }, }, }); ``` ## Resources - [Redux Toolkit Documentation](https://redux-toolkit.js.org/) - [Zustand GitHub](https://github.com/pmndrs/zustand) - [Jotai Documentation](https://jotai.org/) - [TanStack Query](https://tanstack.com/query)
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

modern-javascript-patterns

Master ES6+ features including async/await, destructuring, spread

coding
⭐1
# Modern JavaScript Patterns Comprehensive guide for mastering modern JavaScript (ES6+) features, functional programming patterns, and best practices for writing clean, maintainable, and performant code. ## When to Use This Skill - Refactoring legacy JavaScript to modern syntax - Implementing functional programming patterns - Optimizing JavaScript performance - Writing maintainable and readable code - Working with asynchronous operations - Building modern web applications - Migrating from callbacks to Promises/async-await - Implementing data transformation pipelines ## ES6+ Core Features ### 1. Arrow Functions **Syntax and Use Cases:** ```javascript // Traditional function function add(a, b) { return a + b; } // Arrow function const add = (a, b) => a + b; // Single parameter (parentheses optional) const double = (x) => x * 2; // No parameters const getRandom = () => Math.random(); // Multiple statements (need curly braces) const processUser = (user) => { const normalized = user.name.toLowerCase(); return { ...user, name: normalized }; }; // Returning objects (wrap in parentheses) const createUser = (name, age) => ({ name, age }); ``` **Lexical 'this' Binding:** ```javascript class Counter { constructor() { this.count = 0; } // Arrow function preserves 'this' context increment = () => { this.count++; }; // Traditional function loses 'this' in callbacks incrementTraditional() { setTimeout(function () { this.count++; // 'this' is undefined }, 1000); } // Arrow function maintains 'this' incrementArrow() { setTimeout(() => { this.count++; // 'this' refers to Counter instance }, 1000); } } ``` ### 2. Destructuring **Object Destructuring:** ```javascript const user = { id: 1, name: "John Doe", email: "john@example.com", address: { city: "New York", country: "USA", }, }; // Basic destructuring const { name, email } = user; // Rename variables const { name: userName, email: userEmail } = user; // Default values const { age = 25 } = user; // Nested destructuring const { address: { city, country }, } = user; // Rest operator const { id, ...userWithoutId } = user; // Function parameters function greet({ name, age = 18 }) { console.log(`Hello ${name}, you are ${age}`); } greet(user); ``` **Array Destructuring:** ```javascript const numbers = [1, 2, 3, 4, 5]; // Basic destructuring const [first, second] = numbers; // Skip elements const [, , third] = numbers; // Rest operator const [head, ...tail] = numbers; // Swapping variables let a = 1, b = 2; [a, b] = [b, a]; // Function return values function getCoordinates() { return [10, 20]; } const [x, y] = getCoordinates(); // Default values const [one, two, three = 0] = [1, 2]; ``` ### 3. Spread and Rest Operators **Spread Operator:** ```javascript // Array spreading const arr1 = [1, 2, 3]; const arr2 = [4, 5, 6]; const combined = [...arr1, ...arr2]; // Object spreading const defaults = { theme: "dark", lang: "en" }; const userPrefs = { theme: "light" }; const settings = { ...defaults, ...userPrefs }; // Function arguments const numbers = [1, 2, 3]; Math.max(...numbers); // Copying arrays/objects (shallow copy) const copy = [...arr1]; const objCopy = { ...user }; // Adding items immutably const newArr = [...arr1, 4, 5]; const newObj = { ...user, age: 30 }; ``` **Rest Parameters:** ```javascript // Collect function arguments function sum(...numbers) { return numbers.reduce((total, num) => total + num, 0); } sum(1, 2, 3, 4, 5); // With regular parameters function greet(greeting, ...names) { return `${greeting} ${names.join(", ")}`; } greet("Hello", "John", "Jane", "Bob"); // Object rest const { id, ...userData } = user; // Array rest const [first, ...rest] = [1, 2, 3, 4, 5]; ``` ### 4. Template Literals ```javascript // Basic usage const name = "John"; const greeting = `Hello, ${name}!`; // Multi-line strings const html = ` <div> <h1>${title}</h1> <p>${content}</p> </div> `; // Expression evaluation const price = 19.99; const total = `Total: $${(price * 1.2).toFixed(2)}`; // Tagged template literals function highlight(strings, ...values) { return strings.reduce((result, str, i) => { const value = values[i] || ""; return result + str + `<mark>${value}</mark>`; }, ""); } const name = "John"; const age = 30; const html = highlight`Name: ${name}, Age: ${age}`; // Output: "Name: <mark>John</mark>, Age: <mark>30</mark>" ``` ### 5. Enhanced Object Literals ```javascript const name = "John"; const age = 30; // Shorthand property names const user = { name, age }; // Shorthand method names const calculator = { add(a, b) { return a + b; }, subtract(a, b) { return a - b; }, }; // Computed property names const field = "email"; const user = { name: "John", [field]: "john@example.com", [`get${field.charAt(0).toUpperCase()}${field.slice(1)}`]() { return this[field]; }, }; // Dynamic property creation const createUser = (name, ...props) => { return props.reduce( (user, [key, value]) => ({ ...user, [key]: value, }), { name }, ); }; const user = createUser("John", ["age", 30], ["email", "john@example.com"]); ``` ## Asynchronous Patterns ### 1. Promises **Creating and Using Promises:** ```javascript // Creating a promise const fetchUser = (id) => { return new Promise((resolve, reject) => { setTimeout(() => { if (id > 0) { resolve({ id, name: "John" }); } else { reject(new Error("Invalid ID")); } }, 1000); }); }; // Using promises fetchUser(1) .then((user) => console.log(user)) .catch((error) => console.error(error)) .finally(() => console.log("Done")); // Chaining promises fetchUser(1) .then((user) => fetchUserPosts(user.id)) .then((posts) => processPosts(posts)) .then((result) => console.log(result)) .catch((error) => console.error(error)); ``` **Promise Combinators:** ```javascript // Promise.all - Wait for all promises const promises = [fetchUser(1), fetchUser(2), fetchUser(3)]; Promise.all(promises) .then((users) => console.log(users)) .catch((error) => console.error("At least one failed:", error)); // Promise.allSettled - Wait for all, regardless of outcome Promise.allSettled(promises).then((results) => { results.forEach((result) => { if (result.status === "fulfilled") { console.log("Success:", result.value); } else { console.log("Error:", result.reason); } }); }); // Promise.race - First to complete Promise.race(promises) .then((winner) => console.log("First:", winner)) .catch((error) => console.error(error)); // Promise.any - First to succeed Promise.any(promises) .then((first) => console.log("First success:", first)) .catch((error) => console.error("All failed:", error)); ``` ### 2. Async/Await **Basic Usage:** ```javascript // Async function always returns a Promise async function fetchUser(id) { const response = await fetch(`/api/users/${id}`); const user = await response.json(); return user; } // Error handling with try/catch async function getUserData(id) { try { const user = await fetchUser(id); const posts = await fetchUserPosts(user.id); return { user, posts }; } catch (error) { console.error("Error fetching data:", error); throw error; } } // Sequential vs Parallel execution async function sequential() { const user1 = await fetchUser(1); // Wait const user2 = await fetchUser(2); // Then wait return [user1, user2]; } async function parallel() { const [user1, user2] = await Promise.all([fetchUser(1), fetchUser(2)]); return [user1, user2]; } ``` **Advanced Patterns:** ```javascript // Async IIFE (async () => { const result = await someAsyncOperation(); console.log(result); })(); // Async iteration async function processUsers(userIds) { for (const id of userIds) { const user = await fetchUser(id); await processUser(user); } } // Top-level await (ES2022) const config = await fetch("/config.json").then((r) => r.json()); // Retry logic async function fetchWithRetry(url, retries = 3) { for (let i = 0; i < retries; i++) { try { return await fetch(url); } catch (error) { if (i === retries - 1) throw error; await new Promise((resolve) => setTimeout(resolve, 1000 * (i + 1))); } } } // Timeout wrapper async function withTimeout(promise, ms) { const timeout = new Promise((_, reject) => setTimeout(() => reject(new Error("Timeout")), ms), ); return Promise.race([promise, timeout]); } ``` ## Functional Programming Patterns ### 1. Array Methods **Map, Filter, Reduce:** ```javascript const users = [ { id: 1, name: "John", age: 30, active: true }, { id: 2, name: "Jane", age: 25, active: false }, { id: 3, name: "Bob", age: 35, active: true }, ]; // Map - Transform array const names = users.map((user) => user.name); const upperNames = users.map((user) => user.name.toUpperCase()); // Filter - Select elements const activeUsers = users.filter((user) => user.active); const adults = users.filter((user) => user.age >= 18); // Reduce - Aggregate data const totalAge = users.reduce((sum, user) => sum + user.age, 0); const avgAge = totalAge / users.length; // Group by property const byActive = users.reduce((groups, user) => { const key = user.active ? "active" : "inactive"; return { ...groups, [key]: [...(groups[key] || []), user], }; }, {}); // Chaining methods const result = users .filter((user) => user.active) .map((user) => user.name) .sort() .join(", "); ``` **Advanced Array Methods:** ```javascript // Find - First matching element const user = users.find((u) => u.id === 2); // FindIndex - Index of first match const index = users.findIndex((u) => u.name === "Jane"); // Some - At least one matches const hasActive = users.some((u) => u.active); // Every - All match const allAdults = users.every((u) => u.age >= 18); // FlatMap - Map and flatten const userTags = [ { name: "John", tags: ["admin", "user"] }, { name: "Jane", tags: ["user"] }, ]; const allTags = userTags.flatMap((u) => u.tags); // From - Create array from iterable const str = "hello"; const chars = Array.from(str); const numbers = Array.from({ length: 5 }, (_, i) => i + 1); // Of - Create array from arguments const arr = Array.of(1, 2, 3); ``` ### 2. Higher-Order Functions **Functions as Arguments:** ```javascript // Custom forEach function forEach(array, callback) { for (let i = 0; i < array.length; i++) { callback(array[i], i, array); } } // Custom map function map(array, transform) { const result = []; for (const item of array) { result.push(transform(item)); } return result; } // Custom filter function filter(array, predicate) { const result = []; for (const item of array) { if (predicate(item)) { result.push(item); } } return result; } ``` **Functions Returning Functions:** ```javascript // Currying const multiply = (a) => (b) => a * b; const double = multiply(2); const triple = multiply(3); console.log(double(5)); // 10 console.log(triple(5)); // 15 // Partial application function partial(fn, ...args) { return (...moreArgs) => fn(...args, ...moreArgs); } const add = (a, b, c) => a + b + c; const add5 = partial(add, 5); console.log(add5(3, 2)); // 10 // Memoization function memoize(fn) { const cache = new Map(); return (...args) => { const key = JSON.stringify(args); if (cache.has(key)) { return cache.get(key); } const result = fn(...args); cache.set(key, result); return result; }; } const fibonacci = memoize((n) => { if (n <= 1) return n; return fibonacci(n - 1) + fibonacci(n - 2); }); ``` ### 3. Composition and Piping ```javascript // Function composition const compose = (...fns) => (x) => fns.reduceRight((acc, fn) => fn(acc), x); const pipe = (...fns) => (x) => fns.reduce((acc, fn) => fn(acc), x); // Example usage const addOne = (x) => x + 1; const double = (x) => x * 2; const square = (x) => x * x; const composed = compose(square, double, addOne); console.log(composed(3)); // ((3 + 1) * 2)^2 = 64 const piped = pipe(addOne, double, square); console.log(piped(3)); // ((3 + 1) * 2)^2 = 64 // Practical example const processUser = pipe( (user) => ({ ...user, name: user.name.trim() }), (user) => ({ ...user, email: user.email.toLowerCase() }), (user) => ({ ...user, age: parseInt(user.age) }), ); const user = processUser({ name: " John ", email: "JOHN@EXAMPLE.COM", age: "30", }); ``` ### 4. Pure Functions and Immutability ```javascript // Impure function (modifies input) function addItemImpure(cart, item) { cart.items.push(item); cart.total += item.price; return cart; } // Pure function (no side effects) function addItemPure(cart, item) { return { ...cart, items: [...cart.items, item], total: cart.total + item.price, }; } // Immutable array operations const numbers = [1, 2, 3, 4, 5]; // Add to array const withSix = [...numbers, 6]; // Remove from array const withoutThree = numbers.filter((n) => n !== 3); // Update array element const doubled = numbers.map((n) => (n === 3 ? n * 2 : n)); // Immutable object operations const user = { name: "John", age: 30 }; // Update property const olderUser = { ...user, age: 31 }; // Add property const withEmail = { ...user, email: "john@example.com" }; // Remove property const { age, ...withoutAge } = user; // Deep cloning (simple approach) const deepClone = (obj) => JSON.parse(JSON.stringify(obj)); // Better deep cloning const structuredClone = (obj) => globalThis.structuredClone(obj); ``` ## Modern Class Features ```javascript // Class syntax class User { // Private fields #password; // Public fields id; name; // Static field static count = 0; constructor(id, name, password) { this.id = id; this.name = name; this.#password = password; User.count++; } // Public method greet() { return `Hello, ${this.name}`; } // Private method #hashPassword(password) { return `hashed_${password}`; } // Getter get displayName() { return this.name.toUpperCase(); } // Setter set password(newPassword) { this.#password = this.#hashPassword(newPassword); } // Static method static create(id, name, password) { return new User(id, name, password); } } // Inheritance class Admin extends User { constructor(id, name, password, role) { super(id, name, password); this.role = role; } greet() { return `${super.greet()}, I'm an admin`; } } ``` ## Modules (ES6) ```javascript // Exporting // math.js export const PI = 3.14159; export function add(a, b) { return a + b; } export class Calculator { // ... } // Default export export default function multiply(a, b) { return a * b; } // Importing // app.js import multiply, { PI, add, Calculator } from "./math.js"; // Rename imports import { add as sum } from "./math.js"; // Import all import * as Math from "./math.js"; // Dynamic imports const module = await import("./math.js"); const { add } = await import("./math.js"); // Conditional loading if (condition) { const module = await import("./feature.js"); module.init(); } ``` ## Iterators and Generators ```javascript // Custom iterator const range = { from: 1, to: 5, [Symbol.iterator]() { return { current: this.from, last: this.to, next() { if (this.current <= this.last) { return { done: false, value: this.current++ }; } else { return { done: true }; } }, }; }, }; for (const num of range) { console.log(num); // 1, 2, 3, 4, 5 } // Generator function function* rangeGenerator(from, to) { for (let i = from; i <= to; i++) { yield i; } } for (const num of rangeGenerator(1, 5)) { console.log(num); } // Infinite generator function* fibonacci() { let [prev, curr] = [0, 1]; while (true) { yield curr; [prev, curr] = [curr, prev + curr]; } } // Async generator async function* fetchPages(url) { let page = 1; while (true) { const response = await fetch(`${url}?page=${page}`); const data = await response.json(); if (data.length === 0) break; yield data; page++; } } for await (const page of fetchPages("/api/users")) { console.log(page); } ``` ## Modern Operators ```javascript // Optional chaining const user = { name: "John", address: { city: "NYC" } }; const city = user?.address?.city; const zipCode = user?.address?.zipCode; // undefined // Function call const result = obj.method?.(); // Array access const first = arr?.[0]; // Nullish coalescing const value = null ?? "default"; // 'default' const value = undefined ?? "default"; // 'default' const value = 0 ?? "default"; // 0 (not 'default') const value = "" ?? "default"; // '' (not 'default') // Logical assignment let a = null; a ??= "default"; // a = 'default' let b = 5; b ??= 10; // b = 5 (unchanged) let obj = { count: 0 }; obj.count ||= 1; // obj.count = 1 obj.count &&= 2; // obj.count = 2 ``` ## Performance Optimization ```javascript // Debounce function debounce(fn, delay) { let timeoutId; return (...args) => { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn(...args), delay); }; } const searchDebounced = debounce(search, 300); // Throttle function throttle(fn, limit) { let inThrottle; return (...args) => { if (!inThrottle) { fn(...args); inThrottle = true; setTimeout(() => (inThrottle = false), limit); } }; } const scrollThrottled = throttle(handleScroll, 100); // Lazy evaluation function* lazyMap(iterable, transform) { for (const item of iterable) { yield transform(item); } } // Use only what you need const numbers = [1, 2, 3, 4, 5]; const doubled = lazyMap(numbers, (x) => x * 2); const first = doubled.next().value; // Only computes first value ``` ## Best Practices 1. **Use const by default**: Only use let when reassignment is needed 2. **Prefer arrow functions**: Especially for callbacks 3. **Use template literals**: Instead of string concatenation 4. **Destructure objects and arrays**: For cleaner code 5. **Use async/await**: Instead of Promise chains 6. **Avoid mutating data**: Use spread operator and array methods 7. **Use optional chaining**: Prevent "Cannot read property of undefined" 8. **Use nullish coalescing**: For default values 9. **Prefer array methods**: Over traditional loops 10. **Use modules**: For better code organization 11. **Write pure functions**: Easier to test and reason about 12. **Use meaningful variable names**: Self-documenting code 13. **Keep functions small**: Single responsibility principle 14. **Handle errors properly**: Use try/catch with async/await 15. **Use strict mode**: `'use strict'` for better error catching ## Common Pitfalls 1. **this binding confusion**: Use arrow functions or bind() 2. **Async/await without error handling**: Always use try/catch 3. **Promise creation
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

embedding-strategies

Select and optimize embedding models for semantic search and RAG

coding
⭐1
# Embedding Strategies Guide to selecting and optimizing embedding models for vector search applications. ## When to Use This Skill - Choosing embedding models for RAG - Optimizing chunking strategies - Fine-tuning embeddings for domains - Comparing embedding model performance - Reducing embedding dimensions - Handling multilingual content ## Core Concepts ### 1. Embedding Model Comparison (2026) | Model | Dimensions | Max Tokens | Best For | | -------------------------- | ---------- | ---------- | ----------------------------------- | | **voyage-3-large** | 1024 | 32000 | Claude apps (Anthropic recommended) | | **voyage-3** | 1024 | 32000 | Claude apps, cost-effective | | **voyage-code-3** | 1024 | 32000 | Code search | | **voyage-finance-2** | 1024 | 32000 | Financial documents | | **voyage-law-2** | 1024 | 32000 | Legal documents | | **text-embedding-3-large** | 3072 | 8191 | OpenAI apps, high accuracy | | **text-embedding-3-small** | 1536 | 8191 | OpenAI apps, cost-effective | | **bge-large-en-v1.5** | 1024 | 512 | Open source, local deployment | | **all-MiniLM-L6-v2** | 384 | 256 | Fast, lightweight | | **multilingual-e5-large** | 1024 | 512 | Multi-language | ### 2. Embedding Pipeline ``` Document β†’ Chunking β†’ Preprocessing β†’ Embedding Model β†’ Vector ↓ [Overlap, Size] [Clean, Normalize] [API/Local] ``` ## Templates ### Template 1: Voyage AI Embeddings (Recommended for Claude) ```python from langchain_voyageai import VoyageAIEmbeddings from typing import List import os # Initialize Voyage AI embeddings (recommended by Anthropic for Claude) embeddings = VoyageAIEmbeddings( model="voyage-3-large", voyage_api_key=os.environ.get("VOYAGE_API_KEY") ) def get_embeddings(texts: List[str]) -> List[List[float]]: """Get embeddings from Voyage AI.""" return embeddings.embed_documents(texts) def get_query_embedding(query: str) -> List[float]: """Get single query embedding.""" return embeddings.embed_query(query) # Specialized models for domains code_embeddings = VoyageAIEmbeddings(model="voyage-code-3") finance_embeddings = VoyageAIEmbeddings(model="voyage-finance-2") legal_embeddings = VoyageAIEmbeddings(model="voyage-law-2") ``` ### Template 2: OpenAI Embeddings ```python from openai import OpenAI from typing import List import numpy as np client = OpenAI() def get_embeddings( texts: List[str], model: str = "text-embedding-3-small", dimensions: int = None ) -> List[List[float]]: """Get embeddings from OpenAI with optional dimension reduction.""" # Handle batching for large lists batch_size = 100 all_embeddings = [] for i in range(0, len(texts), batch_size): batch = texts[i:i + batch_size] kwargs = {"input": batch, "model": model} if dimensions: # Matryoshka dimensionality reduction kwargs["dimensions"] = dimensions response = client.embeddings.create(**kwargs) embeddings = [item.embedding for item in response.data] all_embeddings.extend(embeddings) return all_embeddings def get_embedding(text: str, **kwargs) -> List[float]: """Get single embedding.""" return get_embeddings([text], **kwargs)[0] # Dimension reduction with Matryoshka embeddings def get_reduced_embedding(text: str, dimensions: int = 512) -> List[float]: """Get embedding with reduced dimensions (Matryoshka).""" return get_embedding( text, model="text-embedding-3-small", dimensions=dimensions ) ``` ### Template 3: Local Embeddings with Sentence Transformers ```python from sentence_transformers import SentenceTransformer from typing import List, Optional import numpy as np class LocalEmbedder: """Local embedding with sentence-transformers.""" def __init__( self, model_name: str = "BAAI/bge-large-en-v1.5", device: str = "cuda" ): self.model = SentenceTransformer(model_name, device=device) self.model_name = model_name def embed( self, texts: List[str], normalize: bool = True, show_progress: bool = False ) -> np.ndarray: """Embed texts with optional normalization.""" embeddings = self.model.encode( texts, normalize_embeddings=normalize, show_progress_bar=show_progress, convert_to_numpy=True ) return embeddings def embed_query(self, query: str) -> np.ndarray: """Embed a query with appropriate prefix for retrieval models.""" # BGE and similar models benefit from query prefix if "bge" in self.model_name.lower(): query = f"Represent this sentence for searching relevant passages: {query}" return self.embed([query])[0] def embed_documents(self, documents: List[str]) -> np.ndarray: """Embed documents for indexing.""" return self.embed(documents) # E5 model with instructions class E5Embedder: def __init__(self, model_name: str = "intfloat/multilingual-e5-large"): self.model = SentenceTransformer(model_name) def embed_query(self, query: str) -> np.ndarray: """E5 requires 'query:' prefix for queries.""" return self.model.encode(f"query: {query}") def embed_document(self, document: str) -> np.ndarray: """E5 requires 'passage:' prefix for documents.""" return self.model.encode(f"passage: {document}") ``` ### Template 4: Chunking Strategies ```python from typing import List, Tuple import re def chunk_by_tokens( text: str, chunk_size: int = 512, chunk_overlap: int = 50, tokenizer=None ) -> List[str]: """Chunk text by token count.""" import tiktoken tokenizer = tokenizer or tiktoken.get_encoding("cl100k_base") tokens = tokenizer.encode(text) chunks = [] start = 0 while start < len(tokens): end = start + chunk_size chunk_tokens = tokens[start:end] chunk_text = tokenizer.decode(chunk_tokens) chunks.append(chunk_text) start = end - chunk_overlap return chunks def chunk_by_sentences( text: str, max_chunk_size: int = 1000, min_chunk_size: int = 100 ) -> List[str]: """Chunk text by sentences, respecting size limits.""" import nltk sentences = nltk.sent_tokenize(text) chunks = [] current_chunk = [] current_size = 0 for sentence in sentences: sentence_size = len(sentence) if current_size + sentence_size > max_chunk_size and current_chunk: chunks.append(" ".join(current_chunk)) current_chunk = [] current_size = 0 current_chunk.append(sentence) current_size += sentence_size if current_chunk: chunks.append(" ".join(current_chunk)) return chunks def chunk_by_semantic_sections( text: str, headers_pattern: str = r'^#{1,3}\s+.+$' ) -> List[Tuple[str, str]]: """Chunk markdown by headers, preserving hierarchy.""" lines = text.split('\n') chunks = [] current_header = "" current_content = [] for line in lines: if re.match(headers_pattern, line, re.MULTILINE): if current_content: chunks.append((current_header, '\n'.join(current_content))) current_header = line current_content = [] else: current_content.append(line) if current_content: chunks.append((current_header, '\n'.join(current_content))) return chunks def recursive_character_splitter( text: str, chunk_size: int = 1000, chunk_overlap: int = 200, separators: List[str] = None ) -> List[str]: """LangChain-style recursive splitter.""" separators = separators or ["\n\n", "\n", ". ", " ", ""] def split_text(text: str, separators: List[str]) -> List[str]: if not text: return [] separator = separators[0] remaining_separators = separators[1:] if separator == "": # Character-level split return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - chunk_overlap)] splits = text.split(separator) chunks = [] current_chunk = [] current_length = 0 for split in splits: split_length = len(split) + len(separator) if current_length + split_length > chunk_size and current_chunk: chunk_text = separator.join(current_chunk) # Recursively split if still too large if len(chunk_text) > chunk_size and remaining_separators: chunks.extend(split_text(chunk_text, remaining_separators)) else: chunks.append(chunk_text) # Start new chunk with overlap overlap_splits = [] overlap_length = 0 for s in reversed(current_chunk): if overlap_length + len(s) <= chunk_overlap: overlap_splits.insert(0, s) overlap_length += len(s) else: break current_chunk = overlap_splits current_length = overlap_length current_chunk.append(split) current_length += split_length if current_chunk: chunks.append(separator.join(current_chunk)) return chunks return split_text(text, separators) ``` ### Template 5: Domain-Specific Embedding Pipeline ```python import re from typing import List, Optional from dataclasses import dataclass @dataclass class EmbeddedDocument: id: str document_id: str chunk_index: int text: str embedding: List[float] metadata: dict class DomainEmbeddingPipeline: """Pipeline for domain-specific embeddings.""" def __init__( self, embedding_model: str = "voyage-3-large", chunk_size: int = 512, chunk_overlap: int = 50, preprocessing_fn=None ): self.embeddings = VoyageAIEmbeddings(model=embedding_model) self.chunk_size = chunk_size self.chunk_overlap = chunk_overlap self.preprocess = preprocessing_fn or self._default_preprocess def _default_preprocess(self, text: str) -> str: """Default preprocessing.""" # Remove excessive whitespace text = re.sub(r'\s+', ' ', text) # Remove special characters (customize for your domain) text = re.sub(r'[^\w\s.,!?-]', '', text) return text.strip() async def process_documents( self, documents: List[dict], id_field: str = "id", content_field: str = "content", metadata_fields: Optional[List[str]] = None ) -> List[EmbeddedDocument]: """Process documents for vector storage.""" processed = [] for doc in documents: content = doc[content_field] doc_id = doc[id_field] # Preprocess cleaned = self.preprocess(content) # Chunk chunks = chunk_by_tokens( cleaned, self.chunk_size, self.chunk_overlap ) # Create embeddings embeddings = await self.embeddings.aembed_documents(chunks) # Create records for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)): metadata = {"document_id": doc_id, "chunk_index": i} # Add specified metadata fields if metadata_fields: for field in metadata_fields: if field in doc: metadata[field] = doc[field] processed.append(EmbeddedDocument( id=f"{doc_id}_chunk_{i}", document_id=doc_id, chunk_index=i, text=chunk, embedding=embedding, metadata=metadata )) return processed # Code-specific pipeline class CodeEmbeddingPipeline: """Specialized pipeline for code embeddings.""" def __init__(self): # Use Voyage's code-specific model self.embeddings = VoyageAIEmbeddings(model="voyage-code-3") def chunk_code(self, code: str, language: str) -> List[dict]: """Chunk code by functions/classes using tree-sitter.""" try: import tree_sitter_languages parser = tree_sitter_languages.get_parser(language) tree = parser.parse(bytes(code, "utf8")) chunks = [] # Extract function and class definitions self._extract_nodes(tree.root_node, code, chunks) return chunks except ImportError: # Fallback to simple chunking return [{"text": code, "type": "module"}] def _extract_nodes(self, node, source_code: str, chunks: list): """Recursively extract function/class definitions.""" if node.type in ['function_definition', 'class_definition', 'method_definition']: text = source_code[node.start_byte:node.end_byte] chunks.append({ "text": text, "type": node.type, "name": self._get_name(node), "start_line": node.start_point[0], "end_line": node.end_point[0] }) for child in node.children: self._extract_nodes(child, source_code, chunks) def _get_name(self, node) -> str: """Extract name from function/class node.""" for child in node.children: if child.type == 'identifier' or child.type == 'name': return child.text.decode('utf8') return "unknown" async def embed_with_context( self, chunk: str, context: str = "" ) -> List[float]: """Embed code with surrounding context.""" if context: combined = f"Context: {context}\n\nCode:\n{chunk}" else: combined = chunk return await self.embeddings.aembed_query(combined) ``` ### Template 6: Embedding Quality Evaluation ```python import numpy as np from typing import List, Dict def evaluate_retrieval_quality( queries: List[str], relevant_docs: List[List[str]], # List of relevant doc IDs per query retrieved_docs: List[List[str]], # List of retrieved doc IDs per query k: int = 10 ) -> Dict[str, float]: """Evaluate embedding quality for retrieval.""" def precision_at_k(relevant: set, retrieved: List[str], k: int) -> float: retrieved_k = retrieved[:k] relevant_retrieved = len(set(retrieved_k) & relevant) return relevant_retrieved / k if k > 0 else 0 def recall_at_k(relevant: set, retrieved: List[str], k: int) -> float: retrieved_k = retrieved[:k] relevant_retrieved = len(set(retrieved_k) & relevant) return relevant_retrieved / len(relevant) if relevant else 0 def mrr(relevant: set, retrieved: List[str]) -> float: for i, doc in enumerate(retrieved): if doc in relevant: return 1 / (i + 1) return 0 def ndcg_at_k(relevant: set, retrieved: List[str], k: int) -> float: dcg = sum( 1 / np.log2(i + 2) if doc in relevant else 0 for i, doc in enumerate(retrieved[:k]) ) ideal_dcg = sum(1 / np.log2(i + 2) for i in range(min(len(relevant), k))) return dcg / ideal_dcg if ideal_dcg > 0 else 0 metrics = { f"precision@{k}": [], f"recall@{k}": [], "mrr": [], f"ndcg@{k}": [] } for relevant, retrieved in zip(relevant_docs, retrieved_docs): relevant_set = set(relevant) metrics[f"precision@{k}"].append(precision_at_k(relevant_set, retrieved, k)) metrics[f"recall@{k}"].append(recall_at_k(relevant_set, retrieved, k)) metrics["mrr"].append(mrr(relevant_set, retrieved)) metrics[f"ndcg@{k}"].append(ndcg_at_k(relevant_set, retrieved, k)) return {name: np.mean(values) for name, values in metrics.items()} def compute_embedding_similarity( embeddings1: np.ndarray, embeddings2: np.ndarray, metric: str = "cosine" ) -> np.ndarray: """Compute similarity matrix between embedding sets.""" if metric == "cosine": # Normalize and compute dot product norm1 = embeddings1 / np.linalg.norm(embeddings1, axis=1, keepdims=True) norm2 = embeddings2 / np.linalg.norm(embeddings2, axis=1, keepdims=True) return norm1 @ norm2.T elif metric == "euclidean": from scipy.spatial.distance import cdist return -cdist(embeddings1, embeddings2, metric='euclidean') elif metric == "dot": return embeddings1 @ embeddings2.T else: raise ValueError(f"Unknown metric: {metric}") def compare_embedding_models( texts: List[str], models: Dict[str, callable], queries: List[str], relevant_indices: List[List[int]], k: int = 5 ) -> Dict[str, Dict[str, float]]: """Compare multiple embedding models on retrieval quality.""" results = {} for model_name, embed_fn in models.items(): # Embed all texts doc_embeddings = np.array(embed_fn(texts)) retrieved_per_query = [] for query in queries: query_embedding = np.array(embed_fn([query])[0]) # Compute similarities similarities = compute_embedding_similarity( query_embedding.reshape(1, -1), doc_embeddings, metric="cosine" )[0] # Get top-k indices top_k_indices = np.argsort(similarities)[::-1][:k] retrieved_per_query.append([str(i) for i in top_k_indices]) # Convert relevant indices to string IDs relevant_docs = [[str(i) for i in indices] for indices in relevant_indices] results[model_name] = evaluate_retrieval_quality( queries, relevant_docs, retrieved_per_query, k ) return results ``` ## Best Practices ### Do's - **Match model to use case**: Code vs prose vs multilingual - **Chunk thoughtfully**: Preserve semantic boundaries - **Normalize embeddings**: For cosine similarity search - **Batch requests**: More efficient than one-by-one - **Cache embeddings**: Avoid recomputing for static content - **Use Voyage AI for Claude apps**: Recommended by Anthropic ### Don'ts - **Don't ignore token limits**: Truncation loses information - **Don't mix embedding models**: Incompatible vector spaces - **Don't skip preprocessing**: Garbage in, garbage out - **Don't over-chunk**: Lose important context - **Don't forget metadata**: Essential for filtering and debugging ## Resources - [Voyage AI Documentation](https://docs.voyageai.com/) - [OpenAI Embeddings Guide](https://platform.openai.com/docs/guides/embeddings) - [Sentence Transformers](https://www.sbert.net/) - [MTEB Benchmar
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

team-composition-patterns

Design optimal agent team compositions with sizing heuristics,

coding
⭐1
# Team Composition Patterns Best practices for composing multi-agent teams, selecting team sizes, choosing agent types, and configuring display modes for Claude Code's Agent Teams feature. ## When to Use This Skill - Deciding how many teammates to spawn for a task - Choosing between preset team configurations - Selecting the right agent type (subagent_type) for each role - Configuring teammate display modes (tmux, iTerm2, in-process) - Building custom team compositions for non-standard workflows ## Team Sizing Heuristics | Complexity | Team Size | When to Use | | ------------ | --------- | ----------------------------------------------------------- | | Simple | 1-2 | Single-dimension review, isolated bug, small feature | | Moderate | 2-3 | Multi-file changes, 2-3 concerns, medium features | | Complex | 3-4 | Cross-cutting concerns, large features, deep debugging | | Very Complex | 4-5 | Full-stack features, comprehensive reviews, systemic issues | **Rule of thumb**: Start with the smallest team that covers all required dimensions. Adding teammates increases coordination overhead. ## Preset Team Compositions ### Review Team - **Size**: 3 reviewers - **Agents**: 3x `team-reviewer` - **Default dimensions**: security, performance, architecture - **Use when**: Code changes need multi-dimensional quality assessment ### Debug Team - **Size**: 3 investigators - **Agents**: 3x `team-debugger` - **Default hypotheses**: 3 competing hypotheses - **Use when**: Bug has multiple plausible root causes ### Feature Team - **Size**: 3 (1 lead + 2 implementers) - **Agents**: 1x `team-lead` + 2x `team-implementer` - **Use when**: Feature can be decomposed into parallel work streams ### Fullstack Team - **Size**: 4 (1 lead + 3 implementers) - **Agents**: 1x `team-lead` + 1x frontend `team-implementer` + 1x backend `team-implementer` + 1x test `team-implementer` - **Use when**: Feature spans frontend, backend, and test layers ### Research Team - **Size**: 3 researchers - **Agents**: 3x `general-purpose` - **Default areas**: Each assigned a different research question, module, or topic - **Capabilities**: Codebase search (Grep, Glob, Read), web search (WebSearch, WebFetch) - **Use when**: Need to understand a codebase, research libraries, compare approaches, or gather information from code and web sources in parallel ### Security Team - **Size**: 4 reviewers - **Agents**: 4x `team-reviewer` - **Default dimensions**: OWASP/vulnerabilities, auth/access control, dependencies/supply chain, secrets/configuration - **Use when**: Comprehensive security audit covering multiple attack surfaces ### Migration Team - **Size**: 4 (1 lead + 2 implementers + 1 reviewer) - **Agents**: 1x `team-lead` + 2x `team-implementer` + 1x `team-reviewer` - **Use when**: Large codebase migration (framework upgrade, language port, API version bump) requiring parallel work with correctness verification ## Agent Type Selection When spawning teammates with the Task tool, choose `subagent_type` based on what tools the teammate needs: | Agent Type | Tools Available | Use For | | ------------------------------ | ----------------------------------------- | ---------------------------------------------------------- | | `general-purpose` | All tools (Read, Write, Edit, Bash, etc.) | Implementation, debugging, any task requiring file changes | | `Explore` | Read-only tools (Read, Grep, Glob) | Research, code exploration, analysis | | `Plan` | Read-only tools | Architecture planning, task decomposition | | `agent-teams:team-reviewer` | All tools | Code review with structured findings | | `agent-teams:team-debugger` | All tools | Hypothesis-driven investigation | | `agent-teams:team-implementer` | All tools | Building features within file ownership boundaries | | `agent-teams:team-lead` | All tools | Team orchestration and coordination | **Key distinction**: Read-only agents (Explore, Plan) cannot modify files. Never assign implementation tasks to read-only agents. ## Display Mode Configuration Configure in `~/.claude/settings.json`: ```json { "teammateMode": "tmux" } ``` | Mode | Behavior | Best For | | -------------- | ------------------------------ | ------------------------------------------------- | | `"tmux"` | Each teammate in a tmux pane | Development workflows, monitoring multiple agents | | `"iterm2"` | Each teammate in an iTerm2 tab | macOS users who prefer iTerm2 | | `"in-process"` | All teammates in same process | Simple tasks, CI/CD environments | ## Custom Team Guidelines When building custom teams: 1. **Every team needs a coordinator** β€” Either designate a `team-lead` or have the user coordinate directly 2. **Match roles to agent types** β€” Use specialized agents (reviewer, debugger, implementer) when available 3. **Avoid duplicate roles** β€” Two agents doing the same thing wastes resources 4. **Define boundaries upfront** β€” Each teammate needs clear ownership of files or responsibilities 5. **Keep it small** β€” 2-4 teammates is the sweet spot; 5+ requires significant coordination overhead
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

langchain-architecture

Design LLM applications using LangChain 1.x and LangGraph for

coding
⭐1
# LangChain & LangGraph Architecture Master modern LangChain 1.x and LangGraph for building sophisticated LLM applications with agents, state management, memory, and tool integration. ## When to Use This Skill - Building autonomous AI agents with tool access - Implementing complex multi-step LLM workflows - Managing conversation memory and state - Integrating LLMs with external data sources and APIs - Creating modular, reusable LLM application components - Implementing document processing pipelines - Building production-grade LLM applications ## Package Structure (LangChain 1.x) ``` langchain (1.2.x) # High-level orchestration langchain-core (1.2.x) # Core abstractions (messages, prompts, tools) langchain-community # Third-party integrations langgraph # Agent orchestration and state management langchain-openai # OpenAI integrations langchain-anthropic # Anthropic/Claude integrations langchain-voyageai # Voyage AI embeddings langchain-pinecone # Pinecone vector store ``` ## Core Concepts ### 1. LangGraph Agents LangGraph is the standard for building agents in 2026. It provides: **Key Features:** - **StateGraph**: Explicit state management with typed state - **Durable Execution**: Agents persist through failures - **Human-in-the-Loop**: Inspect and modify state at any point - **Memory**: Short-term and long-term memory across sessions - **Checkpointing**: Save and resume agent state **Agent Patterns:** - **ReAct**: Reasoning + Acting with `create_react_agent` - **Plan-and-Execute**: Separate planning and execution nodes - **Multi-Agent**: Supervisor routing between specialized agents - **Tool-Calling**: Structured tool invocation with Pydantic schemas ### 2. State Management LangGraph uses TypedDict for explicit state: ```python from typing import Annotated, TypedDict from langgraph.graph import MessagesState # Simple message-based state class AgentState(MessagesState): """Extends MessagesState with custom fields.""" context: Annotated[list, "retrieved documents"] # Custom state for complex agents class CustomState(TypedDict): messages: Annotated[list, "conversation history"] context: Annotated[dict, "retrieved context"] current_step: str results: list ``` ### 3. Memory Systems Modern memory implementations: - **ConversationBufferMemory**: Stores all messages (short conversations) - **ConversationSummaryMemory**: Summarizes older messages (long conversations) - **ConversationTokenBufferMemory**: Token-based windowing - **VectorStoreRetrieverMemory**: Semantic similarity retrieval - **LangGraph Checkpointers**: Persistent state across sessions ### 4. Document Processing Loading, transforming, and storing documents: **Components:** - **Document Loaders**: Load from various sources - **Text Splitters**: Chunk documents intelligently - **Vector Stores**: Store and retrieve embeddings - **Retrievers**: Fetch relevant documents ### 5. Callbacks & Tracing LangSmith is the standard for observability: - Request/response logging - Token usage tracking - Latency monitoring - Error tracking - Trace visualization ## Quick Start ### Modern ReAct Agent with LangGraph ```python from langgraph.prebuilt import create_react_agent from langgraph.checkpoint.memory import MemorySaver from langchain_anthropic import ChatAnthropic from langchain_core.tools import tool import ast import operator # Initialize LLM (Claude Sonnet 4.6 recommended) llm = ChatAnthropic(model="claude-sonnet-4-6", temperature=0) # Define tools with Pydantic schemas @tool def search_database(query: str) -> str: """Search internal database for information.""" # Your database search logic return f"Results for: {query}" @tool def calculate(expression: str) -> str: """Safely evaluate a mathematical expression. Supports: +, -, *, /, **, %, parentheses Example: '(2 + 3) * 4' returns '20' """ # Safe math evaluation using ast allowed_operators = { ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv, ast.Pow: operator.pow, ast.Mod: operator.mod, ast.USub: operator.neg, } def _eval(node): if isinstance(node, ast.Constant): return node.value elif isinstance(node, ast.BinOp): left = _eval(node.left) right = _eval(node.right) return allowed_operators[type(node.op)](left, right) elif isinstance(node, ast.UnaryOp): operand = _eval(node.operand) return allowed_operators[type(node.op)](operand) else: raise ValueError(f"Unsupported operation: {type(node)}") try: tree = ast.parse(expression, mode='eval') return str(_eval(tree.body)) except Exception as e: return f"Error: {e}" tools = [search_database, calculate] # Create checkpointer for memory persistence checkpointer = MemorySaver() # Create ReAct agent agent = create_react_agent( llm, tools, checkpointer=checkpointer ) # Run agent with thread ID for memory config = {"configurable": {"thread_id": "user-123"}} result = await agent.ainvoke( {"messages": [("user", "Search for Python tutorials and calculate 25 * 4")]}, config=config ) ``` ## Architecture Patterns ### Pattern 1: RAG with LangGraph ```python from langgraph.graph import StateGraph, START, END from langchain_anthropic import ChatAnthropic from langchain_voyageai import VoyageAIEmbeddings from langchain_pinecone import PineconeVectorStore from langchain_core.documents import Document from langchain_core.prompts import ChatPromptTemplate from typing import TypedDict, Annotated class RAGState(TypedDict): question: str context: Annotated[list[Document], "retrieved documents"] answer: str # Initialize components llm = ChatAnthropic(model="claude-sonnet-4-6") embeddings = VoyageAIEmbeddings(model="voyage-3-large") vectorstore = PineconeVectorStore(index_name="docs", embedding=embeddings) retriever = vectorstore.as_retriever(search_kwargs={"k": 4}) # Define nodes async def retrieve(state: RAGState) -> RAGState: """Retrieve relevant documents.""" docs = await retriever.ainvoke(state["question"]) return {"context": docs} async def generate(state: RAGState) -> RAGState: """Generate answer from context.""" prompt = ChatPromptTemplate.from_template( """Answer based on the context below. If you cannot answer, say so. Context: {context} Question: {question} Answer:""" ) context_text = "\n\n".join(doc.page_content for doc in state["context"]) response = await llm.ainvoke( prompt.format(context=context_text, question=state["question"]) ) return {"answer": response.content} # Build graph builder = StateGraph(RAGState) builder.add_node("retrieve", retrieve) builder.add_node("generate", generate) builder.add_edge(START, "retrieve") builder.add_edge("retrieve", "generate") builder.add_edge("generate", END) rag_chain = builder.compile() # Use the chain result = await rag_chain.ainvoke({"question": "What is the main topic?"}) ``` ### Pattern 2: Custom Agent with Structured Tools ```python from langchain_core.tools import StructuredTool from pydantic import BaseModel, Field class SearchInput(BaseModel): """Input for database search.""" query: str = Field(description="Search query") filters: dict = Field(default={}, description="Optional filters") class EmailInput(BaseModel): """Input for sending email.""" recipient: str = Field(description="Email recipient") subject: str = Field(description="Email subject") content: str = Field(description="Email body") async def search_database(query: str, filters: dict = {}) -> str: """Search internal database for information.""" # Your database search logic return f"Results for '{query}' with filters {filters}" async def send_email(recipient: str, subject: str, content: str) -> str: """Send an email to specified recipient.""" # Email sending logic return f"Email sent to {recipient}" tools = [ StructuredTool.from_function( coroutine=search_database, name="search_database", description="Search internal database", args_schema=SearchInput ), StructuredTool.from_function( coroutine=send_email, name="send_email", description="Send an email", args_schema=EmailInput ) ] agent = create_react_agent(llm, tools) ``` ### Pattern 3: Multi-Step Workflow with StateGraph ```python from langgraph.graph import StateGraph, START, END from typing import TypedDict, Literal class WorkflowState(TypedDict): text: str entities: list analysis: str summary: str current_step: str async def extract_entities(state: WorkflowState) -> WorkflowState: """Extract key entities from text.""" prompt = f"Extract key entities from: {state['text']}\n\nReturn as JSON list." response = await llm.ainvoke(prompt) return {"entities": response.content, "current_step": "analyze"} async def analyze_entities(state: WorkflowState) -> WorkflowState: """Analyze extracted entities.""" prompt = f"Analyze these entities: {state['entities']}\n\nProvide insights." response = await llm.ainvoke(prompt) return {"analysis": response.content, "current_step": "summarize"} async def generate_summary(state: WorkflowState) -> WorkflowState: """Generate final summary.""" prompt = f"""Summarize: Entities: {state['entities']} Analysis: {state['analysis']} Provide a concise summary.""" response = await llm.ainvoke(prompt) return {"summary": response.content, "current_step": "complete"} def route_step(state: WorkflowState) -> Literal["analyze", "summarize", "end"]: """Route to next step based on current state.""" step = state.get("current_step", "extract") if step == "analyze": return "analyze" elif step == "summarize": return "summarize" return "end" # Build workflow builder = StateGraph(WorkflowState) builder.add_node("extract", extract_entities) builder.add_node("analyze", analyze_entities) builder.add_node("summarize", generate_summary) builder.add_edge(START, "extract") builder.add_conditional_edges("extract", route_step, { "analyze": "analyze", "summarize": "summarize", "end": END }) builder.add_conditional_edges("analyze", route_step, { "summarize": "summarize", "end": END }) builder.add_edge("summarize", END) workflow = builder.compile() ``` ### Pattern 4: Multi-Agent Orchestration ```python from langgraph.graph import StateGraph, START, END from langgraph.prebuilt import create_react_agent from langchain_core.messages import HumanMessage from typing import Literal class MultiAgentState(TypedDict): messages: list next_agent: str # Create specialized agents researcher = create_react_agent(llm, research_tools) writer = create_react_agent(llm, writing_tools) reviewer = create_react_agent(llm, review_tools) async def supervisor(state: MultiAgentState) -> MultiAgentState: """Route to appropriate agent based on task.""" prompt = f"""Based on the conversation, which agent should handle this? Options: - researcher: For finding information - writer: For creating content - reviewer: For reviewing and editing - FINISH: Task is complete Messages: {state['messages']} Respond with just the agent name.""" response = await llm.ainvoke(prompt) return {"next_agent": response.content.strip().lower()} def route_to_agent(state: MultiAgentState) -> Literal["researcher", "writer", "reviewer", "end"]: """Route based on supervisor decision.""" next_agent = state.get("next_agent", "").lower() if next_agent == "finish": return "end" return next_agent if next_agent in ["researcher", "writer", "reviewer"] else "end" # Build multi-agent graph builder = StateGraph(MultiAgentState) builder.add_node("supervisor", supervisor) builder.add_node("researcher", researcher) builder.add_node("writer", writer) builder.add_node("reviewer", reviewer) builder.add_edge(START, "supervisor") builder.add_conditional_edges("supervisor", route_to_agent, { "researcher": "researcher", "writer": "writer", "reviewer": "reviewer", "end": END }) # Each agent returns to supervisor for agent in ["researcher", "writer", "reviewer"]: builder.add_edge(agent, "supervisor") multi_agent = builder.compile() ``` ## Memory Management ### Token-Based Memory with LangGraph ```python from langgraph.checkpoint.memory import MemorySaver from langgraph.prebuilt import create_react_agent # In-memory checkpointer (development) checkpointer = MemorySaver() # Create agent with persistent memory agent = create_react_agent(llm, tools, checkpointer=checkpointer) # Each thread_id maintains separate conversation config = {"configurable": {"thread_id": "session-abc123"}} # Messages persist across invocations with same thread_id result1 = await agent.ainvoke({"messages": [("user", "My name is Alice")]}, config) result2 = await agent.ainvoke({"messages": [("user", "What's my name?")]}, config) # Agent remembers: "Your name is Alice" ``` ### Production Memory with PostgreSQL ```python from langgraph.checkpoint.postgres import PostgresSaver # Production checkpointer checkpointer = PostgresSaver.from_conn_string( "postgresql://user:pass@localhost/langgraph" ) agent = create_react_agent(llm, tools, checkpointer=checkpointer) ``` ### Vector Store Memory for Long-Term Context ```python from langchain_community.vectorstores import Chroma from langchain_voyageai import VoyageAIEmbeddings embeddings = VoyageAIEmbeddings(model="voyage-3-large") memory_store = Chroma( collection_name="conversation_memory", embedding_function=embeddings, persist_directory="./memory_db" ) async def retrieve_relevant_memory(query: str, k: int = 5) -> list: """Retrieve relevant past conversations.""" docs = await memory_store.asimilarity_search(query, k=k) return [doc.page_content for doc in docs] async def store_memory(content: str, metadata: dict = {}): """Store conversation in long-term memory.""" await memory_store.aadd_texts([content], metadatas=[metadata]) ``` ## Callback System & LangSmith ### LangSmith Tracing ```python import os from langchain_anthropic import ChatAnthropic # Enable LangSmith tracing os.environ["LANGCHAIN_TRACING_V2"] = "true" os.environ["LANGCHAIN_API_KEY"] = "your-api-key" os.environ["LANGCHAIN_PROJECT"] = "my-project" # All LangChain/LangGraph operations are automatically traced llm = ChatAnthropic(model="claude-sonnet-4-6") ``` ### Custom Callback Handler ```python from langchain_core.callbacks import BaseCallbackHandler from typing import Any, Dict, List class CustomCallbackHandler(BaseCallbackHandler): def on_llm_start( self, serialized: Dict[str, Any], prompts: List[str], **kwargs ) -> None: print(f"LLM started with {len(prompts)} prompts") def on_llm_end(self, response, **kwargs) -> None: print(f"LLM completed: {len(response.generations)} generations") def on_llm_error(self, error: Exception, **kwargs) -> None: print(f"LLM error: {error}") def on_tool_start( self, serialized: Dict[str, Any], input_str: str, **kwargs ) -> None: print(f"Tool started: {serialized.get('name')}") def on_tool_end(self, output: str, **kwargs) -> None: print(f"Tool completed: {output[:100]}...") # Use callbacks result = await agent.ainvoke( {"messages": [("user", "query")]}, config={"callbacks": [CustomCallbackHandler()]} ) ``` ## Streaming Responses ```python from langchain_anthropic import ChatAnthropic llm = ChatAnthropic(model="claude-sonnet-4-6", streaming=True) # Stream tokens async for chunk in llm.astream("Tell me a story"): print(chunk.content, end="", flush=True) # Stream agent events async for event in agent.astream_events( {"messages": [("user", "Search and summarize")]}, version="v2" ): if event["event"] == "on_chat_model_stream": print(event["data"]["chunk"].content, end="") elif event["event"] == "on_tool_start": print(f"\n[Using tool: {event['name']}]") ``` ## Testing Strategies ```python import pytest from unittest.mock import AsyncMock, patch @pytest.mark.asyncio async def test_agent_tool_selection(): """Test agent selects correct tool.""" with patch.object(llm, 'ainvoke') as mock_llm: mock_llm.return_value = AsyncMock(content="Using search_database") result = await agent.ainvoke({ "messages": [("user", "search for documents")] }) # Verify tool was called assert "search_database" in str(result) @pytest.mark.asyncio async def test_memory_persistence(): """Test memory persists across invocations.""" config = {"configurable": {"thread_id": "test-thread"}} # First message await agent.ainvoke( {"messages": [("user", "Remember: the code is 12345")]}, config ) # Second message should remember result = await agent.ainvoke( {"messages": [("user", "What was the code?")]}, config ) assert "12345" in result["messages"][-1].content ``` ## Performance Optimization ### 1. Caching with Redis ```python from langchain_community.cache import RedisCache from langchain_core.globals import set_llm_cache import redis redis_client = redis.Redis.from_url("redis://localhost:6379") set_llm_cache(RedisCache(redis_client)) ``` ### 2. Async Batch Processing ```python import asyncio from langchain_core.documents import Document async def process_documents(documents: list[Document]) -> list: """Process documents in parallel.""" tasks = [process_single(doc) for doc in documents] return await asyncio.gather(*tasks) async def process_single(doc: Document) -> dict: """Process a single document.""" chunks = text_splitter.split_documents([doc]) embeddings = await embeddings_model.aembed_documents( [c.page_content for c in chunks] ) return {"doc_id": doc.metadata.get("id"), "embeddings": embeddings} ``` ### 3. Connection Pooling ```python from langchain_pinecone import PineconeVectorStore from pinecone import Pinecone # Reuse Pinecone client pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"]) index = pc.Index("my-index") # Create vector store with existing index vectorstore = PineconeVectorStore(index=index, embedding=embeddings) ``` ## Resources - [LangChain Documentation](https://python.langchain.com/docs/) - [LangGraph Documentation](https://langchain-ai.github.io/langgraph/) - [LangSmith Platform](https://smith.langchain.com/) - [LangChain GitHub](https://github.com/langchain-ai/langchain) - [LangGraph GitHub](https://github.com/langchain-ai/langgraph) ## Common Pitfalls 1. **Using Deprecated APIs**: Use LangGraph for agents, not `initialize_agent` 2. **Memory Overflow**: Use checkpointers with TTL for long-running agents 3. **Poor Tool Descriptions**: Clear descriptions help LLM select correct tools 4. **Context Window Exceeded**: Use summarization or sliding window memory 5. **No Error Handling**: Wrap too
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered
πŸ€–system promptβ€’7 months ago

rag-implementation

Build Retrieval-Augmented Generation (RAG) systems for LLM

coding
⭐1
# RAG Implementation Master Retrieval-Augmented Generation (RAG) to build LLM applications that provide accurate, grounded responses using external knowledge sources. ## When to Use This Skill - Building Q&A systems over proprietary documents - Creating chatbots with current, factual information - Implementing semantic search with natural language queries - Reducing hallucinations with grounded responses - Enabling LLMs to access domain-specific knowledge - Building documentation assistants - Creating research tools with source citation ## Core Components ### 1. Vector Databases **Purpose**: Store and retrieve document embeddings efficiently **Options:** - **Pinecone**: Managed, scalable, serverless - **Weaviate**: Open-source, hybrid search, GraphQL - **Milvus**: High performance, on-premise - **Chroma**: Lightweight, easy to use, local development - **Qdrant**: Fast, filtered search, Rust-based - **pgvector**: PostgreSQL extension, SQL integration ### 2. Embeddings **Purpose**: Convert text to numerical vectors for similarity search **Models (2026):** | Model | Dimensions | Best For | |-------|------------|----------| | **voyage-3-large** | 1024 | Claude apps (Anthropic recommended) | | **voyage-code-3** | 1024 | Code search | | **text-embedding-3-large** | 3072 | OpenAI apps, high accuracy | | **text-embedding-3-small** | 1536 | OpenAI apps, cost-effective | | **bge-large-en-v1.5** | 1024 | Open source, local deployment | | **multilingual-e5-large** | 1024 | Multi-language support | ### 3. Retrieval Strategies **Approaches:** - **Dense Retrieval**: Semantic similarity via embeddings - **Sparse Retrieval**: Keyword matching (BM25, TF-IDF) - **Hybrid Search**: Combine dense + sparse with weighted fusion - **Multi-Query**: Generate multiple query variations - **HyDE**: Generate hypothetical documents for better retrieval ### 4. Reranking **Purpose**: Improve retrieval quality by reordering results **Methods:** - **Cross-Encoders**: BERT-based reranking (ms-marco-MiniLM) - **Cohere Rerank**: API-based reranking - **Maximal Marginal Relevance (MMR)**: Diversity + relevance - **LLM-based**: Use LLM to score relevance ## Quick Start with LangGraph ```python from langgraph.graph import StateGraph, START, END from langchain_anthropic import ChatAnthropic from langchain_voyageai import VoyageAIEmbeddings from langchain_pinecone import PineconeVectorStore from langchain_core.documents import Document from langchain_core.prompts import ChatPromptTemplate from langchain_text_splitters import RecursiveCharacterTextSplitter from typing import TypedDict, Annotated class RAGState(TypedDict): question: str context: list[Document] answer: str # Initialize components llm = ChatAnthropic(model="claude-sonnet-4-6") embeddings = VoyageAIEmbeddings(model="voyage-3-large") vectorstore = PineconeVectorStore(index_name="docs", embedding=embeddings) retriever = vectorstore.as_retriever(search_kwargs={"k": 4}) # RAG prompt rag_prompt = ChatPromptTemplate.from_template( """Answer based on the context below. If you cannot answer, say so. Context: {context} Question: {question} Answer:""" ) async def retrieve(state: RAGState) -> RAGState: """Retrieve relevant documents.""" docs = await retriever.ainvoke(state["question"]) return {"context": docs} async def generate(state: RAGState) -> RAGState: """Generate answer from context.""" context_text = "\n\n".join(doc.page_content for doc in state["context"]) messages = rag_prompt.format_messages( context=context_text, question=state["question"] ) response = await llm.ainvoke(messages) return {"answer": response.content} # Build RAG graph builder = StateGraph(RAGState) builder.add_node("retrieve", retrieve) builder.add_node("generate", generate) builder.add_edge(START, "retrieve") builder.add_edge("retrieve", "generate") builder.add_edge("generate", END) rag_chain = builder.compile() # Use result = await rag_chain.ainvoke({"question": "What are the main features?"}) print(result["answer"]) ``` ## Advanced RAG Patterns ### Pattern 1: Hybrid Search with RRF ```python from langchain_community.retrievers import BM25Retriever from langchain.retrievers import EnsembleRetriever # Sparse retriever (BM25 for keyword matching) bm25_retriever = BM25Retriever.from_documents(documents) bm25_retriever.k = 10 # Dense retriever (embeddings for semantic search) dense_retriever = vectorstore.as_retriever(search_kwargs={"k": 10}) # Combine with Reciprocal Rank Fusion weights ensemble_retriever = EnsembleRetriever( retrievers=[bm25_retriever, dense_retriever], weights=[0.3, 0.7] # 30% keyword, 70% semantic ) ``` ### Pattern 2: Multi-Query Retrieval ```python from langchain.retrievers.multi_query import MultiQueryRetriever # Generate multiple query perspectives for better recall multi_query_retriever = MultiQueryRetriever.from_llm( retriever=vectorstore.as_retriever(search_kwargs={"k": 5}), llm=llm ) # Single query β†’ multiple variations β†’ combined results results = await multi_query_retriever.ainvoke("What is the main topic?") ``` ### Pattern 3: Contextual Compression ```python from langchain.retrievers import ContextualCompressionRetriever from langchain.retrievers.document_compressors import LLMChainExtractor # Compressor extracts only relevant portions compressor = LLMChainExtractor.from_llm(llm) compression_retriever = ContextualCompressionRetriever( base_compressor=compressor, base_retriever=vectorstore.as_retriever(search_kwargs={"k": 10}) ) # Returns only relevant parts of documents compressed_docs = await compression_retriever.ainvoke("specific query") ``` ### Pattern 4: Parent Document Retriever ```python from langchain.retrievers import ParentDocumentRetriever from langchain.storage import InMemoryStore from langchain_text_splitters import RecursiveCharacterTextSplitter # Small chunks for precise retrieval, large chunks for context child_splitter = RecursiveCharacterTextSplitter(chunk_size=400, chunk_overlap=50) parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200) # Store for parent documents docstore = InMemoryStore() parent_retriever = ParentDocumentRetriever( vectorstore=vectorstore, docstore=docstore, child_splitter=child_splitter, parent_splitter=parent_splitter ) # Add documents (splits children, stores parents) await parent_retriever.aadd_documents(documents) # Retrieval returns parent documents with full context results = await parent_retriever.ainvoke("query") ``` ### Pattern 5: HyDE (Hypothetical Document Embeddings) ```python from langchain_core.prompts import ChatPromptTemplate class HyDEState(TypedDict): question: str hypothetical_doc: str context: list[Document] answer: str hyde_prompt = ChatPromptTemplate.from_template( """Write a detailed passage that would answer this question: Question: {question} Passage:""" ) async def generate_hypothetical(state: HyDEState) -> HyDEState: """Generate hypothetical document for better retrieval.""" messages = hyde_prompt.format_messages(question=state["question"]) response = await llm.ainvoke(messages) return {"hypothetical_doc": response.content} async def retrieve_with_hyde(state: HyDEState) -> HyDEState: """Retrieve using hypothetical document.""" # Use hypothetical doc for retrieval instead of original query docs = await retriever.ainvoke(state["hypothetical_doc"]) return {"context": docs} # Build HyDE RAG graph builder = StateGraph(HyDEState) builder.add_node("hypothetical", generate_hypothetical) builder.add_node("retrieve", retrieve_with_hyde) builder.add_node("generate", generate) builder.add_edge(START, "hypothetical") builder.add_edge("hypothetical", "retrieve") builder.add_edge("retrieve", "generate") builder.add_edge("generate", END) hyde_rag = builder.compile() ``` ## Document Chunking Strategies ### Recursive Character Text Splitter ```python from langchain_text_splitters import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter( chunk_size=1000, chunk_overlap=200, length_function=len, separators=["\n\n", "\n", ". ", " ", ""] # Try in order ) chunks = splitter.split_documents(documents) ``` ### Token-Based Splitting ```python from langchain_text_splitters import TokenTextSplitter splitter = TokenTextSplitter( chunk_size=512, chunk_overlap=50, encoding_name="cl100k_base" # OpenAI tiktoken encoding ) ``` ### Semantic Chunking ```python from langchain_experimental.text_splitter import SemanticChunker splitter = SemanticChunker( embeddings=embeddings, breakpoint_threshold_type="percentile", breakpoint_threshold_amount=95 ) ``` ### Markdown Header Splitter ```python from langchain_text_splitters import MarkdownHeaderTextSplitter headers_to_split_on = [ ("#", "Header 1"), ("##", "Header 2"), ("###", "Header 3"), ] splitter = MarkdownHeaderTextSplitter( headers_to_split_on=headers_to_split_on, strip_headers=False ) ``` ## Vector Store Configurations ### Pinecone (Serverless) ```python from pinecone import Pinecone, ServerlessSpec from langchain_pinecone import PineconeVectorStore # Initialize Pinecone client pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"]) # Create index if needed if "my-index" not in pc.list_indexes().names(): pc.create_index( name="my-index", dimension=1024, # voyage-3-large dimensions metric="cosine", spec=ServerlessSpec(cloud="aws", region="us-east-1") ) # Create vector store index = pc.Index("my-index") vectorstore = PineconeVectorStore(index=index, embedding=embeddings) ``` ### Weaviate ```python import weaviate from langchain_weaviate import WeaviateVectorStore client = weaviate.connect_to_local() # or connect_to_weaviate_cloud() vectorstore = WeaviateVectorStore( client=client, index_name="Documents", text_key="content", embedding=embeddings ) ``` ### Chroma (Local Development) ```python from langchain_chroma import Chroma vectorstore = Chroma( collection_name="my_collection", embedding_function=embeddings, persist_directory="./chroma_db" ) ``` ### pgvector (PostgreSQL) ```python from langchain_postgres.vectorstores import PGVector connection_string = "postgresql+psycopg://user:pass@localhost:5432/vectordb" vectorstore = PGVector( embeddings=embeddings, collection_name="documents", connection=connection_string, ) ``` ## Retrieval Optimization ### 1. Metadata Filtering ```python from langchain_core.documents import Document # Add metadata during indexing docs_with_metadata = [] for doc in documents: doc.metadata.update({ "source": doc.metadata.get("source", "unknown"), "category": determine_category(doc.page_content), "date": datetime.now().isoformat() }) docs_with_metadata.append(doc) # Filter during retrieval results = await vectorstore.asimilarity_search( "query", filter={"category": "technical"}, k=5 ) ``` ### 2. Maximal Marginal Relevance (MMR) ```python # Balance relevance with diversity results = await vectorstore.amax_marginal_relevance_search( "query", k=5, fetch_k=20, # Fetch 20, return top 5 diverse lambda_mult=0.5 # 0=max diversity, 1=max relevance ) ``` ### 3. Reranking with Cross-Encoder ```python from sentence_transformers import CrossEncoder reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2') async def retrieve_and_rerank(query: str, k: int = 5) -> list[Document]: # Get initial results candidates = await vectorstore.asimilarity_search(query, k=20) # Rerank pairs = [[query, doc.page_content] for doc in candidates] scores = reranker.predict(pairs) # Sort by score and take top k ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True) return [doc for doc, score in ranked[:k]] ``` ### 4. Cohere Rerank ```python from langchain.retrievers import CohereRerank from langchain_cohere import CohereRerank reranker = CohereRerank(model="rerank-english-v3.0", top_n=5) # Wrap retriever with reranking reranked_retriever = ContextualCompressionRetriever( base_compressor=reranker, base_retriever=vectorstore.as_retriever(search_kwargs={"k": 20}) ) ``` ## Prompt Engineering for RAG ### Contextual Prompt with Citations ```python rag_prompt = ChatPromptTemplate.from_template( """Answer the question based on the context below. Include citations using [1], [2], etc. If you cannot answer based on the context, say "I don't have enough information." Context: {context} Question: {question} Instructions: 1. Use only information from the context 2. Cite sources with [1], [2] format 3. If uncertain, express uncertainty Answer (with citations):""" ) ``` ### Structured Output for RAG ```python from pydantic import BaseModel, Field class RAGResponse(BaseModel): answer: str = Field(description="The answer based on context") confidence: float = Field(description="Confidence score 0-1") sources: list[str] = Field(description="Source document IDs used") reasoning: str = Field(description="Brief reasoning for the answer") # Use with structured output structured_llm = llm.with_structured_output(RAGResponse) ``` ## Evaluation Metrics ```python from typing import TypedDict class RAGEvalMetrics(TypedDict): retrieval_precision: float # Relevant docs / retrieved docs retrieval_recall: float # Retrieved relevant / total relevant answer_relevance: float # Answer addresses question faithfulness: float # Answer grounded in context context_relevance: float # Context relevant to question async def evaluate_rag_system( rag_chain, test_cases: list[dict] ) -> RAGEvalMetrics: """Evaluate RAG system on test cases.""" metrics = {k: [] for k in RAGEvalMetrics.__annotations__} for test in test_cases: result = await rag_chain.ainvoke({"question": test["question"]}) # Retrieval metrics retrieved_ids = {doc.metadata["id"] for doc in result["context"]} relevant_ids = set(test["relevant_doc_ids"]) precision = len(retrieved_ids & relevant_ids) / len(retrieved_ids) recall = len(retrieved_ids & relevant_ids) / len(relevant_ids) metrics["retrieval_precision"].append(precision) metrics["retrieval_recall"].append(recall) # Use LLM-as-judge for quality metrics quality = await evaluate_answer_quality( question=test["question"], answer=result["answer"], context=result["context"], expected=test.get("expected_answer") ) metrics["answer_relevance"].append(quality["relevance"]) metrics["faithfulness"].append(quality["faithfulness"]) metrics["context_relevance"].append(quality["context_relevance"]) return {k: sum(v) / len(v) for k, v in metrics.items()} ``` ## Resources - [LangChain RAG Tutorial](https://python.langchain.com/docs/tutorials/rag/) - [LangGraph RAG Examples](https://langchain-ai.github.io/langgraph/tutorials/rag/) - [Pinecone Best Practices](https://docs.pinecone.io/guides/get-started/overview) - [Voyage AI Embeddings](https://docs.voyageai.com/) - [RAG Evaluation Guide](https://docs.ragas.io/) ## Best Practices 1. **Chunk Size**: Balance between context (larger) and specificity (smaller) - typically 500-1000 tokens 2. **Overlap**: Use 10-20% overlap to preserve context at boundaries 3. **Metadata**: Include source, page, timestamp for filtering and debugging 4. **Hybrid Search**: Combine semantic and keyword search for best recall 5. **Reranking**: Use cross-encoder reranking for precision-critical applications 6. **Citations**: Always return source documents for transparency 7. **Evaluation**: Continuously test retrieval quality and answer accuracy 8. **Monitoring**: Track retrieval metrics and latency in production ## Common Issues - **Poor Retrieval**: Check embedding quality, chunk size, query formulation - **Irrelevant Results**: Add metadata filtering, use hybrid search, rerank - **Missing Information**: Ensure documents are properly indexed, check chunking - **Slow Queries**: Optimize vector store, use caching, reduce k - **Hallucinations**: Improve grounding prompt, add verification step - **Context Too Long**: Use compression or parent document retriever
πŸ‘0
πŸ‘οΈ0
πŸ€– Auto-discovered