./browse/prompts

# Context-Driven Development Guide for implementing and maintaining context as a managed artifact alongside code, enabling consistent AI interactions and team alignment through structured project documentation. ## When to Use This Skill - Setting up new projects with Conductor - Understanding the relationship between context artifacts - Maintaining consistency across AI-assisted development sessions - Onboarding team members to an existing Conductor project - Deciding when to update context documents - Managing greenfield vs brownfield project contexts ## Core Philosophy Context-Driven Development treats project context as a first-class artifact managed alongside code. Instead of relying on ad-hoc prompts or scattered documentation, establish a persistent, structured foundation that informs all AI interactions. Key principles: 1. **Context precedes code**: Define what you're building and how before implementation 2. **Living documentation**: Context artifacts evolve with the project 3. **Single source of truth**: One canonical location for each type of information 4. **AI alignment**: Consistent context produces consistent AI behavior ## The Workflow Follow the **Context → Spec & Plan → Implement** workflow: 1. **Context Phase**: Establish or verify project context artifacts exist and are current 2. **Specification Phase**: Define requirements and acceptance criteria for work units 3. **Planning Phase**: Break specifications into phased, actionable tasks 4. **Implementation Phase**: Execute tasks following established workflow patterns ## Artifact Relationships ### product.md - Defines WHAT and WHY Purpose: Captures product vision, goals, target users, and business context. Contents: - Product name and one-line description - Problem statement and solution approach - Target user personas - Core features and capabilities - Success metrics and KPIs - Product roadmap (high-level) Update when: - Product vision or goals change - New major features are planned - Target audience shifts - Business priorities evolve ### product-guidelines.md - Defines HOW to Communicate Purpose: Establishes brand voice, messaging standards, and communication patterns. Contents: - Brand voice and tone guidelines - Terminology and glossary - Error message conventions - User-facing copy standards - Documentation style Update when: - Brand guidelines change - New terminology is introduced - Communication patterns need refinement ### tech-stack.md - Defines WITH WHAT Purpose: Documents technology choices, dependencies, and architectural decisions. Contents: - Primary languages and frameworks - Key dependencies with versions - Infrastructure and deployment targets - Development tools and environment - Testing frameworks - Code quality tools Update when: - Adding new dependencies - Upgrading major versions - Changing infrastructure - Adopting new tools or patterns ### workflow.md - Defines HOW to Work Purpose: Establishes development practices, quality gates, and team workflows. Contents: - Development methodology (TDD, etc.) - Git workflow and commit conventions - Code review requirements - Testing requirements and coverage targets - Quality assurance gates - Deployment procedures Update when: - Team practices evolve - Quality standards change - New workflow patterns are adopted ### tracks.md - Tracks WHAT'S HAPPENING Purpose: Registry of all work units with status and metadata. Contents: - Active tracks with current status - Completed tracks with completion dates - Track metadata (type, priority, assignee) - Links to individual track directories Update when: - New tracks are created - Track status changes - Tracks are completed or archived See [references/artifact-templates.md](references/artifact-templates.md) for copy-paste starter templates. ## Context Maintenance Principles ### Keep Artifacts Synchronized Ensure changes in one artifact reflect in related documents: - New feature in product.md → Update tech-stack.md if new dependencies needed - Completed track → Update product.md to reflect new capabilities - Workflow change → Update all affected track plans ### Update tech-stack.md When Adding Dependencies Before adding any new dependency: 1. Check if existing dependencies solve the need 2. Document the rationale for new dependencies 3. Add version constraints 4. Note any configuration requirements ### Update product.md When Features Complete After completing a feature track: 1. Move feature from "planned" to "implemented" in product.md 2. Update any affected success metrics 3. Document any scope changes from original plan ### Verify Context Before Implementation Before starting any track: 1. Read all context artifacts 2. Flag any outdated information 3. Propose updates before proceeding 4. Confirm context accuracy with stakeholders ## Greenfield vs Brownfield Handling ### Greenfield Projects (New) For new projects: 1. Run `/conductor:setup` to create all artifacts interactively 2. Answer questions about product vision, tech preferences, and workflow 3. Generate initial style guides for chosen languages 4. Create empty tracks registry Characteristics: - Full control over context structure - Define standards before code exists - Establish patterns early ### Brownfield Projects (Existing) For existing codebases: 1. Run `/conductor:setup` with existing codebase detection 2. System analyzes existing code, configs, and documentation 3. Pre-populate artifacts based on discovered patterns 4. Review and refine generated context Characteristics: - Extract implicit context from existing code - Reconcile existing patterns with desired patterns - Document technical debt and modernization plans - Preserve working patterns while establishing standards ## Benefits ### Team Alignment - New team members onboard faster with explicit context - Consistent terminology and conventions across the team - Shared understanding of product goals and technical decisions ### AI Consistency - AI assistants produce aligned outputs across sessions - Reduced need to re-explain context in each interaction - Predictable behavior based on documented standards ### Institutional Memory - Decisions and rationale are preserved - Context survives team changes - Historical context informs future decisions ### Quality Assurance - Standards are explicit and verifiable - Deviations from context are detectable - Quality gates are documented and enforceable ## Directory Structure ``` conductor/ ├── index.md # Navigation hub linking all artifacts ├── product.md # Product vision and goals ├── product-guidelines.md # Communication standards ├── tech-stack.md # Technology preferences ├── workflow.md # Development practices ├── tracks.md # Work unit registry ├── setup_state.json # Resumable setup state ├── code_styleguides/ # Language-specific conventions │ ├── python.md │ ├── typescript.md │ └── ... └── tracks/ └── <track-id>/ ├── spec.md ├── plan.md ├── metadata.json └── index.md ``` ## Context Lifecycle 1. **Creation**: Initial setup via `/conductor:setup` 2. **Validation**: Verify before each track 3. **Evolution**: Update as project grows 4. **Synchronization**: Keep artifacts aligned 5. **Archival**: Document historical decisions ## Context Validation Checklist Before starting implementation on any track, validate context: ### Product Context - [ ] product.md reflects current product vision - [ ] Target users are accurately described - [ ] Feature list is up to date - [ ] Success metrics are defined ### Technical Context - [ ] tech-stack.md lists all current dependencies - [ ] Version numbers are accurate - [ ] Infrastructure targets are correct - [ ] Development tools are documented ### Workflow Context - [ ] workflow.md describes current practices - [ ] Quality gates are defined - [ ] Coverage targets are specified - [ ] Commit conventions are documented ### Track Context - [ ] tracks.md shows all active work - [ ] No stale or abandoned tracks - [ ] Dependencies between tracks are noted ## Common Anti-Patterns Avoid these context management mistakes: ### Stale Context Problem: Context documents become outdated and misleading. Solution: Update context as part of each track's completion process. ### Context Sprawl Problem: Information scattered across multiple locations. Solution: Use the defined artifact structure; resist creating new document types. ### Implicit Context Problem: Relying on knowledge not captured in artifacts. Solution: If you reference something repeatedly, add it to the appropriate artifact. ### Context Hoarding Problem: One person maintains context without team input. Solution: Review context artifacts in pull requests; make updates collaborative. ### Over-Specification Problem: Context becomes so detailed it's impossible to maintain. Solution: Keep artifacts focused on decisions that affect AI behavior and team alignment. ## Integration with Development Tools ### IDE Integration Configure your IDE to display context files prominently: - Pin conductor/product.md for quick reference - Add tech-stack.md to project notes - Create snippets for common patterns from style guides ### Git Hooks Consider pre-commit hooks that: - Warn when dependencies change without tech-stack.md update - Remind to update product.md when feature branches merge - Validate context artifact syntax ### CI/CD Integration Include context validation in pipelines: - Check tech-stack.md matches actual dependencies - Verify links in context documents resolve - Ensure tracks.md status matches git branch state ## Session Continuity Conductor supports multi-session development through context persistence: ### Starting a New Session 1. Read index.md to orient yourself 2. Check tracks.md for active work 3. Review relevant track's plan.md for current task 4. Verify context artifacts are current ### Ending a Session 1. Update plan.md with current progress 2. Note any blockers or decisions made 3. Commit in-progress work with clear status 4. Update tracks.md if status changed ### Handling Interruptions If interrupted mid-task: 1. Mark task as `[~]` with note about stopping point 2. Commit work-in-progress to feature branch 3. Document any uncommitted decisions in plan.md ## Best Practices 1. **Read context first**: Always read relevant artifacts before starting work 2. **Small updates**: Make incremental context changes, not massive rewrites 3. **Link decisions**: Reference context when making implementation choices 4. **Version context**: Commit context changes alongside code changes 5. **Review context**: Include context artifact reviews in code reviews 6. **Validate regularly**: Run context validation checklist before major work 7. **Communicate changes**: Notify team when context artifacts change significantly 8. **Preserve history**: Use git to track context evolution over time 9. **Question staleness**: If context feels wrong, investigate and update 10. **Keep it actionable**: Every context item should inform a decision or behavior

# Authentication & Authorization Implementation Patterns Build secure, scalable authentication and authorization systems using industry-standard patterns and modern best practices. ## When to Use This Skill - Implementing user authentication systems - Securing REST or GraphQL APIs - Adding OAuth2/social login - Implementing role-based access control (RBAC) - Designing session management - Migrating authentication systems - Debugging auth issues - Implementing SSO or multi-tenancy ## Core Concepts ### 1. Authentication vs Authorization **Authentication (AuthN)**: Who are you? - Verifying identity (username/password, OAuth, biometrics) - Issuing credentials (sessions, tokens) - Managing login/logout **Authorization (AuthZ)**: What can you do? - Permission checking - Role-based access control (RBAC) - Resource ownership validation - Policy enforcement ### 2. Authentication Strategies **Session-Based:** - Server stores session state - Session ID in cookie - Traditional, simple, stateful **Token-Based (JWT):** - Stateless, self-contained - Scales horizontally - Can store claims **OAuth2/OpenID Connect:** - Delegate authentication - Social login (Google, GitHub) - Enterprise SSO ## JWT Authentication ### Pattern 1: JWT Implementation ```typescript // JWT structure: header.payload.signature import jwt from "jsonwebtoken"; import { Request, Response, NextFunction } from "express"; interface JWTPayload { userId: string; email: string; role: string; iat: number; exp: number; } // Generate JWT function generateTokens(userId: string, email: string, role: string) { const accessToken = jwt.sign( { userId, email, role }, process.env.JWT_SECRET!, { expiresIn: "15m" }, // Short-lived ); const refreshToken = jwt.sign( { userId }, process.env.JWT_REFRESH_SECRET!, { expiresIn: "7d" }, // Long-lived ); return { accessToken, refreshToken }; } // Verify JWT function verifyToken(token: string): JWTPayload { try { return jwt.verify(token, process.env.JWT_SECRET!) as JWTPayload; } catch (error) { if (error instanceof jwt.TokenExpiredError) { throw new Error("Token expired"); } if (error instanceof jwt.JsonWebTokenError) { throw new Error("Invalid token"); } throw error; } } // Middleware function authenticate(req: Request, res: Response, next: NextFunction) { const authHeader = req.headers.authorization; if (!authHeader?.startsWith("Bearer ")) { return res.status(401).json({ error: "No token provided" }); } const token = authHeader.substring(7); try { const payload = verifyToken(token); req.user = payload; // Attach user to request next(); } catch (error) { return res.status(401).json({ error: "Invalid token" }); } } // Usage app.get("/api/profile", authenticate, (req, res) => { res.json({ user: req.user }); }); ``` ### Pattern 2: Refresh Token Flow ```typescript interface StoredRefreshToken { token: string; userId: string; expiresAt: Date; createdAt: Date; } class RefreshTokenService { // Store refresh token in database async storeRefreshToken(userId: string, refreshToken: string) { const expiresAt = new Date(Date.now() + 7 * 24 * 60 * 60 * 1000); await db.refreshTokens.create({ token: await hash(refreshToken), // Hash before storing userId, expiresAt, }); } // Refresh access token async refreshAccessToken(refreshToken: string) { // Verify refresh token let payload; try { payload = jwt.verify(refreshToken, process.env.JWT_REFRESH_SECRET!) as { userId: string; }; } catch { throw new Error("Invalid refresh token"); } // Check if token exists in database const storedToken = await db.refreshTokens.findOne({ where: { token: await hash(refreshToken), userId: payload.userId, expiresAt: { $gt: new Date() }, }, }); if (!storedToken) { throw new Error("Refresh token not found or expired"); } // Get user const user = await db.users.findById(payload.userId); if (!user) { throw new Error("User not found"); } // Generate new access token const accessToken = jwt.sign( { userId: user.id, email: user.email, role: user.role }, process.env.JWT_SECRET!, { expiresIn: "15m" }, ); return { accessToken }; } // Revoke refresh token (logout) async revokeRefreshToken(refreshToken: string) { await db.refreshTokens.deleteOne({ token: await hash(refreshToken), }); } // Revoke all user tokens (logout all devices) async revokeAllUserTokens(userId: string) { await db.refreshTokens.deleteMany({ userId }); } } // API endpoints app.post("/api/auth/refresh", async (req, res) => { const { refreshToken } = req.body; try { const { accessToken } = await refreshTokenService.refreshAccessToken(refreshToken); res.json({ accessToken }); } catch (error) { res.status(401).json({ error: "Invalid refresh token" }); } }); app.post("/api/auth/logout", authenticate, async (req, res) => { const { refreshToken } = req.body; await refreshTokenService.revokeRefreshToken(refreshToken); res.json({ message: "Logged out successfully" }); }); ``` ## Session-Based Authentication ### Pattern 1: Express Session ```typescript import session from "express-session"; import RedisStore from "connect-redis"; import { createClient } from "redis"; // Setup Redis for session storage const redisClient = createClient({ url: process.env.REDIS_URL, }); await redisClient.connect(); app.use( session({ store: new RedisStore({ client: redisClient }), secret: process.env.SESSION_SECRET!, resave: false, saveUninitialized: false, cookie: { secure: process.env.NODE_ENV === "production", // HTTPS only httpOnly: true, // No JavaScript access maxAge: 24 * 60 * 60 * 1000, // 24 hours sameSite: "strict", // CSRF protection }, }), ); // Login app.post("/api/auth/login", async (req, res) => { const { email, password } = req.body; const user = await db.users.findOne({ email }); if (!user || !(await verifyPassword(password, user.passwordHash))) { return res.status(401).json({ error: "Invalid credentials" }); } // Store user in session req.session.userId = user.id; req.session.role = user.role; res.json({ user: { id: user.id, email: user.email, role: user.role } }); }); // Session middleware function requireAuth(req: Request, res: Response, next: NextFunction) { if (!req.session.userId) { return res.status(401).json({ error: "Not authenticated" }); } next(); } // Protected route app.get("/api/profile", requireAuth, async (req, res) => { const user = await db.users.findById(req.session.userId); res.json({ user }); }); // Logout app.post("/api/auth/logout", (req, res) => { req.session.destroy((err) => { if (err) { return res.status(500).json({ error: "Logout failed" }); } res.clearCookie("connect.sid"); res.json({ message: "Logged out successfully" }); }); }); ``` ## OAuth2 / Social Login ### Pattern 1: OAuth2 with Passport.js ```typescript import passport from "passport"; import { Strategy as GoogleStrategy } from "passport-google-oauth20"; import { Strategy as GitHubStrategy } from "passport-github2"; // Google OAuth passport.use( new GoogleStrategy( { clientID: process.env.GOOGLE_CLIENT_ID!, clientSecret: process.env.GOOGLE_CLIENT_SECRET!, callbackURL: "/api/auth/google/callback", }, async (accessToken, refreshToken, profile, done) => { try { // Find or create user let user = await db.users.findOne({ googleId: profile.id, }); if (!user) { user = await db.users.create({ googleId: profile.id, email: profile.emails?.[0]?.value, name: profile.displayName, avatar: profile.photos?.[0]?.value, }); } return done(null, user); } catch (error) { return done(error, undefined); } }, ), ); // Routes app.get( "/api/auth/google", passport.authenticate("google", { scope: ["profile", "email"], }), ); app.get( "/api/auth/google/callback", passport.authenticate("google", { session: false }), (req, res) => { // Generate JWT const tokens = generateTokens(req.user.id, req.user.email, req.user.role); // Redirect to frontend with token res.redirect( `${process.env.FRONTEND_URL}/auth/callback?token=${tokens.accessToken}`, ); }, ); ``` ## Authorization Patterns ### Pattern 1: Role-Based Access Control (RBAC) ```typescript enum Role { USER = "user", MODERATOR = "moderator", ADMIN = "admin", } const roleHierarchy: Record<Role, Role[]> = { [Role.ADMIN]: [Role.ADMIN, Role.MODERATOR, Role.USER], [Role.MODERATOR]: [Role.MODERATOR, Role.USER], [Role.USER]: [Role.USER], }; function hasRole(userRole: Role, requiredRole: Role): boolean { return roleHierarchy[userRole].includes(requiredRole); } // Middleware function requireRole(...roles: Role[]) { return (req: Request, res: Response, next: NextFunction) => { if (!req.user) { return res.status(401).json({ error: "Not authenticated" }); } if (!roles.some((role) => hasRole(req.user.role, role))) { return res.status(403).json({ error: "Insufficient permissions" }); } next(); }; } // Usage app.delete( "/api/users/:id", authenticate, requireRole(Role.ADMIN), async (req, res) => { // Only admins can delete users await db.users.delete(req.params.id); res.json({ message: "User deleted" }); }, ); ``` ### Pattern 2: Permission-Based Access Control ```typescript enum Permission { READ_USERS = "read:users", WRITE_USERS = "write:users", DELETE_USERS = "delete:users", READ_POSTS = "read:posts", WRITE_POSTS = "write:posts", } const rolePermissions: Record<Role, Permission[]> = { [Role.USER]: [Permission.READ_POSTS, Permission.WRITE_POSTS], [Role.MODERATOR]: [ Permission.READ_POSTS, Permission.WRITE_POSTS, Permission.READ_USERS, ], [Role.ADMIN]: Object.values(Permission), }; function hasPermission(userRole: Role, permission: Permission): boolean { return rolePermissions[userRole]?.includes(permission) ?? false; } function requirePermission(...permissions: Permission[]) { return (req: Request, res: Response, next: NextFunction) => { if (!req.user) { return res.status(401).json({ error: "Not authenticated" }); } const hasAllPermissions = permissions.every((permission) => hasPermission(req.user.role, permission), ); if (!hasAllPermissions) { return res.status(403).json({ error: "Insufficient permissions" }); } next(); }; } // Usage app.get( "/api/users", authenticate, requirePermission(Permission.READ_USERS), async (req, res) => { const users = await db.users.findAll(); res.json({ users }); }, ); ``` ### Pattern 3: Resource Ownership ```typescript // Check if user owns resource async function requireOwnership( resourceType: "post" | "comment", resourceIdParam: string = "id", ) { return async (req: Request, res: Response, next: NextFunction) => { if (!req.user) { return res.status(401).json({ error: "Not authenticated" }); } const resourceId = req.params[resourceIdParam]; // Admins can access anything if (req.user.role === Role.ADMIN) { return next(); } // Check ownership let resource; if (resourceType === "post") { resource = await db.posts.findById(resourceId); } else if (resourceType === "comment") { resource = await db.comments.findById(resourceId); } if (!resource) { return res.status(404).json({ error: "Resource not found" }); } if (resource.userId !== req.user.userId) { return res.status(403).json({ error: "Not authorized" }); } next(); }; } // Usage app.put( "/api/posts/:id", authenticate, requireOwnership("post"), async (req, res) => { // User can only update their own posts const post = await db.posts.update(req.params.id, req.body); res.json({ post }); }, ); ``` ## Security Best Practices ### Pattern 1: Password Security ```typescript import bcrypt from "bcrypt"; import { z } from "zod"; // Password validation schema const passwordSchema = z .string() .min(12, "Password must be at least 12 characters") .regex(/[A-Z]/, "Password must contain uppercase letter") .regex(/[a-z]/, "Password must contain lowercase letter") .regex(/[0-9]/, "Password must contain number") .regex(/[^A-Za-z0-9]/, "Password must contain special character"); // Hash password async function hashPassword(password: string): Promise<string> { const saltRounds = 12; // 2^12 iterations return bcrypt.hash(password, saltRounds); } // Verify password async function verifyPassword( password: string, hash: string, ): Promise<boolean> { return bcrypt.compare(password, hash); } // Registration with password validation app.post("/api/auth/register", async (req, res) => { try { const { email, password } = req.body; // Validate password passwordSchema.parse(password); // Check if user exists const existingUser = await db.users.findOne({ email }); if (existingUser) { return res.status(400).json({ error: "Email already registered" }); } // Hash password const passwordHash = await hashPassword(password); // Create user const user = await db.users.create({ email, passwordHash, }); // Generate tokens const tokens = generateTokens(user.id, user.email, user.role); res.status(201).json({ user: { id: user.id, email: user.email }, ...tokens, }); } catch (error) { if (error instanceof z.ZodError) { return res.status(400).json({ error: error.errors[0].message }); } res.status(500).json({ error: "Registration failed" }); } }); ``` ### Pattern 2: Rate Limiting ```typescript import rateLimit from "express-rate-limit"; import RedisStore from "rate-limit-redis"; // Login rate limiter const loginLimiter = rateLimit({ store: new RedisStore({ client: redisClient }), windowMs: 15 * 60 * 1000, // 15 minutes max: 5, // 5 attempts message: "Too many login attempts, please try again later", standardHeaders: true, legacyHeaders: false, }); // API rate limiter const apiLimiter = rateLimit({ windowMs: 60 * 1000, // 1 minute max: 100, // 100 requests per minute standardHeaders: true, }); // Apply to routes app.post("/api/auth/login", loginLimiter, async (req, res) => { // Login logic }); app.use("/api/", apiLimiter); ``` ## Best Practices 1. **Never Store Plain Passwords**: Always hash with bcrypt/argon2 2. **Use HTTPS**: Encrypt data in transit 3. **Short-Lived Access Tokens**: 15-30 minutes max 4. **Secure Cookies**: httpOnly, secure, sameSite flags 5. **Validate All Input**: Email format, password strength 6. **Rate Limit Auth Endpoints**: Prevent brute force attacks 7. **Implement CSRF Protection**: For session-based auth 8. **Rotate Secrets Regularly**: JWT secrets, session secrets 9. **Log Security Events**: Login attempts, failed auth 10. **Use MFA When Possible**: Extra security layer ## Common Pitfalls - **Weak Passwords**: Enforce strong password policies - **JWT in localStorage**: Vulnerable to XSS, use httpOnly cookies - **No Token Expiration**: Tokens should expire - **Client-Side Auth Checks Only**: Always validate server-side - **Insecure Password Reset**: Use secure tokens with expiration - **No Rate Limiting**: Vulnerable to brute force - **Trusting Client Data**: Always validate on server ## Resources - **references/jwt-best-practices.md**: JWT implementation guide - **references/oauth2-flows.md**: OAuth2 flow diagrams and examples - **references/session-security.md**: Secure session management - **assets/auth-security-checklist.md**: Security review checklist - **assets/password-policy-template.md**: Password requirements template - **scripts/token-validator.ts**: JWT validation utility

# .NET Backend Development Patterns Master C#/.NET patterns for building production-grade APIs, MCP servers, and enterprise backends with modern best practices (2024/2025). ## When to Use This Skill - Developing new .NET Web APIs or MCP servers - Reviewing C# code for quality and performance - Designing service architectures with dependency injection - Implementing caching strategies with Redis - Writing unit and integration tests - Optimizing database access with EF Core or Dapper - Configuring applications with IOptions pattern - Handling errors and implementing resilience patterns ## Core Concepts ### 1. Project Structure (Clean Architecture) ``` src/ ├── Domain/ # Core business logic (no dependencies) │ ├── Entities/ │ ├── Interfaces/ │ ├── Exceptions/ │ └── ValueObjects/ ├── Application/ # Use cases, DTOs, validation │ ├── Services/ │ ├── DTOs/ │ ├── Validators/ │ └── Interfaces/ ├── Infrastructure/ # External implementations │ ├── Data/ # EF Core, Dapper repositories │ ├── Caching/ # Redis, Memory cache │ ├── External/ # HTTP clients, third-party APIs │ └── DependencyInjection/ # Service registration └── Api/ # Entry point ├── Controllers/ # Or MinimalAPI endpoints ├── Middleware/ ├── Filters/ └── Program.cs ``` ### 2. Dependency Injection Patterns ```csharp // Service registration by lifetime public static class ServiceCollectionExtensions { public static IServiceCollection AddApplicationServices( this IServiceCollection services, IConfiguration configuration) { // Scoped: One instance per HTTP request services.AddScoped<IProductService, ProductService>(); services.AddScoped<IOrderService, OrderService>(); // Singleton: One instance for app lifetime services.AddSingleton<ICacheService, RedisCacheService>(); services.AddSingleton<IConnectionMultiplexer>(_ => ConnectionMultiplexer.Connect(configuration["Redis:Connection"]!)); // Transient: New instance every time services.AddTransient<IValidator<CreateOrderRequest>, CreateOrderValidator>(); // Options pattern for configuration services.Configure<CatalogOptions>(configuration.GetSection("Catalog")); services.Configure<RedisOptions>(configuration.GetSection("Redis")); // Factory pattern for conditional creation services.AddScoped<IPriceCalculator>(sp => { var options = sp.GetRequiredService<IOptions<PricingOptions>>().Value; return options.UseNewEngine ? sp.GetRequiredService<NewPriceCalculator>() : sp.GetRequiredService<LegacyPriceCalculator>(); }); // Keyed services (.NET 8+) services.AddKeyedScoped<IPaymentProcessor, StripeProcessor>("stripe"); services.AddKeyedScoped<IPaymentProcessor, PayPalProcessor>("paypal"); return services; } } // Usage with keyed services public class CheckoutService { public CheckoutService( [FromKeyedServices("stripe")] IPaymentProcessor stripeProcessor) { _processor = stripeProcessor; } } ``` ### 3. Async/Await Patterns ```csharp // ✅ CORRECT: Async all the way down public async Task<Product> GetProductAsync(string id, CancellationToken ct = default) { return await _repository.GetByIdAsync(id, ct); } // ✅ CORRECT: Parallel execution with WhenAll public async Task<(Stock, Price)> GetStockAndPriceAsync( string productId, CancellationToken ct = default) { var stockTask = _stockService.GetAsync(productId, ct); var priceTask = _priceService.GetAsync(productId, ct); await Task.WhenAll(stockTask, priceTask); return (await stockTask, await priceTask); } // ✅ CORRECT: ConfigureAwait in libraries public async Task<T> LibraryMethodAsync<T>(CancellationToken ct = default) { var result = await _httpClient.GetAsync(url, ct).ConfigureAwait(false); return await result.Content.ReadFromJsonAsync<T>(ct).ConfigureAwait(false); } // ✅ CORRECT: ValueTask for hot paths with caching public ValueTask<Product?> GetCachedProductAsync(string id) { if (_cache.TryGetValue(id, out Product? product)) return ValueTask.FromResult(product); return new ValueTask<Product?>(GetFromDatabaseAsync(id)); } // ❌ WRONG: Blocking on async (deadlock risk) var result = GetProductAsync(id).Result; // NEVER do this var result2 = GetProductAsync(id).GetAwaiter().GetResult(); // Also bad // ❌ WRONG: async void (except event handlers) public async void ProcessOrder() { } // Exceptions are lost // ❌ WRONG: Unnecessary Task.Run for already async code await Task.Run(async () => await GetDataAsync()); // Wastes thread ``` ### 4. Configuration with IOptions ```csharp // Configuration classes public class CatalogOptions { public const string SectionName = "Catalog"; public int DefaultPageSize { get; set; } = 50; public int MaxPageSize { get; set; } = 200; public TimeSpan CacheDuration { get; set; } = TimeSpan.FromMinutes(15); public bool EnableEnrichment { get; set; } = true; } public class RedisOptions { public const string SectionName = "Redis"; public string Connection { get; set; } = "localhost:6379"; public string KeyPrefix { get; set; } = "mcp:"; public int Database { get; set; } = 0; } // appsettings.json { "Catalog": { "DefaultPageSize": 50, "MaxPageSize": 200, "CacheDuration": "00:15:00", "EnableEnrichment": true }, "Redis": { "Connection": "localhost:6379", "KeyPrefix": "mcp:", "Database": 0 } } // Registration services.Configure<CatalogOptions>(configuration.GetSection(CatalogOptions.SectionName)); services.Configure<RedisOptions>(configuration.GetSection(RedisOptions.SectionName)); // Usage with IOptions (singleton, read once at startup) public class CatalogService { private readonly CatalogOptions _options; public CatalogService(IOptions<CatalogOptions> options) { _options = options.Value; } } // Usage with IOptionsSnapshot (scoped, re-reads on each request) public class DynamicService { private readonly CatalogOptions _options; public DynamicService(IOptionsSnapshot<CatalogOptions> options) { _options = options.Value; // Fresh value per request } } // Usage with IOptionsMonitor (singleton, notified on changes) public class MonitoredService { private CatalogOptions _options; public MonitoredService(IOptionsMonitor<CatalogOptions> monitor) { _options = monitor.CurrentValue; monitor.OnChange(newOptions => _options = newOptions); } } ``` ### 5. Result Pattern (Avoiding Exceptions for Flow Control) ```csharp // Generic Result type public class Result<T> { public bool IsSuccess { get; } public T? Value { get; } public string? Error { get; } public string? ErrorCode { get; } private Result(bool isSuccess, T? value, string? error, string? errorCode) { IsSuccess = isSuccess; Value = value; Error = error; ErrorCode = errorCode; } public static Result<T> Success(T value) => new(true, value, null, null); public static Result<T> Failure(string error, string? code = null) => new(false, default, error, code); public Result<TNew> Map<TNew>(Func<T, TNew> mapper) => IsSuccess ? Result<TNew>.Success(mapper(Value!)) : Result<TNew>.Failure(Error!, ErrorCode); public async Task<Result<TNew>> MapAsync<TNew>(Func<T, Task<TNew>> mapper) => IsSuccess ? Result<TNew>.Success(await mapper(Value!)) : Result<TNew>.Failure(Error!, ErrorCode); } // Usage in service public async Task<Result<Order>> CreateOrderAsync(CreateOrderRequest request, CancellationToken ct) { // Validation var validation = await _validator.ValidateAsync(request, ct); if (!validation.IsValid) return Result<Order>.Failure( validation.Errors.First().ErrorMessage, "VALIDATION_ERROR"); // Business rule check var stock = await _stockService.CheckAsync(request.ProductId, request.Quantity, ct); if (!stock.IsAvailable) return Result<Order>.Failure( $"Insufficient stock: {stock.Available} available, {request.Quantity} requested", "INSUFFICIENT_STOCK"); // Create order var order = await _repository.CreateAsync(request.ToEntity(), ct); return Result<Order>.Success(order); } // Usage in controller/endpoint app.MapPost("/orders", async ( CreateOrderRequest request, IOrderService orderService, CancellationToken ct) => { var result = await orderService.CreateOrderAsync(request, ct); return result.IsSuccess ? Results.Created($"/orders/{result.Value!.Id}", result.Value) : Results.BadRequest(new { error = result.Error, code = result.ErrorCode }); }); ``` ## Data Access Patterns ### Entity Framework Core ```csharp // DbContext configuration public class AppDbContext : DbContext { public DbSet<Product> Products => Set<Product>(); public DbSet<Order> Orders => Set<Order>(); protected override void OnModelCreating(ModelBuilder modelBuilder) { // Apply all configurations from assembly modelBuilder.ApplyConfigurationsFromAssembly(typeof(AppDbContext).Assembly); // Global query filters modelBuilder.Entity<Product>().HasQueryFilter(p => !p.IsDeleted); } } // Entity configuration public class ProductConfiguration : IEntityTypeConfiguration<Product> { public void Configure(EntityTypeBuilder<Product> builder) { builder.ToTable("Products"); builder.HasKey(p => p.Id); builder.Property(p => p.Id).HasMaxLength(40); builder.Property(p => p.Name).HasMaxLength(200).IsRequired(); builder.Property(p => p.Price).HasPrecision(18, 2); builder.HasIndex(p => p.Sku).IsUnique(); builder.HasIndex(p => new { p.CategoryId, p.Name }); builder.HasMany(p => p.OrderItems) .WithOne(oi => oi.Product) .HasForeignKey(oi => oi.ProductId); } } // Repository with EF Core public class ProductRepository : IProductRepository { private readonly AppDbContext _context; public async Task<Product?> GetByIdAsync(string id, CancellationToken ct = default) { return await _context.Products .AsNoTracking() .FirstOrDefaultAsync(p => p.Id == id, ct); } public async Task<IReadOnlyList<Product>> SearchAsync( ProductSearchCriteria criteria, CancellationToken ct = default) { var query = _context.Products.AsNoTracking(); if (!string.IsNullOrWhiteSpace(criteria.SearchTerm)) query = query.Where(p => EF.Functions.Like(p.Name, $"%{criteria.SearchTerm}%")); if (criteria.CategoryId.HasValue) query = query.Where(p => p.CategoryId == criteria.CategoryId); if (criteria.MinPrice.HasValue) query = query.Where(p => p.Price >= criteria.MinPrice); if (criteria.MaxPrice.HasValue) query = query.Where(p => p.Price <= criteria.MaxPrice); return await query .OrderBy(p => p.Name) .Skip((criteria.Page - 1) * criteria.PageSize) .Take(criteria.PageSize) .ToListAsync(ct); } } ``` ### Dapper for Performance ```csharp public class DapperProductRepository : IProductRepository { private readonly IDbConnection _connection; public async Task<Product?> GetByIdAsync(string id, CancellationToken ct = default) { const string sql = """ SELECT Id, Name, Sku, Price, CategoryId, Stock, CreatedAt FROM Products WHERE Id = @Id AND IsDeleted = 0 """; return await _connection.QueryFirstOrDefaultAsync<Product>( new CommandDefinition(sql, new { Id = id }, cancellationToken: ct)); } public async Task<IReadOnlyList<Product>> SearchAsync( ProductSearchCriteria criteria, CancellationToken ct = default) { var sql = new StringBuilder(""" SELECT Id, Name, Sku, Price, CategoryId, Stock, CreatedAt FROM Products WHERE IsDeleted = 0 """); var parameters = new DynamicParameters(); if (!string.IsNullOrWhiteSpace(criteria.SearchTerm)) { sql.Append(" AND Name LIKE @SearchTerm"); parameters.Add("SearchTerm", $"%{criteria.SearchTerm}%"); } if (criteria.CategoryId.HasValue) { sql.Append(" AND CategoryId = @CategoryId"); parameters.Add("CategoryId", criteria.CategoryId); } if (criteria.MinPrice.HasValue) { sql.Append(" AND Price >= @MinPrice"); parameters.Add("MinPrice", criteria.MinPrice); } if (criteria.MaxPrice.HasValue) { sql.Append(" AND Price <= @MaxPrice"); parameters.Add("MaxPrice", criteria.MaxPrice); } sql.Append(" ORDER BY Name OFFSET @Offset ROWS FETCH NEXT @PageSize ROWS ONLY"); parameters.Add("Offset", (criteria.Page - 1) * criteria.PageSize); parameters.Add("PageSize", criteria.PageSize); var results = await _connection.QueryAsync<Product>( new CommandDefinition(sql.ToString(), parameters, cancellationToken: ct)); return results.ToList(); } // Multi-mapping for related data public async Task<Order?> GetOrderWithItemsAsync(int orderId, CancellationToken ct = default) { const string sql = """ SELECT o.*, oi.*, p.* FROM Orders o LEFT JOIN OrderItems oi ON o.Id = oi.OrderId LEFT JOIN Products p ON oi.ProductId = p.Id WHERE o.Id = @OrderId """; var orderDictionary = new Dictionary<int, Order>(); await _connection.QueryAsync<Order, OrderItem, Product, Order>( new CommandDefinition(sql, new { OrderId = orderId }, cancellationToken: ct), (order, item, product) => { if (!orderDictionary.TryGetValue(order.Id, out var existingOrder)) { existingOrder = order; existingOrder.Items = new List<OrderItem>(); orderDictionary.Add(order.Id, existingOrder); } if (item != null) { item.Product = product; existingOrder.Items.Add(item); } return existingOrder; }, splitOn: "Id,Id"); return orderDictionary.Values.FirstOrDefault(); } } ``` ## Caching Patterns ### Multi-Level Cache with Redis ```csharp public class CachedProductService : IProductService { private readonly IProductRepository _repository; private readonly IMemoryCache _memoryCache; private readonly IDistributedCache _distributedCache; private readonly ILogger<CachedProductService> _logger; private static readonly TimeSpan MemoryCacheDuration = TimeSpan.FromMinutes(1); private static readonly TimeSpan DistributedCacheDuration = TimeSpan.FromMinutes(15); public async Task<Product?> GetByIdAsync(string id, CancellationToken ct = default) { var cacheKey = $"product:{id}"; // L1: Memory cache (in-process, fastest) if (_memoryCache.TryGetValue(cacheKey, out Product? cached)) { _logger.LogDebug("L1 cache hit for {CacheKey}", cacheKey); return cached; } // L2: Distributed cache (Redis) var distributed = await _distributedCache.GetStringAsync(cacheKey, ct); if (distributed != null) { _logger.LogDebug("L2 cache hit for {CacheKey}", cacheKey); var product = JsonSerializer.Deserialize<Product>(distributed); // Populate L1 _memoryCache.Set(cacheKey, product, MemoryCacheDuration); return product; } // L3: Database _logger.LogDebug("Cache miss for {CacheKey}, fetching from database", cacheKey); var fromDb = await _repository.GetByIdAsync(id, ct); if (fromDb != null) { var serialized = JsonSerializer.Serialize(fromDb); // Populate both caches await _distributedCache.SetStringAsync( cacheKey, serialized, new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = DistributedCacheDuration }, ct); _memoryCache.Set(cacheKey, fromDb, MemoryCacheDuration); } return fromDb; } public async Task InvalidateAsync(string id, CancellationToken ct = default) { var cacheKey = $"product:{id}"; _memoryCache.Remove(cacheKey); await _distributedCache.RemoveAsync(cacheKey, ct); _logger.LogInformation("Invalidated cache for {CacheKey}", cacheKey); } } // Stale-while-revalidate pattern public class StaleWhileRevalidateCache<T> { private readonly IDistributedCache _cache; private readonly TimeSpan _freshDuration; private readonly TimeSpan _staleDuration; public async Task<T?> GetOrCreateAsync( string key, Func<CancellationToken, Task<T>> factory, CancellationToken ct = default) { var cached = await _cache.GetStringAsync(key, ct); if (cached != null) { var entry = JsonSerializer.Deserialize<CacheEntry<T>>(cached)!; if (entry.IsStale && !entry.IsExpired) { // Return stale data immediately, refresh in background _ = Task.Run(async () => { var fresh = await factory(CancellationToken.None); await SetAsync(key, fresh, CancellationToken.None); }); } if (!entry.IsExpired) return entry.Value; } // Cache miss or expired var value = await factory(ct); await SetAsync(key, value, ct); return value; } private record CacheEntry<TValue>(TValue Value, DateTime CreatedAt) { public bool IsStale => DateTime.UtcNow - CreatedAt > _freshDuration; public bool IsExpired => DateTime.UtcNow - CreatedAt > _staleDuration; } } ``` ## Testing Patterns ### Unit Tests with xUnit and Moq ```csharp public class OrderServiceTests { private readonly Mock<IOrderRepository> _mockRepository; private readonly Mock<IStockService> _mockStockService; private readonly Mock<IValidator<CreateOrderRequest>> _mockValidator; private readonly OrderService _sut; // System Under Test public OrderServiceTests() { _mockRepository = new Mock<IOrderRepository>(); _mockStockService = new Mock<IStockService>(); _mockValidator = new Mock<IValidator<CreateOrderRequest>>(); // Default: validation passes _mockValidator .Setup(v => v.ValidateAsync(It.IsAny<CreateOrderRequest>(), It.IsAny<

# Angular Migration Master AngularJS to Angular migration, including hybrid apps, component conversion, dependency injection changes, and routing migration. ## When to Use This Skill - Migrating AngularJS (1.x) applications to Angular (2+) - Running hybrid AngularJS/Angular applications - Converting directives to components - Modernizing dependency injection - Migrating routing systems - Updating to latest Angular versions - Implementing Angular best practices ## Migration Strategies ### 1. Big Bang (Complete Rewrite) - Rewrite entire app in Angular - Parallel development - Switch over at once - **Best for:** Small apps, green field projects ### 2. Incremental (Hybrid Approach) - Run AngularJS and Angular side-by-side - Migrate feature by feature - ngUpgrade for interop - **Best for:** Large apps, continuous delivery ### 3. Vertical Slice - Migrate one feature completely - New features in Angular, maintain old in AngularJS - Gradually replace - **Best for:** Medium apps, distinct features ## Hybrid App Setup ```typescript // main.ts - Bootstrap hybrid app import { platformBrowserDynamic } from "@angular/platform-browser-dynamic"; import { UpgradeModule } from "@angular/upgrade/static"; import { AppModule } from "./app/app.module"; platformBrowserDynamic() .bootstrapModule(AppModule) .then((platformRef) => { const upgrade = platformRef.injector.get(UpgradeModule); // Bootstrap AngularJS upgrade.bootstrap(document.body, ["myAngularJSApp"], { strictDi: true }); }); ``` ```typescript // app.module.ts import { NgModule } from "@angular/core"; import { BrowserModule } from "@angular/platform-browser"; import { UpgradeModule } from "@angular/upgrade/static"; @NgModule({ imports: [BrowserModule, UpgradeModule], }) export class AppModule { constructor(private upgrade: UpgradeModule) {} ngDoBootstrap() { // Bootstrapped manually in main.ts } } ``` ## Component Migration ### AngularJS Controller → Angular Component ```javascript // Before: AngularJS controller angular .module("myApp") .controller("UserController", function ($scope, UserService) { $scope.user = {}; $scope.loadUser = function (id) { UserService.getUser(id).then(function (user) { $scope.user = user; }); }; $scope.saveUser = function () { UserService.saveUser($scope.user); }; }); ``` ```typescript // After: Angular component import { Component, OnInit } from "@angular/core"; import { UserService } from "./user.service"; @Component({ selector: "app-user", template: ` <div> <h2>{{ user.name }}</h2> <button (click)="saveUser()">Save</button> </div> `, }) export class UserComponent implements OnInit { user: any = {}; constructor(private userService: UserService) {} ngOnInit() { this.loadUser(1); } loadUser(id: number) { this.userService.getUser(id).subscribe((user) => { this.user = user; }); } saveUser() { this.userService.saveUser(this.user); } } ``` ### AngularJS Directive → Angular Component ```javascript // Before: AngularJS directive angular.module("myApp").directive("userCard", function () { return { restrict: "E", scope: { user: "=", onDelete: "&", }, template: ` <div class="card"> <h3>{{ user.name }}</h3> <button ng-click="onDelete()">Delete</button> </div> `, }; }); ``` ```typescript // After: Angular component import { Component, Input, Output, EventEmitter } from "@angular/core"; @Component({ selector: "app-user-card", template: ` <div class="card"> <h3>{{ user.name }}</h3> <button (click)="delete.emit()">Delete</button> </div> `, }) export class UserCardComponent { @Input() user: any; @Output() delete = new EventEmitter<void>(); } // Usage: <app-user-card [user]="user" (delete)="handleDelete()"></app-user-card> ``` ## Service Migration ```javascript // Before: AngularJS service angular.module("myApp").factory("UserService", function ($http) { return { getUser: function (id) { return $http.get("/api/users/" + id); }, saveUser: function (user) { return $http.post("/api/users", user); }, }; }); ``` ```typescript // After: Angular service import { Injectable } from "@angular/core"; import { HttpClient } from "@angular/common/http"; import { Observable } from "rxjs"; @Injectable({ providedIn: "root", }) export class UserService { constructor(private http: HttpClient) {} getUser(id: number): Observable<any> { return this.http.get(`/api/users/${id}`); } saveUser(user: any): Observable<any> { return this.http.post("/api/users", user); } } ``` ## Dependency Injection Changes ### Downgrading Angular → AngularJS ```typescript // Angular service import { Injectable } from "@angular/core"; @Injectable({ providedIn: "root" }) export class NewService { getData() { return "data from Angular"; } } // Make available to AngularJS import { downgradeInjectable } from "@angular/upgrade/static"; angular.module("myApp").factory("newService", downgradeInjectable(NewService)); // Use in AngularJS angular.module("myApp").controller("OldController", function (newService) { console.log(newService.getData()); }); ``` ### Upgrading AngularJS → Angular ```typescript // AngularJS service angular.module('myApp').factory('oldService', function() { return { getData: function() { return 'data from AngularJS'; } }; }); // Make available to Angular import { InjectionToken } from '@angular/core'; export const OLD_SERVICE = new InjectionToken<any>('oldService'); @NgModule({ providers: [ { provide: OLD_SERVICE, useFactory: (i: any) => i.get('oldService'), deps: ['$injector'] } ] }) // Use in Angular @Component({...}) export class NewComponent { constructor(@Inject(OLD_SERVICE) private oldService: any) { console.log(this.oldService.getData()); } } ``` ## Routing Migration ```javascript // Before: AngularJS routing angular.module("myApp").config(function ($routeProvider) { $routeProvider .when("/users", { template: "<user-list></user-list>", }) .when("/users/:id", { template: "<user-detail></user-detail>", }); }); ``` ```typescript // After: Angular routing import { NgModule } from "@angular/core"; import { RouterModule, Routes } from "@angular/router"; const routes: Routes = [ { path: "users", component: UserListComponent }, { path: "users/:id", component: UserDetailComponent }, ]; @NgModule({ imports: [RouterModule.forRoot(routes)], exports: [RouterModule], }) export class AppRoutingModule {} ``` ## Forms Migration ```html <!-- Before: AngularJS --> <form name="userForm" ng-submit="saveUser()"> <input type="text" ng-model="user.name" required /> <input type="email" ng-model="user.email" required /> <button ng-disabled="userForm.$invalid">Save</button> </form> ``` ```typescript // After: Angular (Template-driven) @Component({ template: ` <form #userForm="ngForm" (ngSubmit)="saveUser()"> <input type="text" [(ngModel)]="user.name" name="name" required> <input type="email" [(ngModel)]="user.email" name="email" required> <button [disabled]="userForm.invalid">Save</button> </form> ` }) // Or Reactive Forms (preferred) import { FormBuilder, FormGroup, Validators } from '@angular/forms'; @Component({ template: ` <form [formGroup]="userForm" (ngSubmit)="saveUser()"> <input formControlName="name"> <input formControlName="email"> <button [disabled]="userForm.invalid">Save</button> </form> ` }) export class UserFormComponent { userForm: FormGroup; constructor(private fb: FormBuilder) { this.userForm = this.fb.group({ name: ['', Validators.required], email: ['', [Validators.required, Validators.email]] }); } saveUser() { console.log(this.userForm.value); } } ``` ## Migration Timeline ``` Phase 1: Setup (1-2 weeks) - Install Angular CLI - Set up hybrid app - Configure build tools - Set up testing Phase 2: Infrastructure (2-4 weeks) - Migrate services - Migrate utilities - Set up routing - Migrate shared components Phase 3: Feature Migration (varies) - Migrate feature by feature - Test thoroughly - Deploy incrementally Phase 4: Cleanup (1-2 weeks) - Remove AngularJS code - Remove ngUpgrade - Optimize bundle - Final testing ``` ## Resources - **references/hybrid-mode.md**: Hybrid app patterns - **references/component-migration.md**: Component conversion guide - **references/dependency-injection.md**: DI migration strategies - **references/routing.md**: Routing migration - **assets/hybrid-bootstrap.ts**: Hybrid app template - **assets/migration-timeline.md**: Project planning - **scripts/analyze-angular-app.sh**: App analysis script ## Best Practices 1. **Start with Services**: Migrate services first (easier) 2. **Incremental Approach**: Feature-by-feature migration 3. **Test Continuously**: Test at every step 4. **Use TypeScript**: Migrate to TypeScript early 5. **Follow Style Guide**: Angular style guide from day 1 6. **Optimize Later**: Get it working, then optimize 7. **Document**: Keep migration notes ## Common Pitfalls - Not setting up hybrid app correctly - Migrating UI before logic - Ignoring change detection differences - Not handling scope properly - Mixing patterns (AngularJS + Angular) - Inadequate testing

# Dependency Upgrade Master major dependency version upgrades, compatibility analysis, staged upgrade strategies, and comprehensive testing approaches. ## When to Use This Skill - Upgrading major framework versions - Updating security-vulnerable dependencies - Modernizing legacy dependencies - Resolving dependency conflicts - Planning incremental upgrade paths - Testing compatibility matrices - Automating dependency updates ## Semantic Versioning Review ``` MAJOR.MINOR.PATCH (e.g., 2.3.1) MAJOR: Breaking changes MINOR: New features, backward compatible PATCH: Bug fixes, backward compatible ^2.3.1 = >=2.3.1 <3.0.0 (minor updates) ~2.3.1 = >=2.3.1 <2.4.0 (patch updates) 2.3.1 = exact version ``` ## Dependency Analysis ### Audit Dependencies ```bash # npm npm outdated npm audit npm audit fix # yarn yarn outdated yarn audit # Check for major updates npx npm-check-updates npx npm-check-updates -u # Update package.json ``` ### Analyze Dependency Tree ```bash # See why a package is installed npm ls package-name yarn why package-name # Find duplicate packages npm dedupe yarn dedupe # Visualize dependencies npx madge --image graph.png src/ ``` ## Compatibility Matrix ```javascript // compatibility-matrix.js const compatibilityMatrix = { react: { "16.x": { "react-dom": "^16.0.0", "react-router-dom": "^5.0.0", "@testing-library/react": "^11.0.0", }, "17.x": { "react-dom": "^17.0.0", "react-router-dom": "^5.0.0 || ^6.0.0", "@testing-library/react": "^12.0.0", }, "18.x": { "react-dom": "^18.0.0", "react-router-dom": "^6.0.0", "@testing-library/react": "^13.0.0", }, }, }; function checkCompatibility(packages) { // Validate package versions against matrix } ``` ## Staged Upgrade Strategy ### Phase 1: Planning ```bash # 1. Identify current versions npm list --depth=0 # 2. Check for breaking changes # Read CHANGELOG.md and MIGRATION.md # 3. Create upgrade plan echo "Upgrade order: 1. TypeScript 2. React 3. React Router 4. Testing libraries 5. Build tools" > UPGRADE_PLAN.md ``` ### Phase 2: Incremental Updates ```bash # Don't upgrade everything at once! # Step 1: Update TypeScript npm install typescript@latest # Test npm run test npm run build # Step 2: Update React (one major version at a time) npm install react@17 react-dom@17 # Test again npm run test # Step 3: Continue with other packages npm install react-router-dom@6 # And so on... ``` ### Phase 3: Validation ```javascript // tests/compatibility.test.js describe("Dependency Compatibility", () => { it("should have compatible React versions", () => { const reactVersion = require("react/package.json").version; const reactDomVersion = require("react-dom/package.json").version; expect(reactVersion).toBe(reactDomVersion); }); it("should not have peer dependency warnings", () => { // Run npm ls and check for warnings }); }); ``` ## Breaking Change Handling ### Identifying Breaking Changes ```bash # Check the changelog directly curl https://raw.githubusercontent.com/facebook/react/master/CHANGELOG.md ``` ### Codemod for Automated Fixes ```bash # Run jscodeshift with transform URL npx jscodeshift -t <transform-url> <path> # Example: Rename unsafe lifecycle methods npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js src/ # For TypeScript files npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js --parser=tsx src/ # Dry run to preview changes npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js --dry src/ ``` ### Custom Migration Script ```javascript // migration-script.js const fs = require("fs"); const glob = require("glob"); glob("src/**/*.tsx", (err, files) => { files.forEach((file) => { let content = fs.readFileSync(file, "utf8"); // Replace old API with new API content = content.replace( /componentWillMount/g, "UNSAFE_componentWillMount", ); // Update imports content = content.replace( /import { Component } from 'react'/g, "import React, { Component } from 'react'", ); fs.writeFileSync(file, content); }); }); ``` ## Testing Strategy ### Unit Tests ```javascript // Ensure tests pass before and after upgrade npm run test // Update test utilities if needed npm install @testing-library/react@latest ``` ### Integration Tests ```javascript // tests/integration/app.test.js describe("App Integration", () => { it("should render without crashing", () => { render(<App />); }); it("should handle navigation", () => { const { getByText } = render(<App />); fireEvent.click(getByText("Navigate")); expect(screen.getByText("New Page")).toBeInTheDocument(); }); }); ``` ### Visual Regression Tests ```javascript // visual-regression.test.js describe("Visual Regression", () => { it("should match snapshot", () => { const { container } = render(<App />); expect(container.firstChild).toMatchSnapshot(); }); }); ``` ### E2E Tests ```javascript // cypress/e2e/app.cy.js describe("E2E Tests", () => { it("should complete user flow", () => { cy.visit("/"); cy.get('[data-testid="login"]').click(); cy.get('input[name="email"]').type("user@example.com"); cy.get('button[type="submit"]').click(); cy.url().should("include", "/dashboard"); }); }); ``` ## Automated Dependency Updates ### Renovate Configuration ```json // renovate.json { "extends": ["config:base"], "packageRules": [ { "matchUpdateTypes": ["minor", "patch"], "automerge": true }, { "matchUpdateTypes": ["major"], "automerge": false, "labels": ["major-update"] } ], "schedule": ["before 3am on Monday"], "timezone": "America/New_York" } ``` ### Dependabot Configuration ```yaml # .github/dependabot.yml version: 2 updates: - package-ecosystem: "npm" directory: "/" schedule: interval: "weekly" open-pull-requests-limit: 5 reviewers: - "team-leads" commit-message: prefix: "chore" include: "scope" ``` ## Rollback Plan ```javascript // rollback.sh #!/bin/bash # Save current state git stash git checkout -b upgrade-branch # Attempt upgrade npm install package@latest # Run tests if npm run test; then echo "Upgrade successful" git add package.json package-lock.json git commit -m "chore: upgrade package" else echo "Upgrade failed, rolling back" git checkout main git branch -D upgrade-branch npm install # Restore from package-lock.json fi ``` ## Common Upgrade Patterns ### Lock File Management ```bash # npm npm install --package-lock-only # Update lock file only npm ci # Clean install from lock file # yarn yarn install --frozen-lockfile # CI mode yarn upgrade-interactive # Interactive upgrades ``` ### Peer Dependency Resolution ```bash # npm 7+: strict peer dependencies npm install --legacy-peer-deps # Ignore peer deps # npm 8+: override peer dependencies npm install --force ``` ### Workspace Upgrades ```bash # Update all workspace packages npm install --workspaces # Update specific workspace npm install package@latest --workspace=packages/app ``` ## Resources - **references/semver.md**: Semantic versioning guide - **references/compatibility-matrix.md**: Common compatibility issues - **references/staged-upgrades.md**: Incremental upgrade strategies - **references/testing-strategy.md**: Comprehensive testing approaches - **assets/upgrade-checklist.md**: Step-by-step checklist - **assets/compatibility-matrix.csv**: Version compatibility table - **scripts/audit-dependencies.sh**: Dependency audit script ## Best Practices 1. **Read Changelogs**: Understand what changed 2. **Upgrade Incrementally**: One major version at a time 3. **Test Thoroughly**: Unit, integration, E2E tests 4. **Check Peer Dependencies**: Resolve conflicts early 5. **Use Lock Files**: Ensure reproducible installs 6. **Automate Updates**: Use Renovate or Dependabot 7. **Monitor**: Watch for runtime errors post-upgrade 8. **Document**: Keep upgrade notes ## Upgrade Checklist ```markdown Pre-Upgrade: - [ ] Review current dependency versions - [ ] Read changelogs for breaking changes - [ ] Create feature branch - [ ] Backup current state (git tag) - [ ] Run full test suite (baseline) During Upgrade: - [ ] Upgrade one dependency at a time - [ ] Update peer dependencies - [ ] Fix TypeScript errors - [ ] Update tests if needed - [ ] Run test suite after each upgrade - [ ] Check bundle size impact Post-Upgrade: - [ ] Full regression testing - [ ] Performance testing - [ ] Update documentation - [ ] Deploy to staging - [ ] Monitor for errors - [ ] Deploy to production ``` ## Common Pitfalls - Upgrading all dependencies at once - Not testing after each upgrade - Ignoring peer dependency warnings - Forgetting to update lock file - Not reading breaking change notes - Skipping major versions - Not having rollback plan

# React Modernization Master React version upgrades, class to hooks migration, concurrent features adoption, and codemods for automated transformation. ## When to Use This Skill - Upgrading React applications to latest versions - Migrating class components to functional components with hooks - Adopting concurrent React features (Suspense, transitions) - Applying codemods for automated refactoring - Modernizing state management patterns - Updating to TypeScript - Improving performance with React 18+ features ## Version Upgrade Path ### React 16 → 17 → 18 **Breaking Changes by Version:** **React 17:** - Event delegation changes - No event pooling - Effect cleanup timing - JSX transform (no React import needed) **React 18:** - Automatic batching - Concurrent rendering - Strict Mode changes (double invocation) - New root API - Suspense on server ## Class to Hooks Migration ### State Management ```javascript // Before: Class component class Counter extends React.Component { constructor(props) { super(props); this.state = { count: 0, name: "", }; } increment = () => { this.setState({ count: this.state.count + 1 }); }; render() { return ( <div> <p>Count: {this.state.count}</p> <button onClick={this.increment}>Increment</button> </div> ); } } // After: Functional component with hooks function Counter() { const [count, setCount] = useState(0); const [name, setName] = useState(""); const increment = () => { setCount(count + 1); }; return ( <div> <p>Count: {count}</p> <button onClick={increment}>Increment</button> </div> ); } ``` ### Lifecycle Methods to Hooks ```javascript // Before: Lifecycle methods class DataFetcher extends React.Component { state = { data: null, loading: true }; componentDidMount() { this.fetchData(); } componentDidUpdate(prevProps) { if (prevProps.id !== this.props.id) { this.fetchData(); } } componentWillUnmount() { this.cancelRequest(); } fetchData = async () => { const data = await fetch(`/api/${this.props.id}`); this.setState({ data, loading: false }); }; cancelRequest = () => { // Cleanup }; render() { if (this.state.loading) return <div>Loading...</div>; return <div>{this.state.data}</div>; } } // After: useEffect hook function DataFetcher({ id }) { const [data, setData] = useState(null); const [loading, setLoading] = useState(true); useEffect(() => { let cancelled = false; const fetchData = async () => { try { const response = await fetch(`/api/${id}`); const result = await response.json(); if (!cancelled) { setData(result); setLoading(false); } } catch (error) { if (!cancelled) { console.error(error); } } }; fetchData(); // Cleanup function return () => { cancelled = true; }; }, [id]); // Re-run when id changes if (loading) return <div>Loading...</div>; return <div>{data}</div>; } ``` ### Context and HOCs to Hooks ```javascript // Before: Context consumer and HOC const ThemeContext = React.createContext(); class ThemedButton extends React.Component { static contextType = ThemeContext; render() { return ( <button style={{ background: this.context.theme }}> {this.props.children} </button> ); } } // After: useContext hook function ThemedButton({ children }) { const { theme } = useContext(ThemeContext); return <button style={{ background: theme }}>{children}</button>; } // Before: HOC for data fetching function withUser(Component) { return class extends React.Component { state = { user: null }; componentDidMount() { fetchUser().then((user) => this.setState({ user })); } render() { return <Component {...this.props} user={this.state.user} />; } }; } // After: Custom hook function useUser() { const [user, setUser] = useState(null); useEffect(() => { fetchUser().then(setUser); }, []); return user; } function UserProfile() { const user = useUser(); if (!user) return <div>Loading...</div>; return <div>{user.name}</div>; } ``` ## React 18 Concurrent Features ### New Root API ```javascript // Before: React 17 import ReactDOM from "react-dom"; ReactDOM.render(<App />, document.getElementById("root")); // After: React 18 import { createRoot } from "react-dom/client"; const root = createRoot(document.getElementById("root")); root.render(<App />); ``` ### Automatic Batching ```javascript // React 18: All updates are batched function handleClick() { setCount((c) => c + 1); setFlag((f) => !f); // Only one re-render (batched) } // Even in async: setTimeout(() => { setCount((c) => c + 1); setFlag((f) => !f); // Still batched in React 18! }, 1000); // Opt out if needed import { flushSync } from "react-dom"; flushSync(() => { setCount((c) => c + 1); }); // Re-render happens here setFlag((f) => !f); // Another re-render ``` ### Transitions ```javascript import { useState, useTransition } from "react"; function SearchResults() { const [query, setQuery] = useState(""); const [results, setResults] = useState([]); const [isPending, startTransition] = useTransition(); const handleChange = (e) => { // Urgent: Update input immediately setQuery(e.target.value); // Non-urgent: Update results (can be interrupted) startTransition(() => { setResults(searchResults(e.target.value)); }); }; return ( <> <input value={query} onChange={handleChange} /> {isPending && <Spinner />} <Results data={results} /> </> ); } ``` ### Suspense for Data Fetching ```javascript import { Suspense } from "react"; // Resource-based data fetching (with React 18) const resource = fetchProfileData(); function ProfilePage() { return ( <Suspense fallback={<Loading />}> <ProfileDetails /> <Suspense fallback={<Loading />}> <ProfileTimeline /> </Suspense> </Suspense> ); } function ProfileDetails() { // This will suspend if data not ready const user = resource.user.read(); return <h1>{user.name}</h1>; } function ProfileTimeline() { const posts = resource.posts.read(); return <Timeline posts={posts} />; } ``` ## Codemods for Automation ### Run React Codemods ```bash # Rename unsafe lifecycle methods npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js src/ # Update React imports (React 17+) npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/update-react-imports.js src/ # Add error boundaries npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/error-boundaries.js src/ # For TypeScript files npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js --parser=tsx src/ # Dry run to preview changes npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js --dry --print src/ # Class to Hooks (third-party) npx codemod react/hooks/convert-class-to-function src/ ``` ### Custom Codemod Example ```javascript // custom-codemod.js module.exports = function (file, api) { const j = api.jscodeshift; const root = j(file.source); // Find setState calls root .find(j.CallExpression, { callee: { type: "MemberExpression", property: { name: "setState" }, }, }) .forEach((path) => { // Transform to useState // ... transformation logic }); return root.toSource(); }; // Run: jscodeshift -t custom-codemod.js src/ ``` ## Performance Optimization ### useMemo and useCallback ```javascript function ExpensiveComponent({ items, filter }) { // Memoize expensive calculation const filteredItems = useMemo(() => { return items.filter((item) => item.category === filter); }, [items, filter]); // Memoize callback to prevent child re-renders const handleClick = useCallback((id) => { console.log("Clicked:", id); }, []); // No dependencies, never changes return <List items={filteredItems} onClick={handleClick} />; } // Child component with memo const List = React.memo(({ items, onClick }) => { return items.map((item) => ( <Item key={item.id} item={item} onClick={onClick} /> )); }); ``` ### Code Splitting ```javascript import { lazy, Suspense } from "react"; // Lazy load components const Dashboard = lazy(() => import("./Dashboard")); const Settings = lazy(() => import("./Settings")); function App() { return ( <Suspense fallback={<Loading />}> <Routes> <Route path="/dashboard" element={<Dashboard />} /> <Route path="/settings" element={<Settings />} /> </Routes> </Suspense> ); } ``` ## TypeScript Migration ```typescript // Before: JavaScript function Button({ onClick, children }) { return <button onClick={onClick}>{children}</button>; } // After: TypeScript interface ButtonProps { onClick: () => void; children: React.ReactNode; } function Button({ onClick, children }: ButtonProps) { return <button onClick={onClick}>{children}</button>; } // Generic components interface ListProps<T> { items: T[]; renderItem: (item: T) => React.ReactNode; } function List<T>({ items, renderItem }: ListProps<T>) { return <>{items.map(renderItem)}</>; } ``` ## Migration Checklist ```markdown ### Pre-Migration - [ ] Update dependencies incrementally (not all at once) - [ ] Review breaking changes in release notes - [ ] Set up testing suite - [ ] Create feature branch ### Class → Hooks Migration - [ ] Identify class components to migrate - [ ] Start with leaf components (no children) - [ ] Convert state to useState - [ ] Convert lifecycle to useEffect - [ ] Convert context to useContext - [ ] Extract custom hooks - [ ] Test thoroughly ### React 18 Upgrade - [ ] Update to React 17 first (if needed) - [ ] Update react and react-dom to 18 - [ ] Update @types/react if using TypeScript - [ ] Change to createRoot API - [ ] Test with StrictMode (double invocation) - [ ] Address concurrent rendering issues - [ ] Adopt Suspense/Transitions where beneficial ### Performance - [ ] Identify performance bottlenecks - [ ] Add React.memo where appropriate - [ ] Use useMemo/useCallback for expensive operations - [ ] Implement code splitting - [ ] Optimize re-renders ### Testing - [ ] Update test utilities (React Testing Library) - [ ] Test with React 18 features - [ ] Check for warnings in console - [ ] Performance testing ``` ## Resources - **references/breaking-changes.md**: Version-specific breaking changes - **references/codemods.md**: Codemod usage guide - **references/hooks-migration.md**: Comprehensive hooks patterns - **references/concurrent-features.md**: React 18 concurrent features - **assets/codemod-config.json**: Codemod configurations - **assets/migration-checklist.md**: Step-by-step checklist - **scripts/apply-codemods.sh**: Automated codemod script ## Best Practices 1. **Incremental Migration**: Don't migrate everything at once 2. **Test Thoroughly**: Comprehensive testing at each step 3. **Use Codemods**: Automate repetitive transformations 4. **Start Simple**: Begin with leaf components 5. **Leverage StrictMode**: Catch issues early 6. **Monitor Performance**: Measure before and after 7. **Document Changes**: Keep migration log ## Common Pitfalls - Forgetting useEffect dependencies - Over-using useMemo/useCallback - Not handling cleanup in useEffect - Mixing class and functional patterns - Ignoring StrictMode warnings - Breaking change assumptions

# Next.js App Router Patterns Comprehensive patterns for Next.js 14+ App Router architecture, Server Components, and modern full-stack React development. ## When to Use This Skill - Building new Next.js applications with App Router - Migrating from Pages Router to App Router - Implementing Server Components and streaming - Setting up parallel and intercepting routes - Optimizing data fetching and caching - Building full-stack features with Server Actions ## Core Concepts ### 1. Rendering Modes | Mode | Where | When to Use | | --------------------- | ------------ | ----------------------------------------- | | **Server Components** | Server only | Data fetching, heavy computation, secrets | | **Client Components** | Browser | Interactivity, hooks, browser APIs | | **Static** | Build time | Content that rarely changes | | **Dynamic** | Request time | Personalized or real-time data | | **Streaming** | Progressive | Large pages, slow data sources | ### 2. File Conventions ``` app/ ├── layout.tsx # Shared UI wrapper ├── page.tsx # Route UI ├── loading.tsx # Loading UI (Suspense) ├── error.tsx # Error boundary ├── not-found.tsx # 404 UI ├── route.ts # API endpoint ├── template.tsx # Re-mounted layout ├── default.tsx # Parallel route fallback └── opengraph-image.tsx # OG image generation ``` ## Quick Start ```typescript // app/layout.tsx import { Inter } from 'next/font/google' import { Providers } from './providers' const inter = Inter({ subsets: ['latin'] }) export const metadata = { title: { default: 'My App', template: '%s | My App' }, description: 'Built with Next.js App Router', } export default function RootLayout({ children, }: { children: React.ReactNode }) { return ( <html lang="en" suppressHydrationWarning> <body className={inter.className}> <Providers>{children}</Providers> </body> </html> ) } // app/page.tsx - Server Component by default async function getProducts() { const res = await fetch('https://api.example.com/products', { next: { revalidate: 3600 }, // ISR: revalidate every hour }) return res.json() } export default async function HomePage() { const products = await getProducts() return ( <main> <h1>Products</h1> <ProductGrid products={products} /> </main> ) } ``` ## Patterns ### Pattern 1: Server Components with Data Fetching ```typescript // app/products/page.tsx import { Suspense } from 'react' import { ProductList, ProductListSkeleton } from '@/components/products' import { FilterSidebar } from '@/components/filters' interface SearchParams { category?: string sort?: 'price' | 'name' | 'date' page?: string } export default async function ProductsPage({ searchParams, }: { searchParams: Promise<SearchParams> }) { const params = await searchParams return ( <div className="flex gap-8"> <FilterSidebar /> <Suspense key={JSON.stringify(params)} fallback={<ProductListSkeleton />} > <ProductList category={params.category} sort={params.sort} page={Number(params.page) || 1} /> </Suspense> </div> ) } // components/products/ProductList.tsx - Server Component async function getProducts(filters: ProductFilters) { const res = await fetch( `${process.env.API_URL}/products?${new URLSearchParams(filters)}`, { next: { tags: ['products'] } } ) if (!res.ok) throw new Error('Failed to fetch products') return res.json() } export async function ProductList({ category, sort, page }: ProductFilters) { const { products, totalPages } = await getProducts({ category, sort, page }) return ( <div> <div className="grid grid-cols-3 gap-4"> {products.map((product) => ( <ProductCard key={product.id} product={product} /> ))} </div> <Pagination currentPage={page} totalPages={totalPages} /> </div> ) } ``` ### Pattern 2: Client Components with 'use client' ```typescript // components/products/AddToCartButton.tsx 'use client' import { useState, useTransition } from 'react' import { addToCart } from '@/app/actions/cart' export function AddToCartButton({ productId }: { productId: string }) { const [isPending, startTransition] = useTransition() const [error, setError] = useState<string | null>(null) const handleClick = () => { setError(null) startTransition(async () => { const result = await addToCart(productId) if (result.error) { setError(result.error) } }) } return ( <div> <button onClick={handleClick} disabled={isPending} className="btn-primary" > {isPending ? 'Adding...' : 'Add to Cart'} </button> {error && <p className="text-red-500 text-sm">{error}</p>} </div> ) } ``` ### Pattern 3: Server Actions ```typescript // app/actions/cart.ts "use server"; import { revalidateTag } from "next/cache"; import { cookies } from "next/headers"; import { redirect } from "next/navigation"; export async function addToCart(productId: string) { const cookieStore = await cookies(); const sessionId = cookieStore.get("session")?.value; if (!sessionId) { redirect("/login"); } try { await db.cart.upsert({ where: { sessionId_productId: { sessionId, productId } }, update: { quantity: { increment: 1 } }, create: { sessionId, productId, quantity: 1 }, }); revalidateTag("cart"); return { success: true }; } catch (error) { return { error: "Failed to add item to cart" }; } } export async function checkout(formData: FormData) { const address = formData.get("address") as string; const payment = formData.get("payment") as string; // Validate if (!address || !payment) { return { error: "Missing required fields" }; } // Process order const order = await processOrder({ address, payment }); // Redirect to confirmation redirect(`/orders/${order.id}/confirmation`); } ``` ### Pattern 4: Parallel Routes ```typescript // app/dashboard/layout.tsx export default function DashboardLayout({ children, analytics, team, }: { children: React.ReactNode analytics: React.ReactNode team: React.ReactNode }) { return ( <div className="dashboard-grid"> <main>{children}</main> <aside className="analytics-panel">{analytics}</aside> <aside className="team-panel">{team}</aside> </div> ) } // app/dashboard/@analytics/page.tsx export default async function AnalyticsSlot() { const stats = await getAnalytics() return <AnalyticsChart data={stats} /> } // app/dashboard/@analytics/loading.tsx export default function AnalyticsLoading() { return <ChartSkeleton /> } // app/dashboard/@team/page.tsx export default async function TeamSlot() { const members = await getTeamMembers() return <TeamList members={members} /> } ``` ### Pattern 5: Intercepting Routes (Modal Pattern) ```typescript // File structure for photo modal // app/ // ├── @modal/ // │ ├── (.)photos/[id]/page.tsx # Intercept // │ └── default.tsx // ├── photos/ // │ └── [id]/page.tsx # Full page // └── layout.tsx // app/@modal/(.)photos/[id]/page.tsx import { Modal } from '@/components/Modal' import { PhotoDetail } from '@/components/PhotoDetail' export default async function PhotoModal({ params, }: { params: Promise<{ id: string }> }) { const { id } = await params const photo = await getPhoto(id) return ( <Modal> <PhotoDetail photo={photo} /> </Modal> ) } // app/photos/[id]/page.tsx - Full page version export default async function PhotoPage({ params, }: { params: Promise<{ id: string }> }) { const { id } = await params const photo = await getPhoto(id) return ( <div className="photo-page"> <PhotoDetail photo={photo} /> <RelatedPhotos photoId={id} /> </div> ) } // app/layout.tsx export default function RootLayout({ children, modal, }: { children: React.ReactNode modal: React.ReactNode }) { return ( <html> <body> {children} {modal} </body> </html> ) } ``` ### Pattern 6: Streaming with Suspense ```typescript // app/product/[id]/page.tsx import { Suspense } from 'react' export default async function ProductPage({ params, }: { params: Promise<{ id: string }> }) { const { id } = await params // This data loads first (blocking) const product = await getProduct(id) return ( <div> {/* Immediate render */} <ProductHeader product={product} /> {/* Stream in reviews */} <Suspense fallback={<ReviewsSkeleton />}> <Reviews productId={id} /> </Suspense> {/* Stream in recommendations */} <Suspense fallback={<RecommendationsSkeleton />}> <Recommendations productId={id} /> </Suspense> </div> ) } // These components fetch their own data async function Reviews({ productId }: { productId: string }) { const reviews = await getReviews(productId) // Slow API return <ReviewList reviews={reviews} /> } async function Recommendations({ productId }: { productId: string }) { const products = await getRecommendations(productId) // ML-based, slow return <ProductCarousel products={products} /> } ``` ### Pattern 7: Route Handlers (API Routes) ```typescript // app/api/products/route.ts import { NextRequest, NextResponse } from "next/server"; export async function GET(request: NextRequest) { const searchParams = request.nextUrl.searchParams; const category = searchParams.get("category"); const products = await db.product.findMany({ where: category ? { category } : undefined, take: 20, }); return NextResponse.json(products); } export async function POST(request: NextRequest) { const body = await request.json(); const product = await db.product.create({ data: body, }); return NextResponse.json(product, { status: 201 }); } // app/api/products/[id]/route.ts export async function GET( request: NextRequest, { params }: { params: Promise<{ id: string }> }, ) { const { id } = await params; const product = await db.product.findUnique({ where: { id } }); if (!product) { return NextResponse.json({ error: "Product not found" }, { status: 404 }); } return NextResponse.json(product); } ``` ### Pattern 8: Metadata and SEO ```typescript // app/products/[slug]/page.tsx import { Metadata } from 'next' import { notFound } from 'next/navigation' type Props = { params: Promise<{ slug: string }> } export async function generateMetadata({ params }: Props): Promise<Metadata> { const { slug } = await params const product = await getProduct(slug) if (!product) return {} return { title: product.name, description: product.description, openGraph: { title: product.name, description: product.description, images: [{ url: product.image, width: 1200, height: 630 }], }, twitter: { card: 'summary_large_image', title: product.name, description: product.description, images: [product.image], }, } } export async function generateStaticParams() { const products = await db.product.findMany({ select: { slug: true } }) return products.map((p) => ({ slug: p.slug })) } export default async function ProductPage({ params }: Props) { const { slug } = await params const product = await getProduct(slug) if (!product) notFound() return <ProductDetail product={product} /> } ``` ## Caching Strategies ### Data Cache ```typescript // No cache (always fresh) fetch(url, { cache: "no-store" }); // Cache forever (static) fetch(url, { cache: "force-cache" }); // ISR - revalidate after 60 seconds fetch(url, { next: { revalidate: 60 } }); // Tag-based invalidation fetch(url, { next: { tags: ["products"] } }); // Invalidate via Server Action ("use server"); import { revalidateTag, revalidatePath } from "next/cache"; export async function updateProduct(id: string, data: ProductData) { await db.product.update({ where: { id }, data }); revalidateTag("products"); revalidatePath("/products"); } ``` ## Best Practices ### Do's - **Start with Server Components** - Add 'use client' only when needed - **Colocate data fetching** - Fetch data where it's used - **Use Suspense boundaries** - Enable streaming for slow data - **Leverage parallel routes** - Independent loading states - **Use Server Actions** - For mutations with progressive enhancement ### Don'ts - **Don't pass serializable data** - Server → Client boundary limitations - **Don't use hooks in Server Components** - No useState, useEffect - **Don't fetch in Client Components** - Use Server Components or React Query - **Don't over-nest layouts** - Each layout adds to the component tree - **Don't ignore loading states** - Always provide loading.tsx or Suspense ## Resources - [Next.js App Router Documentation](https://nextjs.org/docs/app) - [Server Components RFC](https://github.com/reactjs/rfcs/blob/main/text/0188-server-components.md) - [Vercel Templates](https://vercel.com/templates/next.js)

# React State Management Comprehensive guide to modern React state management patterns, from local component state to global stores and server state synchronization. ## When to Use This Skill - Setting up global state management in a React app - Choosing between Redux Toolkit, Zustand, or Jotai - Managing server state with React Query or SWR - Implementing optimistic updates - Debugging state-related issues - Migrating from legacy Redux to modern patterns ## Core Concepts ### 1. State Categories | Type | Description | Solutions | | ---------------- | ---------------------------- | ----------------------------- | | **Local State** | Component-specific, UI state | useState, useReducer | | **Global State** | Shared across components | Redux Toolkit, Zustand, Jotai | | **Server State** | Remote data, caching | React Query, SWR, RTK Query | | **URL State** | Route parameters, search | React Router, nuqs | | **Form State** | Input values, validation | React Hook Form, Formik | ### 2. Selection Criteria ``` Small app, simple state → Zustand or Jotai Large app, complex state → Redux Toolkit Heavy server interaction → React Query + light client state Atomic/granular updates → Jotai ``` ## Quick Start ### Zustand (Simplest) ```typescript // store/useStore.ts import { create } from 'zustand' import { devtools, persist } from 'zustand/middleware' interface AppState { user: User | null theme: 'light' | 'dark' setUser: (user: User | null) => void toggleTheme: () => void } export const useStore = create<AppState>()( devtools( persist( (set) => ({ user: null, theme: 'light', setUser: (user) => set({ user }), toggleTheme: () => set((state) => ({ theme: state.theme === 'light' ? 'dark' : 'light' })), }), { name: 'app-storage' } ) ) ) // Usage in component function Header() { const { user, theme, toggleTheme } = useStore() return ( <header className={theme}> {user?.name} <button onClick={toggleTheme}>Toggle Theme</button> </header> ) } ``` ## Patterns ### Pattern 1: Redux Toolkit with TypeScript ```typescript // store/index.ts import { configureStore } from "@reduxjs/toolkit"; import { TypedUseSelectorHook, useDispatch, useSelector } from "react-redux"; import userReducer from "./slices/userSlice"; import cartReducer from "./slices/cartSlice"; export const store = configureStore({ reducer: { user: userReducer, cart: cartReducer, }, middleware: (getDefaultMiddleware) => getDefaultMiddleware({ serializableCheck: { ignoredActions: ["persist/PERSIST"], }, }), }); export type RootState = ReturnType<typeof store.getState>; export type AppDispatch = typeof store.dispatch; // Typed hooks export const useAppDispatch: () => AppDispatch = useDispatch; export const useAppSelector: TypedUseSelectorHook<RootState> = useSelector; ``` ```typescript // store/slices/userSlice.ts import { createSlice, createAsyncThunk, PayloadAction } from "@reduxjs/toolkit"; interface User { id: string; email: string; name: string; } interface UserState { current: User | null; status: "idle" | "loading" | "succeeded" | "failed"; error: string | null; } const initialState: UserState = { current: null, status: "idle", error: null, }; export const fetchUser = createAsyncThunk( "user/fetchUser", async (userId: string, { rejectWithValue }) => { try { const response = await fetch(`/api/users/${userId}`); if (!response.ok) throw new Error("Failed to fetch user"); return await response.json(); } catch (error) { return rejectWithValue((error as Error).message); } }, ); const userSlice = createSlice({ name: "user", initialState, reducers: { setUser: (state, action: PayloadAction<User>) => { state.current = action.payload; state.status = "succeeded"; }, clearUser: (state) => { state.current = null; state.status = "idle"; }, }, extraReducers: (builder) => { builder .addCase(fetchUser.pending, (state) => { state.status = "loading"; state.error = null; }) .addCase(fetchUser.fulfilled, (state, action) => { state.status = "succeeded"; state.current = action.payload; }) .addCase(fetchUser.rejected, (state, action) => { state.status = "failed"; state.error = action.payload as string; }); }, }); export const { setUser, clearUser } = userSlice.actions; export default userSlice.reducer; ``` ### Pattern 2: Zustand with Slices (Scalable) ```typescript // store/slices/createUserSlice.ts import { StateCreator } from "zustand"; export interface UserSlice { user: User | null; isAuthenticated: boolean; login: (credentials: Credentials) => Promise<void>; logout: () => void; } export const createUserSlice: StateCreator< UserSlice & CartSlice, // Combined store type [], [], UserSlice > = (set, get) => ({ user: null, isAuthenticated: false, login: async (credentials) => { const user = await authApi.login(credentials); set({ user, isAuthenticated: true }); }, logout: () => { set({ user: null, isAuthenticated: false }); // Can access other slices // get().clearCart() }, }); // store/index.ts import { create } from "zustand"; import { createUserSlice, UserSlice } from "./slices/createUserSlice"; import { createCartSlice, CartSlice } from "./slices/createCartSlice"; type StoreState = UserSlice & CartSlice; export const useStore = create<StoreState>()((...args) => ({ ...createUserSlice(...args), ...createCartSlice(...args), })); // Selective subscriptions (prevents unnecessary re-renders) export const useUser = () => useStore((state) => state.user); export const useCart = () => useStore((state) => state.cart); ``` ### Pattern 3: Jotai for Atomic State ```typescript // atoms/userAtoms.ts import { atom } from 'jotai' import { atomWithStorage } from 'jotai/utils' // Basic atom export const userAtom = atom<User | null>(null) // Derived atom (computed) export const isAuthenticatedAtom = atom((get) => get(userAtom) !== null) // Atom with localStorage persistence export const themeAtom = atomWithStorage<'light' | 'dark'>('theme', 'light') // Async atom export const userProfileAtom = atom(async (get) => { const user = get(userAtom) if (!user) return null const response = await fetch(`/api/users/${user.id}/profile`) return response.json() }) // Write-only atom (action) export const logoutAtom = atom(null, (get, set) => { set(userAtom, null) set(cartAtom, []) localStorage.removeItem('token') }) // Usage function Profile() { const [user] = useAtom(userAtom) const [, logout] = useAtom(logoutAtom) const [profile] = useAtom(userProfileAtom) // Suspense-enabled return ( <Suspense fallback={<Skeleton />}> <ProfileContent profile={profile} onLogout={logout} /> </Suspense> ) } ``` ### Pattern 4: React Query for Server State ```typescript // hooks/useUsers.ts import { useQuery, useMutation, useQueryClient } from "@tanstack/react-query"; // Query keys factory export const userKeys = { all: ["users"] as const, lists: () => [...userKeys.all, "list"] as const, list: (filters: UserFilters) => [...userKeys.lists(), filters] as const, details: () => [...userKeys.all, "detail"] as const, detail: (id: string) => [...userKeys.details(), id] as const, }; // Fetch hook export function useUsers(filters: UserFilters) { return useQuery({ queryKey: userKeys.list(filters), queryFn: () => fetchUsers(filters), staleTime: 5 * 60 * 1000, // 5 minutes gcTime: 30 * 60 * 1000, // 30 minutes (formerly cacheTime) }); } // Single user hook export function useUser(id: string) { return useQuery({ queryKey: userKeys.detail(id), queryFn: () => fetchUser(id), enabled: !!id, // Don't fetch if no id }); } // Mutation with optimistic update export function useUpdateUser() { const queryClient = useQueryClient(); return useMutation({ mutationFn: updateUser, onMutate: async (newUser) => { // Cancel outgoing refetches await queryClient.cancelQueries({ queryKey: userKeys.detail(newUser.id), }); // Snapshot previous value const previousUser = queryClient.getQueryData( userKeys.detail(newUser.id), ); // Optimistically update queryClient.setQueryData(userKeys.detail(newUser.id), newUser); return { previousUser }; }, onError: (err, newUser, context) => { // Rollback on error queryClient.setQueryData( userKeys.detail(newUser.id), context?.previousUser, ); }, onSettled: (data, error, variables) => { // Refetch after mutation queryClient.invalidateQueries({ queryKey: userKeys.detail(variables.id), }); }, }); } ``` ### Pattern 5: Combining Client + Server State ```typescript // Zustand for client state const useUIStore = create<UIState>((set) => ({ sidebarOpen: true, modal: null, toggleSidebar: () => set((s) => ({ sidebarOpen: !s.sidebarOpen })), openModal: (modal) => set({ modal }), closeModal: () => set({ modal: null }), })) // React Query for server state function Dashboard() { const { sidebarOpen, toggleSidebar } = useUIStore() const { data: users, isLoading } = useUsers({ active: true }) const { data: stats } = useStats() if (isLoading) return <DashboardSkeleton /> return ( <div className={sidebarOpen ? 'with-sidebar' : ''}> <Sidebar open={sidebarOpen} onToggle={toggleSidebar} /> <main> <StatsCards stats={stats} /> <UserTable users={users} /> </main> </div> ) } ``` ## Best Practices ### Do's - **Colocate state** - Keep state as close to where it's used as possible - **Use selectors** - Prevent unnecessary re-renders with selective subscriptions - **Normalize data** - Flatten nested structures for easier updates - **Type everything** - Full TypeScript coverage prevents runtime errors - **Separate concerns** - Server state (React Query) vs client state (Zustand) ### Don'ts - **Don't over-globalize** - Not everything needs to be in global state - **Don't duplicate server state** - Let React Query manage it - **Don't mutate directly** - Always use immutable updates - **Don't store derived data** - Compute it instead - **Don't mix paradigms** - Pick one primary solution per category ## Migration Guides ### From Legacy Redux to RTK ```typescript // Before (legacy Redux) const ADD_TODO = "ADD_TODO"; const addTodo = (text) => ({ type: ADD_TODO, payload: text }); function todosReducer(state = [], action) { switch (action.type) { case ADD_TODO: return [...state, { text: action.payload, completed: false }]; default: return state; } } // After (Redux Toolkit) const todosSlice = createSlice({ name: "todos", initialState: [], reducers: { addTodo: (state, action: PayloadAction<string>) => { // Immer allows "mutations" state.push({ text: action.payload, completed: false }); }, }, }); ``` ## Resources - [Redux Toolkit Documentation](https://redux-toolkit.js.org/) - [Zustand GitHub](https://github.com/pmndrs/zustand) - [Jotai Documentation](https://jotai.org/) - [TanStack Query](https://tanstack.com/query)

# JavaScript Testing Patterns Comprehensive guide for implementing robust testing strategies in JavaScript/TypeScript applications using modern testing frameworks and best practices. ## When to Use This Skill - Setting up test infrastructure for new projects - Writing unit tests for functions and classes - Creating integration tests for APIs and services - Implementing end-to-end tests for user flows - Mocking external dependencies and APIs - Testing React, Vue, or other frontend components - Implementing test-driven development (TDD) - Setting up continuous testing in CI/CD pipelines ## Testing Frameworks ### Jest - Full-Featured Testing Framework **Setup:** ```typescript // jest.config.ts import type { Config } from "jest"; const config: Config = { preset: "ts-jest", testEnvironment: "node", roots: ["<rootDir>/src"], testMatch: ["**/__tests__/**/*.ts", "**/?(*.)+(spec|test).ts"], collectCoverageFrom: [ "src/**/*.ts", "!src/**/*.d.ts", "!src/**/*.interface.ts", ], coverageThreshold: { global: { branches: 80, functions: 80, lines: 80, statements: 80, }, }, setupFilesAfterEnv: ["<rootDir>/src/test/setup.ts"], }; export default config; ``` ### Vitest - Fast, Vite-Native Testing **Setup:** ```typescript // vitest.config.ts import { defineConfig } from "vitest/config"; export default defineConfig({ test: { globals: true, environment: "node", coverage: { provider: "v8", reporter: ["text", "json", "html"], exclude: ["**/*.d.ts", "**/*.config.ts", "**/dist/**"], }, setupFiles: ["./src/test/setup.ts"], }, }); ``` ## Unit Testing Patterns ### Pattern 1: Testing Pure Functions ```typescript // utils/calculator.ts export function add(a: number, b: number): number { return a + b; } export function divide(a: number, b: number): number { if (b === 0) { throw new Error("Division by zero"); } return a / b; } // utils/calculator.test.ts import { describe, it, expect } from "vitest"; import { add, divide } from "./calculator"; describe("Calculator", () => { describe("add", () => { it("should add two positive numbers", () => { expect(add(2, 3)).toBe(5); }); it("should add negative numbers", () => { expect(add(-2, -3)).toBe(-5); }); it("should handle zero", () => { expect(add(0, 5)).toBe(5); expect(add(5, 0)).toBe(5); }); }); describe("divide", () => { it("should divide two numbers", () => { expect(divide(10, 2)).toBe(5); }); it("should handle decimal results", () => { expect(divide(5, 2)).toBe(2.5); }); it("should throw error when dividing by zero", () => { expect(() => divide(10, 0)).toThrow("Division by zero"); }); }); }); ``` ### Pattern 2: Testing Classes ```typescript // services/user.service.ts export class UserService { private users: Map<string, User> = new Map(); create(user: User): User { if (this.users.has(user.id)) { throw new Error("User already exists"); } this.users.set(user.id, user); return user; } findById(id: string): User | undefined { return this.users.get(id); } update(id: string, updates: Partial<User>): User { const user = this.users.get(id); if (!user) { throw new Error("User not found"); } const updated = { ...user, ...updates }; this.users.set(id, updated); return updated; } delete(id: string): boolean { return this.users.delete(id); } } // services/user.service.test.ts import { describe, it, expect, beforeEach } from "vitest"; import { UserService } from "./user.service"; describe("UserService", () => { let service: UserService; beforeEach(() => { service = new UserService(); }); describe("create", () => { it("should create a new user", () => { const user = { id: "1", name: "John", email: "john@example.com" }; const created = service.create(user); expect(created).toEqual(user); expect(service.findById("1")).toEqual(user); }); it("should throw error if user already exists", () => { const user = { id: "1", name: "John", email: "john@example.com" }; service.create(user); expect(() => service.create(user)).toThrow("User already exists"); }); }); describe("update", () => { it("should update existing user", () => { const user = { id: "1", name: "John", email: "john@example.com" }; service.create(user); const updated = service.update("1", { name: "Jane" }); expect(updated.name).toBe("Jane"); expect(updated.email).toBe("john@example.com"); }); it("should throw error if user not found", () => { expect(() => service.update("999", { name: "Jane" })).toThrow( "User not found", ); }); }); }); ``` ### Pattern 3: Testing Async Functions ```typescript // services/api.service.ts export class ApiService { async fetchUser(id: string): Promise<User> { const response = await fetch(`https://api.example.com/users/${id}`); if (!response.ok) { throw new Error("User not found"); } return response.json(); } async createUser(user: CreateUserDTO): Promise<User> { const response = await fetch("https://api.example.com/users", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(user), }); return response.json(); } } // services/api.service.test.ts import { describe, it, expect, vi, beforeEach } from "vitest"; import { ApiService } from "./api.service"; // Mock fetch globally global.fetch = vi.fn(); describe("ApiService", () => { let service: ApiService; beforeEach(() => { service = new ApiService(); vi.clearAllMocks(); }); describe("fetchUser", () => { it("should fetch user successfully", async () => { const mockUser = { id: "1", name: "John", email: "john@example.com" }; (fetch as any).mockResolvedValueOnce({ ok: true, json: async () => mockUser, }); const user = await service.fetchUser("1"); expect(user).toEqual(mockUser); expect(fetch).toHaveBeenCalledWith("https://api.example.com/users/1"); }); it("should throw error if user not found", async () => { (fetch as any).mockResolvedValueOnce({ ok: false, }); await expect(service.fetchUser("999")).rejects.toThrow("User not found"); }); }); describe("createUser", () => { it("should create user successfully", async () => { const newUser = { name: "John", email: "john@example.com" }; const createdUser = { id: "1", ...newUser }; (fetch as any).mockResolvedValueOnce({ ok: true, json: async () => createdUser, }); const user = await service.createUser(newUser); expect(user).toEqual(createdUser); expect(fetch).toHaveBeenCalledWith( "https://api.example.com/users", expect.objectContaining({ method: "POST", body: JSON.stringify(newUser), }), ); }); }); }); ``` ## Mocking Patterns ### Pattern 1: Mocking Modules ```typescript // services/email.service.ts import nodemailer from "nodemailer"; export class EmailService { private transporter = nodemailer.createTransport({ host: process.env.SMTP_HOST, port: 587, auth: { user: process.env.SMTP_USER, pass: process.env.SMTP_PASS, }, }); async sendEmail(to: string, subject: string, html: string) { await this.transporter.sendMail({ from: process.env.EMAIL_FROM, to, subject, html, }); } } // services/email.service.test.ts import { describe, it, expect, vi, beforeEach } from "vitest"; import { EmailService } from "./email.service"; vi.mock("nodemailer", () => ({ default: { createTransport: vi.fn(() => ({ sendMail: vi.fn().mockResolvedValue({ messageId: "123" }), })), }, })); describe("EmailService", () => { let service: EmailService; beforeEach(() => { service = new EmailService(); }); it("should send email successfully", async () => { await service.sendEmail( "test@example.com", "Test Subject", "<p>Test Body</p>", ); expect(service["transporter"].sendMail).toHaveBeenCalledWith( expect.objectContaining({ to: "test@example.com", subject: "Test Subject", }), ); }); }); ``` ### Pattern 2: Dependency Injection for Testing ```typescript // services/user.service.ts export interface IUserRepository { findById(id: string): Promise<User | null>; create(user: User): Promise<User>; } export class UserService { constructor(private userRepository: IUserRepository) {} async getUser(id: string): Promise<User> { const user = await this.userRepository.findById(id); if (!user) { throw new Error("User not found"); } return user; } async createUser(userData: CreateUserDTO): Promise<User> { // Business logic here const user = { id: generateId(), ...userData }; return this.userRepository.create(user); } } // services/user.service.test.ts import { describe, it, expect, vi, beforeEach } from "vitest"; import { UserService, IUserRepository } from "./user.service"; describe("UserService", () => { let service: UserService; let mockRepository: IUserRepository; beforeEach(() => { mockRepository = { findById: vi.fn(), create: vi.fn(), }; service = new UserService(mockRepository); }); describe("getUser", () => { it("should return user if found", async () => { const mockUser = { id: "1", name: "John", email: "john@example.com" }; vi.mocked(mockRepository.findById).mockResolvedValue(mockUser); const user = await service.getUser("1"); expect(user).toEqual(mockUser); expect(mockRepository.findById).toHaveBeenCalledWith("1"); }); it("should throw error if user not found", async () => { vi.mocked(mockRepository.findById).mockResolvedValue(null); await expect(service.getUser("999")).rejects.toThrow("User not found"); }); }); describe("createUser", () => { it("should create user successfully", async () => { const userData = { name: "John", email: "john@example.com" }; const createdUser = { id: "1", ...userData }; vi.mocked(mockRepository.create).mockResolvedValue(createdUser); const user = await service.createUser(userData); expect(user).toEqual(createdUser); expect(mockRepository.create).toHaveBeenCalled(); }); }); }); ``` ### Pattern 3: Spying on Functions ```typescript // utils/logger.ts export const logger = { info: (message: string) => console.log(`INFO: ${message}`), error: (message: string) => console.error(`ERROR: ${message}`), }; // services/order.service.ts import { logger } from "../utils/logger"; export class OrderService { async processOrder(orderId: string): Promise<void> { logger.info(`Processing order ${orderId}`); // Process order logic logger.info(`Order ${orderId} processed successfully`); } } // services/order.service.test.ts import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; import { OrderService } from "./order.service"; import { logger } from "../utils/logger"; describe("OrderService", () => { let service: OrderService; let loggerSpy: any; beforeEach(() => { service = new OrderService(); loggerSpy = vi.spyOn(logger, "info"); }); afterEach(() => { loggerSpy.mockRestore(); }); it("should log order processing", async () => { await service.processOrder("123"); expect(loggerSpy).toHaveBeenCalledWith("Processing order 123"); expect(loggerSpy).toHaveBeenCalledWith("Order 123 processed successfully"); expect(loggerSpy).toHaveBeenCalledTimes(2); }); }); ``` ## Integration Testing ### Pattern 1: API Integration Tests ```typescript // tests/integration/user.api.test.ts import request from "supertest"; import { app } from "../../src/app"; import { pool } from "../../src/config/database"; describe("User API Integration Tests", () => { beforeAll(async () => { // Setup test database await pool.query("CREATE TABLE IF NOT EXISTS users (...)"); }); afterAll(async () => { // Cleanup await pool.query("DROP TABLE IF EXISTS users"); await pool.end(); }); beforeEach(async () => { // Clear data before each test await pool.query("TRUNCATE TABLE users CASCADE"); }); describe("POST /api/users", () => { it("should create a new user", async () => { const userData = { name: "John Doe", email: "john@example.com", password: "password123", }; const response = await request(app) .post("/api/users") .send(userData) .expect(201); expect(response.body).toMatchObject({ name: userData.name, email: userData.email, }); expect(response.body).toHaveProperty("id"); expect(response.body).not.toHaveProperty("password"); }); it("should return 400 if email is invalid", async () => { const userData = { name: "John Doe", email: "invalid-email", password: "password123", }; const response = await request(app) .post("/api/users") .send(userData) .expect(400); expect(response.body).toHaveProperty("error"); }); it("should return 409 if email already exists", async () => { const userData = { name: "John Doe", email: "john@example.com", password: "password123", }; await request(app).post("/api/users").send(userData); const response = await request(app) .post("/api/users") .send(userData) .expect(409); expect(response.body.error).toContain("already exists"); }); }); describe("GET /api/users/:id", () => { it("should get user by id", async () => { const createResponse = await request(app).post("/api/users").send({ name: "John Doe", email: "john@example.com", password: "password123", }); const userId = createResponse.body.id; const response = await request(app) .get(`/api/users/${userId}`) .expect(200); expect(response.body).toMatchObject({ id: userId, name: "John Doe", email: "john@example.com", }); }); it("should return 404 if user not found", async () => { await request(app).get("/api/users/999").expect(404); }); }); describe("Authentication", () => { it("should require authentication for protected routes", async () => { await request(app).get("/api/users/me").expect(401); }); it("should allow access with valid token", async () => { // Create user and login await request(app).post("/api/users").send({ name: "John Doe", email: "john@example.com", password: "password123", }); const loginResponse = await request(app).post("/api/auth/login").send({ email: "john@example.com", password: "password123", }); const token = loginResponse.body.token; const response = await request(app) .get("/api/users/me") .set("Authorization", `Bearer ${token}`) .expect(200); expect(response.body.email).toBe("john@example.com"); }); }); }); ``` ### Pattern 2: Database Integration Tests ```typescript // tests/integration/user.repository.test.ts import { describe, it, expect, beforeAll, afterAll, beforeEach } from "vitest"; import { Pool } from "pg"; import { UserRepository } from "../../src/repositories/user.repository"; describe("UserRepository Integration Tests", () => { let pool: Pool; let repository: UserRepository; beforeAll(async () => { pool = new Pool({ host: "localhost", port: 5432, database: "test_db", user: "test_user", password: "test_password", }); repository = new UserRepository(pool); // Create tables await pool.query(` CREATE TABLE IF NOT EXISTS users ( id SERIAL PRIMARY KEY, name VARCHAR(255) NOT NULL, email VARCHAR(255) UNIQUE NOT NULL, password VARCHAR(255) NOT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ) `); }); afterAll(async () => { await pool.query("DROP TABLE IF EXISTS users"); await pool.end(); }); beforeEach(async () => { await pool.query("TRUNCATE TABLE users CASCADE"); }); it("should create a user", async () => { const user = await repository.create({ name: "John Doe", email: "john@example.com", password: "hashed_password", }); expect(user).toHaveProperty("id"); expect(user.name).toBe("John Doe"); expect(user.email).toBe("john@example.com"); }); it("should find user by email", async () => { await repository.create({ name: "John Doe", email: "john@example.com", password: "hashed_password", }); const user = await repository.findByEmail("john@example.com"); expect(user).toBeTruthy(); expect(user?.name).toBe("John Doe"); }); it("should return null if user not found", async () => { const user = await repository.findByEmail("nonexistent@example.com"); expect(user).toBeNull(); }); }); ``` ## Frontend Testing with Testing Library ### Pattern 1: React Component Testing ```typescript // components/UserForm.tsx import { useState } from 'react'; interface Props { onSubmit: (user: { name: string; email: string }) => void; } export function UserForm({ onSubmit }: Props) { const [name, setName] = useState(''); const [email, setEmail] = useState(''); const handleSubmit = (e: React.FormEvent) => { e.preventDefault(); onSubmit({ name, email }); }; return ( <form onSubmit={handleSubmit}> <input type="text" placeholder="Name" value={name} onChange={(e) => setName(e.target.value)} data-testid="name-input" /> <input type="email" placeholder="Email" value={email} onChange={(e) => setEmail(e.target.value)} data-testid="email-input" /> <button type="submit">Submit</button> </form> ); } // components/UserForm.test.tsx import { render, screen, fireEvent } from '@testing-library/react'; import { describe, it, expect, vi } from 'vitest'; import { UserForm } from './UserForm'; describe('UserForm', () => { it('should render form inputs', () => { render(<UserForm onSubmit={vi.fn()} />); expect(screen.getByPlaceholderText('Name')).toBeInTheDocument(); expect(screen.getByPlaceholderText('Email')).toBeInTheDocument(); expect(screen.getByRole('button', { name: 'Submit' })).toBeInTheDocument(); }); it('should update input values', () => { render(<UserForm onSubmit={vi.fn()} />); const nameInput = screen.getByTestId('name-input') as HTMLInputElement; const emailInput = screen.getByTestId('email-input') as HTMLInputElement; fireEvent.change(nameInput, { target: { value: 'John Doe' } }); fireEvent.change(emailInpu

# Modern JavaScript Patterns Comprehensive guide for mastering modern JavaScript (ES6+) features, functional programming patterns, and best practices for writing clean, maintainable, and performant code. ## When to Use This Skill - Refactoring legacy JavaScript to modern syntax - Implementing functional programming patterns - Optimizing JavaScript performance - Writing maintainable and readable code - Working with asynchronous operations - Building modern web applications - Migrating from callbacks to Promises/async-await - Implementing data transformation pipelines ## ES6+ Core Features ### 1. Arrow Functions **Syntax and Use Cases:** ```javascript // Traditional function function add(a, b) { return a + b; } // Arrow function const add = (a, b) => a + b; // Single parameter (parentheses optional) const double = (x) => x * 2; // No parameters const getRandom = () => Math.random(); // Multiple statements (need curly braces) const processUser = (user) => { const normalized = user.name.toLowerCase(); return { ...user, name: normalized }; }; // Returning objects (wrap in parentheses) const createUser = (name, age) => ({ name, age }); ``` **Lexical 'this' Binding:** ```javascript class Counter { constructor() { this.count = 0; } // Arrow function preserves 'this' context increment = () => { this.count++; }; // Traditional function loses 'this' in callbacks incrementTraditional() { setTimeout(function () { this.count++; // 'this' is undefined }, 1000); } // Arrow function maintains 'this' incrementArrow() { setTimeout(() => { this.count++; // 'this' refers to Counter instance }, 1000); } } ``` ### 2. Destructuring **Object Destructuring:** ```javascript const user = { id: 1, name: "John Doe", email: "john@example.com", address: { city: "New York", country: "USA", }, }; // Basic destructuring const { name, email } = user; // Rename variables const { name: userName, email: userEmail } = user; // Default values const { age = 25 } = user; // Nested destructuring const { address: { city, country }, } = user; // Rest operator const { id, ...userWithoutId } = user; // Function parameters function greet({ name, age = 18 }) { console.log(`Hello ${name}, you are ${age}`); } greet(user); ``` **Array Destructuring:** ```javascript const numbers = [1, 2, 3, 4, 5]; // Basic destructuring const [first, second] = numbers; // Skip elements const [, , third] = numbers; // Rest operator const [head, ...tail] = numbers; // Swapping variables let a = 1, b = 2; [a, b] = [b, a]; // Function return values function getCoordinates() { return [10, 20]; } const [x, y] = getCoordinates(); // Default values const [one, two, three = 0] = [1, 2]; ``` ### 3. Spread and Rest Operators **Spread Operator:** ```javascript // Array spreading const arr1 = [1, 2, 3]; const arr2 = [4, 5, 6]; const combined = [...arr1, ...arr2]; // Object spreading const defaults = { theme: "dark", lang: "en" }; const userPrefs = { theme: "light" }; const settings = { ...defaults, ...userPrefs }; // Function arguments const numbers = [1, 2, 3]; Math.max(...numbers); // Copying arrays/objects (shallow copy) const copy = [...arr1]; const objCopy = { ...user }; // Adding items immutably const newArr = [...arr1, 4, 5]; const newObj = { ...user, age: 30 }; ``` **Rest Parameters:** ```javascript // Collect function arguments function sum(...numbers) { return numbers.reduce((total, num) => total + num, 0); } sum(1, 2, 3, 4, 5); // With regular parameters function greet(greeting, ...names) { return `${greeting} ${names.join(", ")}`; } greet("Hello", "John", "Jane", "Bob"); // Object rest const { id, ...userData } = user; // Array rest const [first, ...rest] = [1, 2, 3, 4, 5]; ``` ### 4. Template Literals ```javascript // Basic usage const name = "John"; const greeting = `Hello, ${name}!`; // Multi-line strings const html = ` <div> <h1>${title}</h1> <p>${content}</p> </div> `; // Expression evaluation const price = 19.99; const total = `Total: $${(price * 1.2).toFixed(2)}`; // Tagged template literals function highlight(strings, ...values) { return strings.reduce((result, str, i) => { const value = values[i] || ""; return result + str + `<mark>${value}</mark>`; }, ""); } const name = "John"; const age = 30; const html = highlight`Name: ${name}, Age: ${age}`; // Output: "Name: <mark>John</mark>, Age: <mark>30</mark>" ``` ### 5. Enhanced Object Literals ```javascript const name = "John"; const age = 30; // Shorthand property names const user = { name, age }; // Shorthand method names const calculator = { add(a, b) { return a + b; }, subtract(a, b) { return a - b; }, }; // Computed property names const field = "email"; const user = { name: "John", [field]: "john@example.com", [`get${field.charAt(0).toUpperCase()}${field.slice(1)}`]() { return this[field]; }, }; // Dynamic property creation const createUser = (name, ...props) => { return props.reduce( (user, [key, value]) => ({ ...user, [key]: value, }), { name }, ); }; const user = createUser("John", ["age", 30], ["email", "john@example.com"]); ``` ## Asynchronous Patterns ### 1. Promises **Creating and Using Promises:** ```javascript // Creating a promise const fetchUser = (id) => { return new Promise((resolve, reject) => { setTimeout(() => { if (id > 0) { resolve({ id, name: "John" }); } else { reject(new Error("Invalid ID")); } }, 1000); }); }; // Using promises fetchUser(1) .then((user) => console.log(user)) .catch((error) => console.error(error)) .finally(() => console.log("Done")); // Chaining promises fetchUser(1) .then((user) => fetchUserPosts(user.id)) .then((posts) => processPosts(posts)) .then((result) => console.log(result)) .catch((error) => console.error(error)); ``` **Promise Combinators:** ```javascript // Promise.all - Wait for all promises const promises = [fetchUser(1), fetchUser(2), fetchUser(3)]; Promise.all(promises) .then((users) => console.log(users)) .catch((error) => console.error("At least one failed:", error)); // Promise.allSettled - Wait for all, regardless of outcome Promise.allSettled(promises).then((results) => { results.forEach((result) => { if (result.status === "fulfilled") { console.log("Success:", result.value); } else { console.log("Error:", result.reason); } }); }); // Promise.race - First to complete Promise.race(promises) .then((winner) => console.log("First:", winner)) .catch((error) => console.error(error)); // Promise.any - First to succeed Promise.any(promises) .then((first) => console.log("First success:", first)) .catch((error) => console.error("All failed:", error)); ``` ### 2. Async/Await **Basic Usage:** ```javascript // Async function always returns a Promise async function fetchUser(id) { const response = await fetch(`/api/users/${id}`); const user = await response.json(); return user; } // Error handling with try/catch async function getUserData(id) { try { const user = await fetchUser(id); const posts = await fetchUserPosts(user.id); return { user, posts }; } catch (error) { console.error("Error fetching data:", error); throw error; } } // Sequential vs Parallel execution async function sequential() { const user1 = await fetchUser(1); // Wait const user2 = await fetchUser(2); // Then wait return [user1, user2]; } async function parallel() { const [user1, user2] = await Promise.all([fetchUser(1), fetchUser(2)]); return [user1, user2]; } ``` **Advanced Patterns:** ```javascript // Async IIFE (async () => { const result = await someAsyncOperation(); console.log(result); })(); // Async iteration async function processUsers(userIds) { for (const id of userIds) { const user = await fetchUser(id); await processUser(user); } } // Top-level await (ES2022) const config = await fetch("/config.json").then((r) => r.json()); // Retry logic async function fetchWithRetry(url, retries = 3) { for (let i = 0; i < retries; i++) { try { return await fetch(url); } catch (error) { if (i === retries - 1) throw error; await new Promise((resolve) => setTimeout(resolve, 1000 * (i + 1))); } } } // Timeout wrapper async function withTimeout(promise, ms) { const timeout = new Promise((_, reject) => setTimeout(() => reject(new Error("Timeout")), ms), ); return Promise.race([promise, timeout]); } ``` ## Functional Programming Patterns ### 1. Array Methods **Map, Filter, Reduce:** ```javascript const users = [ { id: 1, name: "John", age: 30, active: true }, { id: 2, name: "Jane", age: 25, active: false }, { id: 3, name: "Bob", age: 35, active: true }, ]; // Map - Transform array const names = users.map((user) => user.name); const upperNames = users.map((user) => user.name.toUpperCase()); // Filter - Select elements const activeUsers = users.filter((user) => user.active); const adults = users.filter((user) => user.age >= 18); // Reduce - Aggregate data const totalAge = users.reduce((sum, user) => sum + user.age, 0); const avgAge = totalAge / users.length; // Group by property const byActive = users.reduce((groups, user) => { const key = user.active ? "active" : "inactive"; return { ...groups, [key]: [...(groups[key] || []), user], }; }, {}); // Chaining methods const result = users .filter((user) => user.active) .map((user) => user.name) .sort() .join(", "); ``` **Advanced Array Methods:** ```javascript // Find - First matching element const user = users.find((u) => u.id === 2); // FindIndex - Index of first match const index = users.findIndex((u) => u.name === "Jane"); // Some - At least one matches const hasActive = users.some((u) => u.active); // Every - All match const allAdults = users.every((u) => u.age >= 18); // FlatMap - Map and flatten const userTags = [ { name: "John", tags: ["admin", "user"] }, { name: "Jane", tags: ["user"] }, ]; const allTags = userTags.flatMap((u) => u.tags); // From - Create array from iterable const str = "hello"; const chars = Array.from(str); const numbers = Array.from({ length: 5 }, (_, i) => i + 1); // Of - Create array from arguments const arr = Array.of(1, 2, 3); ``` ### 2. Higher-Order Functions **Functions as Arguments:** ```javascript // Custom forEach function forEach(array, callback) { for (let i = 0; i < array.length; i++) { callback(array[i], i, array); } } // Custom map function map(array, transform) { const result = []; for (const item of array) { result.push(transform(item)); } return result; } // Custom filter function filter(array, predicate) { const result = []; for (const item of array) { if (predicate(item)) { result.push(item); } } return result; } ``` **Functions Returning Functions:** ```javascript // Currying const multiply = (a) => (b) => a * b; const double = multiply(2); const triple = multiply(3); console.log(double(5)); // 10 console.log(triple(5)); // 15 // Partial application function partial(fn, ...args) { return (...moreArgs) => fn(...args, ...moreArgs); } const add = (a, b, c) => a + b + c; const add5 = partial(add, 5); console.log(add5(3, 2)); // 10 // Memoization function memoize(fn) { const cache = new Map(); return (...args) => { const key = JSON.stringify(args); if (cache.has(key)) { return cache.get(key); } const result = fn(...args); cache.set(key, result); return result; }; } const fibonacci = memoize((n) => { if (n <= 1) return n; return fibonacci(n - 1) + fibonacci(n - 2); }); ``` ### 3. Composition and Piping ```javascript // Function composition const compose = (...fns) => (x) => fns.reduceRight((acc, fn) => fn(acc), x); const pipe = (...fns) => (x) => fns.reduce((acc, fn) => fn(acc), x); // Example usage const addOne = (x) => x + 1; const double = (x) => x * 2; const square = (x) => x * x; const composed = compose(square, double, addOne); console.log(composed(3)); // ((3 + 1) * 2)^2 = 64 const piped = pipe(addOne, double, square); console.log(piped(3)); // ((3 + 1) * 2)^2 = 64 // Practical example const processUser = pipe( (user) => ({ ...user, name: user.name.trim() }), (user) => ({ ...user, email: user.email.toLowerCase() }), (user) => ({ ...user, age: parseInt(user.age) }), ); const user = processUser({ name: " John ", email: "JOHN@EXAMPLE.COM", age: "30", }); ``` ### 4. Pure Functions and Immutability ```javascript // Impure function (modifies input) function addItemImpure(cart, item) { cart.items.push(item); cart.total += item.price; return cart; } // Pure function (no side effects) function addItemPure(cart, item) { return { ...cart, items: [...cart.items, item], total: cart.total + item.price, }; } // Immutable array operations const numbers = [1, 2, 3, 4, 5]; // Add to array const withSix = [...numbers, 6]; // Remove from array const withoutThree = numbers.filter((n) => n !== 3); // Update array element const doubled = numbers.map((n) => (n === 3 ? n * 2 : n)); // Immutable object operations const user = { name: "John", age: 30 }; // Update property const olderUser = { ...user, age: 31 }; // Add property const withEmail = { ...user, email: "john@example.com" }; // Remove property const { age, ...withoutAge } = user; // Deep cloning (simple approach) const deepClone = (obj) => JSON.parse(JSON.stringify(obj)); // Better deep cloning const structuredClone = (obj) => globalThis.structuredClone(obj); ``` ## Modern Class Features ```javascript // Class syntax class User { // Private fields #password; // Public fields id; name; // Static field static count = 0; constructor(id, name, password) { this.id = id; this.name = name; this.#password = password; User.count++; } // Public method greet() { return `Hello, ${this.name}`; } // Private method #hashPassword(password) { return `hashed_${password}`; } // Getter get displayName() { return this.name.toUpperCase(); } // Setter set password(newPassword) { this.#password = this.#hashPassword(newPassword); } // Static method static create(id, name, password) { return new User(id, name, password); } } // Inheritance class Admin extends User { constructor(id, name, password, role) { super(id, name, password); this.role = role; } greet() { return `${super.greet()}, I'm an admin`; } } ``` ## Modules (ES6) ```javascript // Exporting // math.js export const PI = 3.14159; export function add(a, b) { return a + b; } export class Calculator { // ... } // Default export export default function multiply(a, b) { return a * b; } // Importing // app.js import multiply, { PI, add, Calculator } from "./math.js"; // Rename imports import { add as sum } from "./math.js"; // Import all import * as Math from "./math.js"; // Dynamic imports const module = await import("./math.js"); const { add } = await import("./math.js"); // Conditional loading if (condition) { const module = await import("./feature.js"); module.init(); } ``` ## Iterators and Generators ```javascript // Custom iterator const range = { from: 1, to: 5, [Symbol.iterator]() { return { current: this.from, last: this.to, next() { if (this.current <= this.last) { return { done: false, value: this.current++ }; } else { return { done: true }; } }, }; }, }; for (const num of range) { console.log(num); // 1, 2, 3, 4, 5 } // Generator function function* rangeGenerator(from, to) { for (let i = from; i <= to; i++) { yield i; } } for (const num of rangeGenerator(1, 5)) { console.log(num); } // Infinite generator function* fibonacci() { let [prev, curr] = [0, 1]; while (true) { yield curr; [prev, curr] = [curr, prev + curr]; } } // Async generator async function* fetchPages(url) { let page = 1; while (true) { const response = await fetch(`${url}?page=${page}`); const data = await response.json(); if (data.length === 0) break; yield data; page++; } } for await (const page of fetchPages("/api/users")) { console.log(page); } ``` ## Modern Operators ```javascript // Optional chaining const user = { name: "John", address: { city: "NYC" } }; const city = user?.address?.city; const zipCode = user?.address?.zipCode; // undefined // Function call const result = obj.method?.(); // Array access const first = arr?.[0]; // Nullish coalescing const value = null ?? "default"; // 'default' const value = undefined ?? "default"; // 'default' const value = 0 ?? "default"; // 0 (not 'default') const value = "" ?? "default"; // '' (not 'default') // Logical assignment let a = null; a ??= "default"; // a = 'default' let b = 5; b ??= 10; // b = 5 (unchanged) let obj = { count: 0 }; obj.count ||= 1; // obj.count = 1 obj.count &&= 2; // obj.count = 2 ``` ## Performance Optimization ```javascript // Debounce function debounce(fn, delay) { let timeoutId; return (...args) => { clearTimeout(timeoutId); timeoutId = setTimeout(() => fn(...args), delay); }; } const searchDebounced = debounce(search, 300); // Throttle function throttle(fn, limit) { let inThrottle; return (...args) => { if (!inThrottle) { fn(...args); inThrottle = true; setTimeout(() => (inThrottle = false), limit); } }; } const scrollThrottled = throttle(handleScroll, 100); // Lazy evaluation function* lazyMap(iterable, transform) { for (const item of iterable) { yield transform(item); } } // Use only what you need const numbers = [1, 2, 3, 4, 5]; const doubled = lazyMap(numbers, (x) => x * 2); const first = doubled.next().value; // Only computes first value ``` ## Best Practices 1. **Use const by default**: Only use let when reassignment is needed 2. **Prefer arrow functions**: Especially for callbacks 3. **Use template literals**: Instead of string concatenation 4. **Destructure objects and arrays**: For cleaner code 5. **Use async/await**: Instead of Promise chains 6. **Avoid mutating data**: Use spread operator and array methods 7. **Use optional chaining**: Prevent "Cannot read property of undefined" 8. **Use nullish coalescing**: For default values 9. **Prefer array methods**: Over traditional loops 10. **Use modules**: For better code organization 11. **Write pure functions**: Easier to test and reason about 12. **Use meaningful variable names**: Self-documenting code 13. **Keep functions small**: Single responsibility principle 14. **Handle errors properly**: Use try/catch with async/await 15. **Use strict mode**: `'use strict'` for better error catching ## Common Pitfalls 1. **this binding confusion**: Use arrow functions or bind() 2. **Async/await without error handling**: Always use try/catch 3. **Promise creation

# Node.js Backend Patterns Comprehensive guidance for building scalable, maintainable, and production-ready Node.js backend applications with modern frameworks, architectural patterns, and best practices. ## When to Use This Skill - Building REST APIs or GraphQL servers - Creating microservices with Node.js - Implementing authentication and authorization - Designing scalable backend architectures - Setting up middleware and error handling - Integrating databases (SQL and NoSQL) - Building real-time applications with WebSockets - Implementing background job processing ## Core Frameworks ### Express.js - Minimalist Framework **Basic Setup:** ```typescript import express, { Request, Response, NextFunction } from "express"; import helmet from "helmet"; import cors from "cors"; import compression from "compression"; const app = express(); // Security middleware app.use(helmet()); app.use(cors({ origin: process.env.ALLOWED_ORIGINS?.split(",") })); app.use(compression()); // Body parsing app.use(express.json({ limit: "10mb" })); app.use(express.urlencoded({ extended: true, limit: "10mb" })); // Request logging app.use((req: Request, res: Response, next: NextFunction) => { console.log(`${req.method} ${req.path}`); next(); }); const PORT = process.env.PORT || 3000; app.listen(PORT, () => { console.log(`Server running on port ${PORT}`); }); ``` ### Fastify - High Performance Framework **Basic Setup:** ```typescript import Fastify from "fastify"; import helmet from "@fastify/helmet"; import cors from "@fastify/cors"; import compress from "@fastify/compress"; const fastify = Fastify({ logger: { level: process.env.LOG_LEVEL || "info", transport: { target: "pino-pretty", options: { colorize: true }, }, }, }); // Plugins await fastify.register(helmet); await fastify.register(cors, { origin: true }); await fastify.register(compress); // Type-safe routes with schema validation fastify.post<{ Body: { name: string; email: string }; Reply: { id: string; name: string }; }>( "/users", { schema: { body: { type: "object", required: ["name", "email"], properties: { name: { type: "string", minLength: 1 }, email: { type: "string", format: "email" }, }, }, }, }, async (request, reply) => { const { name, email } = request.body; return { id: "123", name }; }, ); await fastify.listen({ port: 3000, host: "0.0.0.0" }); ``` ## Architectural Patterns ### Pattern 1: Layered Architecture **Structure:** ``` src/ ├── controllers/ # Handle HTTP requests/responses ├── services/ # Business logic ├── repositories/ # Data access layer ├── models/ # Data models ├── middleware/ # Express/Fastify middleware ├── routes/ # Route definitions ├── utils/ # Helper functions ├── config/ # Configuration └── types/ # TypeScript types ``` **Controller Layer:** ```typescript // controllers/user.controller.ts import { Request, Response, NextFunction } from "express"; import { UserService } from "../services/user.service"; import { CreateUserDTO, UpdateUserDTO } from "../types/user.types"; export class UserController { constructor(private userService: UserService) {} async createUser(req: Request, res: Response, next: NextFunction) { try { const userData: CreateUserDTO = req.body; const user = await this.userService.createUser(userData); res.status(201).json(user); } catch (error) { next(error); } } async getUser(req: Request, res: Response, next: NextFunction) { try { const { id } = req.params; const user = await this.userService.getUserById(id); res.json(user); } catch (error) { next(error); } } async updateUser(req: Request, res: Response, next: NextFunction) { try { const { id } = req.params; const updates: UpdateUserDTO = req.body; const user = await this.userService.updateUser(id, updates); res.json(user); } catch (error) { next(error); } } async deleteUser(req: Request, res: Response, next: NextFunction) { try { const { id } = req.params; await this.userService.deleteUser(id); res.status(204).send(); } catch (error) { next(error); } } } ``` **Service Layer:** ```typescript // services/user.service.ts import { UserRepository } from "../repositories/user.repository"; import { CreateUserDTO, UpdateUserDTO, User } from "../types/user.types"; import { NotFoundError, ValidationError } from "../utils/errors"; import bcrypt from "bcrypt"; export class UserService { constructor(private userRepository: UserRepository) {} async createUser(userData: CreateUserDTO): Promise<User> { // Validation const existingUser = await this.userRepository.findByEmail(userData.email); if (existingUser) { throw new ValidationError("Email already exists"); } // Hash password const hashedPassword = await bcrypt.hash(userData.password, 10); // Create user const user = await this.userRepository.create({ ...userData, password: hashedPassword, }); // Remove password from response const { password, ...userWithoutPassword } = user; return userWithoutPassword as User; } async getUserById(id: string): Promise<User> { const user = await this.userRepository.findById(id); if (!user) { throw new NotFoundError("User not found"); } const { password, ...userWithoutPassword } = user; return userWithoutPassword as User; } async updateUser(id: string, updates: UpdateUserDTO): Promise<User> { const user = await this.userRepository.update(id, updates); if (!user) { throw new NotFoundError("User not found"); } const { password, ...userWithoutPassword } = user; return userWithoutPassword as User; } async deleteUser(id: string): Promise<void> { const deleted = await this.userRepository.delete(id); if (!deleted) { throw new NotFoundError("User not found"); } } } ``` **Repository Layer:** ```typescript // repositories/user.repository.ts import { Pool } from "pg"; import { CreateUserDTO, UpdateUserDTO, UserEntity } from "../types/user.types"; export class UserRepository { constructor(private db: Pool) {} async create( userData: CreateUserDTO & { password: string }, ): Promise<UserEntity> { const query = ` INSERT INTO users (name, email, password) VALUES ($1, $2, $3) RETURNING id, name, email, password, created_at, updated_at `; const { rows } = await this.db.query(query, [ userData.name, userData.email, userData.password, ]); return rows[0]; } async findById(id: string): Promise<UserEntity | null> { const query = "SELECT * FROM users WHERE id = $1"; const { rows } = await this.db.query(query, [id]); return rows[0] || null; } async findByEmail(email: string): Promise<UserEntity | null> { const query = "SELECT * FROM users WHERE email = $1"; const { rows } = await this.db.query(query, [email]); return rows[0] || null; } async update(id: string, updates: UpdateUserDTO): Promise<UserEntity | null> { const fields = Object.keys(updates); const values = Object.values(updates); const setClause = fields .map((field, idx) => `${field} = $${idx + 2}`) .join(", "); const query = ` UPDATE users SET ${setClause}, updated_at = CURRENT_TIMESTAMP WHERE id = $1 RETURNING * `; const { rows } = await this.db.query(query, [id, ...values]); return rows[0] || null; } async delete(id: string): Promise<boolean> { const query = "DELETE FROM users WHERE id = $1"; const { rowCount } = await this.db.query(query, [id]); return rowCount > 0; } } ``` ### Pattern 2: Dependency Injection **DI Container:** ```typescript // di-container.ts import { Pool } from "pg"; import { UserRepository } from "./repositories/user.repository"; import { UserService } from "./services/user.service"; import { UserController } from "./controllers/user.controller"; import { AuthService } from "./services/auth.service"; class Container { private instances = new Map<string, any>(); register<T>(key: string, factory: () => T): void { this.instances.set(key, factory); } resolve<T>(key: string): T { const factory = this.instances.get(key); if (!factory) { throw new Error(`No factory registered for ${key}`); } return factory(); } singleton<T>(key: string, factory: () => T): void { let instance: T; this.instances.set(key, () => { if (!instance) { instance = factory(); } return instance; }); } } export const container = new Container(); // Register dependencies container.singleton( "db", () => new Pool({ host: process.env.DB_HOST, port: parseInt(process.env.DB_PORT || "5432"), database: process.env.DB_NAME, user: process.env.DB_USER, password: process.env.DB_PASSWORD, max: 20, idleTimeoutMillis: 30000, connectionTimeoutMillis: 2000, }), ); container.singleton( "userRepository", () => new UserRepository(container.resolve("db")), ); container.singleton( "userService", () => new UserService(container.resolve("userRepository")), ); container.register( "userController", () => new UserController(container.resolve("userService")), ); container.singleton( "authService", () => new AuthService(container.resolve("userRepository")), ); ``` ## Middleware Patterns ### Authentication Middleware ```typescript // middleware/auth.middleware.ts import { Request, Response, NextFunction } from "express"; import jwt from "jsonwebtoken"; import { UnauthorizedError } from "../utils/errors"; interface JWTPayload { userId: string; email: string; } declare global { namespace Express { interface Request { user?: JWTPayload; } } } export const authenticate = async ( req: Request, res: Response, next: NextFunction, ) => { try { const token = req.headers.authorization?.replace("Bearer ", ""); if (!token) { throw new UnauthorizedError("No token provided"); } const payload = jwt.verify(token, process.env.JWT_SECRET!) as JWTPayload; req.user = payload; next(); } catch (error) { next(new UnauthorizedError("Invalid token")); } }; export const authorize = (...roles: string[]) => { return async (req: Request, res: Response, next: NextFunction) => { if (!req.user) { return next(new UnauthorizedError("Not authenticated")); } // Check if user has required role const hasRole = roles.some((role) => req.user?.roles?.includes(role)); if (!hasRole) { return next(new UnauthorizedError("Insufficient permissions")); } next(); }; }; ``` ### Validation Middleware ```typescript // middleware/validation.middleware.ts import { Request, Response, NextFunction } from "express"; import { AnyZodObject, ZodError } from "zod"; import { ValidationError } from "../utils/errors"; export const validate = (schema: AnyZodObject) => { return async (req: Request, res: Response, next: NextFunction) => { try { await schema.parseAsync({ body: req.body, query: req.query, params: req.params, }); next(); } catch (error) { if (error instanceof ZodError) { const errors = error.errors.map((err) => ({ field: err.path.join("."), message: err.message, })); next(new ValidationError("Validation failed", errors)); } else { next(error); } } }; }; // Usage with Zod import { z } from "zod"; const createUserSchema = z.object({ body: z.object({ name: z.string().min(1), email: z.string().email(), password: z.string().min(8), }), }); router.post("/users", validate(createUserSchema), userController.createUser); ``` ### Rate Limiting Middleware ```typescript // middleware/rate-limit.middleware.ts import rateLimit from "express-rate-limit"; import RedisStore from "rate-limit-redis"; import Redis from "ioredis"; const redis = new Redis({ host: process.env.REDIS_HOST, port: parseInt(process.env.REDIS_PORT || "6379"), }); export const apiLimiter = rateLimit({ store: new RedisStore({ client: redis, prefix: "rl:", }), windowMs: 15 * 60 * 1000, // 15 minutes max: 100, // Limit each IP to 100 requests per windowMs message: "Too many requests from this IP, please try again later", standardHeaders: true, legacyHeaders: false, }); export const authLimiter = rateLimit({ store: new RedisStore({ client: redis, prefix: "rl:auth:", }), windowMs: 15 * 60 * 1000, max: 5, // Stricter limit for auth endpoints skipSuccessfulRequests: true, }); ``` ### Request Logging Middleware ```typescript // middleware/logger.middleware.ts import { Request, Response, NextFunction } from "express"; import pino from "pino"; const logger = pino({ level: process.env.LOG_LEVEL || "info", transport: { target: "pino-pretty", options: { colorize: true }, }, }); export const requestLogger = ( req: Request, res: Response, next: NextFunction, ) => { const start = Date.now(); // Log response when finished res.on("finish", () => { const duration = Date.now() - start; logger.info({ method: req.method, url: req.url, status: res.statusCode, duration: `${duration}ms`, userAgent: req.headers["user-agent"], ip: req.ip, }); }); next(); }; export { logger }; ``` ## Error Handling ### Custom Error Classes ```typescript // utils/errors.ts export class AppError extends Error { constructor( public message: string, public statusCode: number = 500, public isOperational: boolean = true, ) { super(message); Object.setPrototypeOf(this, AppError.prototype); Error.captureStackTrace(this, this.constructor); } } export class ValidationError extends AppError { constructor( message: string, public errors?: any[], ) { super(message, 400); } } export class NotFoundError extends AppError { constructor(message: string = "Resource not found") { super(message, 404); } } export class UnauthorizedError extends AppError { constructor(message: string = "Unauthorized") { super(message, 401); } } export class ForbiddenError extends AppError { constructor(message: string = "Forbidden") { super(message, 403); } } export class ConflictError extends AppError { constructor(message: string) { super(message, 409); } } ``` ### Global Error Handler ```typescript // middleware/error-handler.ts import { Request, Response, NextFunction } from "express"; import { AppError } from "../utils/errors"; import { logger } from "./logger.middleware"; export const errorHandler = ( err: Error, req: Request, res: Response, next: NextFunction, ) => { if (err instanceof AppError) { return res.status(err.statusCode).json({ status: "error", message: err.message, ...(err instanceof ValidationError && { errors: err.errors }), }); } // Log unexpected errors logger.error({ error: err.message, stack: err.stack, url: req.url, method: req.method, }); // Don't leak error details in production const message = process.env.NODE_ENV === "production" ? "Internal server error" : err.message; res.status(500).json({ status: "error", message, }); }; // Async error wrapper export const asyncHandler = ( fn: (req: Request, res: Response, next: NextFunction) => Promise<any>, ) => { return (req: Request, res: Response, next: NextFunction) => { Promise.resolve(fn(req, res, next)).catch(next); }; }; ``` ## Database Patterns ### PostgreSQL with Connection Pool ```typescript // config/database.ts import { Pool, PoolConfig } from "pg"; const poolConfig: PoolConfig = { host: process.env.DB_HOST, port: parseInt(process.env.DB_PORT || "5432"), database: process.env.DB_NAME, user: process.env.DB_USER, password: process.env.DB_PASSWORD, max: 20, idleTimeoutMillis: 30000, connectionTimeoutMillis: 2000, }; export const pool = new Pool(poolConfig); // Test connection pool.on("connect", () => { console.log("Database connected"); }); pool.on("error", (err) => { console.error("Unexpected database error", err); process.exit(-1); }); // Graceful shutdown export const closeDatabase = async () => { await pool.end(); console.log("Database connection closed"); }; ``` ### MongoDB with Mongoose ```typescript // config/mongoose.ts import mongoose from "mongoose"; const connectDB = async () => { try { await mongoose.connect(process.env.MONGODB_URI!, { maxPoolSize: 10, serverSelectionTimeoutMS: 5000, socketTimeoutMS: 45000, }); console.log("MongoDB connected"); } catch (error) { console.error("MongoDB connection error:", error); process.exit(1); } }; mongoose.connection.on("disconnected", () => { console.log("MongoDB disconnected"); }); mongoose.connection.on("error", (err) => { console.error("MongoDB error:", err); }); export { connectDB }; // Model example import { Schema, model, Document } from "mongoose"; interface IUser extends Document { name: string; email: string; password: string; createdAt: Date; updatedAt: Date; } const userSchema = new Schema<IUser>( { name: { type: String, required: true }, email: { type: String, required: true, unique: true }, password: { type: String, required: true }, }, { timestamps: true, }, ); // Indexes userSchema.index({ email: 1 }); export const User = model<IUser>("User", userSchema); ``` ### Transaction Pattern ```typescript // services/order.service.ts import { Pool } from "pg"; export class OrderService { constructor(private db: Pool) {} async createOrder(userId: string, items: any[]) { const client = await this.db.connect(); try { await client.query("BEGIN"); // Create order const orderResult = await client.query( "INSERT INTO orders (user_id, total) VALUES ($1, $2) RETURNING id", [userId, calculateTotal(items)], ); const orderId = orderResult.rows[0].id; // Create order items for (const item of items) { await client.query( "INSERT INTO order_items (order_id, product_id, quantity, price) VALUES ($1, $2, $3, $4)", [orderId, item.productId, item.quantity, item.price], ); // Update inventory await client.query( "UPDATE products SET stock = stock - $1 WHERE id = $2", [item.quantity, item.productId], ); } await client.query("COMMIT"); return orderId; } catch (error) { await client.query("ROLLBACK"); throw error; } finally { client.release(); } } } ``` ## Authentication & Authorization ### JWT Authentication ```typescript // services/auth.service.ts import jwt from

# LangChain & LangGraph Architecture Master modern LangChain 1.x and LangGraph for building sophisticated LLM applications with agents, state management, memory, and tool integration. ## When to Use This Skill - Building autonomous AI agents with tool access - Implementing complex multi-step LLM workflows - Managing conversation memory and state - Integrating LLMs with external data sources and APIs - Creating modular, reusable LLM application components - Implementing document processing pipelines - Building production-grade LLM applications ## Package Structure (LangChain 1.x) ``` langchain (1.2.x) # High-level orchestration langchain-core (1.2.x) # Core abstractions (messages, prompts, tools) langchain-community # Third-party integrations langgraph # Agent orchestration and state management langchain-openai # OpenAI integrations langchain-anthropic # Anthropic/Claude integrations langchain-voyageai # Voyage AI embeddings langchain-pinecone # Pinecone vector store ``` ## Core Concepts ### 1. LangGraph Agents LangGraph is the standard for building agents in 2026. It provides: **Key Features:** - **StateGraph**: Explicit state management with typed state - **Durable Execution**: Agents persist through failures - **Human-in-the-Loop**: Inspect and modify state at any point - **Memory**: Short-term and long-term memory across sessions - **Checkpointing**: Save and resume agent state **Agent Patterns:** - **ReAct**: Reasoning + Acting with `create_react_agent` - **Plan-and-Execute**: Separate planning and execution nodes - **Multi-Agent**: Supervisor routing between specialized agents - **Tool-Calling**: Structured tool invocation with Pydantic schemas ### 2. State Management LangGraph uses TypedDict for explicit state: ```python from typing import Annotated, TypedDict from langgraph.graph import MessagesState # Simple message-based state class AgentState(MessagesState): """Extends MessagesState with custom fields.""" context: Annotated[list, "retrieved documents"] # Custom state for complex agents class CustomState(TypedDict): messages: Annotated[list, "conversation history"] context: Annotated[dict, "retrieved context"] current_step: str results: list ``` ### 3. Memory Systems Modern memory implementations: - **ConversationBufferMemory**: Stores all messages (short conversations) - **ConversationSummaryMemory**: Summarizes older messages (long conversations) - **ConversationTokenBufferMemory**: Token-based windowing - **VectorStoreRetrieverMemory**: Semantic similarity retrieval - **LangGraph Checkpointers**: Persistent state across sessions ### 4. Document Processing Loading, transforming, and storing documents: **Components:** - **Document Loaders**: Load from various sources - **Text Splitters**: Chunk documents intelligently - **Vector Stores**: Store and retrieve embeddings - **Retrievers**: Fetch relevant documents ### 5. Callbacks & Tracing LangSmith is the standard for observability: - Request/response logging - Token usage tracking - Latency monitoring - Error tracking - Trace visualization ## Quick Start ### Modern ReAct Agent with LangGraph ```python from langgraph.prebuilt import create_react_agent from langgraph.checkpoint.memory import MemorySaver from langchain_anthropic import ChatAnthropic from langchain_core.tools import tool import ast import operator # Initialize LLM (Claude Sonnet 4.6 recommended) llm = ChatAnthropic(model="claude-sonnet-4-6", temperature=0) # Define tools with Pydantic schemas @tool def search_database(query: str) -> str: """Search internal database for information.""" # Your database search logic return f"Results for: {query}" @tool def calculate(expression: str) -> str: """Safely evaluate a mathematical expression. Supports: +, -, *, /, **, %, parentheses Example: '(2 + 3) * 4' returns '20' """ # Safe math evaluation using ast allowed_operators = { ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv, ast.Pow: operator.pow, ast.Mod: operator.mod, ast.USub: operator.neg, } def _eval(node): if isinstance(node, ast.Constant): return node.value elif isinstance(node, ast.BinOp): left = _eval(node.left) right = _eval(node.right) return allowed_operators[type(node.op)](left, right) elif isinstance(node, ast.UnaryOp): operand = _eval(node.operand) return allowed_operators[type(node.op)](operand) else: raise ValueError(f"Unsupported operation: {type(node)}") try: tree = ast.parse(expression, mode='eval') return str(_eval(tree.body)) except Exception as e: return f"Error: {e}" tools = [search_database, calculate] # Create checkpointer for memory persistence checkpointer = MemorySaver() # Create ReAct agent agent = create_react_agent( llm, tools, checkpointer=checkpointer ) # Run agent with thread ID for memory config = {"configurable": {"thread_id": "user-123"}} result = await agent.ainvoke( {"messages": [("user", "Search for Python tutorials and calculate 25 * 4")]}, config=config ) ``` ## Architecture Patterns ### Pattern 1: RAG with LangGraph ```python from langgraph.graph import StateGraph, START, END from langchain_anthropic import ChatAnthropic from langchain_voyageai import VoyageAIEmbeddings from langchain_pinecone import PineconeVectorStore from langchain_core.documents import Document from langchain_core.prompts import ChatPromptTemplate from typing import TypedDict, Annotated class RAGState(TypedDict): question: str context: Annotated[list[Document], "retrieved documents"] answer: str # Initialize components llm = ChatAnthropic(model="claude-sonnet-4-6") embeddings = VoyageAIEmbeddings(model="voyage-3-large") vectorstore = PineconeVectorStore(index_name="docs", embedding=embeddings) retriever = vectorstore.as_retriever(search_kwargs={"k": 4}) # Define nodes async def retrieve(state: RAGState) -> RAGState: """Retrieve relevant documents.""" docs = await retriever.ainvoke(state["question"]) return {"context": docs} async def generate(state: RAGState) -> RAGState: """Generate answer from context.""" prompt = ChatPromptTemplate.from_template( """Answer based on the context below. If you cannot answer, say so. Context: {context} Question: {question} Answer:""" ) context_text = "\n\n".join(doc.page_content for doc in state["context"]) response = await llm.ainvoke( prompt.format(context=context_text, question=state["question"]) ) return {"answer": response.content} # Build graph builder = StateGraph(RAGState) builder.add_node("retrieve", retrieve) builder.add_node("generate", generate) builder.add_edge(START, "retrieve") builder.add_edge("retrieve", "generate") builder.add_edge("generate", END) rag_chain = builder.compile() # Use the chain result = await rag_chain.ainvoke({"question": "What is the main topic?"}) ``` ### Pattern 2: Custom Agent with Structured Tools ```python from langchain_core.tools import StructuredTool from pydantic import BaseModel, Field class SearchInput(BaseModel): """Input for database search.""" query: str = Field(description="Search query") filters: dict = Field(default={}, description="Optional filters") class EmailInput(BaseModel): """Input for sending email.""" recipient: str = Field(description="Email recipient") subject: str = Field(description="Email subject") content: str = Field(description="Email body") async def search_database(query: str, filters: dict = {}) -> str: """Search internal database for information.""" # Your database search logic return f"Results for '{query}' with filters {filters}" async def send_email(recipient: str, subject: str, content: str) -> str: """Send an email to specified recipient.""" # Email sending logic return f"Email sent to {recipient}" tools = [ StructuredTool.from_function( coroutine=search_database, name="search_database", description="Search internal database", args_schema=SearchInput ), StructuredTool.from_function( coroutine=send_email, name="send_email", description="Send an email", args_schema=EmailInput ) ] agent = create_react_agent(llm, tools) ``` ### Pattern 3: Multi-Step Workflow with StateGraph ```python from langgraph.graph import StateGraph, START, END from typing import TypedDict, Literal class WorkflowState(TypedDict): text: str entities: list analysis: str summary: str current_step: str async def extract_entities(state: WorkflowState) -> WorkflowState: """Extract key entities from text.""" prompt = f"Extract key entities from: {state['text']}\n\nReturn as JSON list." response = await llm.ainvoke(prompt) return {"entities": response.content, "current_step": "analyze"} async def analyze_entities(state: WorkflowState) -> WorkflowState: """Analyze extracted entities.""" prompt = f"Analyze these entities: {state['entities']}\n\nProvide insights." response = await llm.ainvoke(prompt) return {"analysis": response.content, "current_step": "summarize"} async def generate_summary(state: WorkflowState) -> WorkflowState: """Generate final summary.""" prompt = f"""Summarize: Entities: {state['entities']} Analysis: {state['analysis']} Provide a concise summary.""" response = await llm.ainvoke(prompt) return {"summary": response.content, "current_step": "complete"} def route_step(state: WorkflowState) -> Literal["analyze", "summarize", "end"]: """Route to next step based on current state.""" step = state.get("current_step", "extract") if step == "analyze": return "analyze" elif step == "summarize": return "summarize" return "end" # Build workflow builder = StateGraph(WorkflowState) builder.add_node("extract", extract_entities) builder.add_node("analyze", analyze_entities) builder.add_node("summarize", generate_summary) builder.add_edge(START, "extract") builder.add_conditional_edges("extract", route_step, { "analyze": "analyze", "summarize": "summarize", "end": END }) builder.add_conditional_edges("analyze", route_step, { "summarize": "summarize", "end": END }) builder.add_edge("summarize", END) workflow = builder.compile() ``` ### Pattern 4: Multi-Agent Orchestration ```python from langgraph.graph import StateGraph, START, END from langgraph.prebuilt import create_react_agent from langchain_core.messages import HumanMessage from typing import Literal class MultiAgentState(TypedDict): messages: list next_agent: str # Create specialized agents researcher = create_react_agent(llm, research_tools) writer = create_react_agent(llm, writing_tools) reviewer = create_react_agent(llm, review_tools) async def supervisor(state: MultiAgentState) -> MultiAgentState: """Route to appropriate agent based on task.""" prompt = f"""Based on the conversation, which agent should handle this? Options: - researcher: For finding information - writer: For creating content - reviewer: For reviewing and editing - FINISH: Task is complete Messages: {state['messages']} Respond with just the agent name.""" response = await llm.ainvoke(prompt) return {"next_agent": response.content.strip().lower()} def route_to_agent(state: MultiAgentState) -> Literal["researcher", "writer", "reviewer", "end"]: """Route based on supervisor decision.""" next_agent = state.get("next_agent", "").lower() if next_agent == "finish": return "end" return next_agent if next_agent in ["researcher", "writer", "reviewer"] else "end" # Build multi-agent graph builder = StateGraph(MultiAgentState) builder.add_node("supervisor", supervisor) builder.add_node("researcher", researcher) builder.add_node("writer", writer) builder.add_node("reviewer", reviewer) builder.add_edge(START, "supervisor") builder.add_conditional_edges("supervisor", route_to_agent, { "researcher": "researcher", "writer": "writer", "reviewer": "reviewer", "end": END }) # Each agent returns to supervisor for agent in ["researcher", "writer", "reviewer"]: builder.add_edge(agent, "supervisor") multi_agent = builder.compile() ``` ## Memory Management ### Token-Based Memory with LangGraph ```python from langgraph.checkpoint.memory import MemorySaver from langgraph.prebuilt import create_react_agent # In-memory checkpointer (development) checkpointer = MemorySaver() # Create agent with persistent memory agent = create_react_agent(llm, tools, checkpointer=checkpointer) # Each thread_id maintains separate conversation config = {"configurable": {"thread_id": "session-abc123"}} # Messages persist across invocations with same thread_id result1 = await agent.ainvoke({"messages": [("user", "My name is Alice")]}, config) result2 = await agent.ainvoke({"messages": [("user", "What's my name?")]}, config) # Agent remembers: "Your name is Alice" ``` ### Production Memory with PostgreSQL ```python from langgraph.checkpoint.postgres import PostgresSaver # Production checkpointer checkpointer = PostgresSaver.from_conn_string( "postgresql://user:pass@localhost/langgraph" ) agent = create_react_agent(llm, tools, checkpointer=checkpointer) ``` ### Vector Store Memory for Long-Term Context ```python from langchain_community.vectorstores import Chroma from langchain_voyageai import VoyageAIEmbeddings embeddings = VoyageAIEmbeddings(model="voyage-3-large") memory_store = Chroma( collection_name="conversation_memory", embedding_function=embeddings, persist_directory="./memory_db" ) async def retrieve_relevant_memory(query: str, k: int = 5) -> list: """Retrieve relevant past conversations.""" docs = await memory_store.asimilarity_search(query, k=k) return [doc.page_content for doc in docs] async def store_memory(content: str, metadata: dict = {}): """Store conversation in long-term memory.""" await memory_store.aadd_texts([content], metadatas=[metadata]) ``` ## Callback System & LangSmith ### LangSmith Tracing ```python import os from langchain_anthropic import ChatAnthropic # Enable LangSmith tracing os.environ["LANGCHAIN_TRACING_V2"] = "true" os.environ["LANGCHAIN_API_KEY"] = "your-api-key" os.environ["LANGCHAIN_PROJECT"] = "my-project" # All LangChain/LangGraph operations are automatically traced llm = ChatAnthropic(model="claude-sonnet-4-6") ``` ### Custom Callback Handler ```python from langchain_core.callbacks import BaseCallbackHandler from typing import Any, Dict, List class CustomCallbackHandler(BaseCallbackHandler): def on_llm_start( self, serialized: Dict[str, Any], prompts: List[str], **kwargs ) -> None: print(f"LLM started with {len(prompts)} prompts") def on_llm_end(self, response, **kwargs) -> None: print(f"LLM completed: {len(response.generations)} generations") def on_llm_error(self, error: Exception, **kwargs) -> None: print(f"LLM error: {error}") def on_tool_start( self, serialized: Dict[str, Any], input_str: str, **kwargs ) -> None: print(f"Tool started: {serialized.get('name')}") def on_tool_end(self, output: str, **kwargs) -> None: print(f"Tool completed: {output[:100]}...") # Use callbacks result = await agent.ainvoke( {"messages": [("user", "query")]}, config={"callbacks": [CustomCallbackHandler()]} ) ``` ## Streaming Responses ```python from langchain_anthropic import ChatAnthropic llm = ChatAnthropic(model="claude-sonnet-4-6", streaming=True) # Stream tokens async for chunk in llm.astream("Tell me a story"): print(chunk.content, end="", flush=True) # Stream agent events async for event in agent.astream_events( {"messages": [("user", "Search and summarize")]}, version="v2" ): if event["event"] == "on_chat_model_stream": print(event["data"]["chunk"].content, end="") elif event["event"] == "on_tool_start": print(f"\n[Using tool: {event['name']}]") ``` ## Testing Strategies ```python import pytest from unittest.mock import AsyncMock, patch @pytest.mark.asyncio async def test_agent_tool_selection(): """Test agent selects correct tool.""" with patch.object(llm, 'ainvoke') as mock_llm: mock_llm.return_value = AsyncMock(content="Using search_database") result = await agent.ainvoke({ "messages": [("user", "search for documents")] }) # Verify tool was called assert "search_database" in str(result) @pytest.mark.asyncio async def test_memory_persistence(): """Test memory persists across invocations.""" config = {"configurable": {"thread_id": "test-thread"}} # First message await agent.ainvoke( {"messages": [("user", "Remember: the code is 12345")]}, config ) # Second message should remember result = await agent.ainvoke( {"messages": [("user", "What was the code?")]}, config ) assert "12345" in result["messages"][-1].content ``` ## Performance Optimization ### 1. Caching with Redis ```python from langchain_community.cache import RedisCache from langchain_core.globals import set_llm_cache import redis redis_client = redis.Redis.from_url("redis://localhost:6379") set_llm_cache(RedisCache(redis_client)) ``` ### 2. Async Batch Processing ```python import asyncio from langchain_core.documents import Document async def process_documents(documents: list[Document]) -> list: """Process documents in parallel.""" tasks = [process_single(doc) for doc in documents] return await asyncio.gather(*tasks) async def process_single(doc: Document) -> dict: """Process a single document.""" chunks = text_splitter.split_documents([doc]) embeddings = await embeddings_model.aembed_documents( [c.page_content for c in chunks] ) return {"doc_id": doc.metadata.get("id"), "embeddings": embeddings} ``` ### 3. Connection Pooling ```python from langchain_pinecone import PineconeVectorStore from pinecone import Pinecone # Reuse Pinecone client pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"]) index = pc.Index("my-index") # Create vector store with existing index vectorstore = PineconeVectorStore(index=index, embedding=embeddings) ``` ## Resources - [LangChain Documentation](https://python.langchain.com/docs/) - [LangGraph Documentation](https://langchain-ai.github.io/langgraph/) - [LangSmith Platform](https://smith.langchain.com/) - [LangChain GitHub](https://github.com/langchain-ai/langchain) - [LangGraph GitHub](https://github.com/langchain-ai/langgraph) ## Common Pitfalls 1. **Using Deprecated APIs**: Use LangGraph for agents, not `initialize_agent` 2. **Memory Overflow**: Use checkpointers with TTL for long-running agents 3. **Poor Tool Descriptions**: Clear descriptions help LLM select correct tools 4. **Context Window Exceeded**: Use summarization or sliding window memory 5. **No Error Handling**: Wrap too

# ML Pipeline Workflow Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment. ## Overview This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring. ## When to Use This Skill - Building new ML pipelines from scratch - Designing workflow orchestration for ML systems - Implementing data → model → deployment automation - Setting up reproducible training workflows - Creating DAG-based ML orchestration - Integrating ML components into production systems ## What This Skill Provides ### Core Capabilities 1. **Pipeline Architecture** - End-to-end workflow design - DAG orchestration patterns (Airflow, Dagster, Kubeflow) - Component dependencies and data flow - Error handling and retry strategies 2. **Data Preparation** - Data validation and quality checks - Feature engineering pipelines - Data versioning and lineage - Train/validation/test splitting strategies 3. **Model Training** - Training job orchestration - Hyperparameter management - Experiment tracking integration - Distributed training patterns 4. **Model Validation** - Validation frameworks and metrics - A/B testing infrastructure - Performance regression detection - Model comparison workflows 5. **Deployment Automation** - Model serving patterns - Canary deployments - Blue-green deployment strategies - Rollback mechanisms ### Reference Documentation See the `references/` directory for detailed guides: - **data-preparation.md** - Data cleaning, validation, and feature engineering - **model-training.md** - Training workflows and best practices - **model-validation.md** - Validation strategies and metrics - **model-deployment.md** - Deployment patterns and serving architectures ### Assets and Templates The `assets/` directory contains: - **pipeline-dag.yaml.template** - DAG template for workflow orchestration - **training-config.yaml** - Training configuration template - **validation-checklist.md** - Pre-deployment validation checklist ## Usage Patterns ### Basic Pipeline Setup ```python # 1. Define pipeline stages stages = [ "data_ingestion", "data_validation", "feature_engineering", "model_training", "model_validation", "model_deployment" ] # 2. Configure dependencies # See assets/pipeline-dag.yaml.template for full example ``` ### Production Workflow 1. **Data Preparation Phase** - Ingest raw data from sources - Run data quality checks - Apply feature transformations - Version processed datasets 2. **Training Phase** - Load versioned training data - Execute training jobs - Track experiments and metrics - Save trained models 3. **Validation Phase** - Run validation test suite - Compare against baseline - Generate performance reports - Approve for deployment 4. **Deployment Phase** - Package model artifacts - Deploy to serving infrastructure - Configure monitoring - Validate production traffic ## Best Practices ### Pipeline Design - **Modularity**: Each stage should be independently testable - **Idempotency**: Re-running stages should be safe - **Observability**: Log metrics at every stage - **Versioning**: Track data, code, and model versions - **Failure Handling**: Implement retry logic and alerting ### Data Management - Use data validation libraries (Great Expectations, TFX) - Version datasets with DVC or similar tools - Document feature engineering transformations - Maintain data lineage tracking ### Model Operations - Separate training and serving infrastructure - Use model registries (MLflow, Weights & Biases) - Implement gradual rollouts for new models - Monitor model performance drift - Maintain rollback capabilities ### Deployment Strategies - Start with shadow deployments - Use canary releases for validation - Implement A/B testing infrastructure - Set up automated rollback triggers - Monitor latency and throughput ## Integration Points ### Orchestration Tools - **Apache Airflow**: DAG-based workflow orchestration - **Dagster**: Asset-based pipeline orchestration - **Kubeflow Pipelines**: Kubernetes-native ML workflows - **Prefect**: Modern dataflow automation ### Experiment Tracking - MLflow for experiment tracking and model registry - Weights & Biases for visualization and collaboration - TensorBoard for training metrics ### Deployment Platforms - AWS SageMaker for managed ML infrastructure - Google Vertex AI for GCP deployments - Azure ML for Azure cloud - Kubernetes + KServe for cloud-agnostic serving ## Progressive Disclosure Start with the basics and gradually add complexity: 1. **Level 1**: Simple linear pipeline (data → train → deploy) 2. **Level 2**: Add validation and monitoring stages 3. **Level 3**: Implement hyperparameter tuning 4. **Level 4**: Add A/B testing and gradual rollouts 5. **Level 5**: Multi-model pipelines with ensemble strategies ## Common Patterns ### Batch Training Pipeline ```yaml # See assets/pipeline-dag.yaml.template stages: - name: data_preparation dependencies: [] - name: model_training dependencies: [data_preparation] - name: model_evaluation dependencies: [model_training] - name: model_deployment dependencies: [model_evaluation] ``` ### Real-time Feature Pipeline ```python # Stream processing for real-time features # Combined with batch training # See references/data-preparation.md ``` ### Continuous Training ```python # Automated retraining on schedule # Triggered by data drift detection # See references/model-training.md ``` ## Troubleshooting ### Common Issues - **Pipeline failures**: Check dependencies and data availability - **Training instability**: Review hyperparameters and data quality - **Deployment issues**: Validate model artifacts and serving config - **Performance degradation**: Monitor data drift and model metrics ### Debugging Steps 1. Check pipeline logs for each stage 2. Validate input/output data at boundaries 3. Test components in isolation 4. Review experiment tracking metrics 5. Inspect model artifacts and metadata ## Next Steps After setting up your pipeline: 1. Explore **hyperparameter-tuning** skill for optimization 2. Learn **experiment-tracking-setup** for MLflow/W&B 3. Review **model-deployment-patterns** for serving strategies 4. Implement monitoring with observability tools ## Related Skills - **experiment-tracking-setup**: MLflow and Weights & Biases integration - **hyperparameter-tuning**: Automated hyperparameter optimization - **model-deployment-patterns**: Advanced deployment strategies

# Python Background Jobs & Task Queues Decouple long-running or unreliable work from request/response cycles. Return immediately to the user while background workers handle the heavy lifting asynchronously. ## When to Use This Skill - Processing tasks that take longer than a few seconds - Sending emails, notifications, or webhooks - Generating reports or exporting data - Processing uploads or media transformations - Integrating with unreliable external services - Building event-driven architectures ## Core Concepts ### 1. Task Queue Pattern API accepts request, enqueues a job, returns immediately with a job ID. Workers process jobs asynchronously. ### 2. Idempotency Tasks may be retried on failure. Design for safe re-execution. ### 3. Job State Machine Jobs transition through states: pending → running → succeeded/failed. ### 4. At-Least-Once Delivery Most queues guarantee at-least-once delivery. Your code must handle duplicates. ## Quick Start This skill uses Celery for examples, a widely adopted task queue. Alternatives like RQ, Dramatiq, and cloud-native solutions (AWS SQS, GCP Tasks) are equally valid choices. ```python from celery import Celery app = Celery("tasks", broker="redis://localhost:6379") @app.task def send_email(to: str, subject: str, body: str) -> None: # This runs in a background worker email_client.send(to, subject, body) # In your API handler send_email.delay("user@example.com", "Welcome!", "Thanks for signing up") ``` ## Fundamental Patterns ### Pattern 1: Return Job ID Immediately For operations exceeding a few seconds, return a job ID and process asynchronously. ```python from uuid import uuid4 from dataclasses import dataclass from enum import Enum from datetime import datetime class JobStatus(Enum): PENDING = "pending" RUNNING = "running" SUCCEEDED = "succeeded" FAILED = "failed" @dataclass class Job: id: str status: JobStatus created_at: datetime started_at: datetime | None = None completed_at: datetime | None = None result: dict | None = None error: str | None = None # API endpoint async def start_export(request: ExportRequest) -> JobResponse: """Start export job and return job ID.""" job_id = str(uuid4()) # Persist job record await jobs_repo.create(Job( id=job_id, status=JobStatus.PENDING, created_at=datetime.utcnow(), )) # Enqueue task for background processing await task_queue.enqueue( "export_data", job_id=job_id, params=request.model_dump(), ) # Return immediately with job ID return JobResponse( job_id=job_id, status="pending", poll_url=f"/jobs/{job_id}", ) ``` ### Pattern 2: Celery Task Configuration Configure Celery tasks with proper retry and timeout settings. ```python from celery import Celery app = Celery("tasks", broker="redis://localhost:6379") # Global configuration app.conf.update( task_time_limit=3600, # Hard limit: 1 hour task_soft_time_limit=3000, # Soft limit: 50 minutes task_acks_late=True, # Acknowledge after completion task_reject_on_worker_lost=True, worker_prefetch_multiplier=1, # Don't prefetch too many tasks ) @app.task( bind=True, max_retries=3, default_retry_delay=60, autoretry_for=(ConnectionError, TimeoutError), ) def process_payment(self, payment_id: str) -> dict: """Process payment with automatic retry on transient errors.""" try: result = payment_gateway.charge(payment_id) return {"status": "success", "transaction_id": result.id} except PaymentDeclinedError as e: # Don't retry permanent failures return {"status": "declined", "reason": str(e)} except TransientError as e: # Retry with exponential backoff raise self.retry(exc=e, countdown=2 ** self.request.retries * 60) ``` ### Pattern 3: Make Tasks Idempotent Workers may retry on crash or timeout. Design for safe re-execution. ```python @app.task(bind=True) def process_order(self, order_id: str) -> None: """Process order idempotently.""" order = orders_repo.get(order_id) # Already processed? Return early if order.status == OrderStatus.COMPLETED: logger.info("Order already processed", order_id=order_id) return # Already in progress? Check if we should continue if order.status == OrderStatus.PROCESSING: # Use idempotency key to avoid double-charging pass # Process with idempotency key result = payment_provider.charge( amount=order.total, idempotency_key=f"order-{order_id}", # Critical! ) orders_repo.update(order_id, status=OrderStatus.COMPLETED) ``` **Idempotency Strategies:** 1. **Check-before-write**: Verify state before action 2. **Idempotency keys**: Use unique tokens with external services 3. **Upsert patterns**: `INSERT ... ON CONFLICT UPDATE` 4. **Deduplication window**: Track processed IDs for N hours ### Pattern 4: Job State Management Persist job state transitions for visibility and debugging. ```python class JobRepository: """Repository for managing job state.""" async def create(self, job: Job) -> Job: """Create new job record.""" await self._db.execute( """INSERT INTO jobs (id, status, created_at) VALUES ($1, $2, $3)""", job.id, job.status.value, job.created_at, ) return job async def update_status( self, job_id: str, status: JobStatus, **fields, ) -> None: """Update job status with timestamp.""" updates = {"status": status.value, **fields} if status == JobStatus.RUNNING: updates["started_at"] = datetime.utcnow() elif status in (JobStatus.SUCCEEDED, JobStatus.FAILED): updates["completed_at"] = datetime.utcnow() await self._db.execute( "UPDATE jobs SET status = $1, ... WHERE id = $2", updates, job_id, ) logger.info( "Job status updated", job_id=job_id, status=status.value, ) ``` ## Advanced Patterns ### Pattern 5: Dead Letter Queue Handle permanently failed tasks for manual inspection. ```python @app.task(bind=True, max_retries=3) def process_webhook(self, webhook_id: str, payload: dict) -> None: """Process webhook with DLQ for failures.""" try: result = send_webhook(payload) if not result.success: raise WebhookFailedError(result.error) except Exception as e: if self.request.retries >= self.max_retries: # Move to dead letter queue for manual inspection dead_letter_queue.send({ "task": "process_webhook", "webhook_id": webhook_id, "payload": payload, "error": str(e), "attempts": self.request.retries + 1, "failed_at": datetime.utcnow().isoformat(), }) logger.error( "Webhook moved to DLQ after max retries", webhook_id=webhook_id, error=str(e), ) return # Exponential backoff retry raise self.retry(exc=e, countdown=2 ** self.request.retries * 60) ``` ### Pattern 6: Status Polling Endpoint Provide an endpoint for clients to check job status. ```python from fastapi import FastAPI, HTTPException app = FastAPI() @app.get("/jobs/{job_id}") async def get_job_status(job_id: str) -> JobStatusResponse: """Get current status of a background job.""" job = await jobs_repo.get(job_id) if job is None: raise HTTPException(404, f"Job {job_id} not found") return JobStatusResponse( job_id=job.id, status=job.status.value, created_at=job.created_at, started_at=job.started_at, completed_at=job.completed_at, result=job.result if job.status == JobStatus.SUCCEEDED else None, error=job.error if job.status == JobStatus.FAILED else None, # Helpful for clients is_terminal=job.status in (JobStatus.SUCCEEDED, JobStatus.FAILED), ) ``` ### Pattern 7: Task Chaining and Workflows Compose complex workflows from simple tasks. ```python from celery import chain, group, chord # Simple chain: A → B → C workflow = chain( extract_data.s(source_id), transform_data.s(), load_data.s(destination_id), ) # Parallel execution: A, B, C all at once parallel = group( send_email.s(user_email), send_sms.s(user_phone), update_analytics.s(event_data), ) # Chord: Run tasks in parallel, then a callback # Process all items, then send completion notification workflow = chord( [process_item.s(item_id) for item_id in item_ids], send_completion_notification.s(batch_id), ) workflow.apply_async() ``` ### Pattern 8: Alternative Task Queues Choose the right tool for your needs. **RQ (Redis Queue)**: Simple, Redis-based ```python from rq import Queue from redis import Redis queue = Queue(connection=Redis()) job = queue.enqueue(send_email, "user@example.com", "Subject", "Body") ``` **Dramatiq**: Modern Celery alternative ```python import dramatiq from dramatiq.brokers.redis import RedisBroker dramatiq.set_broker(RedisBroker()) @dramatiq.actor def send_email(to: str, subject: str, body: str) -> None: email_client.send(to, subject, body) ``` **Cloud-native options:** - AWS SQS + Lambda - Google Cloud Tasks - Azure Functions ## Best Practices Summary 1. **Return immediately** - Don't block requests for long operations 2. **Persist job state** - Enable status polling and debugging 3. **Make tasks idempotent** - Safe to retry on any failure 4. **Use idempotency keys** - For external service calls 5. **Set timeouts** - Both soft and hard limits 6. **Implement DLQ** - Capture permanently failed tasks 7. **Log transitions** - Track job state changes 8. **Retry appropriately** - Exponential backoff for transient errors 9. **Don't retry permanent failures** - Validation errors, invalid credentials 10. **Monitor queue depth** - Alert on backlog growth

# Python Code Style & Documentation Consistent code style and clear documentation make codebases maintainable and collaborative. This skill covers modern Python tooling, naming conventions, and documentation standards. ## When to Use This Skill - Setting up linting and formatting for a new project - Writing or reviewing docstrings - Establishing team coding standards - Configuring ruff, mypy, or pyright - Reviewing code for style consistency - Creating project documentation ## Core Concepts ### 1. Automated Formatting Let tools handle formatting debates. Configure once, enforce automatically. ### 2. Consistent Naming Follow PEP 8 conventions with meaningful, descriptive names. ### 3. Documentation as Code Docstrings should be maintained alongside the code they describe. ### 4. Type Annotations Modern Python code should include type hints for all public APIs. ## Quick Start ```bash # Install modern tooling pip install ruff mypy # Configure in pyproject.toml [tool.ruff] line-length = 120 target-version = "py312" # Adjust based on your project's minimum Python version [tool.mypy] strict = true ``` ## Fundamental Patterns ### Pattern 1: Modern Python Tooling Use `ruff` as an all-in-one linter and formatter. It replaces flake8, isort, and black with a single fast tool. ```toml # pyproject.toml [tool.ruff] line-length = 120 target-version = "py312" # Adjust based on your project's minimum Python version [tool.ruff.lint] select = [ "E", # pycodestyle errors "W", # pycodestyle warnings "F", # pyflakes "I", # isort "B", # flake8-bugbear "C4", # flake8-comprehensions "UP", # pyupgrade "SIM", # flake8-simplify ] ignore = ["E501"] # Line length handled by formatter [tool.ruff.format] quote-style = "double" indent-style = "space" ``` Run with: ```bash ruff check --fix . # Lint and auto-fix ruff format . # Format code ``` ### Pattern 2: Type Checking Configuration Configure strict type checking for production code. ```toml # pyproject.toml [tool.mypy] python_version = "3.12" strict = true warn_return_any = true warn_unused_ignores = true disallow_untyped_defs = true disallow_incomplete_defs = true [[tool.mypy.overrides]] module = "tests.*" disallow_untyped_defs = false ``` Alternative: Use `pyright` for faster checking. ```toml [tool.pyright] pythonVersion = "3.12" typeCheckingMode = "strict" ``` ### Pattern 3: Naming Conventions Follow PEP 8 with emphasis on clarity over brevity. **Files and Modules:** ```python # Good: Descriptive snake_case user_repository.py order_processing.py http_client.py # Avoid: Abbreviations usr_repo.py ord_proc.py http_cli.py ``` **Classes and Functions:** ```python # Classes: PascalCase class UserRepository: pass class HTTPClientFactory: # Acronyms stay uppercase pass # Functions and variables: snake_case def get_user_by_email(email: str) -> User | None: retry_count = 3 max_connections = 100 ``` **Constants:** ```python # Module-level constants: SCREAMING_SNAKE_CASE MAX_RETRY_ATTEMPTS = 3 DEFAULT_TIMEOUT_SECONDS = 30 API_BASE_URL = "https://api.example.com" ``` ### Pattern 4: Import Organization Group imports in a consistent order: standard library, third-party, local. ```python # Standard library import os from collections.abc import Callable from typing import Any # Third-party packages import httpx from pydantic import BaseModel from sqlalchemy import Column # Local imports from myproject.models import User from myproject.services import UserService ``` Use absolute imports exclusively: ```python # Preferred from myproject.utils import retry_decorator # Avoid relative imports from ..utils import retry_decorator ``` ## Advanced Patterns ### Pattern 5: Google-Style Docstrings Write docstrings for all public classes, methods, and functions. **Simple Function:** ```python def get_user(user_id: str) -> User: """Retrieve a user by their unique identifier.""" ... ``` **Complex Function:** ```python def process_batch( items: list[Item], max_workers: int = 4, on_progress: Callable[[int, int], None] | None = None, ) -> BatchResult: """Process items concurrently using a worker pool. Processes each item in the batch using the configured number of workers. Progress can be monitored via the optional callback. Args: items: The items to process. Must not be empty. max_workers: Maximum concurrent workers. Defaults to 4. on_progress: Optional callback receiving (completed, total) counts. Returns: BatchResult containing succeeded items and any failures with their associated exceptions. Raises: ValueError: If items is empty. ProcessingError: If the batch cannot be processed. Example: >>> result = process_batch(items, max_workers=8) >>> print(f"Processed {len(result.succeeded)} items") """ ... ``` **Class Docstring:** ```python class UserService: """Service for managing user operations. Provides methods for creating, retrieving, updating, and deleting users with proper validation and error handling. Attributes: repository: The data access layer for user persistence. logger: Logger instance for operation tracking. Example: >>> service = UserService(repository, logger) >>> user = service.create_user(CreateUserInput(...)) """ def __init__(self, repository: UserRepository, logger: Logger) -> None: """Initialize the user service. Args: repository: Data access layer for users. logger: Logger for tracking operations. """ self.repository = repository self.logger = logger ``` ### Pattern 6: Line Length and Formatting Set line length to 120 characters for modern displays while maintaining readability. ```python # Good: Readable line breaks def create_user( email: str, name: str, role: UserRole = UserRole.MEMBER, notify: bool = True, ) -> User: ... # Good: Chain method calls clearly result = ( db.query(User) .filter(User.active == True) .order_by(User.created_at.desc()) .limit(10) .all() ) # Good: Format long strings error_message = ( f"Failed to process user {user_id}: " f"received status {response.status_code} " f"with body {response.text[:100]}" ) ``` ### Pattern 7: Project Documentation **README Structure:** ```markdown # Project Name Brief description of what the project does. ## Installation \`\`\`bash pip install myproject \`\`\` ## Quick Start \`\`\`python from myproject import Client client = Client(api_key="...") result = client.process(data) \`\`\` ## Configuration Document environment variables and configuration options. ## Development \`\`\`bash pip install -e ".[dev]" pytest \`\`\` ``` **CHANGELOG Format (Keep a Changelog):** ```markdown # Changelog ## [Unreleased] ### Added - New feature X ### Changed - Modified behavior of Y ### Fixed - Bug in Z ``` ## Best Practices Summary 1. **Use ruff** - Single tool for linting and formatting 2. **Enable strict mypy** - Catch type errors before runtime 3. **120 character lines** - Modern standard for readability 4. **Descriptive names** - Clarity over brevity 5. **Absolute imports** - More maintainable than relative 6. **Google-style docstrings** - Consistent, readable documentation 7. **Document public APIs** - Every public function needs a docstring 8. **Keep docs updated** - Treat documentation as code 9. **Automate in CI** - Run linters on every commit 10. **Target Python 3.10+** - For new projects, Python 3.12+ is recommended for modern language features

# Python Packaging Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI. ## When to Use This Skill - Creating Python libraries for distribution - Building command-line tools with entry points - Publishing packages to PyPI or private repositories - Setting up Python project structure - Creating installable packages with dependencies - Building wheels and source distributions - Versioning and releasing Python packages - Creating namespace packages - Implementing package metadata and classifiers ## Core Concepts ### 1. Package Structure - **Source layout**: `src/package_name/` (recommended) - **Flat layout**: `package_name/` (simpler but less flexible) - **Package metadata**: pyproject.toml, setup.py, or setup.cfg - **Distribution formats**: wheel (.whl) and source distribution (.tar.gz) ### 2. Modern Packaging Standards - **PEP 517/518**: Build system requirements - **PEP 621**: Metadata in pyproject.toml - **PEP 660**: Editable installs - **pyproject.toml**: Single source of configuration ### 3. Build Backends - **setuptools**: Traditional, widely used - **hatchling**: Modern, opinionated - **flit**: Lightweight, for pure Python - **poetry**: Dependency management + packaging ### 4. Distribution - **PyPI**: Python Package Index (public) - **TestPyPI**: Testing before production - **Private repositories**: JFrog, AWS CodeArtifact, etc. ## Quick Start ### Minimal Package Structure ``` my-package/ ├── pyproject.toml ├── README.md ├── LICENSE ├── src/ │ └── my_package/ │ ├── __init__.py │ └── module.py └── tests/ └── test_module.py ``` ### Minimal pyproject.toml ```toml [build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta" [project] name = "my-package" version = "0.1.0" description = "A short description" authors = [{name = "Your Name", email = "you@example.com"}] readme = "README.md" requires-python = ">=3.8" dependencies = [ "requests>=2.28.0", ] [project.optional-dependencies] dev = [ "pytest>=7.0", "black>=22.0", ] ``` ## Package Structure Patterns ### Pattern 1: Source Layout (Recommended) ``` my-package/ ├── pyproject.toml ├── README.md ├── LICENSE ├── .gitignore ├── src/ │ └── my_package/ │ ├── __init__.py │ ├── core.py │ ├── utils.py │ └── py.typed # For type hints ├── tests/ │ ├── __init__.py │ ├── test_core.py │ └── test_utils.py └── docs/ └── index.md ``` **Advantages:** - Prevents accidentally importing from source - Cleaner test imports - Better isolation **pyproject.toml for source layout:** ```toml [tool.setuptools.packages.find] where = ["src"] ``` ### Pattern 2: Flat Layout ``` my-package/ ├── pyproject.toml ├── README.md ├── my_package/ │ ├── __init__.py │ └── module.py └── tests/ └── test_module.py ``` **Simpler but:** - Can import package without installing - Less professional for libraries ### Pattern 3: Multi-Package Project ``` project/ ├── pyproject.toml ├── packages/ │ ├── package-a/ │ │ └── src/ │ │ └── package_a/ │ └── package-b/ │ └── src/ │ └── package_b/ └── tests/ ``` ## Complete pyproject.toml Examples ### Pattern 4: Full-Featured pyproject.toml ```toml [build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "my-awesome-package" version = "1.0.0" description = "An awesome Python package" readme = "README.md" requires-python = ">=3.8" license = {text = "MIT"} authors = [ {name = "Your Name", email = "you@example.com"}, ] maintainers = [ {name = "Maintainer Name", email = "maintainer@example.com"}, ] keywords = ["example", "package", "awesome"] classifiers = [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", ] dependencies = [ "requests>=2.28.0,<3.0.0", "click>=8.0.0", "pydantic>=2.0.0", ] [project.optional-dependencies] dev = [ "pytest>=7.0.0", "pytest-cov>=4.0.0", "black>=23.0.0", "ruff>=0.1.0", "mypy>=1.0.0", ] docs = [ "sphinx>=5.0.0", "sphinx-rtd-theme>=1.0.0", ] all = [ "my-awesome-package[dev,docs]", ] [project.urls] Homepage = "https://github.com/username/my-awesome-package" Documentation = "https://my-awesome-package.readthedocs.io" Repository = "https://github.com/username/my-awesome-package" "Bug Tracker" = "https://github.com/username/my-awesome-package/issues" Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md" [project.scripts] my-cli = "my_package.cli:main" awesome-tool = "my_package.tools:run" [project.entry-points."my_package.plugins"] plugin1 = "my_package.plugins:plugin1" [tool.setuptools] package-dir = {"" = "src"} zip-safe = false [tool.setuptools.packages.find] where = ["src"] include = ["my_package*"] exclude = ["tests*"] [tool.setuptools.package-data] my_package = ["py.typed", "*.pyi", "data/*.json"] # Black configuration [tool.black] line-length = 100 target-version = ["py38", "py39", "py310", "py311"] include = '\.pyi?$' # Ruff configuration [tool.ruff] line-length = 100 target-version = "py38" [tool.ruff.lint] select = ["E", "F", "I", "N", "W", "UP"] # MyPy configuration [tool.mypy] python_version = "3.8" warn_return_any = true warn_unused_configs = true disallow_untyped_defs = true # Pytest configuration [tool.pytest.ini_options] testpaths = ["tests"] python_files = ["test_*.py"] addopts = "-v --cov=my_package --cov-report=term-missing" # Coverage configuration [tool.coverage.run] source = ["src"] omit = ["*/tests/*"] [tool.coverage.report] exclude_lines = [ "pragma: no cover", "def __repr__", "raise AssertionError", "raise NotImplementedError", ] ``` ### Pattern 5: Dynamic Versioning ```toml [build-system] requires = ["setuptools>=61.0", "setuptools-scm>=8.0"] build-backend = "setuptools.build_meta" [project] name = "my-package" dynamic = ["version"] description = "Package with dynamic version" [tool.setuptools.dynamic] version = {attr = "my_package.__version__"} # Or use setuptools-scm for git-based versioning [tool.setuptools_scm] write_to = "src/my_package/_version.py" ``` **In **init**.py:** ```python # src/my_package/__init__.py __version__ = "1.0.0" # Or with setuptools-scm from importlib.metadata import version __version__ = version("my-package") ``` ## Command-Line Interface (CLI) Patterns ### Pattern 6: CLI with Click ```python # src/my_package/cli.py import click @click.group() @click.version_option() def cli(): """My awesome CLI tool.""" pass @cli.command() @click.argument("name") @click.option("--greeting", default="Hello", help="Greeting to use") def greet(name: str, greeting: str): """Greet someone.""" click.echo(f"{greeting}, {name}!") @cli.command() @click.option("--count", default=1, help="Number of times to repeat") def repeat(count: int): """Repeat a message.""" for i in range(count): click.echo(f"Message {i + 1}") def main(): """Entry point for CLI.""" cli() if __name__ == "__main__": main() ``` **Register in pyproject.toml:** ```toml [project.scripts] my-tool = "my_package.cli:main" ``` **Usage:** ```bash pip install -e . my-tool greet World my-tool greet Alice --greeting="Hi" my-tool repeat --count=3 ``` ### Pattern 7: CLI with argparse ```python # src/my_package/cli.py import argparse import sys def main(): """Main CLI entry point.""" parser = argparse.ArgumentParser( description="My awesome tool", prog="my-tool" ) parser.add_argument( "--version", action="version", version="%(prog)s 1.0.0" ) subparsers = parser.add_subparsers(dest="command", help="Commands") # Add subcommand process_parser = subparsers.add_parser("process", help="Process data") process_parser.add_argument("input_file", help="Input file path") process_parser.add_argument( "--output", "-o", default="output.txt", help="Output file path" ) args = parser.parse_args() if args.command == "process": process_data(args.input_file, args.output) else: parser.print_help() sys.exit(1) def process_data(input_file: str, output_file: str): """Process data from input to output.""" print(f"Processing {input_file} -> {output_file}") if __name__ == "__main__": main() ``` ## Building and Publishing ### Pattern 8: Build Package Locally ```bash # Install build tools pip install build twine # Build distribution python -m build # This creates: # dist/ # my-package-1.0.0.tar.gz (source distribution) # my_package-1.0.0-py3-none-any.whl (wheel) # Check the distribution twine check dist/* ``` ### Pattern 9: Publishing to PyPI ```bash # Install publishing tools pip install twine # Test on TestPyPI first twine upload --repository testpypi dist/* # Install from TestPyPI to test pip install --index-url https://test.pypi.org/simple/ my-package # If all good, publish to PyPI twine upload dist/* ``` **Using API tokens (recommended):** ```bash # Create ~/.pypirc [distutils] index-servers = pypi testpypi [pypi] username = __token__ password = pypi-...your-token... [testpypi] username = __token__ password = pypi-...your-test-token... ``` ### Pattern 10: Automated Publishing with GitHub Actions ```yaml # .github/workflows/publish.yml name: Publish to PyPI on: release: types: [created] jobs: publish: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Set up Python uses: actions/setup-python@v4 with: python-version: "3.11" - name: Install dependencies run: | pip install build twine - name: Build package run: python -m build - name: Check package run: twine check dist/* - name: Publish to PyPI env: TWINE_USERNAME: __token__ TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }} run: twine upload dist/* ``` ## Advanced Patterns ### Pattern 11: Including Data Files ```toml [tool.setuptools.package-data] my_package = [ "data/*.json", "templates/*.html", "static/css/*.css", "py.typed", ] ``` **Accessing data files:** ```python # src/my_package/loader.py from importlib.resources import files import json def load_config(): """Load configuration from package data.""" config_file = files("my_package").joinpath("data/config.json") with config_file.open() as f: return json.load(f) # Python 3.9+ from importlib.resources import files data = files("my_package").joinpath("data/file.txt").read_text() ``` ### Pattern 12: Namespace Packages **For large projects split across multiple repositories:** ``` # Package 1: company-core company/ └── core/ ├── __init__.py └── models.py # Package 2: company-api company/ └── api/ ├── __init__.py └── routes.py ``` **Do NOT include **init**.py in the namespace directory (company/):** ```toml # company-core/pyproject.toml [project] name = "company-core" [tool.setuptools.packages.find] where = ["."] include = ["company.core*"] # company-api/pyproject.toml [project] name = "company-api" [tool.setuptools.packages.find] where = ["."] include = ["company.api*"] ``` **Usage:** ```python # Both packages can be imported under same namespace from company.core import models from company.api import routes ``` ### Pattern 13: C Extensions ```toml [build-system] requires = ["setuptools>=61.0", "wheel", "Cython>=0.29"] build-backend = "setuptools.build_meta" [tool.setuptools] ext-modules = [ {name = "my_package.fast_module", sources = ["src/fast_module.c"]}, ] ``` **Or with setup.py:** ```python # setup.py from setuptools import setup, Extension setup( ext_modules=[ Extension( "my_package.fast_module", sources=["src/fast_module.c"], include_dirs=["src/include"], ) ] ) ``` ## Version Management ### Pattern 14: Semantic Versioning ```python # src/my_package/__init__.py __version__ = "1.2.3" # Semantic versioning: MAJOR.MINOR.PATCH # MAJOR: Breaking changes # MINOR: New features (backward compatible) # PATCH: Bug fixes ``` **Version constraints in dependencies:** ```toml dependencies = [ "requests>=2.28.0,<3.0.0", # Compatible range "click~=8.1.0", # Compatible release (~= 8.1.0 means >=8.1.0,<8.2.0) "pydantic>=2.0", # Minimum version "numpy==1.24.3", # Exact version (avoid if possible) ] ``` ### Pattern 15: Git-Based Versioning ```toml [build-system] requires = ["setuptools>=61.0", "setuptools-scm>=8.0"] build-backend = "setuptools.build_meta" [project] name = "my-package" dynamic = ["version"] [tool.setuptools_scm] write_to = "src/my_package/_version.py" version_scheme = "post-release" local_scheme = "dirty-tag" ``` **Creates versions like:** - `1.0.0` (from git tag) - `1.0.1.dev3+g1234567` (3 commits after tag) ## Testing Installation ### Pattern 16: Editable Install ```bash # Install in development mode pip install -e . # With optional dependencies pip install -e ".[dev]" pip install -e ".[dev,docs]" # Now changes to source code are immediately reflected ``` ### Pattern 17: Testing in Isolated Environment ```bash # Create virtual environment python -m venv test-env source test-env/bin/activate # Linux/Mac # test-env\Scripts\activate # Windows # Install package pip install dist/my_package-1.0.0-py3-none-any.whl # Test it works python -c "import my_package; print(my_package.__version__)" # Test CLI my-tool --help # Cleanup deactivate rm -rf test-env ``` ## Documentation ### Pattern 18: README.md Template ````markdown # My Package [](https://pypi.org/project/my-package/) [](https://pypi.org/project/my-package/) [](https://github.com/username/my-package/actions) Brief description of your package. ## Installation ```bash pip install my-package ``` ```` ## Quick Start ```python from my_package import something result = something.do_stuff() ``` ## Features - Feature 1 - Feature 2 - Feature 3 ## Documentation Full documentation: https://my-package.readthedocs.io ## Development ```bash git clone https://github.com/username/my-package.git cd my-package pip install -e ".[dev]" pytest ``` ## License MIT ```` ## Common Patterns ### Pattern 19: Multi-Architecture Wheels ```yaml # .github/workflows/wheels.yml name: Build wheels on: [push, pull_request] jobs: build_wheels: name: Build wheels on ${{ matrix.os }} runs-on: ${{ matrix.os }} strategy: matrix: os: [ubuntu-latest, windows-latest, macos-latest] steps: - uses: actions/checkout@v3 - name: Build wheels uses: pypa/cibuildwheel@v2.16.2 - uses: actions/upload-artifact@v3 with: path: ./wheelhouse/*.whl ```` ### Pattern 20: Private Package Index ```bash # Install from private index pip install my-package --index-url https://private.pypi.org/simple/ # Or add to pip.conf [global] index-url = https://private.pypi.org/simple/ extra-index-url = https://pypi.org/simple/ # Upload to private index twine upload --repository-url https://private.pypi.org/ dist/* ``` ## File Templates ### .gitignore for Python Packages ```gitignore # Build artifacts build/ dist/ *.egg-info/ *.egg .eggs/ # Python __pycache__/ *.py[cod] *$py.class *.so # Virtual environments venv/ env/ ENV/ # IDE .vscode/ .idea/ *.swp # Testing .pytest_cache/ .coverage htmlcov/ # Distribution *.whl *.tar.gz ``` ### MANIFEST.in ``` # MANIFEST.in include README.md include LICENSE include pyproject.toml recursive-include src/my_package/data *.json recursive-include src/my_package/templates *.html recursive-exclude * __pycache__ recursive-exclude * *.py[co] ``` ## Checklist for Publishing - [ ] Code is tested (pytest passing) - [ ] Documentation is complete (README, docstrings) - [ ] Version number updated - [ ] CHANGELOG.md updated - [ ] License file included - [ ] pyproject.toml is complete - [ ] Package builds without errors - [ ] Installation tested in clean environment - [ ] CLI tools work (if applicable) - [ ] PyPI metadata is correct (classifiers, keywords) - [ ] GitHub repository linked - [ ] Tested on TestPyPI first - [ ] Git tag created for release ## Resources - **Python Packaging Guide**: https://packaging.python.org/ - **PyPI**: https://pypi.org/ - **TestPyPI**: https://test.pypi.org/ - **setuptools documentation**: https://setuptools.pypa.io/ - **build**: https://pypa-build.readthedocs.io/ - **twine**: https://twine.readthedocs.io/ ## Best Practices Summary 1. **Use src/ layout** for cleaner package structure 2. **Use pyproject.toml** for modern packaging 3. **Pin build dependencies** in build-system.requires 4. **Version appropriately** with semantic versioning 5. **Include all metadata** (classifiers, URLs, etc.) 6. **Test installation** in clean environments 7. **Use TestPyPI** before publishing to PyPI 8. **Document thoroughly** with README and docstrings 9. **Include LICENSE** file 10. **Automate publishing** with CI/CD

# Python Type Safety Leverage Python's type system to catch errors at static analysis time. Type annotations serve as enforced documentation that tooling validates automatically. ## When to Use This Skill - Adding type hints to existing code - Creating generic, reusable classes - Defining structural interfaces with protocols - Configuring mypy or pyright for strict checking - Understanding type narrowing and guards - Building type-safe APIs and libraries ## Core Concepts ### 1. Type Annotations Declare expected types for function parameters, return values, and variables. ### 2. Generics Write reusable code that preserves type information across different types. ### 3. Protocols Define structural interfaces without inheritance (duck typing with type safety). ### 4. Type Narrowing Use guards and conditionals to narrow types within code blocks. ## Quick Start ```python def get_user(user_id: str) -> User | None: """Return type makes 'might not exist' explicit.""" ... # Type checker enforces handling None case user = get_user("123") if user is None: raise UserNotFoundError("123") print(user.name) # Type checker knows user is User here ``` ## Fundamental Patterns ### Pattern 1: Annotate All Public Signatures Every public function, method, and class should have type annotations. ```python def get_user(user_id: str) -> User: """Retrieve user by ID.""" ... def process_batch( items: list[Item], max_workers: int = 4, ) -> BatchResult[ProcessedItem]: """Process items concurrently.""" ... class UserRepository: def __init__(self, db: Database) -> None: self._db = db async def find_by_id(self, user_id: str) -> User | None: """Return User if found, None otherwise.""" ... async def find_by_email(self, email: str) -> User | None: ... async def save(self, user: User) -> User: """Save and return user with generated ID.""" ... ``` Use `mypy --strict` or `pyright` in CI to catch type errors early. For existing projects, enable strict mode incrementally using per-module overrides. ### Pattern 2: Use Modern Union Syntax Python 3.10+ provides cleaner union syntax. ```python # Preferred (3.10+) def find_user(user_id: str) -> User | None: ... def parse_value(v: str) -> int | float | str: ... # Older style (still valid, needed for 3.9) from typing import Optional, Union def find_user(user_id: str) -> Optional[User]: ... ``` ### Pattern 3: Type Narrowing with Guards Use conditionals to narrow types for the type checker. ```python def process_user(user_id: str) -> UserData: user = find_user(user_id) if user is None: raise UserNotFoundError(f"User {user_id} not found") # Type checker knows user is User here, not User | None return UserData( name=user.name, email=user.email, ) def process_items(items: list[Item | None]) -> list[ProcessedItem]: # Filter and narrow types valid_items = [item for item in items if item is not None] # valid_items is now list[Item] return [process(item) for item in valid_items] ``` ### Pattern 4: Generic Classes Create type-safe reusable containers. ```python from typing import TypeVar, Generic T = TypeVar("T") E = TypeVar("E", bound=Exception) class Result(Generic[T, E]): """Represents either a success value or an error.""" def __init__( self, value: T | None = None, error: E | None = None, ) -> None: if (value is None) == (error is None): raise ValueError("Exactly one of value or error must be set") self._value = value self._error = error @property def is_success(self) -> bool: return self._error is None @property def is_failure(self) -> bool: return self._error is not None def unwrap(self) -> T: """Get value or raise the error.""" if self._error is not None: raise self._error return self._value # type: ignore[return-value] def unwrap_or(self, default: T) -> T: """Get value or return default.""" if self._error is not None: return default return self._value # type: ignore[return-value] # Usage preserves types def parse_config(path: str) -> Result[Config, ConfigError]: try: return Result(value=Config.from_file(path)) except ConfigError as e: return Result(error=e) result = parse_config("config.yaml") if result.is_success: config = result.unwrap() # Type: Config ``` ## Advanced Patterns ### Pattern 5: Generic Repository Create type-safe data access patterns. ```python from typing import TypeVar, Generic from abc import ABC, abstractmethod T = TypeVar("T") ID = TypeVar("ID") class Repository(ABC, Generic[T, ID]): """Generic repository interface.""" @abstractmethod async def get(self, id: ID) -> T | None: """Get entity by ID.""" ... @abstractmethod async def save(self, entity: T) -> T: """Save and return entity.""" ... @abstractmethod async def delete(self, id: ID) -> bool: """Delete entity, return True if existed.""" ... class UserRepository(Repository[User, str]): """Concrete repository for Users with string IDs.""" async def get(self, id: str) -> User | None: row = await self._db.fetchrow( "SELECT * FROM users WHERE id = $1", id ) return User(**row) if row else None async def save(self, entity: User) -> User: ... async def delete(self, id: str) -> bool: ... ``` ### Pattern 6: TypeVar with Bounds Restrict generic parameters to specific types. ```python from typing import TypeVar from pydantic import BaseModel ModelT = TypeVar("ModelT", bound=BaseModel) def validate_and_create(model_cls: type[ModelT], data: dict) -> ModelT: """Create a validated Pydantic model from dict.""" return model_cls.model_validate(data) # Works with any BaseModel subclass class User(BaseModel): name: str email: str user = validate_and_create(User, {"name": "Alice", "email": "a@b.com"}) # user is typed as User # Type error: str is not a BaseModel subclass result = validate_and_create(str, {"name": "Alice"}) # Error! ``` ### Pattern 7: Protocols for Structural Typing Define interfaces without requiring inheritance. ```python from typing import Protocol, runtime_checkable @runtime_checkable class Serializable(Protocol): """Any class that can be serialized to/from dict.""" def to_dict(self) -> dict: ... @classmethod def from_dict(cls, data: dict) -> "Serializable": ... # User satisfies Serializable without inheriting from it class User: def __init__(self, id: str, name: str) -> None: self.id = id self.name = name def to_dict(self) -> dict: return {"id": self.id, "name": self.name} @classmethod def from_dict(cls, data: dict) -> "User": return cls(id=data["id"], name=data["name"]) def serialize(obj: Serializable) -> str: """Works with any Serializable object.""" return json.dumps(obj.to_dict()) # Works - User matches the protocol serialize(User("1", "Alice")) # Runtime checking with @runtime_checkable isinstance(User("1", "Alice"), Serializable) # True ``` ### Pattern 8: Common Protocol Patterns Define reusable structural interfaces. ```python from typing import Protocol class Closeable(Protocol): """Resource that can be closed.""" def close(self) -> None: ... class AsyncCloseable(Protocol): """Async resource that can be closed.""" async def close(self) -> None: ... class Readable(Protocol): """Object that can be read from.""" def read(self, n: int = -1) -> bytes: ... class HasId(Protocol): """Object with an ID property.""" @property def id(self) -> str: ... class Comparable(Protocol): """Object that supports comparison.""" def __lt__(self, other: "Comparable") -> bool: ... def __le__(self, other: "Comparable") -> bool: ... ``` ### Pattern 9: Type Aliases Create meaningful type names. **Note:** The `type` statement was introduced in Python 3.10 for simple aliases. Generic type statements require Python 3.12+. ```python # Python 3.10+ type statement for simple aliases type UserId = str type UserDict = dict[str, Any] # Python 3.12+ type statement with generics type Handler[T] = Callable[[Request], T] type AsyncHandler[T] = Callable[[Request], Awaitable[T]] # Python 3.9-3.11 style (needed for broader compatibility) from typing import TypeAlias from collections.abc import Callable, Awaitable UserId: TypeAlias = str Handler: TypeAlias = Callable[[Request], Response] # Usage def register_handler(path: str, handler: Handler[Response]) -> None: ... ``` ### Pattern 10: Callable Types Type function parameters and callbacks. ```python from collections.abc import Callable, Awaitable # Sync callback ProgressCallback = Callable[[int, int], None] # (current, total) # Async callback AsyncHandler = Callable[[Request], Awaitable[Response]] # With named parameters (using Protocol) class OnProgress(Protocol): def __call__( self, current: int, total: int, *, message: str = "", ) -> None: ... def process_items( items: list[Item], on_progress: ProgressCallback | None = None, ) -> list[Result]: for i, item in enumerate(items): if on_progress: on_progress(i, len(items)) ... ``` ## Configuration ### Strict Mode Checklist For `mypy --strict` compliance: ```toml # pyproject.toml [tool.mypy] python_version = "3.12" strict = true warn_return_any = true warn_unused_ignores = true disallow_untyped_defs = true disallow_incomplete_defs = true no_implicit_optional = true ``` Incremental adoption goals: - All function parameters annotated - All return types annotated - Class attributes annotated - Minimize `Any` usage (acceptable for truly dynamic data) - Generic collections use type parameters (`list[str]` not `list`) For existing codebases, enable strict mode per-module using `# mypy: strict` or configure per-module overrides in `pyproject.toml`. ## Best Practices Summary 1. **Annotate all public APIs** - Functions, methods, class attributes 2. **Use `T | None`** - Modern union syntax over `Optional[T]` 3. **Run strict type checking** - `mypy --strict` in CI 4. **Use generics** - Preserve type info in reusable code 5. **Define protocols** - Structural typing for interfaces 6. **Narrow types** - Use guards to help the type checker 7. **Bound type vars** - Restrict generics to meaningful types 8. **Create type aliases** - Meaningful names for complex types 9. **Minimize `Any`** - Use specific types or generics. `Any` is acceptable for truly dynamic data or when interfacing with untyped third-party code 10. **Document with types** - Types are enforceable documentation

# UV Package Manager Comprehensive guide to using uv, an extremely fast Python package installer and resolver written in Rust, for modern Python project management and dependency workflows. ## When to Use This Skill - Setting up new Python projects quickly - Managing Python dependencies faster than pip - Creating and managing virtual environments - Installing Python interpreters - Resolving dependency conflicts efficiently - Migrating from pip/pip-tools/poetry - Speeding up CI/CD pipelines - Managing monorepo Python projects - Working with lockfiles for reproducible builds - Optimizing Docker builds with Python dependencies ## Core Concepts ### 1. What is uv? - **Ultra-fast package installer**: 10-100x faster than pip - **Written in Rust**: Leverages Rust's performance - **Drop-in pip replacement**: Compatible with pip workflows - **Virtual environment manager**: Create and manage venvs - **Python installer**: Download and manage Python versions - **Resolver**: Advanced dependency resolution - **Lockfile support**: Reproducible installations ### 2. Key Features - Blazing fast installation speeds - Disk space efficient with global cache - Compatible with pip, pip-tools, poetry - Comprehensive dependency resolution - Cross-platform support (Linux, macOS, Windows) - No Python required for installation - Built-in virtual environment support ### 3. UV vs Traditional Tools - **vs pip**: 10-100x faster, better resolver - **vs pip-tools**: Faster, simpler, better UX - **vs poetry**: Faster, less opinionated, lighter - **vs conda**: Faster, Python-focused ## Installation ### Quick Install ```bash # macOS/Linux curl -LsSf https://astral.sh/uv/install.sh | sh # Windows (PowerShell) powershell -c "irm https://astral.sh/uv/install.ps1 | iex" # Using pip (if you already have Python) pip install uv # Using Homebrew (macOS) brew install uv # Using cargo (if you have Rust) cargo install --git https://github.com/astral-sh/uv uv ``` ### Verify Installation ```bash uv --version # uv 0.x.x ``` ## Quick Start ### Create a New Project ```bash # Create new project with virtual environment uv init my-project cd my-project # Or create in current directory uv init . # Initialize creates: # - .python-version (Python version) # - pyproject.toml (project config) # - README.md # - .gitignore ``` ### Install Dependencies ```bash # Install packages (creates venv if needed) uv add requests pandas # Install dev dependencies uv add --dev pytest black ruff # Install from requirements.txt uv pip install -r requirements.txt # Install from pyproject.toml uv sync ``` ## Virtual Environment Management ### Pattern 1: Creating Virtual Environments ```bash # Create virtual environment with uv uv venv # Create with specific Python version uv venv --python 3.12 # Create with custom name uv venv my-env # Create with system site packages uv venv --system-site-packages # Specify location uv venv /path/to/venv ``` ### Pattern 2: Activating Virtual Environments ```bash # Linux/macOS source .venv/bin/activate # Windows (Command Prompt) .venv\Scripts\activate.bat # Windows (PowerShell) .venv\Scripts\Activate.ps1 # Or use uv run (no activation needed) uv run python script.py uv run pytest ``` ### Pattern 3: Using uv run ```bash # Run Python script (auto-activates venv) uv run python app.py # Run installed CLI tool uv run black . uv run pytest # Run with specific Python version uv run --python 3.11 python script.py # Pass arguments uv run python script.py --arg value ``` ## Package Management ### Pattern 4: Adding Dependencies ```bash # Add package (adds to pyproject.toml) uv add requests # Add with version constraint uv add "django>=4.0,<5.0" # Add multiple packages uv add numpy pandas matplotlib # Add dev dependency uv add --dev pytest pytest-cov # Add optional dependency group uv add --optional docs sphinx # Add from git uv add git+https://github.com/user/repo.git # Add from git with specific ref uv add git+https://github.com/user/repo.git@v1.0.0 # Add from local path uv add ./local-package # Add editable local package uv add -e ./local-package ``` ### Pattern 5: Removing Dependencies ```bash # Remove package uv remove requests # Remove dev dependency uv remove --dev pytest # Remove multiple packages uv remove numpy pandas matplotlib ``` ### Pattern 6: Upgrading Dependencies ```bash # Upgrade specific package uv add --upgrade requests # Upgrade all packages uv sync --upgrade # Upgrade package to latest uv add --upgrade requests # Show what would be upgraded uv tree --outdated ``` ### Pattern 7: Locking Dependencies ```bash # Generate uv.lock file uv lock # Update lock file uv lock --upgrade # Lock without installing uv lock --no-install # Lock specific package uv lock --upgrade-package requests ``` ## Python Version Management ### Pattern 8: Installing Python Versions ```bash # Install Python version uv python install 3.12 # Install multiple versions uv python install 3.11 3.12 3.13 # Install latest version uv python install # List installed versions uv python list # Find available versions uv python list --all-versions ``` ### Pattern 9: Setting Python Version ```bash # Set Python version for project uv python pin 3.12 # This creates/updates .python-version file # Use specific Python version for command uv --python 3.11 run python script.py # Create venv with specific version uv venv --python 3.12 ``` ## Project Configuration ### Pattern 10: pyproject.toml with uv ```toml [project] name = "my-project" version = "0.1.0" description = "My awesome project" readme = "README.md" requires-python = ">=3.8" dependencies = [ "requests>=2.31.0", "pydantic>=2.0.0", "click>=8.1.0", ] [project.optional-dependencies] dev = [ "pytest>=7.4.0", "pytest-cov>=4.1.0", "black>=23.0.0", "ruff>=0.1.0", "mypy>=1.5.0", ] docs = [ "sphinx>=7.0.0", "sphinx-rtd-theme>=1.3.0", ] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.uv] dev-dependencies = [ # Additional dev dependencies managed by uv ] [tool.uv.sources] # Custom package sources my-package = { git = "https://github.com/user/repo.git" } ``` ### Pattern 11: Using uv with Existing Projects ```bash # Migrate from requirements.txt uv add -r requirements.txt # Migrate from poetry # Already have pyproject.toml, just use: uv sync # Export to requirements.txt uv pip freeze > requirements.txt # Export with hashes uv pip freeze --require-hashes > requirements.txt ``` ## Advanced Workflows ### Pattern 12: Monorepo Support ```bash # Project structure # monorepo/ # packages/ # package-a/ # pyproject.toml # package-b/ # pyproject.toml # pyproject.toml (root) # Root pyproject.toml [tool.uv.workspace] members = ["packages/*"] # Install all workspace packages uv sync # Add workspace dependency uv add --path ./packages/package-a ``` ### Pattern 13: CI/CD Integration ```yaml # .github/workflows/test.yml name: Tests on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install uv uses: astral-sh/setup-uv@v2 with: enable-cache: true - name: Set up Python run: uv python install 3.12 - name: Install dependencies run: uv sync --all-extras --dev - name: Run tests run: uv run pytest - name: Run linting run: | uv run ruff check . uv run black --check . ``` ### Pattern 14: Docker Integration ```dockerfile # Dockerfile FROM python:3.12-slim # Install uv COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv # Set working directory WORKDIR /app # Copy dependency files COPY pyproject.toml uv.lock ./ # Install dependencies RUN uv sync --frozen --no-dev # Copy application code COPY . . # Run application CMD ["uv", "run", "python", "app.py"] ``` **Optimized multi-stage build:** ```dockerfile # Multi-stage Dockerfile FROM python:3.12-slim AS builder # Install uv COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv WORKDIR /app # Install dependencies to venv COPY pyproject.toml uv.lock ./ RUN uv sync --frozen --no-dev --no-editable # Runtime stage FROM python:3.12-slim WORKDIR /app # Copy venv from builder COPY --from=builder /app/.venv .venv COPY . . # Use venv ENV PATH="/app/.venv/bin:$PATH" CMD ["python", "app.py"] ``` ### Pattern 15: Lockfile Workflows ```bash # Create lockfile (uv.lock) uv lock # Install from lockfile (exact versions) uv sync --frozen # Update lockfile without installing uv lock --no-install # Upgrade specific package in lock uv lock --upgrade-package requests # Check if lockfile is up to date uv lock --check # Export lockfile to requirements.txt uv export --format requirements-txt > requirements.txt # Export with hashes for security uv export --format requirements-txt --hash > requirements.txt ``` ## Performance Optimization ### Pattern 16: Using Global Cache ```bash # UV automatically uses global cache at: # Linux: ~/.cache/uv # macOS: ~/Library/Caches/uv # Windows: %LOCALAPPDATA%\uv\cache # Clear cache uv cache clean # Check cache size uv cache dir ``` ### Pattern 17: Parallel Installation ```bash # UV installs packages in parallel by default # Control parallelism uv pip install --jobs 4 package1 package2 # No parallel (sequential) uv pip install --jobs 1 package ``` ### Pattern 18: Offline Mode ```bash # Install from cache only (no network) uv pip install --offline package # Sync from lockfile offline uv sync --frozen --offline ``` ## Comparison with Other Tools ### uv vs pip ```bash # pip python -m venv .venv source .venv/bin/activate pip install requests pandas numpy # ~30 seconds # uv uv venv uv add requests pandas numpy # ~2 seconds (10-15x faster) ``` ### uv vs poetry ```bash # poetry poetry init poetry add requests pandas poetry install # ~20 seconds # uv uv init uv add requests pandas uv sync # ~3 seconds (6-7x faster) ``` ### uv vs pip-tools ```bash # pip-tools pip-compile requirements.in pip-sync requirements.txt # ~15 seconds # uv uv lock uv sync --frozen # ~2 seconds (7-8x faster) ``` ## Common Workflows ### Pattern 19: Starting a New Project ```bash # Complete workflow uv init my-project cd my-project # Set Python version uv python pin 3.12 # Add dependencies uv add fastapi uvicorn pydantic # Add dev dependencies uv add --dev pytest black ruff mypy # Create structure mkdir -p src/my_project tests # Run tests uv run pytest # Format code uv run black . uv run ruff check . ``` ### Pattern 20: Maintaining Existing Project ```bash # Clone repository git clone https://github.com/user/project.git cd project # Install dependencies (creates venv automatically) uv sync # Install with dev dependencies uv sync --all-extras # Update dependencies uv lock --upgrade # Run application uv run python app.py # Run tests uv run pytest # Add new dependency uv add new-package # Commit updated files git add pyproject.toml uv.lock git commit -m "Add new-package dependency" ``` ## Tool Integration ### Pattern 21: Pre-commit Hooks ```yaml # .pre-commit-config.yaml repos: - repo: local hooks: - id: uv-lock name: uv lock entry: uv lock language: system pass_filenames: false - id: ruff name: ruff entry: uv run ruff check --fix language: system types: [python] - id: black name: black entry: uv run black language: system types: [python] ``` ### Pattern 22: VS Code Integration ```json // .vscode/settings.json { "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python", "python.terminal.activateEnvironment": true, "python.testing.pytestEnabled": true, "python.testing.pytestArgs": ["-v"], "python.linting.enabled": true, "python.formatting.provider": "black", "[python]": { "editor.defaultFormatter": "ms-python.black-formatter", "editor.formatOnSave": true } } ``` ## Troubleshooting ### Common Issues ```bash # Issue: uv not found # Solution: Add to PATH or reinstall echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.bashrc # Issue: Wrong Python version # Solution: Pin version explicitly uv python pin 3.12 uv venv --python 3.12 # Issue: Dependency conflict # Solution: Check resolution uv lock --verbose # Issue: Cache issues # Solution: Clear cache uv cache clean # Issue: Lockfile out of sync # Solution: Regenerate uv lock --upgrade ``` ## Best Practices ### Project Setup 1. **Always use lockfiles** for reproducibility 2. **Pin Python version** with .python-version 3. **Separate dev dependencies** from production 4. **Use uv run** instead of activating venv 5. **Commit uv.lock** to version control 6. **Use --frozen in CI** for consistent builds 7. **Leverage global cache** for speed 8. **Use workspace** for monorepos 9. **Export requirements.txt** for compatibility 10. **Keep uv updated** for latest features ### Performance Tips ```bash # Use frozen installs in CI uv sync --frozen # Use offline mode when possible uv sync --offline # Parallel operations (automatic) # uv does this by default # Reuse cache across environments # uv shares cache globally # Use lockfiles to skip resolution uv sync --frozen # skips resolution ``` ## Migration Guide ### From pip + requirements.txt ```bash # Before python -m venv .venv source .venv/bin/activate pip install -r requirements.txt # After uv venv uv pip install -r requirements.txt # Or better: uv init uv add -r requirements.txt ``` ### From Poetry ```bash # Before poetry install poetry add requests # After uv sync uv add requests # Keep existing pyproject.toml # uv reads [project] and [tool.poetry] sections ``` ### From pip-tools ```bash # Before pip-compile requirements.in pip-sync requirements.txt # After uv lock uv sync --frozen ``` ## Command Reference ### Essential Commands ```bash # Project management uv init [PATH] # Initialize project uv add PACKAGE # Add dependency uv remove PACKAGE # Remove dependency uv sync # Install dependencies uv lock # Create/update lockfile # Virtual environments uv venv [PATH] # Create venv uv run COMMAND # Run in venv # Python management uv python install VERSION # Install Python uv python list # List installed Pythons uv python pin VERSION # Pin Python version # Package installation (pip-compatible) uv pip install PACKAGE # Install package uv pip uninstall PACKAGE # Uninstall package uv pip freeze # List installed uv pip list # List packages # Utility uv cache clean # Clear cache uv cache dir # Show cache location uv --version # Show version ``` ## Resources - **Official documentation**: https://docs.astral.sh/uv/ - **GitHub repository**: https://github.com/astral-sh/uv - **Astral blog**: https://astral.sh/blog - **Migration guides**: https://docs.astral.sh/uv/guides/ - **Comparison with other tools**: https://docs.astral.sh/uv/pip/compatibility/ ## Best Practices Summary 1. **Use uv for all new projects** - Start with `uv init` 2. **Commit lockfiles** - Ensure reproducible builds 3. **Pin Python versions** - Use .python-version 4. **Use uv run** - Avoid manual venv activation 5. **Leverage caching** - Let uv manage global cache 6. **Use --frozen in CI** - Exact reproduction 7. **Keep uv updated** - Fast-moving project 8. **Use workspaces** - For monorepo projects 9. **Export for compatibility** - Generate requirements.txt when needed 10. **Read the docs** - uv is feature-rich and evolving

# Bash Defensive Patterns Comprehensive guidance for writing production-ready Bash scripts using defensive programming techniques, error handling, and safety best practices to prevent common pitfalls and ensure reliability. ## When to Use This Skill - Writing production automation scripts - Building CI/CD pipeline scripts - Creating system administration utilities - Developing error-resilient deployment automation - Writing scripts that must handle edge cases safely - Building maintainable shell script libraries - Implementing comprehensive logging and monitoring - Creating scripts that must work across different platforms ## Core Defensive Principles ### 1. Strict Mode Enable bash strict mode at the start of every script to catch errors early. ```bash #!/bin/bash set -Eeuo pipefail # Exit on error, unset variables, pipe failures ``` **Key flags:** - `set -E`: Inherit ERR trap in functions - `set -e`: Exit on any error (command returns non-zero) - `set -u`: Exit on undefined variable reference - `set -o pipefail`: Pipe fails if any command fails (not just last) ### 2. Error Trapping and Cleanup Implement proper cleanup on script exit or error. ```bash #!/bin/bash set -Eeuo pipefail trap 'echo "Error on line $LINENO"' ERR trap 'echo "Cleaning up..."; rm -rf "$TMPDIR"' EXIT TMPDIR=$(mktemp -d) # Script code here ``` ### 3. Variable Safety Always quote variables to prevent word splitting and globbing issues. ```bash # Wrong - unsafe cp $source $dest # Correct - safe cp "$source" "$dest" # Required variables - fail with message if unset : "${REQUIRED_VAR:?REQUIRED_VAR is not set}" ``` ### 4. Array Handling Use arrays safely for complex data handling. ```bash # Safe array iteration declare -a items=("item 1" "item 2" "item 3") for item in "${items[@]}"; do echo "Processing: $item" done # Reading output into array safely mapfile -t lines < <(some_command) readarray -t numbers < <(seq 1 10) ``` ### 5. Conditional Safety Use `[[ ]]` for Bash-specific features, `[ ]` for POSIX. ```bash # Bash - safer if [[ -f "$file" && -r "$file" ]]; then content=$(<"$file") fi # POSIX - portable if [ -f "$file" ] && [ -r "$file" ]; then content=$(cat "$file") fi # Test for existence before operations if [[ -z "${VAR:-}" ]]; then echo "VAR is not set or is empty" fi ``` ## Fundamental Patterns ### Pattern 1: Safe Script Directory Detection ```bash #!/bin/bash set -Eeuo pipefail # Correctly determine script directory SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")" echo "Script location: $SCRIPT_DIR/$SCRIPT_NAME" ``` ### Pattern 2: Comprehensive Function Templat ```bash #!/bin/bash set -Eeuo pipefail # Prefix for functions: handle_*, process_*, check_*, validate_* # Include documentation and error handling validate_file() { local -r file="$1" local -r message="${2:-File not found: $file}" if [[ ! -f "$file" ]]; then echo "ERROR: $message" >&2 return 1 fi return 0 } process_files() { local -r input_dir="$1" local -r output_dir="$2" # Validate inputs [[ -d "$input_dir" ]] || { echo "ERROR: input_dir not a directory" >&2; return 1; } # Create output directory if needed mkdir -p "$output_dir" || { echo "ERROR: Cannot create output_dir" >&2; return 1; } # Process files safely while IFS= read -r -d '' file; do echo "Processing: $file" # Do work done < <(find "$input_dir" -maxdepth 1 -type f -print0) return 0 } ``` ### Pattern 3: Safe Temporary File Handling ```bash #!/bin/bash set -Eeuo pipefail trap 'rm -rf -- "$TMPDIR"' EXIT # Create temporary directory TMPDIR=$(mktemp -d) || { echo "ERROR: Failed to create temp directory" >&2; exit 1; } # Create temporary files in directory TMPFILE1="$TMPDIR/temp1.txt" TMPFILE2="$TMPDIR/temp2.txt" # Use temporary files touch "$TMPFILE1" "$TMPFILE2" echo "Temp files created in: $TMPDIR" ``` ### Pattern 4: Robust Argument Parsing ```bash #!/bin/bash set -Eeuo pipefail # Default values VERBOSE=false DRY_RUN=false OUTPUT_FILE="" THREADS=4 usage() { cat <<EOF Usage: $0 [OPTIONS] Options: -v, --verbose Enable verbose output -d, --dry-run Run without making changes -o, --output FILE Output file path -j, --jobs NUM Number of parallel jobs -h, --help Show this help message EOF exit "${1:-0}" } # Parse arguments while [[ $# -gt 0 ]]; do case "$1" in -v|--verbose) VERBOSE=true shift ;; -d|--dry-run) DRY_RUN=true shift ;; -o|--output) OUTPUT_FILE="$2" shift 2 ;; -j|--jobs) THREADS="$2" shift 2 ;; -h|--help) usage 0 ;; --) shift break ;; *) echo "ERROR: Unknown option: $1" >&2 usage 1 ;; esac done # Validate required arguments [[ -n "$OUTPUT_FILE" ]] || { echo "ERROR: -o/--output is required" >&2; usage 1; } ``` ### Pattern 5: Structured Logging ```bash #!/bin/bash set -Eeuo pipefail # Logging functions log_info() { echo "[$(date +'%Y-%m-%d %H:%M:%S')] INFO: $*" >&2 } log_warn() { echo "[$(date +'%Y-%m-%d %H:%M:%S')] WARN: $*" >&2 } log_error() { echo "[$(date +'%Y-%m-%d %H:%M:%S')] ERROR: $*" >&2 } log_debug() { if [[ "${DEBUG:-0}" == "1" ]]; then echo "[$(date +'%Y-%m-%d %H:%M:%S')] DEBUG: $*" >&2 fi } # Usage log_info "Starting script" log_debug "Debug information" log_warn "Warning message" log_error "Error occurred" ``` ### Pattern 6: Process Orchestration with Signals ```bash #!/bin/bash set -Eeuo pipefail # Track background processes PIDS=() cleanup() { log_info "Shutting down..." # Terminate all background processes for pid in "${PIDS[@]}"; do if kill -0 "$pid" 2>/dev/null; then kill -TERM "$pid" 2>/dev/null || true fi done # Wait for graceful shutdown for pid in "${PIDS[@]}"; do wait "$pid" 2>/dev/null || true done } trap cleanup SIGTERM SIGINT # Start background tasks background_task & PIDS+=($!) another_task & PIDS+=($!) # Wait for all background processes wait ``` ### Pattern 7: Safe File Operations ```bash #!/bin/bash set -Eeuo pipefail # Use -i flag to move safely without overwriting safe_move() { local -r source="$1" local -r dest="$2" if [[ ! -e "$source" ]]; then echo "ERROR: Source does not exist: $source" >&2 return 1 fi if [[ -e "$dest" ]]; then echo "ERROR: Destination already exists: $dest" >&2 return 1 fi mv "$source" "$dest" } # Safe directory cleanup safe_rmdir() { local -r dir="$1" if [[ ! -d "$dir" ]]; then echo "ERROR: Not a directory: $dir" >&2 return 1 fi # Use -I flag to prompt before rm (BSD/GNU compatible) rm -rI -- "$dir" } # Atomic file writes atomic_write() { local -r target="$1" local -r tmpfile tmpfile=$(mktemp) || return 1 # Write to temp file first cat > "$tmpfile" # Atomic rename mv "$tmpfile" "$target" } ``` ### Pattern 8: Idempotent Script Design ```bash #!/bin/bash set -Eeuo pipefail # Check if resource already exists ensure_directory() { local -r dir="$1" if [[ -d "$dir" ]]; then log_info "Directory already exists: $dir" return 0 fi mkdir -p "$dir" || { log_error "Failed to create directory: $dir" return 1 } log_info "Created directory: $dir" } # Ensure configuration state ensure_config() { local -r config_file="$1" local -r default_value="$2" if [[ ! -f "$config_file" ]]; then echo "$default_value" > "$config_file" log_info "Created config: $config_file" fi } # Rerunning script multiple times should be safe ensure_directory "/var/cache/myapp" ensure_config "/etc/myapp/config" "DEBUG=false" ``` ### Pattern 9: Safe Command Substitution ```bash #!/bin/bash set -Eeuo pipefail # Use $() instead of backticks name=$(<"$file") # Modern, safe variable assignment from file output=$(command -v python3) # Get command location safely # Handle command substitution with error checking result=$(command -v node) || { log_error "node command not found" return 1 } # For multiple lines mapfile -t lines < <(grep "pattern" "$file") # NUL-safe iteration while IFS= read -r -d '' file; do echo "Processing: $file" done < <(find /path -type f -print0) ``` ### Pattern 10: Dry-Run Support ```bash #!/bin/bash set -Eeuo pipefail DRY_RUN="${DRY_RUN:-false}" run_cmd() { if [[ "$DRY_RUN" == "true" ]]; then echo "[DRY RUN] Would execute: $*" return 0 fi "$@" } # Usage run_cmd cp "$source" "$dest" run_cmd rm "$file" run_cmd chown "$owner" "$target" ``` ## Advanced Defensive Techniques ### Named Parameters Pattern ```bash #!/bin/bash set -Eeuo pipefail process_data() { local input_file="" local output_dir="" local format="json" # Parse named parameters while [[ $# -gt 0 ]]; do case "$1" in --input=*) input_file="${1#*=}" ;; --output=*) output_dir="${1#*=}" ;; --format=*) format="${1#*=}" ;; *) echo "ERROR: Unknown parameter: $1" >&2 return 1 ;; esac shift done # Validate required parameters [[ -n "$input_file" ]] || { echo "ERROR: --input is required" >&2; return 1; } [[ -n "$output_dir" ]] || { echo "ERROR: --output is required" >&2; return 1; } } ``` ### Dependency Checking ```bash #!/bin/bash set -Eeuo pipefail check_dependencies() { local -a missing_deps=() local -a required=("jq" "curl" "git") for cmd in "${required[@]}"; do if ! command -v "$cmd" &>/dev/null; then missing_deps+=("$cmd") fi done if [[ ${#missing_deps[@]} -gt 0 ]]; then echo "ERROR: Missing required commands: ${missing_deps[*]}" >&2 return 1 fi } check_dependencies ``` ## Best Practices Summary 1. **Always use strict mode** - `set -Eeuo pipefail` 2. **Quote all variables** - `"$variable"` prevents word splitting 3. **Use [[]] conditionals** - More robust than [ ] 4. **Implement error trapping** - Catch and handle errors gracefully 5. **Validate all inputs** - Check file existence, permissions, formats 6. **Use functions for reusability** - Prefix with meaningful names 7. **Implement structured logging** - Include timestamps and levels 8. **Support dry-run mode** - Allow users to preview changes 9. **Handle temporary files safely** - Use mktemp, cleanup with trap 10. **Design for idempotency** - Scripts should be safe to rerun 11. **Document requirements** - List dependencies and minimum versions 12. **Test error paths** - Ensure error handling works correctly 13. **Use `command -v`** - Safer than `which` for checking executables 14. **Prefer printf over echo** - More predictable across systems ## Resources - **Bash Strict Mode**: http://redsymbol.net/articles/unofficial-bash-strict-mode/ - **Google Shell Style Guide**: https://google.github.io/styleguide/shellguide.html - **Defensive BASH Programming**: https://www.lifepipe.net/

# Memory Safety Patterns Cross-language patterns for memory-safe programming including RAII, ownership, smart pointers, and resource management. ## When to Use This Skill - Writing memory-safe systems code - Managing resources (files, sockets, memory) - Preventing use-after-free and leaks - Implementing RAII patterns - Choosing between languages for safety - Debugging memory issues ## Core Concepts ### 1. Memory Bug Categories | Bug Type | Description | Prevention | | -------------------- | -------------------------------- | ----------------- | | **Use-after-free** | Access freed memory | Ownership, RAII | | **Double-free** | Free same memory twice | Smart pointers | | **Memory leak** | Never free memory | RAII, GC | | **Buffer overflow** | Write past buffer end | Bounds checking | | **Dangling pointer** | Pointer to freed memory | Lifetime tracking | | **Data race** | Concurrent unsynchronized access | Ownership, Sync | ### 2. Safety Spectrum ``` Manual (C) → Smart Pointers (C++) → Ownership (Rust) → GC (Go, Java) Less safe More safe More control Less control ``` ## Patterns by Language ### Pattern 1: RAII in C++ ```cpp // RAII: Resource Acquisition Is Initialization // Resource lifetime tied to object lifetime #include <memory> #include <fstream> #include <mutex> // File handle with RAII class FileHandle { public: explicit FileHandle(const std::string& path) : file_(path) { if (!file_.is_open()) { throw std::runtime_error("Failed to open file"); } } // Destructor automatically closes file ~FileHandle() = default; // fstream closes in its destructor // Delete copy (prevent double-close) FileHandle(const FileHandle&) = delete; FileHandle& operator=(const FileHandle&) = delete; // Allow move FileHandle(FileHandle&&) = default; FileHandle& operator=(FileHandle&&) = default; void write(const std::string& data) { file_ << data; } private: std::fstream file_; }; // Lock guard (RAII for mutexes) class Database { public: void update(const std::string& key, const std::string& value) { std::lock_guard<std::mutex> lock(mutex_); // Released on scope exit data_[key] = value; } std::string get(const std::string& key) { std::shared_lock<std::shared_mutex> lock(shared_mutex_); return data_[key]; } private: std::mutex mutex_; std::shared_mutex shared_mutex_; std::map<std::string, std::string> data_; }; // Transaction with rollback (RAII) template<typename T> class Transaction { public: explicit Transaction(T& target) : target_(target), backup_(target), committed_(false) {} ~Transaction() { if (!committed_) { target_ = backup_; // Rollback } } void commit() { committed_ = true; } T& get() { return target_; } private: T& target_; T backup_; bool committed_; }; ``` ### Pattern 2: Smart Pointers in C++ ```cpp #include <memory> // unique_ptr: Single ownership class Engine { public: void start() { /* ... */ } }; class Car { public: Car() : engine_(std::make_unique<Engine>()) {} void start() { engine_->start(); } // Transfer ownership std::unique_ptr<Engine> extractEngine() { return std::move(engine_); } private: std::unique_ptr<Engine> engine_; }; // shared_ptr: Shared ownership class Node { public: std::string data; std::shared_ptr<Node> next; // Use weak_ptr to break cycles std::weak_ptr<Node> parent; }; void sharedPtrExample() { auto node1 = std::make_shared<Node>(); auto node2 = std::make_shared<Node>(); node1->next = node2; node2->parent = node1; // Weak reference prevents cycle // Access weak_ptr if (auto parent = node2->parent.lock()) { // parent is valid shared_ptr } } // Custom deleter for resources class Socket { public: static void close(int* fd) { if (fd && *fd >= 0) { ::close(*fd); delete fd; } } }; auto createSocket() { int fd = socket(AF_INET, SOCK_STREAM, 0); return std::unique_ptr<int, decltype(&Socket::close)>( new int(fd), &Socket::close ); } // make_unique/make_shared best practices void bestPractices() { // Good: Exception safe, single allocation auto ptr = std::make_shared<Widget>(); // Bad: Two allocations, not exception safe std::shared_ptr<Widget> ptr2(new Widget()); // For arrays auto arr = std::make_unique<int[]>(10); } ``` ### Pattern 3: Ownership in Rust ```rust // Move semantics (default) fn move_example() { let s1 = String::from("hello"); let s2 = s1; // s1 is MOVED, no longer valid // println!("{}", s1); // Compile error! println!("{}", s2); } // Borrowing (references) fn borrow_example() { let s = String::from("hello"); // Immutable borrow (multiple allowed) let len = calculate_length(&s); println!("{} has length {}", s, len); // Mutable borrow (only one allowed) let mut s = String::from("hello"); change(&mut s); } fn calculate_length(s: &String) -> usize { s.len() } // s goes out of scope, but doesn't drop since borrowed fn change(s: &mut String) { s.push_str(", world"); } // Lifetimes: Compiler tracks reference validity fn longest<'a>(x: &'a str, y: &'a str) -> &'a str { if x.len() > y.len() { x } else { y } } // Struct with references needs lifetime annotation struct ImportantExcerpt<'a> { part: &'a str, } impl<'a> ImportantExcerpt<'a> { fn level(&self) -> i32 { 3 } // Lifetime elision: compiler infers 'a for &self fn announce_and_return_part(&self, announcement: &str) -> &str { println!("Attention: {}", announcement); self.part } } // Interior mutability use std::cell::{Cell, RefCell}; use std::rc::Rc; struct Stats { count: Cell<i32>, // Copy types data: RefCell<Vec<String>>, // Non-Copy types } impl Stats { fn increment(&self) { self.count.set(self.count.get() + 1); } fn add_data(&self, item: String) { self.data.borrow_mut().push(item); } } // Rc for shared ownership (single-threaded) fn rc_example() { let data = Rc::new(vec![1, 2, 3]); let data2 = Rc::clone(&data); // Increment reference count println!("Count: {}", Rc::strong_count(&data)); // 2 } // Arc for shared ownership (thread-safe) use std::sync::Arc; use std::thread; fn arc_example() { let data = Arc::new(vec![1, 2, 3]); let handles: Vec<_> = (0..3) .map(|_| { let data = Arc::clone(&data); thread::spawn(move || { println!("{:?}", data); }) }) .collect(); for handle in handles { handle.join().unwrap(); } } ``` ### Pattern 4: Safe Resource Management in C ```c // C doesn't have RAII, but we can use patterns #include <stdlib.h> #include <stdio.h> // Pattern: goto cleanup int process_file(const char* path) { FILE* file = NULL; char* buffer = NULL; int result = -1; file = fopen(path, "r"); if (!file) { goto cleanup; } buffer = malloc(1024); if (!buffer) { goto cleanup; } // Process file... result = 0; cleanup: if (buffer) free(buffer); if (file) fclose(file); return result; } // Pattern: Opaque pointer with create/destroy typedef struct Context Context; Context* context_create(void); void context_destroy(Context* ctx); int context_process(Context* ctx, const char* data); // Implementation struct Context { int* data; size_t size; FILE* log; }; Context* context_create(void) { Context* ctx = calloc(1, sizeof(Context)); if (!ctx) return NULL; ctx->data = malloc(100 * sizeof(int)); if (!ctx->data) { free(ctx); return NULL; } ctx->log = fopen("log.txt", "w"); if (!ctx->log) { free(ctx->data); free(ctx); return NULL; } return ctx; } void context_destroy(Context* ctx) { if (ctx) { if (ctx->log) fclose(ctx->log); if (ctx->data) free(ctx->data); free(ctx); } } // Pattern: Cleanup attribute (GCC/Clang extension) #define AUTO_FREE __attribute__((cleanup(auto_free_func))) void auto_free_func(void** ptr) { free(*ptr); } void auto_free_example(void) { AUTO_FREE char* buffer = malloc(1024); // buffer automatically freed at end of scope } ``` ### Pattern 5: Bounds Checking ```cpp // C++: Use containers instead of raw arrays #include <vector> #include <array> #include <span> void safe_array_access() { std::vector<int> vec = {1, 2, 3, 4, 5}; // Safe: throws std::out_of_range try { int val = vec.at(10); } catch (const std::out_of_range& e) { // Handle error } // Unsafe but faster (no bounds check) int val = vec[2]; // Modern C++20: std::span for array views std::span<int> view(vec); // Iterators are bounds-safe for (int& x : view) { x *= 2; } } // Fixed-size arrays void fixed_array() { std::array<int, 5> arr = {1, 2, 3, 4, 5}; // Compile-time size known static_assert(arr.size() == 5); // Safe access int val = arr.at(2); } ``` ```rust // Rust: Bounds checking by default fn rust_bounds_checking() { let vec = vec![1, 2, 3, 4, 5]; // Runtime bounds check (panics if out of bounds) let val = vec[2]; // Explicit option (no panic) match vec.get(10) { Some(val) => println!("Got {}", val), None => println!("Index out of bounds"), } // Iterators (no bounds checking needed) for val in &vec { println!("{}", val); } // Slices are bounds-checked let slice = &vec[1..3]; // [2, 3] } ``` ### Pattern 6: Preventing Data Races ```cpp // C++: Thread-safe shared state #include <mutex> #include <shared_mutex> #include <atomic> class ThreadSafeCounter { public: void increment() { // Atomic operations count_.fetch_add(1, std::memory_order_relaxed); } int get() const { return count_.load(std::memory_order_relaxed); } private: std::atomic<int> count_{0}; }; class ThreadSafeMap { public: void write(const std::string& key, int value) { std::unique_lock lock(mutex_); data_[key] = value; } std::optional<int> read(const std::string& key) { std::shared_lock lock(mutex_); auto it = data_.find(key); if (it != data_.end()) { return it->second; } return std::nullopt; } private: mutable std::shared_mutex mutex_; std::map<std::string, int> data_; }; ``` ```rust // Rust: Data race prevention at compile time use std::sync::{Arc, Mutex, RwLock}; use std::sync::atomic::{AtomicI32, Ordering}; use std::thread; // Atomic for simple types fn atomic_example() { let counter = Arc::new(AtomicI32::new(0)); let handles: Vec<_> = (0..10) .map(|_| { let counter = Arc::clone(&counter); thread::spawn(move || { counter.fetch_add(1, Ordering::SeqCst); }) }) .collect(); for handle in handles { handle.join().unwrap(); } println!("Counter: {}", counter.load(Ordering::SeqCst)); } // Mutex for complex types fn mutex_example() { let data = Arc::new(Mutex::new(vec![])); let handles: Vec<_> = (0..10) .map(|i| { let data = Arc::clone(&data); thread::spawn(move || { let mut vec = data.lock().unwrap(); vec.push(i); }) }) .collect(); for handle in handles { handle.join().unwrap(); } } // RwLock for read-heavy workloads fn rwlock_example() { let data = Arc::new(RwLock::new(HashMap::new())); // Multiple readers OK let read_guard = data.read().unwrap(); // Writer blocks readers let write_guard = data.write().unwrap(); } ``` ## Best Practices ### Do's - **Prefer RAII** - Tie resource lifetime to scope - **Use smart pointers** - Avoid raw pointers in C++ - **Understand ownership** - Know who owns what - **Check bounds** - Use safe access methods - **Use tools** - AddressSanitizer, Valgrind, Miri ### Don'ts - **Don't use raw pointers** - Unless interfacing with C - **Don't return local references** - Dangling pointer - **Don't ignore compiler warnings** - They catch bugs - **Don't use `unsafe` carelessly** - In Rust, minimize it - **Don't assume thread safety** - Be explicit ## Debugging Tools ```bash # AddressSanitizer (Clang/GCC) clang++ -fsanitize=address -g source.cpp # Valgrind valgrind --leak-check=full ./program # Rust Miri (undefined behavior detector) cargo +nightly miri run # ThreadSanitizer clang++ -fsanitize=thread -g source.cpp ``` ## Resources - [C++ Core Guidelines](https://isocpp.github.io/CppCoreGuidelines/) - [Rust Ownership](https://doc.rust-lang.org/book/ch04-00-understanding-ownership.html) - [AddressSanitizer](https://clang.llvm.org/docs/AddressSanitizer.html)