Skip to main content
EVOKORE// BROWSE
>

./browse/prompts

9 NODES
🤖system prompt•7 months ago

architecture-patterns

Implement proven backend architecture patterns including Clean

coding
⭐1
# Architecture Patterns Master proven backend architecture patterns including Clean Architecture, Hexagonal Architecture, and Domain-Driven Design to build maintainable, testable, and scalable systems. ## When to Use This Skill - Designing new backend systems from scratch - Refactoring monolithic applications for better maintainability - Establishing architecture standards for your team - Migrating from tightly coupled to loosely coupled architectures - Implementing domain-driven design principles - Creating testable and mockable codebases - Planning microservices decomposition ## Core Concepts ### 1. Clean Architecture (Uncle Bob) **Layers (dependency flows inward):** - **Entities**: Core business models - **Use Cases**: Application business rules - **Interface Adapters**: Controllers, presenters, gateways - **Frameworks & Drivers**: UI, database, external services **Key Principles:** - Dependencies point inward - Inner layers know nothing about outer layers - Business logic independent of frameworks - Testable without UI, database, or external services ### 2. Hexagonal Architecture (Ports and Adapters) **Components:** - **Domain Core**: Business logic - **Ports**: Interfaces defining interactions - **Adapters**: Implementations of ports (database, REST, message queue) **Benefits:** - Swap implementations easily (mock for testing) - Technology-agnostic core - Clear separation of concerns ### 3. Domain-Driven Design (DDD) **Strategic Patterns:** - **Bounded Contexts**: Separate models for different domains - **Context Mapping**: How contexts relate - **Ubiquitous Language**: Shared terminology **Tactical Patterns:** - **Entities**: Objects with identity - **Value Objects**: Immutable objects defined by attributes - **Aggregates**: Consistency boundaries - **Repositories**: Data access abstraction - **Domain Events**: Things that happened ## Clean Architecture Pattern ### Directory Structure ``` app/ ├── domain/ # Entities & business rules │ ├── entities/ │ │ ├── user.py │ │ └── order.py │ ├── value_objects/ │ │ ├── email.py │ │ └── money.py │ └── interfaces/ # Abstract interfaces │ ├── user_repository.py │ └── payment_gateway.py ├── use_cases/ # Application business rules │ ├── create_user.py │ ├── process_order.py │ └── send_notification.py ├── adapters/ # Interface implementations │ ├── repositories/ │ │ ├── postgres_user_repository.py │ │ └── redis_cache_repository.py │ ├── controllers/ │ │ └── user_controller.py │ └── gateways/ │ ├── stripe_payment_gateway.py │ └── sendgrid_email_gateway.py └── infrastructure/ # Framework & external concerns ├── database.py ├── config.py └── logging.py ``` ### Implementation Example ```python # domain/entities/user.py from dataclasses import dataclass from datetime import datetime from typing import Optional @dataclass class User: """Core user entity - no framework dependencies.""" id: str email: str name: str created_at: datetime is_active: bool = True def deactivate(self): """Business rule: deactivating user.""" self.is_active = False def can_place_order(self) -> bool: """Business rule: active users can order.""" return self.is_active # domain/interfaces/user_repository.py from abc import ABC, abstractmethod from typing import Optional, List from domain.entities.user import User class IUserRepository(ABC): """Port: defines contract, no implementation.""" @abstractmethod async def find_by_id(self, user_id: str) -> Optional[User]: pass @abstractmethod async def find_by_email(self, email: str) -> Optional[User]: pass @abstractmethod async def save(self, user: User) -> User: pass @abstractmethod async def delete(self, user_id: str) -> bool: pass # use_cases/create_user.py from domain.entities.user import User from domain.interfaces.user_repository import IUserRepository from dataclasses import dataclass from datetime import datetime import uuid @dataclass class CreateUserRequest: email: str name: str @dataclass class CreateUserResponse: user: User success: bool error: Optional[str] = None class CreateUserUseCase: """Use case: orchestrates business logic.""" def __init__(self, user_repository: IUserRepository): self.user_repository = user_repository async def execute(self, request: CreateUserRequest) -> CreateUserResponse: # Business validation existing = await self.user_repository.find_by_email(request.email) if existing: return CreateUserResponse( user=None, success=False, error="Email already exists" ) # Create entity user = User( id=str(uuid.uuid4()), email=request.email, name=request.name, created_at=datetime.now(), is_active=True ) # Persist saved_user = await self.user_repository.save(user) return CreateUserResponse( user=saved_user, success=True ) # adapters/repositories/postgres_user_repository.py from domain.interfaces.user_repository import IUserRepository from domain.entities.user import User from typing import Optional import asyncpg class PostgresUserRepository(IUserRepository): """Adapter: PostgreSQL implementation.""" def __init__(self, pool: asyncpg.Pool): self.pool = pool async def find_by_id(self, user_id: str) -> Optional[User]: async with self.pool.acquire() as conn: row = await conn.fetchrow( "SELECT * FROM users WHERE id = $1", user_id ) return self._to_entity(row) if row else None async def find_by_email(self, email: str) -> Optional[User]: async with self.pool.acquire() as conn: row = await conn.fetchrow( "SELECT * FROM users WHERE email = $1", email ) return self._to_entity(row) if row else None async def save(self, user: User) -> User: async with self.pool.acquire() as conn: await conn.execute( """ INSERT INTO users (id, email, name, created_at, is_active) VALUES ($1, $2, $3, $4, $5) ON CONFLICT (id) DO UPDATE SET email = $2, name = $3, is_active = $5 """, user.id, user.email, user.name, user.created_at, user.is_active ) return user async def delete(self, user_id: str) -> bool: async with self.pool.acquire() as conn: result = await conn.execute( "DELETE FROM users WHERE id = $1", user_id ) return result == "DELETE 1" def _to_entity(self, row) -> User: """Map database row to entity.""" return User( id=row["id"], email=row["email"], name=row["name"], created_at=row["created_at"], is_active=row["is_active"] ) # adapters/controllers/user_controller.py from fastapi import APIRouter, Depends, HTTPException from use_cases.create_user import CreateUserUseCase, CreateUserRequest from pydantic import BaseModel router = APIRouter() class CreateUserDTO(BaseModel): email: str name: str @router.post("/users") async def create_user( dto: CreateUserDTO, use_case: CreateUserUseCase = Depends(get_create_user_use_case) ): """Controller: handles HTTP concerns only.""" request = CreateUserRequest(email=dto.email, name=dto.name) response = await use_case.execute(request) if not response.success: raise HTTPException(status_code=400, detail=response.error) return {"user": response.user} ``` ## Hexagonal Architecture Pattern ```python # Core domain (hexagon center) class OrderService: """Domain service - no infrastructure dependencies.""" def __init__( self, order_repository: OrderRepositoryPort, payment_gateway: PaymentGatewayPort, notification_service: NotificationPort ): self.orders = order_repository self.payments = payment_gateway self.notifications = notification_service async def place_order(self, order: Order) -> OrderResult: # Business logic if not order.is_valid(): return OrderResult(success=False, error="Invalid order") # Use ports (interfaces) payment = await self.payments.charge( amount=order.total, customer=order.customer_id ) if not payment.success: return OrderResult(success=False, error="Payment failed") order.mark_as_paid() saved_order = await self.orders.save(order) await self.notifications.send( to=order.customer_email, subject="Order confirmed", body=f"Order {order.id} confirmed" ) return OrderResult(success=True, order=saved_order) # Ports (interfaces) class OrderRepositoryPort(ABC): @abstractmethod async def save(self, order: Order) -> Order: pass class PaymentGatewayPort(ABC): @abstractmethod async def charge(self, amount: Money, customer: str) -> PaymentResult: pass class NotificationPort(ABC): @abstractmethod async def send(self, to: str, subject: str, body: str): pass # Adapters (implementations) class StripePaymentAdapter(PaymentGatewayPort): """Primary adapter: connects to Stripe API.""" def __init__(self, api_key: str): self.stripe = stripe self.stripe.api_key = api_key async def charge(self, amount: Money, customer: str) -> PaymentResult: try: charge = self.stripe.Charge.create( amount=amount.cents, currency=amount.currency, customer=customer ) return PaymentResult(success=True, transaction_id=charge.id) except stripe.error.CardError as e: return PaymentResult(success=False, error=str(e)) class MockPaymentAdapter(PaymentGatewayPort): """Test adapter: no external dependencies.""" async def charge(self, amount: Money, customer: str) -> PaymentResult: return PaymentResult(success=True, transaction_id="mock-123") ``` ## Domain-Driven Design Pattern ```python # Value Objects (immutable) from dataclasses import dataclass from typing import Optional @dataclass(frozen=True) class Email: """Value object: validated email.""" value: str def __post_init__(self): if "@" not in self.value: raise ValueError("Invalid email") @dataclass(frozen=True) class Money: """Value object: amount with currency.""" amount: int # cents currency: str def add(self, other: "Money") -> "Money": if self.currency != other.currency: raise ValueError("Currency mismatch") return Money(self.amount + other.amount, self.currency) # Entities (with identity) class Order: """Entity: has identity, mutable state.""" def __init__(self, id: str, customer: Customer): self.id = id self.customer = customer self.items: List[OrderItem] = [] self.status = OrderStatus.PENDING self._events: List[DomainEvent] = [] def add_item(self, product: Product, quantity: int): """Business logic in entity.""" item = OrderItem(product, quantity) self.items.append(item) self._events.append(ItemAddedEvent(self.id, item)) def total(self) -> Money: """Calculated property.""" return sum(item.subtotal() for item in self.items) def submit(self): """State transition with business rules.""" if not self.items: raise ValueError("Cannot submit empty order") if self.status != OrderStatus.PENDING: raise ValueError("Order already submitted") self.status = OrderStatus.SUBMITTED self._events.append(OrderSubmittedEvent(self.id)) # Aggregates (consistency boundary) class Customer: """Aggregate root: controls access to entities.""" def __init__(self, id: str, email: Email): self.id = id self.email = email self._addresses: List[Address] = [] self._orders: List[str] = [] # Order IDs, not full objects def add_address(self, address: Address): """Aggregate enforces invariants.""" if len(self._addresses) >= 5: raise ValueError("Maximum 5 addresses allowed") self._addresses.append(address) @property def primary_address(self) -> Optional[Address]: return next((a for a in self._addresses if a.is_primary), None) # Domain Events @dataclass class OrderSubmittedEvent: order_id: str occurred_at: datetime = field(default_factory=datetime.now) # Repository (aggregate persistence) class OrderRepository: """Repository: persist/retrieve aggregates.""" async def find_by_id(self, order_id: str) -> Optional[Order]: """Reconstitute aggregate from storage.""" pass async def save(self, order: Order): """Persist aggregate and publish events.""" await self._persist(order) await self._publish_events(order._events) order._events.clear() ``` ## Resources - **references/clean-architecture-guide.md**: Detailed layer breakdown - **references/hexagonal-architecture-guide.md**: Ports and adapters patterns - **references/ddd-tactical-patterns.md**: Entities, value objects, aggregates - **assets/clean-architecture-template/**: Complete project structure - **assets/ddd-examples/**: Domain modeling examples ## Best Practices 1. **Dependency Rule**: Dependencies always point inward 2. **Interface Segregation**: Small, focused interfaces 3. **Business Logic in Domain**: Keep frameworks out of core 4. **Test Independence**: Core testable without infrastructure 5. **Bounded Contexts**: Clear domain boundaries 6. **Ubiquitous Language**: Consistent terminology 7. **Thin Controllers**: Delegate to use cases 8. **Rich Domain Models**: Behavior with data ## Common Pitfalls - **Anemic Domain**: Entities with only data, no behavior - **Framework Coupling**: Business logic depends on frameworks - **Fat Controllers**: Business logic in controllers - **Repository Leakage**: Exposing ORM objects - **Missing Abstractions**: Concrete dependencies in core - **Over-Engineering**: Clean architecture for simple CRUD
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

dbt-transformation-patterns

Master dbt (data build tool) for analytics engineering with model

data
⭐1
# dbt Transformation Patterns Production-ready patterns for dbt (data build tool) including model organization, testing strategies, documentation, and incremental processing. ## When to Use This Skill - Building data transformation pipelines with dbt - Organizing models into staging, intermediate, and marts layers - Implementing data quality tests - Creating incremental models for large datasets - Documenting data models and lineage - Setting up dbt project structure ## Core Concepts ### 1. Model Layers (Medallion Architecture) ``` sources/ Raw data definitions ↓ staging/ 1:1 with source, light cleaning ↓ intermediate/ Business logic, joins, aggregations ↓ marts/ Final analytics tables ``` ### 2. Naming Conventions | Layer | Prefix | Example | | ------------ | -------------- | ----------------------------- | | Staging | `stg_` | `stg_stripe__payments` | | Intermediate | `int_` | `int_payments_pivoted` | | Marts | `dim_`, `fct_` | `dim_customers`, `fct_orders` | ## Quick Start ```yaml # dbt_project.yml name: "analytics" version: "1.0.0" profile: "analytics" model-paths: ["models"] analysis-paths: ["analyses"] test-paths: ["tests"] seed-paths: ["seeds"] macro-paths: ["macros"] vars: start_date: "2020-01-01" models: analytics: staging: +materialized: view +schema: staging intermediate: +materialized: ephemeral marts: +materialized: table +schema: analytics ``` ``` # Project structure models/ ├── staging/ │ ├── stripe/ │ │ ├── _stripe__sources.yml │ │ ├── _stripe__models.yml │ │ ├── stg_stripe__customers.sql │ │ └── stg_stripe__payments.sql │ └── shopify/ │ ├── _shopify__sources.yml │ └── stg_shopify__orders.sql ├── intermediate/ │ └── finance/ │ └── int_payments_pivoted.sql └── marts/ ├── core/ │ ├── _core__models.yml │ ├── dim_customers.sql │ └── fct_orders.sql └── finance/ └── fct_revenue.sql ``` ## Patterns ### Pattern 1: Source Definitions ```yaml # models/staging/stripe/_stripe__sources.yml version: 2 sources: - name: stripe description: Raw Stripe data loaded via Fivetran database: raw schema: stripe loader: fivetran loaded_at_field: _fivetran_synced freshness: warn_after: { count: 12, period: hour } error_after: { count: 24, period: hour } tables: - name: customers description: Stripe customer records columns: - name: id description: Primary key tests: - unique - not_null - name: email description: Customer email - name: created description: Account creation timestamp - name: payments description: Stripe payment transactions columns: - name: id tests: - unique - not_null - name: customer_id tests: - not_null - relationships: to: source('stripe', 'customers') field: id ``` ### Pattern 2: Staging Models ```sql -- models/staging/stripe/stg_stripe__customers.sql with source as ( select * from {{ source('stripe', 'customers') }} ), renamed as ( select -- ids id as customer_id, -- strings lower(email) as email, name as customer_name, -- timestamps created as created_at, -- metadata _fivetran_synced as _loaded_at from source ) select * from renamed ``` ```sql -- models/staging/stripe/stg_stripe__payments.sql {{ config( materialized='incremental', unique_key='payment_id', on_schema_change='append_new_columns' ) }} with source as ( select * from {{ source('stripe', 'payments') }} {% if is_incremental() %} where _fivetran_synced > (select max(_loaded_at) from {{ this }}) {% endif %} ), renamed as ( select -- ids id as payment_id, customer_id, invoice_id, -- amounts (convert cents to dollars) amount / 100.0 as amount, amount_refunded / 100.0 as amount_refunded, -- status status as payment_status, -- timestamps created as created_at, -- metadata _fivetran_synced as _loaded_at from source ) select * from renamed ``` ### Pattern 3: Intermediate Models ```sql -- models/intermediate/finance/int_payments_pivoted_to_customer.sql with payments as ( select * from {{ ref('stg_stripe__payments') }} ), customers as ( select * from {{ ref('stg_stripe__customers') }} ), payment_summary as ( select customer_id, count(*) as total_payments, count(case when payment_status = 'succeeded' then 1 end) as successful_payments, sum(case when payment_status = 'succeeded' then amount else 0 end) as total_amount_paid, min(created_at) as first_payment_at, max(created_at) as last_payment_at from payments group by customer_id ) select customers.customer_id, customers.email, customers.created_at as customer_created_at, coalesce(payment_summary.total_payments, 0) as total_payments, coalesce(payment_summary.successful_payments, 0) as successful_payments, coalesce(payment_summary.total_amount_paid, 0) as lifetime_value, payment_summary.first_payment_at, payment_summary.last_payment_at from customers left join payment_summary using (customer_id) ``` ### Pattern 4: Mart Models (Dimensions and Facts) ```sql -- models/marts/core/dim_customers.sql {{ config( materialized='table', unique_key='customer_id' ) }} with customers as ( select * from {{ ref('int_payments_pivoted_to_customer') }} ), orders as ( select * from {{ ref('stg_shopify__orders') }} ), order_summary as ( select customer_id, count(*) as total_orders, sum(total_price) as total_order_value, min(created_at) as first_order_at, max(created_at) as last_order_at from orders group by customer_id ), final as ( select -- surrogate key {{ dbt_utils.generate_surrogate_key(['customers.customer_id']) }} as customer_key, -- natural key customers.customer_id, -- attributes customers.email, customers.customer_created_at, -- payment metrics customers.total_payments, customers.successful_payments, customers.lifetime_value, customers.first_payment_at, customers.last_payment_at, -- order metrics coalesce(order_summary.total_orders, 0) as total_orders, coalesce(order_summary.total_order_value, 0) as total_order_value, order_summary.first_order_at, order_summary.last_order_at, -- calculated fields case when customers.lifetime_value >= 1000 then 'high' when customers.lifetime_value >= 100 then 'medium' else 'low' end as customer_tier, -- timestamps current_timestamp as _loaded_at from customers left join order_summary using (customer_id) ) select * from final ``` ```sql -- models/marts/core/fct_orders.sql {{ config( materialized='incremental', unique_key='order_id', incremental_strategy='merge' ) }} with orders as ( select * from {{ ref('stg_shopify__orders') }} {% if is_incremental() %} where updated_at > (select max(updated_at) from {{ this }}) {% endif %} ), customers as ( select * from {{ ref('dim_customers') }} ), final as ( select -- keys orders.order_id, customers.customer_key, orders.customer_id, -- dimensions orders.order_status, orders.fulfillment_status, orders.payment_status, -- measures orders.subtotal, orders.tax, orders.shipping, orders.total_price, orders.total_discount, orders.item_count, -- timestamps orders.created_at, orders.updated_at, orders.fulfilled_at, -- metadata current_timestamp as _loaded_at from orders left join customers on orders.customer_id = customers.customer_id ) select * from final ``` ### Pattern 5: Testing and Documentation ```yaml # models/marts/core/_core__models.yml version: 2 models: - name: dim_customers description: Customer dimension with payment and order metrics columns: - name: customer_key description: Surrogate key for the customer dimension tests: - unique - not_null - name: customer_id description: Natural key from source system tests: - unique - not_null - name: email description: Customer email address tests: - not_null - name: customer_tier description: Customer value tier based on lifetime value tests: - accepted_values: values: ["high", "medium", "low"] - name: lifetime_value description: Total amount paid by customer tests: - dbt_utils.expression_is_true: expression: ">= 0" - name: fct_orders description: Order fact table with all order transactions tests: - dbt_utils.recency: datepart: day field: created_at interval: 1 columns: - name: order_id tests: - unique - not_null - name: customer_key tests: - not_null - relationships: to: ref('dim_customers') field: customer_key ``` ### Pattern 6: Macros and DRY Code ```sql -- macros/cents_to_dollars.sql {% macro cents_to_dollars(column_name, precision=2) %} round({{ column_name }} / 100.0, {{ precision }}) {% endmacro %} -- macros/generate_schema_name.sql {% macro generate_schema_name(custom_schema_name, node) %} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none -%} {{ default_schema }} {%- else -%} {{ default_schema }}_{{ custom_schema_name }} {%- endif -%} {% endmacro %} -- macros/limit_data_in_dev.sql {% macro limit_data_in_dev(column_name, days=3) %} {% if target.name == 'dev' %} where {{ column_name }} >= dateadd(day, -{{ days }}, current_date) {% endif %} {% endmacro %} -- Usage in model select * from {{ ref('stg_orders') }} {{ limit_data_in_dev('created_at') }} ``` ### Pattern 7: Incremental Strategies ```sql -- Delete+Insert (default for most warehouses) {{ config( materialized='incremental', unique_key='id', incremental_strategy='delete+insert' ) }} -- Merge (best for late-arriving data) {{ config( materialized='incremental', unique_key='id', incremental_strategy='merge', merge_update_columns=['status', 'amount', 'updated_at'] ) }} -- Insert Overwrite (partition-based) {{ config( materialized='incremental', incremental_strategy='insert_overwrite', partition_by={ "field": "created_date", "data_type": "date", "granularity": "day" } ) }} select *, date(created_at) as created_date from {{ ref('stg_events') }} {% if is_incremental() %} where created_date >= dateadd(day, -3, current_date) {% endif %} ``` ## dbt Commands ```bash # Development dbt run # Run all models dbt run --select staging # Run staging models only dbt run --select +fct_orders # Run fct_orders and its upstream dbt run --select fct_orders+ # Run fct_orders and its downstream dbt run --full-refresh # Rebuild incremental models # Testing dbt test # Run all tests dbt test --select stg_stripe # Test specific models dbt build # Run + test in DAG order # Documentation dbt docs generate # Generate docs dbt docs serve # Serve docs locally # Debugging dbt compile # Compile SQL without running dbt debug # Test connection dbt ls --select tag:critical # List models by tag ``` ## Best Practices ### Do's - **Use staging layer** - Clean data once, use everywhere - **Test aggressively** - Not null, unique, relationships - **Document everything** - Column descriptions, model descriptions - **Use incremental** - For tables > 1M rows - **Version control** - dbt project in Git ### Don'ts - **Don't skip staging** - Raw → mart is tech debt - **Don't hardcode dates** - Use `{{ var('start_date') }}` - **Don't repeat logic** - Extract to macros - **Don't test in prod** - Use dev target - **Don't ignore freshness** - Monitor source data ## Resources - [dbt Documentation](https://docs.getdbt.com/) - [dbt Best Practices](https://docs.getdbt.com/guides/best-practices) - [dbt-utils Package](https://hub.getdbt.com/dbt-labs/dbt_utils/latest/) - [dbt Discourse](https://discourse.getdbt.com/)
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

e2e-testing-patterns

Master end-to-end testing with Playwright and Cypress to build

coding
⭐1
# E2E Testing Patterns Build reliable, fast, and maintainable end-to-end test suites that provide confidence to ship code quickly and catch regressions before users do. ## When to Use This Skill - Implementing end-to-end test automation - Debugging flaky or unreliable tests - Testing critical user workflows - Setting up CI/CD test pipelines - Testing across multiple browsers - Validating accessibility requirements - Testing responsive designs - Establishing E2E testing standards ## Core Concepts ### 1. E2E Testing Fundamentals **What to Test with E2E:** - Critical user journeys (login, checkout, signup) - Complex interactions (drag-and-drop, multi-step forms) - Cross-browser compatibility - Real API integration - Authentication flows **What NOT to Test with E2E:** - Unit-level logic (use unit tests) - API contracts (use integration tests) - Edge cases (too slow) - Internal implementation details ### 2. Test Philosophy **The Testing Pyramid:** ``` /\ /E2E\ ← Few, focused on critical paths /─────\ /Integr\ ← More, test component interactions /────────\ /Unit Tests\ ← Many, fast, isolated /────────────\ ``` **Best Practices:** - Test user behavior, not implementation - Keep tests independent - Make tests deterministic - Optimize for speed - Use data-testid, not CSS selectors ## Playwright Patterns ### Setup and Configuration ```typescript // playwright.config.ts import { defineConfig, devices } from "@playwright/test"; export default defineConfig({ testDir: "./e2e", timeout: 30000, expect: { timeout: 5000, }, fullyParallel: true, forbidOnly: !!process.env.CI, retries: process.env.CI ? 2 : 0, workers: process.env.CI ? 1 : undefined, reporter: [["html"], ["junit", { outputFile: "results.xml" }]], use: { baseURL: "http://localhost:3000", trace: "on-first-retry", screenshot: "only-on-failure", video: "retain-on-failure", }, projects: [ { name: "chromium", use: { ...devices["Desktop Chrome"] } }, { name: "firefox", use: { ...devices["Desktop Firefox"] } }, { name: "webkit", use: { ...devices["Desktop Safari"] } }, { name: "mobile", use: { ...devices["iPhone 13"] } }, ], }); ``` ### Pattern 1: Page Object Model ```typescript // pages/LoginPage.ts import { Page, Locator } from "@playwright/test"; export class LoginPage { readonly page: Page; readonly emailInput: Locator; readonly passwordInput: Locator; readonly loginButton: Locator; readonly errorMessage: Locator; constructor(page: Page) { this.page = page; this.emailInput = page.getByLabel("Email"); this.passwordInput = page.getByLabel("Password"); this.loginButton = page.getByRole("button", { name: "Login" }); this.errorMessage = page.getByRole("alert"); } async goto() { await this.page.goto("/login"); } async login(email: string, password: string) { await this.emailInput.fill(email); await this.passwordInput.fill(password); await this.loginButton.click(); } async getErrorMessage(): Promise<string> { return (await this.errorMessage.textContent()) ?? ""; } } // Test using Page Object import { test, expect } from "@playwright/test"; import { LoginPage } from "./pages/LoginPage"; test("successful login", async ({ page }) => { const loginPage = new LoginPage(page); await loginPage.goto(); await loginPage.login("user@example.com", "password123"); await expect(page).toHaveURL("/dashboard"); await expect(page.getByRole("heading", { name: "Dashboard" })).toBeVisible(); }); test("failed login shows error", async ({ page }) => { const loginPage = new LoginPage(page); await loginPage.goto(); await loginPage.login("invalid@example.com", "wrong"); const error = await loginPage.getErrorMessage(); expect(error).toContain("Invalid credentials"); }); ``` ### Pattern 2: Fixtures for Test Data ```typescript // fixtures/test-data.ts import { test as base } from "@playwright/test"; type TestData = { testUser: { email: string; password: string; name: string; }; adminUser: { email: string; password: string; }; }; export const test = base.extend<TestData>({ testUser: async ({}, use) => { const user = { email: `test-${Date.now()}@example.com`, password: "Test123!@#", name: "Test User", }; // Setup: Create user in database await createTestUser(user); await use(user); // Teardown: Clean up user await deleteTestUser(user.email); }, adminUser: async ({}, use) => { await use({ email: "admin@example.com", password: process.env.ADMIN_PASSWORD!, }); }, }); // Usage in tests import { test } from "./fixtures/test-data"; test("user can update profile", async ({ page, testUser }) => { await page.goto("/login"); await page.getByLabel("Email").fill(testUser.email); await page.getByLabel("Password").fill(testUser.password); await page.getByRole("button", { name: "Login" }).click(); await page.goto("/profile"); await page.getByLabel("Name").fill("Updated Name"); await page.getByRole("button", { name: "Save" }).click(); await expect(page.getByText("Profile updated")).toBeVisible(); }); ``` ### Pattern 3: Waiting Strategies ```typescript // ❌ Bad: Fixed timeouts await page.waitForTimeout(3000); // Flaky! // ✅ Good: Wait for specific conditions await page.waitForLoadState("networkidle"); await page.waitForURL("/dashboard"); await page.waitForSelector('[data-testid="user-profile"]'); // ✅ Better: Auto-waiting with assertions await expect(page.getByText("Welcome")).toBeVisible(); await expect(page.getByRole("button", { name: "Submit" })).toBeEnabled(); // Wait for API response const responsePromise = page.waitForResponse( (response) => response.url().includes("/api/users") && response.status() === 200, ); await page.getByRole("button", { name: "Load Users" }).click(); const response = await responsePromise; const data = await response.json(); expect(data.users).toHaveLength(10); // Wait for multiple conditions await Promise.all([ page.waitForURL("/success"), page.waitForLoadState("networkidle"), expect(page.getByText("Payment successful")).toBeVisible(), ]); ``` ### Pattern 4: Network Mocking and Interception ```typescript // Mock API responses test("displays error when API fails", async ({ page }) => { await page.route("**/api/users", (route) => { route.fulfill({ status: 500, contentType: "application/json", body: JSON.stringify({ error: "Internal Server Error" }), }); }); await page.goto("/users"); await expect(page.getByText("Failed to load users")).toBeVisible(); }); // Intercept and modify requests test("can modify API request", async ({ page }) => { await page.route("**/api/users", async (route) => { const request = route.request(); const postData = JSON.parse(request.postData() || "{}"); // Modify request postData.role = "admin"; await route.continue({ postData: JSON.stringify(postData), }); }); // Test continues... }); // Mock third-party services test("payment flow with mocked Stripe", async ({ page }) => { await page.route("**/api/stripe/**", (route) => { route.fulfill({ status: 200, body: JSON.stringify({ id: "mock_payment_id", status: "succeeded", }), }); }); // Test payment flow with mocked response }); ``` ## Cypress Patterns ### Setup and Configuration ```typescript // cypress.config.ts import { defineConfig } from "cypress"; export default defineConfig({ e2e: { baseUrl: "http://localhost:3000", viewportWidth: 1280, viewportHeight: 720, video: false, screenshotOnRunFailure: true, defaultCommandTimeout: 10000, requestTimeout: 10000, setupNodeEvents(on, config) { // Implement node event listeners }, }, }); ``` ### Pattern 1: Custom Commands ```typescript // cypress/support/commands.ts declare global { namespace Cypress { interface Chainable { login(email: string, password: string): Chainable<void>; createUser(userData: UserData): Chainable<User>; dataCy(value: string): Chainable<JQuery<HTMLElement>>; } } } Cypress.Commands.add("login", (email: string, password: string) => { cy.visit("/login"); cy.get('[data-testid="email"]').type(email); cy.get('[data-testid="password"]').type(password); cy.get('[data-testid="login-button"]').click(); cy.url().should("include", "/dashboard"); }); Cypress.Commands.add("createUser", (userData: UserData) => { return cy.request("POST", "/api/users", userData).its("body"); }); Cypress.Commands.add("dataCy", (value: string) => { return cy.get(`[data-cy="${value}"]`); }); // Usage cy.login("user@example.com", "password"); cy.dataCy("submit-button").click(); ``` ### Pattern 2: Cypress Intercept ```typescript // Mock API calls cy.intercept("GET", "/api/users", { statusCode: 200, body: [ { id: 1, name: "John" }, { id: 2, name: "Jane" }, ], }).as("getUsers"); cy.visit("/users"); cy.wait("@getUsers"); cy.get('[data-testid="user-list"]').children().should("have.length", 2); // Modify responses cy.intercept("GET", "/api/users", (req) => { req.reply((res) => { // Modify response res.body.users = res.body.users.slice(0, 5); res.send(); }); }); // Simulate slow network cy.intercept("GET", "/api/data", (req) => { req.reply((res) => { res.delay(3000); // 3 second delay res.send(); }); }); ``` ## Advanced Patterns ### Pattern 1: Visual Regression Testing ```typescript // With Playwright import { test, expect } from "@playwright/test"; test("homepage looks correct", async ({ page }) => { await page.goto("/"); await expect(page).toHaveScreenshot("homepage.png", { fullPage: true, maxDiffPixels: 100, }); }); test("button in all states", async ({ page }) => { await page.goto("/components"); const button = page.getByRole("button", { name: "Submit" }); // Default state await expect(button).toHaveScreenshot("button-default.png"); // Hover state await button.hover(); await expect(button).toHaveScreenshot("button-hover.png"); // Disabled state await button.evaluate((el) => el.setAttribute("disabled", "true")); await expect(button).toHaveScreenshot("button-disabled.png"); }); ``` ### Pattern 2: Parallel Testing with Sharding ```typescript // playwright.config.ts export default defineConfig({ projects: [ { name: "shard-1", use: { ...devices["Desktop Chrome"] }, grepInvert: /@slow/, shard: { current: 1, total: 4 }, }, { name: "shard-2", use: { ...devices["Desktop Chrome"] }, shard: { current: 2, total: 4 }, }, // ... more shards ], }); // Run in CI // npx playwright test --shard=1/4 // npx playwright test --shard=2/4 ``` ### Pattern 3: Accessibility Testing ```typescript // Install: npm install @axe-core/playwright import { test, expect } from "@playwright/test"; import AxeBuilder from "@axe-core/playwright"; test("page should not have accessibility violations", async ({ page }) => { await page.goto("/"); const accessibilityScanResults = await new AxeBuilder({ page }) .exclude("#third-party-widget") .analyze(); expect(accessibilityScanResults.violations).toEqual([]); }); test("form is accessible", async ({ page }) => { await page.goto("/signup"); const results = await new AxeBuilder({ page }).include("form").analyze(); expect(results.violations).toEqual([]); }); ``` ## Best Practices 1. **Use Data Attributes**: `data-testid` or `data-cy` for stable selectors 2. **Avoid Brittle Selectors**: Don't rely on CSS classes or DOM structure 3. **Test User Behavior**: Click, type, see - not implementation details 4. **Keep Tests Independent**: Each test should run in isolation 5. **Clean Up Test Data**: Create and destroy test data in each test 6. **Use Page Objects**: Encapsulate page logic 7. **Meaningful Assertions**: Check actual user-visible behavior 8. **Optimize for Speed**: Mock when possible, parallel execution ```typescript // ❌ Bad selectors cy.get(".btn.btn-primary.submit-button").click(); cy.get("div > form > div:nth-child(2) > input").type("text"); // ✅ Good selectors cy.getByRole("button", { name: "Submit" }).click(); cy.getByLabel("Email address").type("user@example.com"); cy.get('[data-testid="email-input"]').type("user@example.com"); ``` ## Common Pitfalls - **Flaky Tests**: Use proper waits, not fixed timeouts - **Slow Tests**: Mock external APIs, use parallel execution - **Over-Testing**: Don't test every edge case with E2E - **Coupled Tests**: Tests should not depend on each other - **Poor Selectors**: Avoid CSS classes and nth-child - **No Cleanup**: Clean up test data after each test - **Testing Implementation**: Test user behavior, not internals ## Debugging Failing Tests ```typescript // Playwright debugging // 1. Run in headed mode npx playwright test --headed // 2. Run in debug mode npx playwright test --debug // 3. Use trace viewer await page.screenshot({ path: 'screenshot.png' }); await page.video()?.saveAs('video.webm'); // 4. Add test.step for better reporting test('checkout flow', async ({ page }) => { await test.step('Add item to cart', async () => { await page.goto('/products'); await page.getByRole('button', { name: 'Add to Cart' }).click(); }); await test.step('Proceed to checkout', async () => { await page.goto('/cart'); await page.getByRole('button', { name: 'Checkout' }).click(); }); }); // 5. Inspect page state await page.pause(); // Pauses execution, opens inspector ``` ## Resources - **references/playwright-best-practices.md**: Playwright-specific patterns - **references/cypress-best-practices.md**: Cypress-specific patterns - **references/flaky-test-debugging.md**: Debugging unreliable tests - **assets/e2e-testing-checklist.md**: What to test with E2E - **assets/selector-strategies.md**: Finding reliable selectors - **scripts/test-analyzer.ts**: Analyze test flakiness and duration
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

dotnet-backend-patterns

Master C#/.NET backend development patterns for building robust

coding
⭐1
# .NET Backend Development Patterns Master C#/.NET patterns for building production-grade APIs, MCP servers, and enterprise backends with modern best practices (2024/2025). ## When to Use This Skill - Developing new .NET Web APIs or MCP servers - Reviewing C# code for quality and performance - Designing service architectures with dependency injection - Implementing caching strategies with Redis - Writing unit and integration tests - Optimizing database access with EF Core or Dapper - Configuring applications with IOptions pattern - Handling errors and implementing resilience patterns ## Core Concepts ### 1. Project Structure (Clean Architecture) ``` src/ ├── Domain/ # Core business logic (no dependencies) │ ├── Entities/ │ ├── Interfaces/ │ ├── Exceptions/ │ └── ValueObjects/ ├── Application/ # Use cases, DTOs, validation │ ├── Services/ │ ├── DTOs/ │ ├── Validators/ │ └── Interfaces/ ├── Infrastructure/ # External implementations │ ├── Data/ # EF Core, Dapper repositories │ ├── Caching/ # Redis, Memory cache │ ├── External/ # HTTP clients, third-party APIs │ └── DependencyInjection/ # Service registration └── Api/ # Entry point ├── Controllers/ # Or MinimalAPI endpoints ├── Middleware/ ├── Filters/ └── Program.cs ``` ### 2. Dependency Injection Patterns ```csharp // Service registration by lifetime public static class ServiceCollectionExtensions { public static IServiceCollection AddApplicationServices( this IServiceCollection services, IConfiguration configuration) { // Scoped: One instance per HTTP request services.AddScoped<IProductService, ProductService>(); services.AddScoped<IOrderService, OrderService>(); // Singleton: One instance for app lifetime services.AddSingleton<ICacheService, RedisCacheService>(); services.AddSingleton<IConnectionMultiplexer>(_ => ConnectionMultiplexer.Connect(configuration["Redis:Connection"]!)); // Transient: New instance every time services.AddTransient<IValidator<CreateOrderRequest>, CreateOrderValidator>(); // Options pattern for configuration services.Configure<CatalogOptions>(configuration.GetSection("Catalog")); services.Configure<RedisOptions>(configuration.GetSection("Redis")); // Factory pattern for conditional creation services.AddScoped<IPriceCalculator>(sp => { var options = sp.GetRequiredService<IOptions<PricingOptions>>().Value; return options.UseNewEngine ? sp.GetRequiredService<NewPriceCalculator>() : sp.GetRequiredService<LegacyPriceCalculator>(); }); // Keyed services (.NET 8+) services.AddKeyedScoped<IPaymentProcessor, StripeProcessor>("stripe"); services.AddKeyedScoped<IPaymentProcessor, PayPalProcessor>("paypal"); return services; } } // Usage with keyed services public class CheckoutService { public CheckoutService( [FromKeyedServices("stripe")] IPaymentProcessor stripeProcessor) { _processor = stripeProcessor; } } ``` ### 3. Async/Await Patterns ```csharp // ✅ CORRECT: Async all the way down public async Task<Product> GetProductAsync(string id, CancellationToken ct = default) { return await _repository.GetByIdAsync(id, ct); } // ✅ CORRECT: Parallel execution with WhenAll public async Task<(Stock, Price)> GetStockAndPriceAsync( string productId, CancellationToken ct = default) { var stockTask = _stockService.GetAsync(productId, ct); var priceTask = _priceService.GetAsync(productId, ct); await Task.WhenAll(stockTask, priceTask); return (await stockTask, await priceTask); } // ✅ CORRECT: ConfigureAwait in libraries public async Task<T> LibraryMethodAsync<T>(CancellationToken ct = default) { var result = await _httpClient.GetAsync(url, ct).ConfigureAwait(false); return await result.Content.ReadFromJsonAsync<T>(ct).ConfigureAwait(false); } // ✅ CORRECT: ValueTask for hot paths with caching public ValueTask<Product?> GetCachedProductAsync(string id) { if (_cache.TryGetValue(id, out Product? product)) return ValueTask.FromResult(product); return new ValueTask<Product?>(GetFromDatabaseAsync(id)); } // ❌ WRONG: Blocking on async (deadlock risk) var result = GetProductAsync(id).Result; // NEVER do this var result2 = GetProductAsync(id).GetAwaiter().GetResult(); // Also bad // ❌ WRONG: async void (except event handlers) public async void ProcessOrder() { } // Exceptions are lost // ❌ WRONG: Unnecessary Task.Run for already async code await Task.Run(async () => await GetDataAsync()); // Wastes thread ``` ### 4. Configuration with IOptions ```csharp // Configuration classes public class CatalogOptions { public const string SectionName = "Catalog"; public int DefaultPageSize { get; set; } = 50; public int MaxPageSize { get; set; } = 200; public TimeSpan CacheDuration { get; set; } = TimeSpan.FromMinutes(15); public bool EnableEnrichment { get; set; } = true; } public class RedisOptions { public const string SectionName = "Redis"; public string Connection { get; set; } = "localhost:6379"; public string KeyPrefix { get; set; } = "mcp:"; public int Database { get; set; } = 0; } // appsettings.json { "Catalog": { "DefaultPageSize": 50, "MaxPageSize": 200, "CacheDuration": "00:15:00", "EnableEnrichment": true }, "Redis": { "Connection": "localhost:6379", "KeyPrefix": "mcp:", "Database": 0 } } // Registration services.Configure<CatalogOptions>(configuration.GetSection(CatalogOptions.SectionName)); services.Configure<RedisOptions>(configuration.GetSection(RedisOptions.SectionName)); // Usage with IOptions (singleton, read once at startup) public class CatalogService { private readonly CatalogOptions _options; public CatalogService(IOptions<CatalogOptions> options) { _options = options.Value; } } // Usage with IOptionsSnapshot (scoped, re-reads on each request) public class DynamicService { private readonly CatalogOptions _options; public DynamicService(IOptionsSnapshot<CatalogOptions> options) { _options = options.Value; // Fresh value per request } } // Usage with IOptionsMonitor (singleton, notified on changes) public class MonitoredService { private CatalogOptions _options; public MonitoredService(IOptionsMonitor<CatalogOptions> monitor) { _options = monitor.CurrentValue; monitor.OnChange(newOptions => _options = newOptions); } } ``` ### 5. Result Pattern (Avoiding Exceptions for Flow Control) ```csharp // Generic Result type public class Result<T> { public bool IsSuccess { get; } public T? Value { get; } public string? Error { get; } public string? ErrorCode { get; } private Result(bool isSuccess, T? value, string? error, string? errorCode) { IsSuccess = isSuccess; Value = value; Error = error; ErrorCode = errorCode; } public static Result<T> Success(T value) => new(true, value, null, null); public static Result<T> Failure(string error, string? code = null) => new(false, default, error, code); public Result<TNew> Map<TNew>(Func<T, TNew> mapper) => IsSuccess ? Result<TNew>.Success(mapper(Value!)) : Result<TNew>.Failure(Error!, ErrorCode); public async Task<Result<TNew>> MapAsync<TNew>(Func<T, Task<TNew>> mapper) => IsSuccess ? Result<TNew>.Success(await mapper(Value!)) : Result<TNew>.Failure(Error!, ErrorCode); } // Usage in service public async Task<Result<Order>> CreateOrderAsync(CreateOrderRequest request, CancellationToken ct) { // Validation var validation = await _validator.ValidateAsync(request, ct); if (!validation.IsValid) return Result<Order>.Failure( validation.Errors.First().ErrorMessage, "VALIDATION_ERROR"); // Business rule check var stock = await _stockService.CheckAsync(request.ProductId, request.Quantity, ct); if (!stock.IsAvailable) return Result<Order>.Failure( $"Insufficient stock: {stock.Available} available, {request.Quantity} requested", "INSUFFICIENT_STOCK"); // Create order var order = await _repository.CreateAsync(request.ToEntity(), ct); return Result<Order>.Success(order); } // Usage in controller/endpoint app.MapPost("/orders", async ( CreateOrderRequest request, IOrderService orderService, CancellationToken ct) => { var result = await orderService.CreateOrderAsync(request, ct); return result.IsSuccess ? Results.Created($"/orders/{result.Value!.Id}", result.Value) : Results.BadRequest(new { error = result.Error, code = result.ErrorCode }); }); ``` ## Data Access Patterns ### Entity Framework Core ```csharp // DbContext configuration public class AppDbContext : DbContext { public DbSet<Product> Products => Set<Product>(); public DbSet<Order> Orders => Set<Order>(); protected override void OnModelCreating(ModelBuilder modelBuilder) { // Apply all configurations from assembly modelBuilder.ApplyConfigurationsFromAssembly(typeof(AppDbContext).Assembly); // Global query filters modelBuilder.Entity<Product>().HasQueryFilter(p => !p.IsDeleted); } } // Entity configuration public class ProductConfiguration : IEntityTypeConfiguration<Product> { public void Configure(EntityTypeBuilder<Product> builder) { builder.ToTable("Products"); builder.HasKey(p => p.Id); builder.Property(p => p.Id).HasMaxLength(40); builder.Property(p => p.Name).HasMaxLength(200).IsRequired(); builder.Property(p => p.Price).HasPrecision(18, 2); builder.HasIndex(p => p.Sku).IsUnique(); builder.HasIndex(p => new { p.CategoryId, p.Name }); builder.HasMany(p => p.OrderItems) .WithOne(oi => oi.Product) .HasForeignKey(oi => oi.ProductId); } } // Repository with EF Core public class ProductRepository : IProductRepository { private readonly AppDbContext _context; public async Task<Product?> GetByIdAsync(string id, CancellationToken ct = default) { return await _context.Products .AsNoTracking() .FirstOrDefaultAsync(p => p.Id == id, ct); } public async Task<IReadOnlyList<Product>> SearchAsync( ProductSearchCriteria criteria, CancellationToken ct = default) { var query = _context.Products.AsNoTracking(); if (!string.IsNullOrWhiteSpace(criteria.SearchTerm)) query = query.Where(p => EF.Functions.Like(p.Name, $"%{criteria.SearchTerm}%")); if (criteria.CategoryId.HasValue) query = query.Where(p => p.CategoryId == criteria.CategoryId); if (criteria.MinPrice.HasValue) query = query.Where(p => p.Price >= criteria.MinPrice); if (criteria.MaxPrice.HasValue) query = query.Where(p => p.Price <= criteria.MaxPrice); return await query .OrderBy(p => p.Name) .Skip((criteria.Page - 1) * criteria.PageSize) .Take(criteria.PageSize) .ToListAsync(ct); } } ``` ### Dapper for Performance ```csharp public class DapperProductRepository : IProductRepository { private readonly IDbConnection _connection; public async Task<Product?> GetByIdAsync(string id, CancellationToken ct = default) { const string sql = """ SELECT Id, Name, Sku, Price, CategoryId, Stock, CreatedAt FROM Products WHERE Id = @Id AND IsDeleted = 0 """; return await _connection.QueryFirstOrDefaultAsync<Product>( new CommandDefinition(sql, new { Id = id }, cancellationToken: ct)); } public async Task<IReadOnlyList<Product>> SearchAsync( ProductSearchCriteria criteria, CancellationToken ct = default) { var sql = new StringBuilder(""" SELECT Id, Name, Sku, Price, CategoryId, Stock, CreatedAt FROM Products WHERE IsDeleted = 0 """); var parameters = new DynamicParameters(); if (!string.IsNullOrWhiteSpace(criteria.SearchTerm)) { sql.Append(" AND Name LIKE @SearchTerm"); parameters.Add("SearchTerm", $"%{criteria.SearchTerm}%"); } if (criteria.CategoryId.HasValue) { sql.Append(" AND CategoryId = @CategoryId"); parameters.Add("CategoryId", criteria.CategoryId); } if (criteria.MinPrice.HasValue) { sql.Append(" AND Price >= @MinPrice"); parameters.Add("MinPrice", criteria.MinPrice); } if (criteria.MaxPrice.HasValue) { sql.Append(" AND Price <= @MaxPrice"); parameters.Add("MaxPrice", criteria.MaxPrice); } sql.Append(" ORDER BY Name OFFSET @Offset ROWS FETCH NEXT @PageSize ROWS ONLY"); parameters.Add("Offset", (criteria.Page - 1) * criteria.PageSize); parameters.Add("PageSize", criteria.PageSize); var results = await _connection.QueryAsync<Product>( new CommandDefinition(sql.ToString(), parameters, cancellationToken: ct)); return results.ToList(); } // Multi-mapping for related data public async Task<Order?> GetOrderWithItemsAsync(int orderId, CancellationToken ct = default) { const string sql = """ SELECT o.*, oi.*, p.* FROM Orders o LEFT JOIN OrderItems oi ON o.Id = oi.OrderId LEFT JOIN Products p ON oi.ProductId = p.Id WHERE o.Id = @OrderId """; var orderDictionary = new Dictionary<int, Order>(); await _connection.QueryAsync<Order, OrderItem, Product, Order>( new CommandDefinition(sql, new { OrderId = orderId }, cancellationToken: ct), (order, item, product) => { if (!orderDictionary.TryGetValue(order.Id, out var existingOrder)) { existingOrder = order; existingOrder.Items = new List<OrderItem>(); orderDictionary.Add(order.Id, existingOrder); } if (item != null) { item.Product = product; existingOrder.Items.Add(item); } return existingOrder; }, splitOn: "Id,Id"); return orderDictionary.Values.FirstOrDefault(); } } ``` ## Caching Patterns ### Multi-Level Cache with Redis ```csharp public class CachedProductService : IProductService { private readonly IProductRepository _repository; private readonly IMemoryCache _memoryCache; private readonly IDistributedCache _distributedCache; private readonly ILogger<CachedProductService> _logger; private static readonly TimeSpan MemoryCacheDuration = TimeSpan.FromMinutes(1); private static readonly TimeSpan DistributedCacheDuration = TimeSpan.FromMinutes(15); public async Task<Product?> GetByIdAsync(string id, CancellationToken ct = default) { var cacheKey = $"product:{id}"; // L1: Memory cache (in-process, fastest) if (_memoryCache.TryGetValue(cacheKey, out Product? cached)) { _logger.LogDebug("L1 cache hit for {CacheKey}", cacheKey); return cached; } // L2: Distributed cache (Redis) var distributed = await _distributedCache.GetStringAsync(cacheKey, ct); if (distributed != null) { _logger.LogDebug("L2 cache hit for {CacheKey}", cacheKey); var product = JsonSerializer.Deserialize<Product>(distributed); // Populate L1 _memoryCache.Set(cacheKey, product, MemoryCacheDuration); return product; } // L3: Database _logger.LogDebug("Cache miss for {CacheKey}, fetching from database", cacheKey); var fromDb = await _repository.GetByIdAsync(id, ct); if (fromDb != null) { var serialized = JsonSerializer.Serialize(fromDb); // Populate both caches await _distributedCache.SetStringAsync( cacheKey, serialized, new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = DistributedCacheDuration }, ct); _memoryCache.Set(cacheKey, fromDb, MemoryCacheDuration); } return fromDb; } public async Task InvalidateAsync(string id, CancellationToken ct = default) { var cacheKey = $"product:{id}"; _memoryCache.Remove(cacheKey); await _distributedCache.RemoveAsync(cacheKey, ct); _logger.LogInformation("Invalidated cache for {CacheKey}", cacheKey); } } // Stale-while-revalidate pattern public class StaleWhileRevalidateCache<T> { private readonly IDistributedCache _cache; private readonly TimeSpan _freshDuration; private readonly TimeSpan _staleDuration; public async Task<T?> GetOrCreateAsync( string key, Func<CancellationToken, Task<T>> factory, CancellationToken ct = default) { var cached = await _cache.GetStringAsync(key, ct); if (cached != null) { var entry = JsonSerializer.Deserialize<CacheEntry<T>>(cached)!; if (entry.IsStale && !entry.IsExpired) { // Return stale data immediately, refresh in background _ = Task.Run(async () => { var fresh = await factory(CancellationToken.None); await SetAsync(key, fresh, CancellationToken.None); }); } if (!entry.IsExpired) return entry.Value; } // Cache miss or expired var value = await factory(ct); await SetAsync(key, value, ct); return value; } private record CacheEntry<TValue>(TValue Value, DateTime CreatedAt) { public bool IsStale => DateTime.UtcNow - CreatedAt > _freshDuration; public bool IsExpired => DateTime.UtcNow - CreatedAt > _staleDuration; } } ``` ## Testing Patterns ### Unit Tests with xUnit and Moq ```csharp public class OrderServiceTests { private readonly Mock<IOrderRepository> _mockRepository; private readonly Mock<IStockService> _mockStockService; private readonly Mock<IValidator<CreateOrderRequest>> _mockValidator; private readonly OrderService _sut; // System Under Test public OrderServiceTests() { _mockRepository = new Mock<IOrderRepository>(); _mockStockService = new Mock<IStockService>(); _mockValidator = new Mock<IValidator<CreateOrderRequest>>(); // Default: validation passes _mockValidator .Setup(v => v.ValidateAsync(It.IsAny<CreateOrderRequest>(), It.IsAny<
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

incident-runbook-templates

Create structured incident response runbooks with step-by-step

coding
⭐1
# Incident Runbook Templates Production-ready templates for incident response runbooks covering detection, triage, mitigation, resolution, and communication. ## When to Use This Skill - Creating incident response procedures - Building service-specific runbooks - Establishing escalation paths - Documenting recovery procedures - Responding to active incidents - Onboarding on-call engineers ## Core Concepts ### 1. Incident Severity Levels | Severity | Impact | Response Time | Example | | -------- | -------------------------- | ----------------- | ----------------------- | | **SEV1** | Complete outage, data loss | 15 min | Production down | | **SEV2** | Major degradation | 30 min | Critical feature broken | | **SEV3** | Minor impact | 2 hours | Non-critical bug | | **SEV4** | Minimal impact | Next business day | Cosmetic issue | ### 2. Runbook Structure ``` 1. Overview & Impact 2. Detection & Alerts 3. Initial Triage 4. Mitigation Steps 5. Root Cause Investigation 6. Resolution Procedures 7. Verification & Rollback 8. Communication Templates 9. Escalation Matrix ``` ## Runbook Templates ### Template 1: Service Outage Runbook ````markdown # [Service Name] Outage Runbook ## Overview **Service**: Payment Processing Service **Owner**: Platform Team **Slack**: #payments-incidents **PagerDuty**: payments-oncall ## Impact Assessment - [ ] Which customers are affected? - [ ] What percentage of traffic is impacted? - [ ] Are there financial implications? - [ ] What's the blast radius? ## Detection ### Alerts - `payment_error_rate > 5%` (PagerDuty) - `payment_latency_p99 > 2s` (Slack) - `payment_success_rate < 95%` (PagerDuty) ### Dashboards - [Payment Service Dashboard](https://grafana/d/payments) - [Error Tracking](https://sentry.io/payments) - [Dependency Status](https://status.stripe.com) ## Initial Triage (First 5 Minutes) ### 1. Assess Scope ```bash # Check service health kubectl get pods -n payments -l app=payment-service # Check recent deployments kubectl rollout history deployment/payment-service -n payments # Check error rates curl -s "http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[5m]))" ``` ```` ### 2. Quick Health Checks - [ ] Can you reach the service? `curl -I https://api.company.com/payments/health` - [ ] Database connectivity? Check connection pool metrics - [ ] External dependencies? Check Stripe, bank API status - [ ] Recent changes? Check deploy history ### 3. Initial Classification | Symptom | Likely Cause | Go To Section | | -------------------- | ------------------- | ------------- | | All requests failing | Service down | Section 4.1 | | High latency | Database/dependency | Section 4.2 | | Partial failures | Code bug | Section 4.3 | | Spike in errors | Traffic surge | Section 4.4 | ## Mitigation Procedures ### 4.1 Service Completely Down ```bash # Step 1: Check pod status kubectl get pods -n payments # Step 2: If pods are crash-looping, check logs kubectl logs -n payments -l app=payment-service --tail=100 # Step 3: Check recent deployments kubectl rollout history deployment/payment-service -n payments # Step 4: ROLLBACK if recent deploy is suspect kubectl rollout undo deployment/payment-service -n payments # Step 5: Scale up if resource constrained kubectl scale deployment/payment-service -n payments --replicas=10 # Step 6: Verify recovery kubectl rollout status deployment/payment-service -n payments ``` ### 4.2 High Latency ```bash # Step 1: Check database connections kubectl exec -n payments deploy/payment-service -- \ curl localhost:8080/metrics | grep db_pool # Step 2: Check slow queries (if DB issue) psql -h $DB_HOST -U $DB_USER -c " SELECT pid, now() - query_start AS duration, query FROM pg_stat_activity WHERE state = 'active' AND duration > interval '5 seconds' ORDER BY duration DESC;" # Step 3: Kill long-running queries if needed psql -h $DB_HOST -U $DB_USER -c "SELECT pg_terminate_backend(pid);" # Step 4: Check external dependency latency curl -w "@curl-format.txt" -o /dev/null -s https://api.stripe.com/v1/health # Step 5: Enable circuit breaker if dependency is slow kubectl set env deployment/payment-service \ STRIPE_CIRCUIT_BREAKER_ENABLED=true -n payments ``` ### 4.3 Partial Failures (Specific Errors) ```bash # Step 1: Identify error pattern kubectl logs -n payments -l app=payment-service --tail=500 | \ grep -i error | sort | uniq -c | sort -rn | head -20 # Step 2: Check error tracking # Go to Sentry: https://sentry.io/payments # Step 3: If specific endpoint, enable feature flag to disable curl -X POST https://api.company.com/internal/feature-flags \ -d '{"flag": "DISABLE_PROBLEMATIC_FEATURE", "enabled": true}' # Step 4: If data issue, check recent data changes psql -h $DB_HOST -c " SELECT * FROM audit_log WHERE table_name = 'payment_methods' AND created_at > now() - interval '1 hour';" ``` ### 4.4 Traffic Surge ```bash # Step 1: Check current request rate kubectl top pods -n payments # Step 2: Scale horizontally kubectl scale deployment/payment-service -n payments --replicas=20 # Step 3: Enable rate limiting kubectl set env deployment/payment-service \ RATE_LIMIT_ENABLED=true \ RATE_LIMIT_RPS=1000 -n payments # Step 4: If attack, block suspicious IPs kubectl apply -f - <<EOF apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: block-suspicious namespace: payments spec: podSelector: matchLabels: app: payment-service ingress: - from: - ipBlock: cidr: 0.0.0.0/0 except: - 192.168.1.0/24 # Suspicious range EOF ``` ## Verification Steps ```bash # Verify service is healthy curl -s https://api.company.com/payments/health | jq # Verify error rate is back to normal curl -s "http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[5m]))" | jq '.data.result[0].value[1]' # Verify latency is acceptable curl -s "http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,sum(rate(http_request_duration_seconds_bucket[5m]))by(le))" | jq # Smoke test critical flows ./scripts/smoke-test-payments.sh ``` ## Rollback Procedures ```bash # Rollback Kubernetes deployment kubectl rollout undo deployment/payment-service -n payments # Rollback database migration (if applicable) ./scripts/db-rollback.sh $MIGRATION_VERSION # Rollback feature flag curl -X POST https://api.company.com/internal/feature-flags \ -d '{"flag": "NEW_PAYMENT_FLOW", "enabled": false}' ``` ## Escalation Matrix | Condition | Escalate To | Contact | | ----------------------------- | ------------------- | ------------------- | | > 15 min unresolved SEV1 | Engineering Manager | @manager (Slack) | | Data breach suspected | Security Team | #security-incidents | | Financial impact > $10k | Finance + Legal | @finance-oncall | | Customer communication needed | Support Lead | @support-lead | ## Communication Templates ### Initial Notification (Internal) ``` 🚨 INCIDENT: Payment Service Degradation Severity: SEV2 Status: Investigating Impact: ~20% of payment requests failing Start Time: [TIME] Incident Commander: [NAME] Current Actions: - Investigating root cause - Scaling up service - Monitoring dashboards Updates in #payments-incidents ``` ### Status Update ``` 📊 UPDATE: Payment Service Incident Status: Mitigating Impact: Reduced to ~5% failure rate Duration: 25 minutes Actions Taken: - Rolled back deployment v2.3.4 → v2.3.3 - Scaled service from 5 → 10 replicas Next Steps: - Continuing to monitor - Root cause analysis in progress ETA to Resolution: ~15 minutes ``` ### Resolution Notification ``` ✅ RESOLVED: Payment Service Incident Duration: 45 minutes Impact: ~5,000 affected transactions Root Cause: Memory leak in v2.3.4 Resolution: - Rolled back to v2.3.3 - Transactions auto-retried successfully Follow-up: - Postmortem scheduled for [DATE] - Bug fix in progress ``` ```` ### Template 2: Database Incident Runbook ```markdown # Database Incident Runbook ## Quick Reference | Issue | Command | |-------|---------| | Check connections | `SELECT count(*) FROM pg_stat_activity;` | | Kill query | `SELECT pg_terminate_backend(pid);` | | Check replication lag | `SELECT extract(epoch from (now() - pg_last_xact_replay_timestamp()));` | | Check locks | `SELECT * FROM pg_locks WHERE NOT granted;` | ## Connection Pool Exhaustion ```sql -- Check current connections SELECT datname, usename, state, count(*) FROM pg_stat_activity GROUP BY datname, usename, state ORDER BY count(*) DESC; -- Identify long-running connections SELECT pid, usename, datname, state, query_start, query FROM pg_stat_activity WHERE state != 'idle' ORDER BY query_start; -- Terminate idle connections SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE state = 'idle' AND query_start < now() - interval '10 minutes'; ```` ## Replication Lag ```sql -- Check lag on replica SELECT CASE WHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN 0 ELSE extract(epoch from now() - pg_last_xact_replay_timestamp()) END AS lag_seconds; -- If lag > 60s, consider: -- 1. Check network between primary/replica -- 2. Check replica disk I/O -- 3. Consider failover if unrecoverable ``` ## Disk Space Critical ```bash # Check disk usage df -h /var/lib/postgresql/data # Find large tables psql -c "SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;" # VACUUM to reclaim space psql -c "VACUUM FULL large_table;" # If emergency, delete old data or expand disk ``` ``` ## Best Practices ### Do's - **Keep runbooks updated** - Review after every incident - **Test runbooks regularly** - Game days, chaos engineering - **Include rollback steps** - Always have an escape hatch - **Document assumptions** - What must be true for steps to work - **Link to dashboards** - Quick access during stress ### Don'ts - **Don't assume knowledge** - Write for 3 AM brain - **Don't skip verification** - Confirm each step worked - **Don't forget communication** - Keep stakeholders informed - **Don't work alone** - Escalate early - **Don't skip postmortems** - Learn from every incident ## Resources - [Google SRE Book - Incident Management](https://sre.google/sre-book/managing-incidents/) - [PagerDuty Incident Response](https://response.pagerduty.com/) - [Atlassian Incident Management](https://www.atlassian.com/incident-management) ```
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

billing-automation

Build automated billing systems for recurring payments, invoicing,

business
⭐1
# Billing Automation Master automated billing systems including recurring billing, invoice generation, dunning management, proration, and tax calculation. ## When to Use This Skill - Implementing SaaS subscription billing - Automating invoice generation and delivery - Managing failed payment recovery (dunning) - Calculating prorated charges for plan changes - Handling sales tax, VAT, and GST - Processing usage-based billing - Managing billing cycles and renewals ## Core Concepts ### 1. Billing Cycles **Common Intervals:** - Monthly (most common for SaaS) - Annual (discounted long-term) - Quarterly - Weekly - Custom (usage-based, per-seat) ### 2. Subscription States ``` trial → active → past_due → canceled → paused → resumed ``` ### 3. Dunning Management Automated process to recover failed payments through: - Retry schedules - Customer notifications - Grace periods - Account restrictions ### 4. Proration Adjusting charges when: - Upgrading/downgrading mid-cycle - Adding/removing seats - Changing billing frequency ## Quick Start ```python from billing import BillingEngine, Subscription # Initialize billing engine billing = BillingEngine() # Create subscription subscription = billing.create_subscription( customer_id="cus_123", plan_id="plan_pro_monthly", billing_cycle_anchor=datetime.now(), trial_days=14 ) # Process billing cycle billing.process_billing_cycle(subscription.id) ``` ## Subscription Lifecycle Management ```python from datetime import datetime, timedelta from enum import Enum class SubscriptionStatus(Enum): TRIAL = "trial" ACTIVE = "active" PAST_DUE = "past_due" CANCELED = "canceled" PAUSED = "paused" class Subscription: def __init__(self, customer_id, plan, billing_cycle_day=None): self.id = generate_id() self.customer_id = customer_id self.plan = plan self.status = SubscriptionStatus.TRIAL self.current_period_start = datetime.now() self.current_period_end = self.current_period_start + timedelta(days=plan.trial_days or 30) self.billing_cycle_day = billing_cycle_day or self.current_period_start.day self.trial_end = datetime.now() + timedelta(days=plan.trial_days) if plan.trial_days else None def start_trial(self, trial_days): """Start trial period.""" self.status = SubscriptionStatus.TRIAL self.trial_end = datetime.now() + timedelta(days=trial_days) self.current_period_end = self.trial_end def activate(self): """Activate subscription after trial or immediately.""" self.status = SubscriptionStatus.ACTIVE self.current_period_start = datetime.now() self.current_period_end = self.calculate_next_billing_date() def mark_past_due(self): """Mark subscription as past due after failed payment.""" self.status = SubscriptionStatus.PAST_DUE # Trigger dunning workflow def cancel(self, at_period_end=True): """Cancel subscription.""" if at_period_end: self.cancel_at_period_end = True # Will cancel when current period ends else: self.status = SubscriptionStatus.CANCELED self.canceled_at = datetime.now() def calculate_next_billing_date(self): """Calculate next billing date based on interval.""" if self.plan.interval == 'month': return self.current_period_start + timedelta(days=30) elif self.plan.interval == 'year': return self.current_period_start + timedelta(days=365) elif self.plan.interval == 'week': return self.current_period_start + timedelta(days=7) ``` ## Billing Cycle Processing ```python class BillingEngine: def process_billing_cycle(self, subscription_id): """Process billing for a subscription.""" subscription = self.get_subscription(subscription_id) # Check if billing is due if datetime.now() < subscription.current_period_end: return # Generate invoice invoice = self.generate_invoice(subscription) # Attempt payment payment_result = self.charge_customer( subscription.customer_id, invoice.total ) if payment_result.success: # Payment successful invoice.mark_paid() subscription.advance_billing_period() self.send_invoice(invoice) else: # Payment failed subscription.mark_past_due() self.start_dunning_process(subscription, invoice) def generate_invoice(self, subscription): """Generate invoice for billing period.""" invoice = Invoice( customer_id=subscription.customer_id, subscription_id=subscription.id, period_start=subscription.current_period_start, period_end=subscription.current_period_end ) # Add subscription line item invoice.add_line_item( description=subscription.plan.name, amount=subscription.plan.amount, quantity=subscription.quantity or 1 ) # Add usage-based charges if applicable if subscription.has_usage_billing: usage_charges = self.calculate_usage_charges(subscription) invoice.add_line_item( description="Usage charges", amount=usage_charges ) # Calculate tax tax = self.calculate_tax(invoice.subtotal, subscription.customer) invoice.tax = tax invoice.finalize() return invoice def charge_customer(self, customer_id, amount): """Charge customer using saved payment method.""" customer = self.get_customer(customer_id) try: # Charge using payment processor charge = stripe.Charge.create( customer=customer.stripe_id, amount=int(amount * 100), # Convert to cents currency='usd' ) return PaymentResult(success=True, transaction_id=charge.id) except stripe.error.CardError as e: return PaymentResult(success=False, error=str(e)) ``` ## Dunning Management ```python class DunningManager: """Manage failed payment recovery.""" def __init__(self): self.retry_schedule = [ {'days': 3, 'email_template': 'payment_failed_first'}, {'days': 7, 'email_template': 'payment_failed_reminder'}, {'days': 14, 'email_template': 'payment_failed_final'} ] def start_dunning_process(self, subscription, invoice): """Start dunning process for failed payment.""" dunning_attempt = DunningAttempt( subscription_id=subscription.id, invoice_id=invoice.id, attempt_number=1, next_retry=datetime.now() + timedelta(days=3) ) # Send initial failure notification self.send_dunning_email(subscription, 'payment_failed_first') # Schedule retries self.schedule_retries(dunning_attempt) def retry_payment(self, dunning_attempt): """Retry failed payment.""" subscription = self.get_subscription(dunning_attempt.subscription_id) invoice = self.get_invoice(dunning_attempt.invoice_id) # Attempt payment again result = self.charge_customer(subscription.customer_id, invoice.total) if result.success: # Payment succeeded invoice.mark_paid() subscription.status = SubscriptionStatus.ACTIVE self.send_dunning_email(subscription, 'payment_recovered') dunning_attempt.mark_resolved() else: # Still failing dunning_attempt.attempt_number += 1 if dunning_attempt.attempt_number < len(self.retry_schedule): # Schedule next retry next_retry_config = self.retry_schedule[dunning_attempt.attempt_number] dunning_attempt.next_retry = datetime.now() + timedelta(days=next_retry_config['days']) self.send_dunning_email(subscription, next_retry_config['email_template']) else: # Exhausted retries, cancel subscription subscription.cancel(at_period_end=False) self.send_dunning_email(subscription, 'subscription_canceled') def send_dunning_email(self, subscription, template): """Send dunning notification to customer.""" customer = self.get_customer(subscription.customer_id) email_content = self.render_template(template, { 'customer_name': customer.name, 'amount_due': subscription.plan.amount, 'update_payment_url': f"https://app.example.com/billing" }) send_email( to=customer.email, subject=email_content['subject'], body=email_content['body'] ) ``` ## Proration ```python class ProrationCalculator: """Calculate prorated charges for plan changes.""" @staticmethod def calculate_proration(old_plan, new_plan, period_start, period_end, change_date): """Calculate proration for plan change.""" # Days in current period total_days = (period_end - period_start).days # Days used on old plan days_used = (change_date - period_start).days # Days remaining on new plan days_remaining = (period_end - change_date).days # Calculate prorated amounts unused_amount = (old_plan.amount / total_days) * days_remaining new_plan_amount = (new_plan.amount / total_days) * days_remaining # Net charge/credit proration = new_plan_amount - unused_amount return { 'old_plan_credit': -unused_amount, 'new_plan_charge': new_plan_amount, 'net_proration': proration, 'days_used': days_used, 'days_remaining': days_remaining } @staticmethod def calculate_seat_proration(current_seats, new_seats, price_per_seat, period_start, period_end, change_date): """Calculate proration for seat changes.""" total_days = (period_end - period_start).days days_remaining = (period_end - change_date).days # Additional seats charge additional_seats = new_seats - current_seats prorated_amount = (additional_seats * price_per_seat / total_days) * days_remaining return { 'additional_seats': additional_seats, 'prorated_charge': max(0, prorated_amount), # No refund for removing seats mid-cycle 'effective_date': change_date } ``` ## Tax Calculation ```python class TaxCalculator: """Calculate sales tax, VAT, GST.""" def __init__(self): # Tax rates by region self.tax_rates = { 'US_CA': 0.0725, # California sales tax 'US_NY': 0.04, # New York sales tax 'GB': 0.20, # UK VAT 'DE': 0.19, # Germany VAT 'FR': 0.20, # France VAT 'AU': 0.10, # Australia GST } def calculate_tax(self, amount, customer): """Calculate applicable tax.""" # Determine tax jurisdiction jurisdiction = self.get_tax_jurisdiction(customer) if not jurisdiction: return 0 # Get tax rate tax_rate = self.tax_rates.get(jurisdiction, 0) # Calculate tax tax = amount * tax_rate return { 'tax_amount': tax, 'tax_rate': tax_rate, 'jurisdiction': jurisdiction, 'tax_type': self.get_tax_type(jurisdiction) } def get_tax_jurisdiction(self, customer): """Determine tax jurisdiction based on customer location.""" if customer.country == 'US': # US: Tax based on customer state return f"US_{customer.state}" elif customer.country in ['GB', 'DE', 'FR']: # EU: VAT return customer.country elif customer.country == 'AU': # Australia: GST return 'AU' else: return None def get_tax_type(self, jurisdiction): """Get type of tax for jurisdiction.""" if jurisdiction.startswith('US_'): return 'Sales Tax' elif jurisdiction in ['GB', 'DE', 'FR']: return 'VAT' elif jurisdiction == 'AU': return 'GST' return 'Tax' def validate_vat_number(self, vat_number, country): """Validate EU VAT number.""" # Use VIES API for validation # Returns True if valid, False otherwise pass ``` ## Invoice Generation ```python class Invoice: def __init__(self, customer_id, subscription_id=None): self.id = generate_invoice_number() self.customer_id = customer_id self.subscription_id = subscription_id self.status = 'draft' self.line_items = [] self.subtotal = 0 self.tax = 0 self.total = 0 self.created_at = datetime.now() def add_line_item(self, description, amount, quantity=1): """Add line item to invoice.""" line_item = { 'description': description, 'unit_amount': amount, 'quantity': quantity, 'total': amount * quantity } self.line_items.append(line_item) self.subtotal += line_item['total'] def finalize(self): """Finalize invoice and calculate total.""" self.total = self.subtotal + self.tax self.status = 'open' self.finalized_at = datetime.now() def mark_paid(self): """Mark invoice as paid.""" self.status = 'paid' self.paid_at = datetime.now() def to_pdf(self): """Generate PDF invoice.""" from reportlab.pdfgen import canvas # Generate PDF # Include: company info, customer info, line items, tax, total pass def to_html(self): """Generate HTML invoice.""" template = """ <!DOCTYPE html> <html> <head><title>Invoice #{invoice_number}</title></head> <body> <h1>Invoice #{invoice_number}</h1> <p>Date: {date}</p> <h2>Bill To:</h2> <p>{customer_name}<br>{customer_address}</p> <table> <tr><th>Description</th><th>Quantity</th><th>Amount</th></tr> {line_items} </table> <p>Subtotal: ${subtotal}</p> <p>Tax: ${tax}</p> <h3>Total: ${total}</h3> </body> </html> """ return template.format( invoice_number=self.id, date=self.created_at.strftime('%Y-%m-%d'), customer_name=self.customer.name, customer_address=self.customer.address, line_items=self.render_line_items(), subtotal=self.subtotal, tax=self.tax, total=self.total ) ``` ## Usage-Based Billing ```python class UsageBillingEngine: """Track and bill for usage.""" def track_usage(self, customer_id, metric, quantity): """Track usage event.""" UsageRecord.create( customer_id=customer_id, metric=metric, quantity=quantity, timestamp=datetime.now() ) def calculate_usage_charges(self, subscription, period_start, period_end): """Calculate charges for usage in billing period.""" usage_records = UsageRecord.get_for_period( subscription.customer_id, period_start, period_end ) total_usage = sum(record.quantity for record in usage_records) # Tiered pricing if subscription.plan.pricing_model == 'tiered': charge = self.calculate_tiered_pricing(total_usage, subscription.plan.tiers) # Per-unit pricing elif subscription.plan.pricing_model == 'per_unit': charge = total_usage * subscription.plan.unit_price # Volume pricing elif subscription.plan.pricing_model == 'volume': charge = self.calculate_volume_pricing(total_usage, subscription.plan.tiers) return charge def calculate_tiered_pricing(self, total_usage, tiers): """Calculate cost using tiered pricing.""" charge = 0 remaining = total_usage for tier in sorted(tiers, key=lambda x: x['up_to']): tier_usage = min(remaining, tier['up_to'] - tier['from']) charge += tier_usage * tier['unit_price'] remaining -= tier_usage if remaining <= 0: break return charge ``` ## Resources - **references/billing-cycles.md**: Billing cycle management - **references/dunning-management.md**: Failed payment recovery - **references/proration.md**: Prorated charge calculations - **references/tax-calculation.md**: Tax/VAT/GST handling - **references/invoice-lifecycle.md**: Invoice state management - **assets/billing-state-machine.yaml**: Billing workflow - **assets/invoice-template.html**: Invoice templates - **assets/dunning-policy.yaml**: Dunning configuration ## Best Practices 1. **Automate Everything**: Minimize manual intervention 2. **Clear Communication**: Notify customers of billing events 3. **Flexible Retry Logic**: Balance recovery with customer experience 4. **Accurate Proration**: Fair calculation for plan changes 5. **Tax Compliance**: Calculate correct tax for jurisdiction 6. **Audit Trail**: Log all billing events 7. **Graceful Degradation**: Handle edge cases without breaking ## Common Pitfalls - **Incorrect Proration**: Not accounting for partial periods - **Missing Tax**: Forgetting to add tax to invoices - **Aggressive Dunning**: Canceling too quickly - **No Notifications**: Not informing customers of failures - **Hardcoded Cycles**: Not supporting custom billing dates
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

pci-compliance

Implement PCI DSS compliance requirements for secure handling of

business
⭐1
# PCI Compliance Master PCI DSS (Payment Card Industry Data Security Standard) compliance for secure payment processing and handling of cardholder data. ## When to Use This Skill - Building payment processing systems - Handling credit card information - Implementing secure payment flows - Conducting PCI compliance audits - Reducing PCI compliance scope - Implementing tokenization and encryption - Preparing for PCI DSS assessments ## PCI DSS Requirements (12 Core Requirements) ### Build and Maintain Secure Network 1. Install and maintain firewall configuration 2. Don't use vendor-supplied defaults for passwords ### Protect Cardholder Data 3. Protect stored cardholder data 4. Encrypt transmission of cardholder data across public networks ### Maintain Vulnerability Management 5. Protect systems against malware 6. Develop and maintain secure systems and applications ### Implement Strong Access Control 7. Restrict access to cardholder data by business need-to-know 8. Identify and authenticate access to system components 9. Restrict physical access to cardholder data ### Monitor and Test Networks 10. Track and monitor all access to network resources and cardholder data 11. Regularly test security systems and processes ### Maintain Information Security Policy 12. Maintain a policy that addresses information security ## Compliance Levels **Level 1**: > 6 million transactions/year (annual ROC required) **Level 2**: 1-6 million transactions/year (annual SAQ) **Level 3**: 20,000-1 million e-commerce transactions/year **Level 4**: < 20,000 e-commerce or < 1 million total transactions ## Data Minimization (Never Store) ```python # NEVER STORE THESE PROHIBITED_DATA = { 'full_track_data': 'Magnetic stripe data', 'cvv': 'Card verification code/value', 'pin': 'PIN or PIN block' } # CAN STORE (if encrypted) ALLOWED_DATA = { 'pan': 'Primary Account Number (card number)', 'cardholder_name': 'Name on card', 'expiration_date': 'Card expiration', 'service_code': 'Service code' } class PaymentData: """Safe payment data handling.""" def __init__(self): self.prohibited_fields = ['cvv', 'cvv2', 'cvc', 'pin'] def sanitize_log(self, data): """Remove sensitive data from logs.""" sanitized = data.copy() # Mask PAN if 'card_number' in sanitized: card = sanitized['card_number'] sanitized['card_number'] = f"{card[:6]}{'*' * (len(card) - 10)}{card[-4:]}" # Remove prohibited data for field in self.prohibited_fields: sanitized.pop(field, None) return sanitized def validate_no_prohibited_storage(self, data): """Ensure no prohibited data is being stored.""" for field in self.prohibited_fields: if field in data: raise SecurityError(f"Attempting to store prohibited field: {field}") ``` ## Tokenization ### Using Payment Processor Tokens ```python import stripe class TokenizedPayment: """Handle payments using tokens (no card data on server).""" @staticmethod def create_payment_method_token(card_details): """Create token from card details (client-side only).""" # THIS SHOULD ONLY BE DONE CLIENT-SIDE WITH STRIPE.JS # NEVER send card details to your server """ // Frontend JavaScript const stripe = Stripe('pk_...'); const {token, error} = await stripe.createToken({ card: { number: '4242424242424242', exp_month: 12, exp_year: 2024, cvc: '123' } }); // Send token.id to server (NOT card details) """ pass @staticmethod def charge_with_token(token_id, amount): """Charge using token (server-side).""" # Your server only sees the token, never the card number stripe.api_key = "sk_..." charge = stripe.Charge.create( amount=amount, currency="usd", source=token_id, # Token instead of card details description="Payment" ) return charge @staticmethod def store_payment_method(customer_id, payment_method_token): """Store payment method as token for future use.""" stripe.Customer.modify( customer_id, source=payment_method_token ) # Store only customer_id and payment_method_id in your database # NEVER store actual card details return { 'customer_id': customer_id, 'has_payment_method': True # DO NOT store: card number, CVV, etc. } ``` ### Custom Tokenization (Advanced) ```python import secrets from cryptography.fernet import Fernet class TokenVault: """Secure token vault for card data (if you must store it).""" def __init__(self, encryption_key): self.cipher = Fernet(encryption_key) self.vault = {} # In production: use encrypted database def tokenize(self, card_data): """Convert card data to token.""" # Generate secure random token token = secrets.token_urlsafe(32) # Encrypt card data encrypted = self.cipher.encrypt(json.dumps(card_data).encode()) # Store token -> encrypted data mapping self.vault[token] = encrypted return token def detokenize(self, token): """Retrieve card data from token.""" encrypted = self.vault.get(token) if not encrypted: raise ValueError("Token not found") # Decrypt decrypted = self.cipher.decrypt(encrypted) return json.loads(decrypted.decode()) def delete_token(self, token): """Remove token from vault.""" self.vault.pop(token, None) ``` ## Encryption ### Data at Rest ```python from cryptography.hazmat.primitives.ciphers.aead import AESGCM import os class EncryptedStorage: """Encrypt data at rest using AES-256-GCM.""" def __init__(self, encryption_key): """Initialize with 256-bit key.""" self.key = encryption_key # Must be 32 bytes def encrypt(self, plaintext): """Encrypt data.""" # Generate random nonce nonce = os.urandom(12) # Encrypt aesgcm = AESGCM(self.key) ciphertext = aesgcm.encrypt(nonce, plaintext.encode(), None) # Return nonce + ciphertext return nonce + ciphertext def decrypt(self, encrypted_data): """Decrypt data.""" # Extract nonce and ciphertext nonce = encrypted_data[:12] ciphertext = encrypted_data[12:] # Decrypt aesgcm = AESGCM(self.key) plaintext = aesgcm.decrypt(nonce, ciphertext, None) return plaintext.decode() # Usage storage = EncryptedStorage(os.urandom(32)) encrypted_pan = storage.encrypt("4242424242424242") # Store encrypted_pan in database ``` ### Data in Transit ```python # Always use TLS 1.2 or higher # Flask/Django example app.config['SESSION_COOKIE_SECURE'] = True # HTTPS only app.config['SESSION_COOKIE_HTTPONLY'] = True app.config['SESSION_COOKIE_SAMESITE'] = 'Strict' # Enforce HTTPS from flask_talisman import Talisman Talisman(app, force_https=True) ``` ## Access Control ```python from functools import wraps from flask import session def require_pci_access(f): """Decorator to restrict access to cardholder data.""" @wraps(f) def decorated_function(*args, **kwargs): user = session.get('user') # Check if user has PCI access role if not user or 'pci_access' not in user.get('roles', []): return {'error': 'Unauthorized access to cardholder data'}, 403 # Log access attempt audit_log( user=user['id'], action='access_cardholder_data', resource=f.__name__ ) return f(*args, **kwargs) return decorated_function @app.route('/api/payment-methods') @require_pci_access def get_payment_methods(): """Retrieve payment methods (restricted access).""" # Only accessible to users with pci_access role pass ``` ## Audit Logging ```python import logging from datetime import datetime class PCIAuditLogger: """PCI-compliant audit logging.""" def __init__(self): self.logger = logging.getLogger('pci_audit') # Configure to write to secure, append-only log def log_access(self, user_id, resource, action, result): """Log access to cardholder data.""" entry = { 'timestamp': datetime.utcnow().isoformat(), 'user_id': user_id, 'resource': resource, 'action': action, 'result': result, 'ip_address': request.remote_addr } self.logger.info(json.dumps(entry)) def log_authentication(self, user_id, success, method): """Log authentication attempt.""" entry = { 'timestamp': datetime.utcnow().isoformat(), 'user_id': user_id, 'event': 'authentication', 'success': success, 'method': method, 'ip_address': request.remote_addr } self.logger.info(json.dumps(entry)) # Usage audit = PCIAuditLogger() audit.log_access(user_id=123, resource='payment_methods', action='read', result='success') ``` ## Security Best Practices ### Input Validation ```python import re def validate_card_number(card_number): """Validate card number format (Luhn algorithm).""" # Remove spaces and dashes card_number = re.sub(r'[\s-]', '', card_number) # Check if all digits if not card_number.isdigit(): return False # Luhn algorithm def luhn_checksum(card_num): def digits_of(n): return [int(d) for d in str(n)] digits = digits_of(card_num) odd_digits = digits[-1::-2] even_digits = digits[-2::-2] checksum = sum(odd_digits) for d in even_digits: checksum += sum(digits_of(d * 2)) return checksum % 10 return luhn_checksum(card_number) == 0 def sanitize_input(user_input): """Sanitize user input to prevent injection.""" # Remove special characters # Validate against expected format # Escape for database queries pass ``` ## PCI DSS SAQ (Self-Assessment Questionnaire) ### SAQ A (Least Requirements) - E-commerce using hosted payment page - No card data on your systems - ~20 questions ### SAQ A-EP - E-commerce with embedded payment form - Uses JavaScript to handle card data - ~180 questions ### SAQ D (Most Requirements) - Store, process, or transmit card data - Full PCI DSS requirements - ~300 questions ## Compliance Checklist ```python PCI_COMPLIANCE_CHECKLIST = { 'network_security': [ 'Firewall configured and maintained', 'No vendor default passwords', 'Network segmentation implemented' ], 'data_protection': [ 'No storage of CVV, track data, or PIN', 'PAN encrypted when stored', 'PAN masked when displayed', 'Encryption keys properly managed' ], 'vulnerability_management': [ 'Anti-virus installed and updated', 'Secure development practices', 'Regular security patches', 'Vulnerability scanning performed' ], 'access_control': [ 'Access restricted by role', 'Unique IDs for all users', 'Multi-factor authentication', 'Physical security measures' ], 'monitoring': [ 'Audit logs enabled', 'Log review process', 'File integrity monitoring', 'Regular security testing' ], 'policy': [ 'Security policy documented', 'Risk assessment performed', 'Security awareness training', 'Incident response plan' ] } ``` ## Resources - **references/data-minimization.md**: Never store prohibited data - **references/tokenization.md**: Tokenization strategies - **references/encryption.md**: Encryption requirements - **references/access-control.md**: Role-based access - **references/audit-logging.md**: Comprehensive logging - **assets/pci-compliance-checklist.md**: Complete checklist - **assets/encrypted-storage.py**: Encryption utilities - **scripts/audit-payment-system.sh**: Compliance audit script ## Common Violations 1. **Storing CVV**: Never store card verification codes 2. **Unencrypted PAN**: Card numbers must be encrypted at rest 3. **Weak Encryption**: Use AES-256 or equivalent 4. **No Access Controls**: Restrict who can access cardholder data 5. **Missing Audit Logs**: Must log all access to payment data 6. **Insecure Transmission**: Always use TLS 1.2+ 7. **Default Passwords**: Change all default credentials 8. **No Security Testing**: Regular penetration testing required ## Reducing PCI Scope 1. **Use Hosted Payments**: Stripe Checkout, PayPal, etc. 2. **Tokenization**: Replace card data with tokens 3. **Network Segmentation**: Isolate cardholder data environment 4. **Outsource**: Use PCI-compliant payment processors 5. **No Storage**: Never store full card details By minimizing systems that touch card data, you reduce compliance burden significantly.
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

stripe-integration

Implement Stripe payment processing for robust, PCI-compliant

business
⭐1
# Stripe Integration Master Stripe payment processing integration for robust, PCI-compliant payment flows including checkout, subscriptions, webhooks, and refunds. ## When to Use This Skill - Implementing payment processing in web/mobile applications - Setting up subscription billing systems - Handling one-time payments and recurring charges - Processing refunds and disputes - Managing customer payment methods - Implementing SCA (Strong Customer Authentication) for European payments - Building marketplace payment flows with Stripe Connect ## Core Concepts ### 1. Payment Flows **Checkout Sessions** - Recommended for most integrations - Supports all UI paths: - Stripe-hosted checkout page - Embedded checkout form - Custom UI with Elements (Payment Element, Express Checkout Element) using `ui_mode='custom'` - Provides built-in checkout capabilities (line items, discounts, tax, shipping, address collection, saved payment methods, and checkout lifecycle events) - Lower integration and maintenance burden than Payment Intents **Payment Intents (Bespoke control)** - You calculate the final amount with taxes, discounts, subscriptions, and currency conversion yourself. - More complex implementation and long-term maintenance burden - Requires Stripe.js for PCI compliance **Setup Intents (Save Payment Methods)** - Collect payment method without charging - Used for subscriptions and future payments - Requires customer confirmation ### 2. Webhooks **Critical Events:** - `payment_intent.succeeded`: Payment completed - `payment_intent.payment_failed`: Payment failed - `customer.subscription.updated`: Subscription changed - `customer.subscription.deleted`: Subscription canceled - `charge.refunded`: Refund processed - `invoice.payment_succeeded`: Subscription payment successful ### 3. Subscriptions **Components:** - **Product**: What you're selling - **Price**: How much and how often - **Subscription**: Customer's recurring payment - **Invoice**: Generated for each billing cycle ### 4. Customer Management - Create and manage customer records - Store multiple payment methods - Track customer metadata - Manage billing details ## Quick Start ```python import stripe stripe.api_key = "sk_test_..." # Create a checkout session session = stripe.checkout.Session.create( line_items=[{ 'price_data': { 'currency': 'usd', 'product_data': { 'name': 'Premium Subscription', }, 'unit_amount': 2000, # $20.00 'recurring': { 'interval': 'month', }, }, 'quantity': 1, }], mode='subscription', success_url='https://yourdomain.com/success?session_id={CHECKOUT_SESSION_ID}', cancel_url='https://yourdomain.com/cancel' ) # Redirect user to session.url print(session.url) ``` ## Payment Implementation Patterns ### Pattern 1: One-Time Payment (Hosted Checkout) ```python def create_checkout_session(amount, currency='usd'): """Create a one-time payment checkout session.""" try: session = stripe.checkout.Session.create( line_items=[{ 'price_data': { 'currency': currency, 'product_data': { 'name': 'Blue T-shirt', 'images': ['https://example.com/product.jpg'], }, 'unit_amount': amount, # Amount in cents }, 'quantity': 1, }], mode='payment', success_url='https://yourdomain.com/success?session_id={CHECKOUT_SESSION_ID}', cancel_url='https://yourdomain.com/cancel', metadata={ 'order_id': 'order_123', 'user_id': 'user_456' } ) return session except stripe.error.StripeError as e: # Handle error print(f"Stripe error: {e.user_message}") raise ``` ### Pattern 2: Elements with Checkout Sessions ```python def create_checkout_session_for_elements(amount, currency='usd'): """Create a checkout session configured for Payment Element.""" session = stripe.checkout.Session.create( mode='payment', ui_mode='custom', line_items=[{ 'price_data': { 'currency': currency, 'product_data': {'name': 'Blue T-shirt'}, 'unit_amount': amount, }, 'quantity': 1, }], return_url='https://yourdomain.com/complete?session_id={CHECKOUT_SESSION_ID}' ) return session.client_secret # Send to frontend ``` ```javascript const stripe = Stripe("pk_test_..."); const appearance = { theme: "stripe" }; const checkout = stripe.initCheckout({ clientSecret, elementsOptions: { appearance }, }); const loadActionsResult = await checkout.loadActions(); if (loadActionsResult.type === "success") { const { actions } = loadActionsResult; const session = actions.getSession(); const button = document.getElementById("pay-button"); const checkoutContainer = document.getElementById("checkout-container"); const emailInput = document.getElementById("email"); const emailErrors = document.getElementById("email-errors"); const errors = document.getElementById("confirm-errors"); // Display a formatted string representing the total amount checkoutContainer.append(`Total: ${session.total.total.amount}`); // Mount Payment Element const paymentElement = checkout.createPaymentElement(); paymentElement.mount("#payment-element"); // Store email for submission emailInput.addEventListener("blur", () => { actions.updateEmail(emailInput.value).then((result) => { if (result.error) emailErrors.textContent = result.error.message; }); }); // Handle form submission button.addEventListener("click", () => { actions.confirm().then((result) => { if (result.type === "error") errors.textContent = result.error.message; }); }); } ``` ### Pattern 3: Elements with Payment Intents Pattern 2 (Elements with Checkout Sessions) is Stripe's recommended approach, but you can also use Payment Intents as an alternative. ```python def create_payment_intent(amount, currency='usd', customer_id=None): """Create a payment intent for bespoke checkout UI with Payment Element.""" intent = stripe.PaymentIntent.create( amount=amount, currency=currency, customer=customer_id, automatic_payment_methods={ 'enabled': True, }, metadata={ 'integration_check': 'accept_a_payment' } ) return intent.client_secret # Send to frontend ``` ```javascript // Mount Payment Element and confirm via Payment Intents const stripe = Stripe("pk_test_..."); const appearance = { theme: "stripe" }; const elements = stripe.elements({ appearance, clientSecret }); const paymentElement = elements.create("payment"); paymentElement.mount("#payment-element"); document.getElementById("pay-button").addEventListener("click", async () => { const { error } = await stripe.confirmPayment({ elements, confirmParams: { return_url: "https://yourdomain.com/complete", }, }); if (error) { document.getElementById("errors").textContent = error.message; } }); ``` ### Pattern 4: Subscription Creation ```python def create_subscription(customer_id, price_id): """Create a subscription for a customer.""" try: subscription = stripe.Subscription.create( customer=customer_id, items=[{'price': price_id}], payment_behavior='default_incomplete', payment_settings={'save_default_payment_method': 'on_subscription'}, expand=['latest_invoice.payment_intent'], ) return { 'subscription_id': subscription.id, 'client_secret': subscription.latest_invoice.payment_intent.client_secret } except stripe.error.StripeError as e: print(f"Subscription creation failed: {e}") raise ``` ### Pattern 5: Customer Portal ```python def create_customer_portal_session(customer_id): """Create a portal session for customers to manage subscriptions.""" session = stripe.billing_portal.Session.create( customer=customer_id, return_url='https://yourdomain.com/account', ) return session.url # Redirect customer here ``` ## Webhook Handling ### Secure Webhook Endpoint ```python from flask import Flask, request import stripe app = Flask(__name__) endpoint_secret = 'whsec_...' @app.route('/webhook', methods=['POST']) def webhook(): payload = request.data sig_header = request.headers.get('Stripe-Signature') try: event = stripe.Webhook.construct_event( payload, sig_header, endpoint_secret ) except ValueError: # Invalid payload return 'Invalid payload', 400 except stripe.error.SignatureVerificationError: # Invalid signature return 'Invalid signature', 400 # Handle the event if event['type'] == 'payment_intent.succeeded': payment_intent = event['data']['object'] handle_successful_payment(payment_intent) elif event['type'] == 'payment_intent.payment_failed': payment_intent = event['data']['object'] handle_failed_payment(payment_intent) elif event['type'] == 'customer.subscription.deleted': subscription = event['data']['object'] handle_subscription_canceled(subscription) return 'Success', 200 def handle_successful_payment(payment_intent): """Process successful payment.""" customer_id = payment_intent.get('customer') amount = payment_intent['amount'] metadata = payment_intent.get('metadata', {}) # Update your database # Send confirmation email # Fulfill order print(f"Payment succeeded: {payment_intent['id']}") def handle_failed_payment(payment_intent): """Handle failed payment.""" error = payment_intent.get('last_payment_error', {}) print(f"Payment failed: {error.get('message')}") # Notify customer # Update order status def handle_subscription_canceled(subscription): """Handle subscription cancellation.""" customer_id = subscription['customer'] # Update user access # Send cancellation email print(f"Subscription canceled: {subscription['id']}") ``` ### Webhook Best Practices ```python import hashlib import hmac def verify_webhook_signature(payload, signature, secret): """Manually verify webhook signature.""" expected_sig = hmac.new( secret.encode('utf-8'), payload, hashlib.sha256 ).hexdigest() return hmac.compare_digest(signature, expected_sig) def handle_webhook_idempotently(event_id, handler): """Ensure webhook is processed exactly once.""" # Check if event already processed if is_event_processed(event_id): return # Process event try: handler() mark_event_processed(event_id) except Exception as e: log_error(e) # Stripe will retry failed webhooks raise ``` ## Customer Management ```python def create_customer(email, name, payment_method_id=None): """Create a Stripe customer.""" customer = stripe.Customer.create( email=email, name=name, payment_method=payment_method_id, invoice_settings={ 'default_payment_method': payment_method_id } if payment_method_id else None, metadata={ 'user_id': '12345' } ) return customer def attach_payment_method(customer_id, payment_method_id): """Attach a payment method to a customer.""" stripe.PaymentMethod.attach( payment_method_id, customer=customer_id ) # Set as default stripe.Customer.modify( customer_id, invoice_settings={ 'default_payment_method': payment_method_id } ) def list_customer_payment_methods(customer_id): """List all payment methods for a customer.""" payment_methods = stripe.PaymentMethod.list( customer=customer_id, type='card' ) return payment_methods.data ``` ## Refund Handling ```python def create_refund(payment_intent_id, amount=None, reason=None): """Create a refund.""" refund_params = { 'payment_intent': payment_intent_id } if amount: refund_params['amount'] = amount # Partial refund if reason: refund_params['reason'] = reason # 'duplicate', 'fraudulent', 'requested_by_customer' refund = stripe.Refund.create(**refund_params) return refund def handle_dispute(charge_id, evidence): """Update dispute with evidence.""" stripe.Dispute.modify( charge_id, evidence={ 'customer_name': evidence.get('customer_name'), 'customer_email_address': evidence.get('customer_email'), 'shipping_documentation': evidence.get('shipping_proof'), 'customer_communication': evidence.get('communication'), } ) ``` ## Testing ```python # Use test mode keys stripe.api_key = "sk_test_..." # Test card numbers TEST_CARDS = { 'success': '4242424242424242', 'declined': '4000000000000002', '3d_secure': '4000002500003155', 'insufficient_funds': '4000000000009995' } def test_payment_flow(): """Test complete payment flow.""" # Create test customer customer = stripe.Customer.create( email="test@example.com" ) # Create payment intent intent = stripe.PaymentIntent.create( amount=1000, automatic_payment_methods={ 'enabled': True }, currency='usd', customer=customer.id ) # Confirm with test card confirmed = stripe.PaymentIntent.confirm( intent.id, payment_method='pm_card_visa' # Test payment method ) assert confirmed.status == 'succeeded' ``` ## Resources - **references/checkout-flows.md**: Detailed checkout implementation - **references/webhook-handling.md**: Webhook security and processing - **references/subscription-management.md**: Subscription lifecycle - **references/customer-management.md**: Customer and payment method handling - **references/invoice-generation.md**: Invoicing and billing - **assets/stripe-client.py**: Production-ready Stripe client wrapper - **assets/webhook-handler.py**: Complete webhook processor - **assets/checkout-config.json**: Checkout configuration templates ## Best Practices 1. **Always Use Webhooks**: Don't rely solely on client-side confirmation 2. **Idempotency**: Handle webhook events idempotently 3. **Error Handling**: Gracefully handle all Stripe errors 4. **Test Mode**: Thoroughly test with test keys before production 5. **Metadata**: Use metadata to link Stripe objects to your database 6. **Monitoring**: Track payment success rates and errors 7. **PCI Compliance**: Never handle raw card data on your server 8. **SCA Ready**: Implement 3D Secure for European payments ## Common Pitfalls - **Not Verifying Webhooks**: Always verify webhook signatures - **Missing Webhook Events**: Handle all relevant webhook events - **Hardcoded Amounts**: Use cents/smallest currency unit - **No Retry Logic**: Implement retries for API calls - **Ignoring Test Mode**: Test all edge cases with test cards
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

python-observability

Python observability patterns including structured logging,

coding
⭐1
# Python Observability Instrument Python applications with structured logs, metrics, and traces. When something breaks in production, you need to answer "what, where, and why" without deploying new code. ## When to Use This Skill - Adding structured logging to applications - Implementing metrics collection with Prometheus - Setting up distributed tracing across services - Propagating correlation IDs through request chains - Debugging production issues - Building observability dashboards ## Core Concepts ### 1. Structured Logging Emit logs as JSON with consistent fields for production environments. Machine-readable logs enable powerful queries and alerts. For local development, consider human-readable formats. ### 2. The Four Golden Signals Track latency, traffic, errors, and saturation for every service boundary. ### 3. Correlation IDs Thread a unique ID through all logs and spans for a single request, enabling end-to-end tracing. ### 4. Bounded Cardinality Keep metric label values bounded. Unbounded labels (like user IDs) explode storage costs. ## Quick Start ```python import structlog structlog.configure( processors=[ structlog.processors.TimeStamper(fmt="iso"), structlog.processors.JSONRenderer(), ], ) logger = structlog.get_logger() logger.info("Request processed", user_id="123", duration_ms=45) ``` ## Fundamental Patterns ### Pattern 1: Structured Logging with Structlog Configure structlog for JSON output with consistent fields. ```python import logging import structlog def configure_logging(log_level: str = "INFO") -> None: """Configure structured logging for the application.""" structlog.configure( processors=[ structlog.contextvars.merge_contextvars, structlog.processors.add_log_level, structlog.processors.TimeStamper(fmt="iso"), structlog.processors.StackInfoRenderer(), structlog.processors.format_exc_info, structlog.processors.JSONRenderer(), ], wrapper_class=structlog.make_filtering_bound_logger( getattr(logging, log_level.upper()) ), context_class=dict, logger_factory=structlog.PrintLoggerFactory(), cache_logger_on_first_use=True, ) # Initialize at application startup configure_logging("INFO") logger = structlog.get_logger() ``` ### Pattern 2: Consistent Log Fields Every log entry should include standard fields for filtering and correlation. ```python import structlog from contextvars import ContextVar # Store correlation ID in context correlation_id: ContextVar[str] = ContextVar("correlation_id", default="") logger = structlog.get_logger() def process_request(request: Request) -> Response: """Process request with structured logging.""" logger.info( "Request received", correlation_id=correlation_id.get(), method=request.method, path=request.path, user_id=request.user_id, ) try: result = handle_request(request) logger.info( "Request completed", correlation_id=correlation_id.get(), status_code=200, duration_ms=elapsed, ) return result except Exception as e: logger.error( "Request failed", correlation_id=correlation_id.get(), error_type=type(e).__name__, error_message=str(e), ) raise ``` ### Pattern 3: Semantic Log Levels Use log levels consistently across the application. | Level | Purpose | Examples | |-------|---------|----------| | `DEBUG` | Development diagnostics | Variable values, internal state | | `INFO` | Request lifecycle, operations | Request start/end, job completion | | `WARNING` | Recoverable anomalies | Retry attempts, fallback used | | `ERROR` | Failures needing attention | Exceptions, service unavailable | ```python # DEBUG: Detailed internal information logger.debug("Cache lookup", key=cache_key, hit=cache_hit) # INFO: Normal operational events logger.info("Order created", order_id=order.id, total=order.total) # WARNING: Abnormal but handled situations logger.warning( "Rate limit approaching", current_rate=950, limit=1000, reset_seconds=30, ) # ERROR: Failures requiring investigation logger.error( "Payment processing failed", order_id=order.id, error=str(e), payment_provider="stripe", ) ``` Never log expected behavior at `ERROR`. A user entering a wrong password is `INFO`, not `ERROR`. ### Pattern 4: Correlation ID Propagation Generate a unique ID at ingress and thread it through all operations. ```python from contextvars import ContextVar import uuid import structlog correlation_id: ContextVar[str] = ContextVar("correlation_id", default="") def set_correlation_id(cid: str | None = None) -> str: """Set correlation ID for current context.""" cid = cid or str(uuid.uuid4()) correlation_id.set(cid) structlog.contextvars.bind_contextvars(correlation_id=cid) return cid # FastAPI middleware example from fastapi import Request async def correlation_middleware(request: Request, call_next): """Middleware to set and propagate correlation ID.""" # Use incoming header or generate new cid = request.headers.get("X-Correlation-ID") or str(uuid.uuid4()) set_correlation_id(cid) response = await call_next(request) response.headers["X-Correlation-ID"] = cid return response ``` Propagate to outbound requests: ```python import httpx async def call_downstream_service(endpoint: str, data: dict) -> dict: """Call downstream service with correlation ID.""" async with httpx.AsyncClient() as client: response = await client.post( endpoint, json=data, headers={"X-Correlation-ID": correlation_id.get()}, ) return response.json() ``` ## Advanced Patterns ### Pattern 5: The Four Golden Signals with Prometheus Track these metrics for every service boundary: ```python from prometheus_client import Counter, Histogram, Gauge # Latency: How long requests take REQUEST_LATENCY = Histogram( "http_request_duration_seconds", "Request latency in seconds", ["method", "endpoint", "status"], buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10], ) # Traffic: Request rate REQUEST_COUNT = Counter( "http_requests_total", "Total HTTP requests", ["method", "endpoint", "status"], ) # Errors: Error rate ERROR_COUNT = Counter( "http_errors_total", "Total HTTP errors", ["method", "endpoint", "error_type"], ) # Saturation: Resource utilization DB_POOL_USAGE = Gauge( "db_connection_pool_used", "Number of database connections in use", ) ``` Instrument your endpoints: ```python import time from functools import wraps def track_request(func): """Decorator to track request metrics.""" @wraps(func) async def wrapper(request: Request, *args, **kwargs): method = request.method endpoint = request.url.path start = time.perf_counter() try: response = await func(request, *args, **kwargs) status = str(response.status_code) return response except Exception as e: status = "500" ERROR_COUNT.labels( method=method, endpoint=endpoint, error_type=type(e).__name__, ).inc() raise finally: duration = time.perf_counter() - start REQUEST_COUNT.labels(method=method, endpoint=endpoint, status=status).inc() REQUEST_LATENCY.labels(method=method, endpoint=endpoint, status=status).observe(duration) return wrapper ``` ### Pattern 6: Bounded Cardinality Avoid labels with unbounded values to prevent metric explosion. ```python # BAD: User ID has potentially millions of values REQUEST_COUNT.labels(method="GET", user_id=user.id) # Don't do this! # GOOD: Bounded values only REQUEST_COUNT.labels(method="GET", endpoint="/users", status="200") # If you need per-user metrics, use a different approach: # - Log the user_id and query logs # - Use a separate analytics system # - Bucket users by type/tier REQUEST_COUNT.labels( method="GET", endpoint="/users", user_tier="premium", # Bounded set of values ) ``` ### Pattern 7: Timed Operations with Context Manager Create a reusable timing context manager for operations. ```python from contextlib import contextmanager import time import structlog logger = structlog.get_logger() @contextmanager def timed_operation(name: str, **extra_fields): """Context manager for timing and logging operations.""" start = time.perf_counter() logger.debug("Operation started", operation=name, **extra_fields) try: yield except Exception as e: elapsed_ms = (time.perf_counter() - start) * 1000 logger.error( "Operation failed", operation=name, duration_ms=round(elapsed_ms, 2), error=str(e), **extra_fields, ) raise else: elapsed_ms = (time.perf_counter() - start) * 1000 logger.info( "Operation completed", operation=name, duration_ms=round(elapsed_ms, 2), **extra_fields, ) # Usage with timed_operation("fetch_user_orders", user_id=user.id): orders = await order_repository.get_by_user(user.id) ``` ### Pattern 8: OpenTelemetry Tracing Set up distributed tracing with OpenTelemetry. **Note:** OpenTelemetry is actively evolving. Check the [official Python documentation](https://opentelemetry.io/docs/languages/python/) for the latest API patterns and best practices. ```python from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter def configure_tracing(service_name: str, otlp_endpoint: str) -> None: """Configure OpenTelemetry tracing.""" provider = TracerProvider() processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=otlp_endpoint)) provider.add_span_processor(processor) trace.set_tracer_provider(provider) tracer = trace.get_tracer(__name__) async def process_order(order_id: str) -> Order: """Process order with tracing.""" with tracer.start_as_current_span("process_order") as span: span.set_attribute("order.id", order_id) with tracer.start_as_current_span("validate_order"): validate_order(order_id) with tracer.start_as_current_span("charge_payment"): charge_payment(order_id) with tracer.start_as_current_span("send_confirmation"): send_confirmation(order_id) return order ``` ## Best Practices Summary 1. **Use structured logging** - JSON logs with consistent fields 2. **Propagate correlation IDs** - Thread through all requests and logs 3. **Track the four golden signals** - Latency, traffic, errors, saturation 4. **Bound label cardinality** - Never use unbounded values as metric labels 5. **Log at appropriate levels** - Don't cry wolf with ERROR 6. **Include context** - User ID, request ID, operation name in logs 7. **Use context managers** - Consistent timing and error handling 8. **Separate concerns** - Observability code shouldn't pollute business logic 9. **Test your observability** - Verify logs and metrics in integration tests 10. **Set up alerts** - Metrics are useless without alerting
👍0
👁️0
🤖 Auto-discovered