Skip to main content
EVOKORE// BROWSE
>

./browse/prompts

29 NODES
šŸ¤–system prompt•7 months ago

fastapi-templates

Create production-ready FastAPI projects with async patterns,

coding
⭐1
# FastAPI Project Templates Production-ready FastAPI project structures with async patterns, dependency injection, middleware, and best practices for building high-performance APIs. ## When to Use This Skill - Starting new FastAPI projects from scratch - Implementing async REST APIs with Python - Building high-performance web services and microservices - Creating async applications with PostgreSQL, MongoDB - Setting up API projects with proper structure and testing ## Core Concepts ### 1. Project Structure **Recommended Layout:** ``` app/ ā”œā”€ā”€ api/ # API routes │ ā”œā”€ā”€ v1/ │ │ ā”œā”€ā”€ endpoints/ │ │ │ ā”œā”€ā”€ users.py │ │ │ ā”œā”€ā”€ auth.py │ │ │ └── items.py │ │ └── router.py │ └── dependencies.py # Shared dependencies ā”œā”€ā”€ core/ # Core configuration │ ā”œā”€ā”€ config.py │ ā”œā”€ā”€ security.py │ └── database.py ā”œā”€ā”€ models/ # Database models │ ā”œā”€ā”€ user.py │ └── item.py ā”œā”€ā”€ schemas/ # Pydantic schemas │ ā”œā”€ā”€ user.py │ └── item.py ā”œā”€ā”€ services/ # Business logic │ ā”œā”€ā”€ user_service.py │ └── auth_service.py ā”œā”€ā”€ repositories/ # Data access │ ā”œā”€ā”€ user_repository.py │ └── item_repository.py └── main.py # Application entry ``` ### 2. Dependency Injection FastAPI's built-in DI system using `Depends`: - Database session management - Authentication/authorization - Shared business logic - Configuration injection ### 3. Async Patterns Proper async/await usage: - Async route handlers - Async database operations - Async background tasks - Async middleware ## Implementation Patterns ### Pattern 1: Complete FastAPI Application ```python # main.py from fastapi import FastAPI, Depends from fastapi.middleware.cors import CORSMiddleware from contextlib import asynccontextmanager @asynccontextmanager async def lifespan(app: FastAPI): """Application lifespan events.""" # Startup await database.connect() yield # Shutdown await database.disconnect() app = FastAPI( title="API Template", version="1.0.0", lifespan=lifespan ) # CORS middleware app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Include routers from app.api.v1.router import api_router app.include_router(api_router, prefix="/api/v1") # core/config.py from pydantic_settings import BaseSettings from functools import lru_cache class Settings(BaseSettings): """Application settings.""" DATABASE_URL: str SECRET_KEY: str ACCESS_TOKEN_EXPIRE_MINUTES: int = 30 API_V1_STR: str = "/api/v1" class Config: env_file = ".env" @lru_cache() def get_settings() -> Settings: return Settings() # core/database.py from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import sessionmaker from app.core.config import get_settings settings = get_settings() engine = create_async_engine( settings.DATABASE_URL, echo=True, future=True ) AsyncSessionLocal = sessionmaker( engine, class_=AsyncSession, expire_on_commit=False ) Base = declarative_base() async def get_db() -> AsyncSession: """Dependency for database session.""" async with AsyncSessionLocal() as session: try: yield session await session.commit() except Exception: await session.rollback() raise finally: await session.close() ``` ### Pattern 2: CRUD Repository Pattern ```python # repositories/base_repository.py from typing import Generic, TypeVar, Type, Optional, List from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy import select from pydantic import BaseModel ModelType = TypeVar("ModelType") CreateSchemaType = TypeVar("CreateSchemaType", bound=BaseModel) UpdateSchemaType = TypeVar("UpdateSchemaType", bound=BaseModel) class BaseRepository(Generic[ModelType, CreateSchemaType, UpdateSchemaType]): """Base repository for CRUD operations.""" def __init__(self, model: Type[ModelType]): self.model = model async def get(self, db: AsyncSession, id: int) -> Optional[ModelType]: """Get by ID.""" result = await db.execute( select(self.model).where(self.model.id == id) ) return result.scalars().first() async def get_multi( self, db: AsyncSession, skip: int = 0, limit: int = 100 ) -> List[ModelType]: """Get multiple records.""" result = await db.execute( select(self.model).offset(skip).limit(limit) ) return result.scalars().all() async def create( self, db: AsyncSession, obj_in: CreateSchemaType ) -> ModelType: """Create new record.""" db_obj = self.model(**obj_in.dict()) db.add(db_obj) await db.flush() await db.refresh(db_obj) return db_obj async def update( self, db: AsyncSession, db_obj: ModelType, obj_in: UpdateSchemaType ) -> ModelType: """Update record.""" update_data = obj_in.dict(exclude_unset=True) for field, value in update_data.items(): setattr(db_obj, field, value) await db.flush() await db.refresh(db_obj) return db_obj async def delete(self, db: AsyncSession, id: int) -> bool: """Delete record.""" obj = await self.get(db, id) if obj: await db.delete(obj) return True return False # repositories/user_repository.py from app.repositories.base_repository import BaseRepository from app.models.user import User from app.schemas.user import UserCreate, UserUpdate class UserRepository(BaseRepository[User, UserCreate, UserUpdate]): """User-specific repository.""" async def get_by_email(self, db: AsyncSession, email: str) -> Optional[User]: """Get user by email.""" result = await db.execute( select(User).where(User.email == email) ) return result.scalars().first() async def is_active(self, db: AsyncSession, user_id: int) -> bool: """Check if user is active.""" user = await self.get(db, user_id) return user.is_active if user else False user_repository = UserRepository(User) ``` ### Pattern 3: Service Layer ```python # services/user_service.py from typing import Optional from sqlalchemy.ext.asyncio import AsyncSession from app.repositories.user_repository import user_repository from app.schemas.user import UserCreate, UserUpdate, User from app.core.security import get_password_hash, verify_password class UserService: """Business logic for users.""" def __init__(self): self.repository = user_repository async def create_user( self, db: AsyncSession, user_in: UserCreate ) -> User: """Create new user with hashed password.""" # Check if email exists existing = await self.repository.get_by_email(db, user_in.email) if existing: raise ValueError("Email already registered") # Hash password user_in_dict = user_in.dict() user_in_dict["hashed_password"] = get_password_hash(user_in_dict.pop("password")) # Create user user = await self.repository.create(db, UserCreate(**user_in_dict)) return user async def authenticate( self, db: AsyncSession, email: str, password: str ) -> Optional[User]: """Authenticate user.""" user = await self.repository.get_by_email(db, email) if not user: return None if not verify_password(password, user.hashed_password): return None return user async def update_user( self, db: AsyncSession, user_id: int, user_in: UserUpdate ) -> Optional[User]: """Update user.""" user = await self.repository.get(db, user_id) if not user: return None if user_in.password: user_in_dict = user_in.dict(exclude_unset=True) user_in_dict["hashed_password"] = get_password_hash( user_in_dict.pop("password") ) user_in = UserUpdate(**user_in_dict) return await self.repository.update(db, user, user_in) user_service = UserService() ``` ### Pattern 4: API Endpoints with Dependencies ```python # api/v1/endpoints/users.py from fastapi import APIRouter, Depends, HTTPException, status from sqlalchemy.ext.asyncio import AsyncSession from typing import List from app.core.database import get_db from app.schemas.user import User, UserCreate, UserUpdate from app.services.user_service import user_service from app.api.dependencies import get_current_user router = APIRouter() @router.post("/", response_model=User, status_code=status.HTTP_201_CREATED) async def create_user( user_in: UserCreate, db: AsyncSession = Depends(get_db) ): """Create new user.""" try: user = await user_service.create_user(db, user_in) return user except ValueError as e: raise HTTPException(status_code=400, detail=str(e)) @router.get("/me", response_model=User) async def read_current_user( current_user: User = Depends(get_current_user) ): """Get current user.""" return current_user @router.get("/{user_id}", response_model=User) async def read_user( user_id: int, db: AsyncSession = Depends(get_db), current_user: User = Depends(get_current_user) ): """Get user by ID.""" user = await user_service.repository.get(db, user_id) if not user: raise HTTPException(status_code=404, detail="User not found") return user @router.patch("/{user_id}", response_model=User) async def update_user( user_id: int, user_in: UserUpdate, db: AsyncSession = Depends(get_db), current_user: User = Depends(get_current_user) ): """Update user.""" if current_user.id != user_id: raise HTTPException(status_code=403, detail="Not authorized") user = await user_service.update_user(db, user_id, user_in) if not user: raise HTTPException(status_code=404, detail="User not found") return user @router.delete("/{user_id}", status_code=status.HTTP_204_NO_CONTENT) async def delete_user( user_id: int, db: AsyncSession = Depends(get_db), current_user: User = Depends(get_current_user) ): """Delete user.""" if current_user.id != user_id: raise HTTPException(status_code=403, detail="Not authorized") deleted = await user_service.repository.delete(db, user_id) if not deleted: raise HTTPException(status_code=404, detail="User not found") ``` ### Pattern 5: Authentication & Authorization ```python # core/security.py from datetime import datetime, timedelta from typing import Optional from jose import JWTError, jwt from passlib.context import CryptContext from app.core.config import get_settings settings = get_settings() pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto") ALGORITHM = "HS256" def create_access_token(data: dict, expires_delta: Optional[timedelta] = None): """Create JWT access token.""" to_encode = data.copy() if expires_delta: expire = datetime.utcnow() + expires_delta else: expire = datetime.utcnow() + timedelta(minutes=15) to_encode.update({"exp": expire}) encoded_jwt = jwt.encode(to_encode, settings.SECRET_KEY, algorithm=ALGORITHM) return encoded_jwt def verify_password(plain_password: str, hashed_password: str) -> bool: """Verify password against hash.""" return pwd_context.verify(plain_password, hashed_password) def get_password_hash(password: str) -> str: """Hash password.""" return pwd_context.hash(password) # api/dependencies.py from fastapi import Depends, HTTPException, status from fastapi.security import OAuth2PasswordBearer from jose import JWTError, jwt from sqlalchemy.ext.asyncio import AsyncSession from app.core.database import get_db from app.core.security import ALGORITHM from app.core.config import get_settings from app.repositories.user_repository import user_repository oauth2_scheme = OAuth2PasswordBearer(tokenUrl=f"{settings.API_V1_STR}/auth/login") async def get_current_user( db: AsyncSession = Depends(get_db), token: str = Depends(oauth2_scheme) ): """Get current authenticated user.""" credentials_exception = HTTPException( status_code=status.HTTP_401_UNAUTHORIZED, detail="Could not validate credentials", headers={"WWW-Authenticate": "Bearer"}, ) try: payload = jwt.decode(token, settings.SECRET_KEY, algorithms=[ALGORITHM]) user_id: int = payload.get("sub") if user_id is None: raise credentials_exception except JWTError: raise credentials_exception user = await user_repository.get(db, user_id) if user is None: raise credentials_exception return user ``` ## Testing ```python # tests/conftest.py import pytest import asyncio from httpx import AsyncClient from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession from sqlalchemy.orm import sessionmaker from app.main import app from app.core.database import get_db, Base TEST_DATABASE_URL = "sqlite+aiosqlite:///:memory:" @pytest.fixture(scope="session") def event_loop(): loop = asyncio.get_event_loop_policy().new_event_loop() yield loop loop.close() @pytest.fixture async def db_session(): engine = create_async_engine(TEST_DATABASE_URL, echo=True) async with engine.begin() as conn: await conn.run_sync(Base.metadata.create_all) AsyncSessionLocal = sessionmaker( engine, class_=AsyncSession, expire_on_commit=False ) async with AsyncSessionLocal() as session: yield session @pytest.fixture async def client(db_session): async def override_get_db(): yield db_session app.dependency_overrides[get_db] = override_get_db async with AsyncClient(app=app, base_url="http://test") as client: yield client # tests/test_users.py import pytest @pytest.mark.asyncio async def test_create_user(client): response = await client.post( "/api/v1/users/", json={ "email": "test@example.com", "password": "testpass123", "name": "Test User" } ) assert response.status_code == 201 data = response.json() assert data["email"] == "test@example.com" assert "id" in data ``` ## Resources - **references/fastapi-architecture.md**: Detailed architecture guide - **references/async-best-practices.md**: Async/await patterns - **references/testing-strategies.md**: Comprehensive testing guide - **assets/project-template/**: Complete FastAPI project - **assets/docker-compose.yml**: Development environment setup ## Best Practices 1. **Async All The Way**: Use async for database, external APIs 2. **Dependency Injection**: Leverage FastAPI's DI system 3. **Repository Pattern**: Separate data access from business logic 4. **Service Layer**: Keep business logic out of routes 5. **Pydantic Schemas**: Strong typing for request/response 6. **Error Handling**: Consistent error responses 7. **Testing**: Test all layers independently ## Common Pitfalls - **Blocking Code in Async**: Using synchronous database drivers - **No Service Layer**: Business logic in route handlers - **Missing Type Hints**: Loses FastAPI's benefits - **Ignoring Sessions**: Not properly managing database sessions - **No Testing**: Skipping integration tests - **Tight Coupling**: Direct database access in routes
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

event-store-design

Design and implement event stores for event-sourced systems. Use

coding
⭐1
# Event Store Design Comprehensive guide to designing event stores for event-sourced applications. ## When to Use This Skill - Designing event sourcing infrastructure - Choosing between event store technologies - Implementing custom event stores - Optimizing event storage and retrieval - Setting up event store schemas - Planning for event store scaling ## Core Concepts ### 1. Event Store Architecture ``` ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ Event Store │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ │ │ Stream 1 │ │ Stream 2 │ │ Stream 3 │ │ │ │ (Aggregate) │ │ (Aggregate) │ │ (Aggregate) │ │ │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ │ │ Event 1 │ │ Event 1 │ │ Event 1 │ │ │ │ Event 2 │ │ Event 2 │ │ Event 2 │ │ │ │ Event 3 │ │ ... │ │ Event 3 │ │ │ │ ... │ │ │ │ Event 4 │ │ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ Global Position: 1 → 2 → 3 → 4 → 5 → 6 → ... │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ``` ### 2. Event Store Requirements | Requirement | Description | | ----------------- | ---------------------------------- | | **Append-only** | Events are immutable, only appends | | **Ordered** | Per-stream and global ordering | | **Versioned** | Optimistic concurrency control | | **Subscriptions** | Real-time event notifications | | **Idempotent** | Handle duplicate writes safely | ## Technology Comparison | Technology | Best For | Limitations | | ---------------- | ------------------------- | -------------------------------- | | **EventStoreDB** | Pure event sourcing | Single-purpose | | **PostgreSQL** | Existing Postgres stack | Manual implementation | | **Kafka** | High-throughput streaming | Not ideal for per-stream queries | | **DynamoDB** | Serverless, AWS-native | Query limitations | | **Marten** | .NET ecosystems | .NET specific | ## Templates ### Template 1: PostgreSQL Event Store Schema ```sql -- Events table CREATE TABLE events ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), stream_id VARCHAR(255) NOT NULL, stream_type VARCHAR(255) NOT NULL, event_type VARCHAR(255) NOT NULL, event_data JSONB NOT NULL, metadata JSONB DEFAULT '{}', version BIGINT NOT NULL, global_position BIGSERIAL, created_at TIMESTAMPTZ DEFAULT NOW(), CONSTRAINT unique_stream_version UNIQUE (stream_id, version) ); -- Index for stream queries CREATE INDEX idx_events_stream_id ON events(stream_id, version); -- Index for global subscription CREATE INDEX idx_events_global_position ON events(global_position); -- Index for event type queries CREATE INDEX idx_events_event_type ON events(event_type); -- Index for time-based queries CREATE INDEX idx_events_created_at ON events(created_at); -- Snapshots table CREATE TABLE snapshots ( stream_id VARCHAR(255) PRIMARY KEY, stream_type VARCHAR(255) NOT NULL, snapshot_data JSONB NOT NULL, version BIGINT NOT NULL, created_at TIMESTAMPTZ DEFAULT NOW() ); -- Subscriptions checkpoint table CREATE TABLE subscription_checkpoints ( subscription_id VARCHAR(255) PRIMARY KEY, last_position BIGINT NOT NULL DEFAULT 0, updated_at TIMESTAMPTZ DEFAULT NOW() ); ``` ### Template 2: Python Event Store Implementation ```python from dataclasses import dataclass, field from datetime import datetime from typing import Any, Optional, List from uuid import UUID, uuid4 import json import asyncpg @dataclass class Event: stream_id: str event_type: str data: dict metadata: dict = field(default_factory=dict) event_id: UUID = field(default_factory=uuid4) version: Optional[int] = None global_position: Optional[int] = None created_at: datetime = field(default_factory=datetime.utcnow) class EventStore: def __init__(self, pool: asyncpg.Pool): self.pool = pool async def append_events( self, stream_id: str, stream_type: str, events: List[Event], expected_version: Optional[int] = None ) -> List[Event]: """Append events to a stream with optimistic concurrency.""" async with self.pool.acquire() as conn: async with conn.transaction(): # Check expected version if expected_version is not None: current = await conn.fetchval( "SELECT MAX(version) FROM events WHERE stream_id = $1", stream_id ) current = current or 0 if current != expected_version: raise ConcurrencyError( f"Expected version {expected_version}, got {current}" ) # Get starting version start_version = await conn.fetchval( "SELECT COALESCE(MAX(version), 0) + 1 FROM events WHERE stream_id = $1", stream_id ) # Insert events saved_events = [] for i, event in enumerate(events): event.version = start_version + i row = await conn.fetchrow( """ INSERT INTO events (id, stream_id, stream_type, event_type, event_data, metadata, version, created_at) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING global_position """, event.event_id, stream_id, stream_type, event.event_type, json.dumps(event.data), json.dumps(event.metadata), event.version, event.created_at ) event.global_position = row['global_position'] saved_events.append(event) return saved_events async def read_stream( self, stream_id: str, from_version: int = 0, limit: int = 1000 ) -> List[Event]: """Read events from a stream.""" async with self.pool.acquire() as conn: rows = await conn.fetch( """ SELECT id, stream_id, event_type, event_data, metadata, version, global_position, created_at FROM events WHERE stream_id = $1 AND version >= $2 ORDER BY version LIMIT $3 """, stream_id, from_version, limit ) return [self._row_to_event(row) for row in rows] async def read_all( self, from_position: int = 0, limit: int = 1000 ) -> List[Event]: """Read all events globally.""" async with self.pool.acquire() as conn: rows = await conn.fetch( """ SELECT id, stream_id, event_type, event_data, metadata, version, global_position, created_at FROM events WHERE global_position > $1 ORDER BY global_position LIMIT $2 """, from_position, limit ) return [self._row_to_event(row) for row in rows] async def subscribe( self, subscription_id: str, handler, from_position: int = 0, batch_size: int = 100 ): """Subscribe to all events from a position.""" # Get checkpoint async with self.pool.acquire() as conn: checkpoint = await conn.fetchval( """ SELECT last_position FROM subscription_checkpoints WHERE subscription_id = $1 """, subscription_id ) position = checkpoint or from_position while True: events = await self.read_all(position, batch_size) if not events: await asyncio.sleep(1) # Poll interval continue for event in events: await handler(event) position = event.global_position # Save checkpoint async with self.pool.acquire() as conn: await conn.execute( """ INSERT INTO subscription_checkpoints (subscription_id, last_position) VALUES ($1, $2) ON CONFLICT (subscription_id) DO UPDATE SET last_position = $2, updated_at = NOW() """, subscription_id, position ) def _row_to_event(self, row) -> Event: return Event( event_id=row['id'], stream_id=row['stream_id'], event_type=row['event_type'], data=json.loads(row['event_data']), metadata=json.loads(row['metadata']), version=row['version'], global_position=row['global_position'], created_at=row['created_at'] ) class ConcurrencyError(Exception): """Raised when optimistic concurrency check fails.""" pass ``` ### Template 3: EventStoreDB Usage ```python from esdbclient import EventStoreDBClient, NewEvent, StreamState import json # Connect client = EventStoreDBClient(uri="esdb://localhost:2113?tls=false") # Append events def append_events(stream_name: str, events: list, expected_revision=None): new_events = [ NewEvent( type=event['type'], data=json.dumps(event['data']).encode(), metadata=json.dumps(event.get('metadata', {})).encode() ) for event in events ] if expected_revision is None: state = StreamState.ANY elif expected_revision == -1: state = StreamState.NO_STREAM else: state = expected_revision return client.append_to_stream( stream_name=stream_name, events=new_events, current_version=state ) # Read stream def read_stream(stream_name: str, from_revision: int = 0): events = client.get_stream( stream_name=stream_name, stream_position=from_revision ) return [ { 'type': event.type, 'data': json.loads(event.data), 'metadata': json.loads(event.metadata) if event.metadata else {}, 'stream_position': event.stream_position, 'commit_position': event.commit_position } for event in events ] # Subscribe to all async def subscribe_to_all(handler, from_position: int = 0): subscription = client.subscribe_to_all(commit_position=from_position) async for event in subscription: await handler({ 'type': event.type, 'data': json.loads(event.data), 'stream_id': event.stream_name, 'position': event.commit_position }) # Category projection ($ce-Category) def read_category(category: str): """Read all events for a category using system projection.""" return read_stream(f"$ce-{category}") ``` ### Template 4: DynamoDB Event Store ```python import boto3 from boto3.dynamodb.conditions import Key from datetime import datetime import json import uuid class DynamoEventStore: def __init__(self, table_name: str): self.dynamodb = boto3.resource('dynamodb') self.table = self.dynamodb.Table(table_name) def append_events(self, stream_id: str, events: list, expected_version: int = None): """Append events with conditional write for concurrency.""" with self.table.batch_writer() as batch: for i, event in enumerate(events): version = (expected_version or 0) + i + 1 item = { 'PK': f"STREAM#{stream_id}", 'SK': f"VERSION#{version:020d}", 'GSI1PK': 'EVENTS', 'GSI1SK': datetime.utcnow().isoformat(), 'event_id': str(uuid.uuid4()), 'stream_id': stream_id, 'event_type': event['type'], 'event_data': json.dumps(event['data']), 'version': version, 'created_at': datetime.utcnow().isoformat() } batch.put_item(Item=item) return events def read_stream(self, stream_id: str, from_version: int = 0): """Read events from a stream.""" response = self.table.query( KeyConditionExpression=Key('PK').eq(f"STREAM#{stream_id}") & Key('SK').gte(f"VERSION#{from_version:020d}") ) return [ { 'event_type': item['event_type'], 'data': json.loads(item['event_data']), 'version': item['version'] } for item in response['Items'] ] # Table definition (CloudFormation/Terraform) """ DynamoDB Table: - PK (Partition Key): String - SK (Sort Key): String - GSI1PK, GSI1SK for global ordering Capacity: On-demand or provisioned based on throughput needs """ ``` ## Best Practices ### Do's - **Use stream IDs that include aggregate type** - `Order-{uuid}` - **Include correlation/causation IDs** - For tracing - **Version events from day one** - Plan for schema evolution - **Implement idempotency** - Use event IDs for deduplication - **Index appropriately** - For your query patterns ### Don'ts - **Don't update or delete events** - They're immutable facts - **Don't store large payloads** - Keep events small - **Don't skip optimistic concurrency** - Prevents data corruption - **Don't ignore backpressure** - Handle slow consumers ## Resources - [EventStoreDB](https://www.eventstore.com/) - [Marten Events](https://martendb.io/events/) - [Event Sourcing Pattern](https://docs.microsoft.com/en-us/azure/architecture/patterns/event-sourcing)
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

solidity-security

Master smart contract security best practices to prevent common

security
⭐1
# Solidity Security Master smart contract security best practices, vulnerability prevention, and secure Solidity development patterns. ## When to Use This Skill - Writing secure smart contracts - Auditing existing contracts for vulnerabilities - Implementing secure DeFi protocols - Preventing reentrancy, overflow, and access control issues - Optimizing gas usage while maintaining security - Preparing contracts for professional audits - Understanding common attack vectors ## Critical Vulnerabilities ### 1. Reentrancy Attacker calls back into your contract before state is updated. **Vulnerable Code:** ```solidity // VULNERABLE TO REENTRANCY contract VulnerableBank { mapping(address => uint256) public balances; function withdraw() public { uint256 amount = balances[msg.sender]; // DANGER: External call before state update (bool success, ) = msg.sender.call{value: amount}(""); require(success); balances[msg.sender] = 0; // Too late! } } ``` **Secure Pattern (Checks-Effects-Interactions):** ```solidity contract SecureBank { mapping(address => uint256) public balances; function withdraw() public { uint256 amount = balances[msg.sender]; require(amount > 0, "Insufficient balance"); // EFFECTS: Update state BEFORE external call balances[msg.sender] = 0; // INTERACTIONS: External call last (bool success, ) = msg.sender.call{value: amount}(""); require(success, "Transfer failed"); } } ``` **Alternative: ReentrancyGuard** ```solidity import "@openzeppelin/contracts/security/ReentrancyGuard.sol"; contract SecureBank is ReentrancyGuard { mapping(address => uint256) public balances; function withdraw() public nonReentrant { uint256 amount = balances[msg.sender]; require(amount > 0, "Insufficient balance"); balances[msg.sender] = 0; (bool success, ) = msg.sender.call{value: amount}(""); require(success, "Transfer failed"); } } ``` ### 2. Integer Overflow/Underflow **Vulnerable Code (Solidity < 0.8.0):** ```solidity // VULNERABLE contract VulnerableToken { mapping(address => uint256) public balances; function transfer(address to, uint256 amount) public { // No overflow check - can wrap around balances[msg.sender] -= amount; // Can underflow! balances[to] += amount; // Can overflow! } } ``` **Secure Pattern (Solidity >= 0.8.0):** ```solidity // Solidity 0.8+ has built-in overflow/underflow checks contract SecureToken { mapping(address => uint256) public balances; function transfer(address to, uint256 amount) public { // Automatically reverts on overflow/underflow balances[msg.sender] -= amount; balances[to] += amount; } } ``` **For Solidity < 0.8.0, use SafeMath:** ```solidity import "@openzeppelin/contracts/utils/math/SafeMath.sol"; contract SecureToken { using SafeMath for uint256; mapping(address => uint256) public balances; function transfer(address to, uint256 amount) public { balances[msg.sender] = balances[msg.sender].sub(amount); balances[to] = balances[to].add(amount); } } ``` ### 3. Access Control **Vulnerable Code:** ```solidity // VULNERABLE: Anyone can call critical functions contract VulnerableContract { address public owner; function withdraw(uint256 amount) public { // No access control! payable(msg.sender).transfer(amount); } } ``` **Secure Pattern:** ```solidity import "@openzeppelin/contracts/access/Ownable.sol"; contract SecureContract is Ownable { function withdraw(uint256 amount) public onlyOwner { payable(owner()).transfer(amount); } } // Or implement custom role-based access contract RoleBasedContract { mapping(address => bool) public admins; modifier onlyAdmin() { require(admins[msg.sender], "Not an admin"); _; } function criticalFunction() public onlyAdmin { // Protected function } } ``` ### 4. Front-Running **Vulnerable:** ```solidity // VULNERABLE TO FRONT-RUNNING contract VulnerableDEX { function swap(uint256 amount, uint256 minOutput) public { // Attacker sees this in mempool and front-runs uint256 output = calculateOutput(amount); require(output >= minOutput, "Slippage too high"); // Perform swap } } ``` **Mitigation:** ```solidity contract SecureDEX { mapping(bytes32 => bool) public usedCommitments; // Step 1: Commit to trade function commitTrade(bytes32 commitment) public { usedCommitments[commitment] = true; } // Step 2: Reveal trade (next block) function revealTrade( uint256 amount, uint256 minOutput, bytes32 secret ) public { bytes32 commitment = keccak256(abi.encodePacked( msg.sender, amount, minOutput, secret )); require(usedCommitments[commitment], "Invalid commitment"); // Perform swap } } ``` ## Security Best Practices ### Checks-Effects-Interactions Pattern ```solidity contract SecurePattern { mapping(address => uint256) public balances; function withdraw(uint256 amount) public { // 1. CHECKS: Validate conditions require(amount <= balances[msg.sender], "Insufficient balance"); require(amount > 0, "Amount must be positive"); // 2. EFFECTS: Update state balances[msg.sender] -= amount; // 3. INTERACTIONS: External calls last (bool success, ) = msg.sender.call{value: amount}(""); require(success, "Transfer failed"); } } ``` ### Pull Over Push Pattern ```solidity // Prefer this (pull) contract SecurePayment { mapping(address => uint256) public pendingWithdrawals; function recordPayment(address recipient, uint256 amount) internal { pendingWithdrawals[recipient] += amount; } function withdraw() public { uint256 amount = pendingWithdrawals[msg.sender]; require(amount > 0, "Nothing to withdraw"); pendingWithdrawals[msg.sender] = 0; payable(msg.sender).transfer(amount); } } // Over this (push) contract RiskyPayment { function distributePayments(address[] memory recipients, uint256[] memory amounts) public { for (uint i = 0; i < recipients.length; i++) { // If any transfer fails, entire batch fails payable(recipients[i]).transfer(amounts[i]); } } } ``` ### Input Validation ```solidity contract SecureContract { function transfer(address to, uint256 amount) public { // Validate inputs require(to != address(0), "Invalid recipient"); require(to != address(this), "Cannot send to contract"); require(amount > 0, "Amount must be positive"); require(amount <= balances[msg.sender], "Insufficient balance"); // Proceed with transfer balances[msg.sender] -= amount; balances[to] += amount; } } ``` ### Emergency Stop (Circuit Breaker) ```solidity import "@openzeppelin/contracts/security/Pausable.sol"; contract EmergencyStop is Pausable, Ownable { function criticalFunction() public whenNotPaused { // Function logic } function emergencyStop() public onlyOwner { _pause(); } function resume() public onlyOwner { _unpause(); } } ``` ## Gas Optimization ### Use `uint256` Instead of Smaller Types ```solidity // More gas efficient contract GasEfficient { uint256 public value; // Optimal function set(uint256 _value) public { value = _value; } } // Less efficient contract GasInefficient { uint8 public value; // Still uses 256-bit slot function set(uint8 _value) public { value = _value; // Extra gas for type conversion } } ``` ### Pack Storage Variables ```solidity // Gas efficient (3 variables in 1 slot) contract PackedStorage { uint128 public a; // Slot 0 uint64 public b; // Slot 0 uint64 public c; // Slot 0 uint256 public d; // Slot 1 } // Gas inefficient (each variable in separate slot) contract UnpackedStorage { uint256 public a; // Slot 0 uint256 public b; // Slot 1 uint256 public c; // Slot 2 uint256 public d; // Slot 3 } ``` ### Use `calldata` Instead of `memory` for Function Arguments ```solidity contract GasOptimized { // More gas efficient function processData(uint256[] calldata data) public pure returns (uint256) { return data[0]; } // Less efficient function processDataMemory(uint256[] memory data) public pure returns (uint256) { return data[0]; } } ``` ### Use Events for Data Storage (When Appropriate) ```solidity contract EventStorage { // Emitting events is cheaper than storage event DataStored(address indexed user, uint256 indexed id, bytes data); function storeData(uint256 id, bytes calldata data) public { emit DataStored(msg.sender, id, data); // Don't store in contract storage unless needed } } ``` ## Common Vulnerabilities Checklist ```solidity // Security Checklist Contract contract SecurityChecklist { /** * [ ] Reentrancy protection (ReentrancyGuard or CEI pattern) * [ ] Integer overflow/underflow (Solidity 0.8+ or SafeMath) * [ ] Access control (Ownable, roles, modifiers) * [ ] Input validation (require statements) * [ ] Front-running mitigation (commit-reveal if applicable) * [ ] Gas optimization (packed storage, calldata) * [ ] Emergency stop mechanism (Pausable) * [ ] Pull over push pattern for payments * [ ] No delegatecall to untrusted contracts * [ ] No tx.origin for authentication (use msg.sender) * [ ] Proper event emission * [ ] External calls at end of function * [ ] Check return values of external calls * [ ] No hardcoded addresses * [ ] Upgrade mechanism (if proxy pattern) */ } ``` ## Testing for Security ```javascript // Hardhat test example const { expect } = require("chai"); const { ethers } = require("hardhat"); describe("Security Tests", function () { it("Should prevent reentrancy attack", async function () { const [attacker] = await ethers.getSigners(); const VictimBank = await ethers.getContractFactory("SecureBank"); const bank = await VictimBank.deploy(); const Attacker = await ethers.getContractFactory("ReentrancyAttacker"); const attackerContract = await Attacker.deploy(bank.address); // Deposit funds await bank.deposit({ value: ethers.utils.parseEther("10") }); // Attempt reentrancy attack await expect( attackerContract.attack({ value: ethers.utils.parseEther("1") }), ).to.be.revertedWith("ReentrancyGuard: reentrant call"); }); it("Should prevent integer overflow", async function () { const Token = await ethers.getContractFactory("SecureToken"); const token = await Token.deploy(); // Attempt overflow await expect(token.transfer(attacker.address, ethers.constants.MaxUint256)) .to.be.reverted; }); it("Should enforce access control", async function () { const [owner, attacker] = await ethers.getSigners(); const Contract = await ethers.getContractFactory("SecureContract"); const contract = await Contract.deploy(); // Attempt unauthorized withdrawal await expect(contract.connect(attacker).withdraw(100)).to.be.revertedWith( "Ownable: caller is not the owner", ); }); }); ``` ## Audit Preparation ```solidity contract WellDocumentedContract { /** * @title Well Documented Contract * @dev Example of proper documentation for audits * @notice This contract handles user deposits and withdrawals */ /// @notice Mapping of user balances mapping(address => uint256) public balances; /** * @dev Deposits ETH into the contract * @notice Anyone can deposit funds */ function deposit() public payable { require(msg.value > 0, "Must send ETH"); balances[msg.sender] += msg.value; } /** * @dev Withdraws user's balance * @notice Follows CEI pattern to prevent reentrancy * @param amount Amount to withdraw in wei */ function withdraw(uint256 amount) public { // CHECKS require(amount <= balances[msg.sender], "Insufficient balance"); // EFFECTS balances[msg.sender] -= amount; // INTERACTIONS (bool success, ) = msg.sender.call{value: amount}(""); require(success, "Transfer failed"); } } ``` ## Resources - **references/reentrancy.md**: Comprehensive reentrancy prevention - **references/access-control.md**: Role-based access patterns - **references/overflow-underflow.md**: SafeMath and integer safety - **references/gas-optimization.md**: Gas saving techniques - **references/vulnerability-patterns.md**: Common vulnerability catalog - **assets/solidity-contracts-templates.sol**: Secure contract templates - **assets/security-checklist.md**: Pre-audit checklist - **scripts/analyze-contract.sh**: Static analysis tools ## Tools for Security Analysis - **Slither**: Static analysis tool - **Mythril**: Security analysis tool - **Echidna**: Fuzzing tool - **Manticore**: Symbolic execution - **Securify**: Automated security scanner ## Common Pitfalls 1. **Using `tx.origin` for Authentication**: Use `msg.sender` instead 2. **Unchecked External Calls**: Always check return values 3. **Delegatecall to Untrusted Contracts**: Can hijack your contract 4. **Floating Pragma**: Pin to specific Solidity version 5. **Missing Events**: Emit events for state changes 6. **Excessive Gas in Loops**: Can hit block gas limit 7. **No Upgrade Path**: Consider proxy patterns if upgrades needed
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

kpi-dashboard-design

Design effective KPI dashboards with metrics selection,

coding
⭐1
# KPI Dashboard Design Comprehensive patterns for designing effective Key Performance Indicator (KPI) dashboards that drive business decisions. ## When to Use This Skill - Designing executive dashboards - Selecting meaningful KPIs - Building real-time monitoring displays - Creating department-specific metrics views - Improving existing dashboard layouts - Establishing metric governance ## Core Concepts ### 1. KPI Framework | Level | Focus | Update Frequency | Audience | | --------------- | ---------------- | ----------------- | ---------- | | **Strategic** | Long-term goals | Monthly/Quarterly | Executives | | **Tactical** | Department goals | Weekly/Monthly | Managers | | **Operational** | Day-to-day | Real-time/Daily | Teams | ### 2. SMART KPIs ``` Specific: Clear definition Measurable: Quantifiable Achievable: Realistic targets Relevant: Aligned to goals Time-bound: Defined period ``` ### 3. Dashboard Hierarchy ``` ā”œā”€ā”€ Executive Summary (1 page) │ ā”œā”€ā”€ 4-6 headline KPIs │ ā”œā”€ā”€ Trend indicators │ └── Key alerts ā”œā”€ā”€ Department Views │ ā”œā”€ā”€ Sales Dashboard │ ā”œā”€ā”€ Marketing Dashboard │ ā”œā”€ā”€ Operations Dashboard │ └── Finance Dashboard └── Detailed Drilldowns ā”œā”€ā”€ Individual metrics └── Root cause analysis ``` ## Common KPIs by Department ### Sales KPIs ```yaml Revenue Metrics: - Monthly Recurring Revenue (MRR) - Annual Recurring Revenue (ARR) - Average Revenue Per User (ARPU) - Revenue Growth Rate Pipeline Metrics: - Sales Pipeline Value - Win Rate - Average Deal Size - Sales Cycle Length Activity Metrics: - Calls/Emails per Rep - Demos Scheduled - Proposals Sent - Close Rate ``` ### Marketing KPIs ```yaml Acquisition: - Cost Per Acquisition (CPA) - Customer Acquisition Cost (CAC) - Lead Volume - Marketing Qualified Leads (MQL) Engagement: - Website Traffic - Conversion Rate - Email Open/Click Rate - Social Engagement ROI: - Marketing ROI - Campaign Performance - Channel Attribution - CAC Payback Period ``` ### Product KPIs ```yaml Usage: - Daily/Monthly Active Users (DAU/MAU) - Session Duration - Feature Adoption Rate - Stickiness (DAU/MAU) Quality: - Net Promoter Score (NPS) - Customer Satisfaction (CSAT) - Bug/Issue Count - Time to Resolution Growth: - User Growth Rate - Activation Rate - Retention Rate - Churn Rate ``` ### Finance KPIs ```yaml Profitability: - Gross Margin - Net Profit Margin - EBITDA - Operating Margin Liquidity: - Current Ratio - Quick Ratio - Cash Flow - Working Capital Efficiency: - Revenue per Employee - Operating Expense Ratio - Days Sales Outstanding - Inventory Turnover ``` ## Dashboard Layout Patterns ### Pattern 1: Executive Summary ``` ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ EXECUTIVE DASHBOARD [Date Range ā–¼] │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ REVENUE │ PROFIT │ CUSTOMERS │ NPS SCORE │ │ $2.4M │ $450K │ 12,450 │ 72 │ │ ā–² 12% │ ā–² 8% │ ā–² 15% │ ā–² 5pts │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ │ │ Revenue Trend │ Revenue by Product │ │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ │ │ /\ /\ │ │ │ ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ 45% │ │ │ │ / \ / \ /\ │ │ │ ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ 32% │ │ │ │ / \/ \ / \ │ │ │ ā–ˆā–ˆā–ˆā–ˆ 18% │ │ │ │ / \/ \ │ │ │ ā–ˆā–ˆ 5% │ │ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ │ │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ šŸ”“ Alert: Churn rate exceeded threshold (>5%) │ │ 🟔 Warning: Support ticket volume 20% above average │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ``` ### Pattern 2: SaaS Metrics Dashboard ``` ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ SAAS METRICS Jan 2024 [Monthly ā–¼] │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ MRR GROWTH │ │ │ MRR │ │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ │ │ $125,000 │ │ │ /── │ │ │ │ ā–² 8% │ │ │ /────/ │ │ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ │ /────/ │ │ │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ │ /────/ │ │ │ │ ARR │ │ │ /────/ │ │ │ │ $1,500,000 │ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ │ │ ā–² 15% │ │ J F M A M J J A S O N D │ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¼ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ UNIT ECONOMICS │ COHORT RETENTION │ │ │ │ │ CAC: $450 │ Month 1: ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ 100% │ │ LTV: $2,700 │ Month 3: ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ 85% │ │ LTV/CAC: 6.0x │ Month 6: ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ 80% │ │ │ Month 12: ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ 72% │ │ Payback: 4 months │ │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ CHURN ANALYSIS │ │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ │ │ Gross │ Net │ Logo │ Expansion │ │ │ │ 4.2% │ 1.8% │ 3.1% │ 2.4% │ │ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ``` ### Pattern 3: Real-time Operations ``` ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ OPERATIONS CENTER Live ā— Last: 10:42:15 │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ SYSTEM HEALTH │ SERVICE STATUS │ │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ │ │ │ CPU MEM DISK │ │ ā— API Gateway Healthy │ │ │ 45% 72% 58% │ │ ā— User Service Healthy │ │ │ ā–ˆā–ˆā–ˆ ā–ˆā–ˆā–ˆā–ˆ ā–ˆā–ˆā–ˆ │ │ ā— Payment Service Degraded │ │ │ ā–ˆā–ˆā–ˆ ā–ˆā–ˆā–ˆā–ˆ ā–ˆā–ˆā–ˆ │ │ ā— Database Healthy │ │ │ ā–ˆā–ˆā–ˆ ā–ˆā–ˆā–ˆā–ˆ ā–ˆā–ˆā–ˆ │ │ ā— Cache Healthy │ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¼ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ REQUEST THROUGHPUT │ ERROR RATE │ │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ │ │ ā–ā–‚ā–ƒā–„ā–…ā–†ā–‡ā–ˆā–‡ā–†ā–…ā–„ā–ƒā–‚ā–ā–‚ā–ƒā–„ā–… │ │ │ ▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁ │ │ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ │ │ Current: 12,450 req/s │ Current: 0.02% │ │ Peak: 18,200 req/s │ Threshold: 1.0% │ ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤ │ RECENT ALERTS │ │ 10:40 🟔 High latency on payment-service (p99 > 500ms) │ │ 10:35 🟢 Resolved: Database connection pool recovered │ │ 10:22 šŸ”“ Payment service circuit breaker tripped │ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ``` ## Implementation Patterns ### SQL for KPI Calculations ```sql -- Monthly Recurring Revenue (MRR) WITH mrr_calculation AS ( SELECT DATE_TRUNC('month', billing_date) AS month, SUM( CASE subscription_interval WHEN 'monthly' THEN amount WHEN 'yearly' THEN amount / 12 WHEN 'quarterly' THEN amount / 3 END ) AS mrr FROM subscriptions WHERE status = 'active' GROUP BY DATE_TRUNC('month', billing_date) ) SELECT month, mrr, LAG(mrr) OVER (ORDER BY month) AS prev_mrr, (mrr - LAG(mrr) OVER (ORDER BY month)) / LAG(mrr) OVER (ORDER BY month) * 100 AS growth_pct FROM mrr_calculation; -- Cohort Retention WITH cohorts AS ( SELECT user_id, DATE_TRUNC('month', created_at) AS cohort_month FROM users ), activity AS ( SELECT user_id, DATE_TRUNC('month', event_date) AS activity_month FROM user_events WHERE event_type = 'active_session' ) SELECT c.cohort_month, EXTRACT(MONTH FROM age(a.activity_month, c.cohort_month)) AS months_since_signup, COUNT(DISTINCT a.user_id) AS active_users, COUNT(DISTINCT a.user_id)::FLOAT / COUNT(DISTINCT c.user_id) * 100 AS retention_rate FROM cohorts c LEFT JOIN activity a ON c.user_id = a.user_id AND a.activity_month >= c.cohort_month GROUP BY c.cohort_month, EXTRACT(MONTH FROM age(a.activity_month, c.cohort_month)) ORDER BY c.cohort_month, months_since_signup; -- Customer Acquisition Cost (CAC) SELECT DATE_TRUNC('month', acquired_date) AS month, SUM(marketing_spend) / NULLIF(COUNT(new_customers), 0) AS cac, SUM(marketing_spend) AS total_spend, COUNT(new_customers) AS customers_acquired FROM ( SELECT DATE_TRUNC('month', u.created_at) AS acquired_date, u.id AS new_customers, m.spend AS marketing_spend FROM users u JOIN marketing_spend m ON DATE_TRUNC('month', u.created_at) = m.month WHERE u.source = 'marketing' ) acquisition GROUP BY DATE_TRUNC('month', acquired_date); ``` ### Python Dashboard Code (Streamlit) ```python import streamlit as st import pandas as pd import plotly.express as px import plotly.graph_objects as go st.set_page_config(page_title="KPI Dashboard", layout="wide") # Header with date filter col1, col2 = st.columns([3, 1]) with col1: st.title("Executive Dashboard") with col2: date_range = st.selectbox( "Period", ["Last 7 Days", "Last 30 Days", "Last Quarter", "YTD"] ) # KPI Cards def metric_card(label, value, delta, prefix="", suffix=""): delta_color = "green" if delta >= 0 else "red" delta_arrow = "ā–²" if delta >= 0 else "ā–¼" st.metric( label=label, value=f"{prefix}{value:,.0f}{suffix}", delta=f"{delta_arrow} {abs(delta):.1f}%" ) col1, col2, col3, col4 = st.columns(4) with col1: metric_card("Revenue", 2400000, 12.5, prefix="$") with col2: metric_card("Customers", 12450, 15.2) with col3: metric_card("NPS Score", 72, 5.0) with col4: metric_card("Churn Rate", 4.2, -0.8, suffix="%") # Charts col1, col2 = st.columns(2) with col1: st.subheader("Revenue Trend") revenue_data = pd.DataFrame({ 'Month': pd.date_range('2024-01-01', periods=12, freq='M'), 'Revenue': [180000, 195000, 210000, 225000, 240000, 255000, 270000, 285000, 300000, 315000, 330000, 345000] }) fig = px.line(revenue_data, x='Month', y='Revenue', line_shape='spline', markers=True) fig.update_layout(height=300) st.plotly_chart(fig, use_container_width=True) with col2: st.subheader("Revenue by Product") product_data = pd.DataFrame({ 'Product': ['Enterprise', 'Professional', 'Starter', 'Other'], 'Revenue': [45, 32, 18, 5] }) fig = px.pie(product_data, values='Revenue', names='Product', hole=0.4) fig.update_layout(height=300) st.plotly_chart(fig, use_container_width=True) # Cohort Heatmap st.subheader("Cohort Retention") cohort_data = pd.DataFrame({ 'Cohort': ['Jan', 'Feb', 'Mar', 'Apr', 'May'], 'M0': [100, 100, 100, 100, 100], 'M1': [85, 87, 84, 86, 88], 'M2': [78, 80, 76, 79, None], 'M3': [72, 74, 70, None, None], 'M4': [68, 70, None, None, None], }) fig = go.Figure(data=go.Heatmap( z=cohort_data.iloc[:, 1:].values, x=['M0', 'M1', 'M2', 'M3', 'M4'], y=cohort_data['Cohort'], colorscale='Blues', text=cohort_data.iloc[:, 1:].values, texttemplate='%{text}%', textfont={"size": 12}, )) fig.update_layout(height=250) st.plotly_chart(fig, use_container_width=True) # Alerts Section st.subheader("Alerts") alerts = [ {"level": "error", "message": "Churn rate exceeded threshold (>5%)"}, {"level": "warning", "message": "Support ticket volume 20% above average"}, ] for alert in alerts: if alert["level"] == "error": st.error(f"šŸ”“ {alert['message']}") elif alert["level"] == "warning": st.warning(f"🟔 {alert['message']}") ``` ## Best Practices ### Do's - **Limit to 5-7 KPIs** - Focus on what matters - **Show context** - Comparisons, trends, targets - **Use consistent colors** - Red=bad, green=good - **Enable drilldown** - From summary to detail - **Update appropriately** - Match metric frequency ### Don'ts - **Don't show vanity metrics** - Focus on actionable data - **Don't overcrowd** - White space aids comprehension - **Don't use 3D charts** - They distort perception - **Don't hide methodology** - Document calculations - **Don't ignore mobile** - Ensure responsive design ## Resources - [Stephen Few's Dashboard Design](https://www.perceptualedge.com/articles/visual_business_intelligence/rules_for_using_color.pdf) - [Edward Tufte's Principles](https://www.edwardtufte.com/tufte/) - [Google Data Studio Gallery](https://datastudio.google.com/gallery)
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

deployment-pipeline-design

Design multi-stage CI/CD pipelines with approval gates, security

coding
⭐1
# Deployment Pipeline Design Architecture patterns for multi-stage CI/CD pipelines with approval gates and deployment strategies. ## Purpose Design robust, secure deployment pipelines that balance speed with safety through proper stage organization and approval workflows. ## When to Use - Design CI/CD architecture - Implement deployment gates - Configure multi-environment pipelines - Establish deployment best practices - Implement progressive delivery ## Pipeline Stages ### Standard Pipeline Flow ``` ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā” ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” │ Build │ → │ Test │ → │ Staging │ → │ Approve│ → │Production│ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ ``` ### Detailed Stage Breakdown 1. **Source** - Code checkout 2. **Build** - Compile, package, containerize 3. **Test** - Unit, integration, security scans 4. **Staging Deploy** - Deploy to staging environment 5. **Integration Tests** - E2E, smoke tests 6. **Approval Gate** - Manual approval required 7. **Production Deploy** - Canary, blue-green, rolling 8. **Verification** - Health checks, monitoring 9. **Rollback** - Automated rollback on failure ## Approval Gate Patterns ### Pattern 1: Manual Approval ```yaml # GitHub Actions production-deploy: needs: staging-deploy environment: name: production url: https://app.example.com runs-on: ubuntu-latest steps: - name: Deploy to production run: | # Deployment commands ``` ### Pattern 2: Time-Based Approval ```yaml # GitLab CI deploy:production: stage: deploy script: - deploy.sh production environment: name: production when: delayed start_in: 30 minutes only: - main ``` ### Pattern 3: Multi-Approver ```yaml # Azure Pipelines stages: - stage: Production dependsOn: Staging jobs: - deployment: Deploy environment: name: production resourceType: Kubernetes strategy: runOnce: preDeploy: steps: - task: ManualValidation@0 inputs: notifyUsers: "team-leads@example.com" instructions: "Review staging metrics before approving" ``` **Reference:** See `assets/approval-gate-template.yml` ## Deployment Strategies ### 1. Rolling Deployment ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: my-app spec: replicas: 10 strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 1 ``` **Characteristics:** - Gradual rollout - Zero downtime - Easy rollback - Best for most applications ### 2. Blue-Green Deployment ```yaml # Blue (current) kubectl apply -f blue-deployment.yaml kubectl label service my-app version=blue # Green (new) kubectl apply -f green-deployment.yaml # Test green environment kubectl label service my-app version=green # Rollback if needed kubectl label service my-app version=blue ``` **Characteristics:** - Instant switchover - Easy rollback - Doubles infrastructure cost temporarily - Good for high-risk deployments ### 3. Canary Deployment ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: my-app spec: replicas: 10 strategy: canary: steps: - setWeight: 10 - pause: { duration: 5m } - setWeight: 25 - pause: { duration: 5m } - setWeight: 50 - pause: { duration: 5m } - setWeight: 100 ``` **Characteristics:** - Gradual traffic shift - Risk mitigation - Real user testing - Requires service mesh or similar ### 4. Feature Flags ```python from flagsmith import Flagsmith flagsmith = Flagsmith(environment_key="API_KEY") if flagsmith.has_feature("new_checkout_flow"): # New code path process_checkout_v2() else: # Existing code path process_checkout_v1() ``` **Characteristics:** - Deploy without releasing - A/B testing - Instant rollback - Granular control ## Pipeline Orchestration ### Multi-Stage Pipeline Example ```yaml name: Production Pipeline on: push: branches: [main] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Build application run: make build - name: Build Docker image run: docker build -t myapp:${{ github.sha }} . - name: Push to registry run: docker push myapp:${{ github.sha }} test: needs: build runs-on: ubuntu-latest steps: - name: Unit tests run: make test - name: Security scan run: trivy image myapp:${{ github.sha }} deploy-staging: needs: test runs-on: ubuntu-latest environment: name: staging steps: - name: Deploy to staging run: kubectl apply -f k8s/staging/ integration-test: needs: deploy-staging runs-on: ubuntu-latest steps: - name: Run E2E tests run: npm run test:e2e deploy-production: needs: integration-test runs-on: ubuntu-latest environment: name: production steps: - name: Canary deployment run: | kubectl apply -f k8s/production/ kubectl argo rollouts promote my-app verify: needs: deploy-production runs-on: ubuntu-latest steps: - name: Health check run: curl -f https://app.example.com/health - name: Notify team run: | curl -X POST ${{ secrets.SLACK_WEBHOOK }} \ -d '{"text":"Production deployment successful!"}' ``` ## Pipeline Best Practices 1. **Fail fast** - Run quick tests first 2. **Parallel execution** - Run independent jobs concurrently 3. **Caching** - Cache dependencies between runs 4. **Artifact management** - Store build artifacts 5. **Environment parity** - Keep environments consistent 6. **Secrets management** - Use secret stores (Vault, etc.) 7. **Deployment windows** - Schedule deployments appropriately 8. **Monitoring integration** - Track deployment metrics 9. **Rollback automation** - Auto-rollback on failures 10. **Documentation** - Document pipeline stages ## Rollback Strategies ### Automated Rollback ```yaml deploy-and-verify: steps: - name: Deploy new version run: kubectl apply -f k8s/ - name: Wait for rollout run: kubectl rollout status deployment/my-app - name: Health check id: health run: | for i in {1..10}; do if curl -sf https://app.example.com/health; then exit 0 fi sleep 10 done exit 1 - name: Rollback on failure if: failure() run: kubectl rollout undo deployment/my-app ``` ### Manual Rollback ```bash # List revision history kubectl rollout history deployment/my-app # Rollback to previous version kubectl rollout undo deployment/my-app # Rollback to specific revision kubectl rollout undo deployment/my-app --to-revision=3 ``` ## Monitoring and Metrics ### Key Pipeline Metrics - **Deployment Frequency** - How often deployments occur - **Lead Time** - Time from commit to production - **Change Failure Rate** - Percentage of failed deployments - **Mean Time to Recovery (MTTR)** - Time to recover from failure - **Pipeline Success Rate** - Percentage of successful runs - **Average Pipeline Duration** - Time to complete pipeline ### Integration with Monitoring ```yaml - name: Post-deployment verification run: | # Wait for metrics stabilization sleep 60 # Check error rate ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=rate(http_errors_total[5m])" | jq '.data.result[0].value[1]') if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then echo "Error rate too high: $ERROR_RATE" exit 1 fi ``` ## Reference Files - `references/pipeline-orchestration.md` - Complex pipeline patterns - `assets/approval-gate-template.yml` - Approval workflow templates ## Related Skills - `github-actions-templates` - For GitHub Actions implementation - `gitlab-ci-patterns` - For GitLab CI implementation - `secrets-management` - For secrets handling
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

github-actions-templates

Create production-ready GitHub Actions workflows for automated

coding
⭐1
# GitHub Actions Templates Production-ready GitHub Actions workflow patterns for testing, building, and deploying applications. ## Purpose Create efficient, secure GitHub Actions workflows for continuous integration and deployment across various tech stacks. ## When to Use - Automate testing and deployment - Build Docker images and push to registries - Deploy to Kubernetes clusters - Run security scans - Implement matrix builds for multiple environments ## Common Workflow Patterns ### Pattern 1: Test Workflow ```yaml name: Test on: push: branches: [main, develop] pull_request: branches: [main] jobs: test: runs-on: ubuntu-latest strategy: matrix: node-version: [18.x, 20.x] steps: - uses: actions/checkout@v4 - name: Use Node.js ${{ matrix.node-version }} uses: actions/setup-node@v4 with: node-version: ${{ matrix.node-version }} cache: "npm" - name: Install dependencies run: npm ci - name: Run linter run: npm run lint - name: Run tests run: npm test - name: Upload coverage uses: codecov/codecov-action@v3 with: files: ./coverage/lcov.info ``` **Reference:** See `assets/test-workflow.yml` ### Pattern 2: Build and Push Docker Image ```yaml name: Build and Push on: push: branches: [main] tags: ["v*"] env: REGISTRY: ghcr.io IMAGE_NAME: ${{ github.repository }} jobs: build: runs-on: ubuntu-latest permissions: contents: read packages: write steps: - uses: actions/checkout@v4 - name: Log in to Container Registry uses: docker/login-action@v3 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Extract metadata id: meta uses: docker/metadata-action@v5 with: images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }} tags: | type=ref,event=branch type=ref,event=pr type=semver,pattern={{version}} type=semver,pattern={{major}}.{{minor}} - name: Build and push uses: docker/build-push-action@v5 with: context: . push: true tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} cache-from: type=gha cache-to: type=gha,mode=max ``` **Reference:** See `assets/deploy-workflow.yml` ### Pattern 3: Deploy to Kubernetes ```yaml name: Deploy to Kubernetes on: push: branches: [main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: us-west-2 - name: Update kubeconfig run: | aws eks update-kubeconfig --name production-cluster --region us-west-2 - name: Deploy to Kubernetes run: | kubectl apply -f k8s/ kubectl rollout status deployment/my-app -n production kubectl get services -n production - name: Verify deployment run: | kubectl get pods -n production kubectl describe deployment my-app -n production ``` ### Pattern 4: Matrix Build ```yaml name: Matrix Build on: [push, pull_request] jobs: build: runs-on: ${{ matrix.os }} strategy: matrix: os: [ubuntu-latest, macos-latest, windows-latest] python-version: ["3.9", "3.10", "3.11", "3.12"] steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Run tests run: pytest ``` **Reference:** See `assets/matrix-build.yml` ## Workflow Best Practices 1. **Use specific action versions** (@v4, not @latest) 2. **Cache dependencies** to speed up builds 3. **Use secrets** for sensitive data 4. **Implement status checks** on PRs 5. **Use matrix builds** for multi-version testing 6. **Set appropriate permissions** 7. **Use reusable workflows** for common patterns 8. **Implement approval gates** for production 9. **Add notification steps** for failures 10. **Use self-hosted runners** for sensitive workloads ## Reusable Workflows ```yaml # .github/workflows/reusable-test.yml name: Reusable Test Workflow on: workflow_call: inputs: node-version: required: true type: string secrets: NPM_TOKEN: required: true jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ inputs.node-version }} - run: npm ci - run: npm test ``` **Use reusable workflow:** ```yaml jobs: call-test: uses: ./.github/workflows/reusable-test.yml with: node-version: "20.x" secrets: NPM_TOKEN: ${{ secrets.NPM_TOKEN }} ``` ## Security Scanning ```yaml name: Security Scan on: push: branches: [main] pull_request: branches: [main] jobs: security: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Run Trivy vulnerability scanner uses: aquasecurity/trivy-action@master with: scan-type: "fs" scan-ref: "." format: "sarif" output: "trivy-results.sarif" - name: Upload Trivy results to GitHub Security uses: github/codeql-action/upload-sarif@v2 with: sarif_file: "trivy-results.sarif" - name: Run Snyk Security Scan uses: snyk/actions/node@master env: SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} ``` ## Deployment with Approvals ```yaml name: Deploy to Production on: push: tags: ["v*"] jobs: deploy: runs-on: ubuntu-latest environment: name: production url: https://app.example.com steps: - uses: actions/checkout@v4 - name: Deploy application run: | echo "Deploying to production..." # Deployment commands here - name: Notify Slack if: success() uses: slackapi/slack-github-action@v1 with: webhook-url: ${{ secrets.SLACK_WEBHOOK }} payload: | { "text": "Deployment to production completed successfully!" } ``` ## Reference Files - `assets/test-workflow.yml` - Testing workflow template - `assets/deploy-workflow.yml` - Deployment workflow template - `assets/matrix-build.yml` - Matrix build template - `references/common-workflows.md` - Common workflow patterns ## Related Skills - `gitlab-ci-patterns` - For GitLab CI workflows - `deployment-pipeline-design` - For pipeline architecture - `secrets-management` - For secrets handling
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

secrets-management

Implement secure secrets management for CI/CD pipelines using

coding
⭐1
# Secrets Management Secure secrets management practices for CI/CD pipelines using Vault, AWS Secrets Manager, and other tools. ## Purpose Implement secure secrets management in CI/CD pipelines without hardcoding sensitive information. ## When to Use - Store API keys and credentials - Manage database passwords - Handle TLS certificates - Rotate secrets automatically - Implement least-privilege access ## Secrets Management Tools ### HashiCorp Vault - Centralized secrets management - Dynamic secrets generation - Secret rotation - Audit logging - Fine-grained access control ### AWS Secrets Manager - AWS-native solution - Automatic rotation - Integration with RDS - CloudFormation support ### Azure Key Vault - Azure-native solution - HSM-backed keys - Certificate management - RBAC integration ### Google Secret Manager - GCP-native solution - Versioning - IAM integration ## HashiCorp Vault Integration ### Setup Vault ```bash # Start Vault dev server vault server -dev # Set environment export VAULT_ADDR='http://127.0.0.1:8200' export VAULT_TOKEN='root' # Enable secrets engine vault secrets enable -path=secret kv-v2 # Store secret vault kv put secret/database/config username=admin password=secret ``` ### GitHub Actions with Vault ```yaml name: Deploy with Vault Secrets on: [push] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Import Secrets from Vault uses: hashicorp/vault-action@v2 with: url: https://vault.example.com:8200 token: ${{ secrets.VAULT_TOKEN }} secrets: | secret/data/database username | DB_USERNAME ; secret/data/database password | DB_PASSWORD ; secret/data/api key | API_KEY - name: Use secrets run: | echo "Connecting to database as $DB_USERNAME" # Use $DB_PASSWORD, $API_KEY ``` ### GitLab CI with Vault ```yaml deploy: image: vault:latest before_script: - export VAULT_ADDR=https://vault.example.com:8200 - export VAULT_TOKEN=$VAULT_TOKEN - apk add curl jq script: - | DB_PASSWORD=$(vault kv get -field=password secret/database/config) API_KEY=$(vault kv get -field=key secret/api/credentials) echo "Deploying with secrets..." # Use $DB_PASSWORD, $API_KEY ``` **Reference:** See `references/vault-setup.md` ## AWS Secrets Manager ### Store Secret ```bash aws secretsmanager create-secret \ --name production/database/password \ --secret-string "super-secret-password" ``` ### Retrieve in GitHub Actions ```yaml - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: us-west-2 - name: Get secret from AWS run: | SECRET=$(aws secretsmanager get-secret-value \ --secret-id production/database/password \ --query SecretString \ --output text) echo "::add-mask::$SECRET" echo "DB_PASSWORD=$SECRET" >> $GITHUB_ENV - name: Use secret run: | # Use $DB_PASSWORD ./deploy.sh ``` ### Terraform with AWS Secrets Manager ```hcl data "aws_secretsmanager_secret_version" "db_password" { secret_id = "production/database/password" } resource "aws_db_instance" "main" { allocated_storage = 100 engine = "postgres" instance_class = "db.t3.large" username = "admin" password = jsondecode(data.aws_secretsmanager_secret_version.db_password.secret_string)["password"] } ``` ## GitHub Secrets ### Organization/Repository Secrets ```yaml - name: Use GitHub secret run: | echo "API Key: ${{ secrets.API_KEY }}" echo "Database URL: ${{ secrets.DATABASE_URL }}" ``` ### Environment Secrets ```yaml deploy: runs-on: ubuntu-latest environment: production steps: - name: Deploy run: | echo "Deploying with ${{ secrets.PROD_API_KEY }}" ``` **Reference:** See `references/github-secrets.md` ## GitLab CI/CD Variables ### Project Variables ```yaml deploy: script: - echo "Deploying with $API_KEY" - echo "Database: $DATABASE_URL" ``` ### Protected and Masked Variables - Protected: Only available in protected branches - Masked: Hidden in job logs - File type: Stored as file ## Best Practices 1. **Never commit secrets** to Git 2. **Use different secrets** per environment 3. **Rotate secrets regularly** 4. **Implement least-privilege access** 5. **Enable audit logging** 6. **Use secret scanning** (GitGuardian, TruffleHog) 7. **Mask secrets in logs** 8. **Encrypt secrets at rest** 9. **Use short-lived tokens** when possible 10. **Document secret requirements** ## Secret Rotation ### Automated Rotation with AWS ```python import boto3 import json def lambda_handler(event, context): client = boto3.client('secretsmanager') # Get current secret response = client.get_secret_value(SecretId='my-secret') current_secret = json.loads(response['SecretString']) # Generate new password new_password = generate_strong_password() # Update database password update_database_password(new_password) # Update secret client.put_secret_value( SecretId='my-secret', SecretString=json.dumps({ 'username': current_secret['username'], 'password': new_password }) ) return {'statusCode': 200} ``` ### Manual Rotation Process 1. Generate new secret 2. Update secret in secret store 3. Update applications to use new secret 4. Verify functionality 5. Revoke old secret ## External Secrets Operator ### Kubernetes Integration ```yaml apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: vault-backend namespace: production spec: provider: vault: server: "https://vault.example.com:8200" path: "secret" version: "v2" auth: kubernetes: mountPath: "kubernetes" role: "production" --- apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: database-credentials namespace: production spec: refreshInterval: 1h secretStoreRef: name: vault-backend kind: SecretStore target: name: database-credentials creationPolicy: Owner data: - secretKey: username remoteRef: key: database/config property: username - secretKey: password remoteRef: key: database/config property: password ``` ## Secret Scanning ### Pre-commit Hook ```bash #!/bin/bash # .git/hooks/pre-commit # Check for secrets with TruffleHog docker run --rm -v "$(pwd):/repo" \ trufflesecurity/trufflehog:latest \ filesystem --directory=/repo if [ $? -ne 0 ]; then echo "āŒ Secret detected! Commit blocked." exit 1 fi ``` ### CI/CD Secret Scanning ```yaml secret-scan: stage: security image: trufflesecurity/trufflehog:latest script: - trufflehog filesystem . allow_failure: false ``` ## Reference Files - `references/vault-setup.md` - HashiCorp Vault configuration - `references/github-secrets.md` - GitHub Secrets best practices ## Related Skills - `github-actions-templates` - For GitHub Actions integration - `gitlab-ci-patterns` - For GitLab CI integration - `deployment-pipeline-design` - For pipeline architecture
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

cost-optimization

Optimize cloud costs through resource rightsizing, tagging

architecture
⭐1
# Cloud Cost Optimization Strategies and patterns for optimizing cloud costs across AWS, Azure, and GCP. ## Purpose Implement systematic cost optimization strategies to reduce cloud spending while maintaining performance and reliability. ## When to Use - Reduce cloud spending - Right-size resources - Implement cost governance - Optimize multi-cloud costs - Meet budget constraints ## Cost Optimization Framework ### 1. Visibility - Implement cost allocation tags - Use cloud cost management tools - Set up budget alerts - Create cost dashboards ### 2. Right-Sizing - Analyze resource utilization - Downsize over-provisioned resources - Use auto-scaling - Remove idle resources ### 3. Pricing Models - Use reserved capacity - Leverage spot/preemptible instances - Implement savings plans - Use committed use discounts ### 4. Architecture Optimization - Use managed services - Implement caching - Optimize data transfer - Use lifecycle policies ## AWS Cost Optimization ### Reserved Instances ``` Savings: 30-72% vs On-Demand Term: 1 or 3 years Payment: All/Partial/No upfront Flexibility: Standard or Convertible ``` ### Savings Plans ``` Compute Savings Plans: 66% savings EC2 Instance Savings Plans: 72% savings Applies to: EC2, Fargate, Lambda Flexible across: Instance families, regions, OS ``` ### Spot Instances ``` Savings: Up to 90% vs On-Demand Best for: Batch jobs, CI/CD, stateless workloads Risk: 2-minute interruption notice Strategy: Mix with On-Demand for resilience ``` ### S3 Cost Optimization ```hcl resource "aws_s3_bucket_lifecycle_configuration" "example" { bucket = aws_s3_bucket.example.id rule { id = "transition-to-ia" status = "Enabled" transition { days = 30 storage_class = "STANDARD_IA" } transition { days = 90 storage_class = "GLACIER" } expiration { days = 365 } } } ``` ## Azure Cost Optimization ### Reserved VM Instances - 1 or 3 year terms - Up to 72% savings - Flexible sizing - Exchangeable ### Azure Hybrid Benefit - Use existing Windows Server licenses - Up to 80% savings with RI - Available for Windows and SQL Server ### Azure Advisor Recommendations - Right-size VMs - Delete unused resources - Use reserved capacity - Optimize storage ## GCP Cost Optimization ### Committed Use Discounts - 1 or 3 year commitment - Up to 57% savings - Applies to vCPUs and memory - Resource-based or spend-based ### Sustained Use Discounts - Automatic discounts - Up to 30% for running instances - No commitment required - Applies to Compute Engine, GKE ### Preemptible VMs - Up to 80% savings - 24-hour maximum runtime - Best for batch workloads ## Tagging Strategy ### AWS Tagging ```hcl locals { common_tags = { Environment = "production" Project = "my-project" CostCenter = "engineering" Owner = "team@example.com" ManagedBy = "terraform" } } resource "aws_instance" "example" { ami = "ami-12345678" instance_type = "t3.medium" tags = merge( local.common_tags, { Name = "web-server" } ) } ``` **Reference:** See `references/tagging-standards.md` ## Cost Monitoring ### Budget Alerts ```hcl # AWS Budget resource "aws_budgets_budget" "monthly" { name = "monthly-budget" budget_type = "COST" limit_amount = "1000" limit_unit = "USD" time_period_start = "2024-01-01_00:00" time_unit = "MONTHLY" notification { comparison_operator = "GREATER_THAN" threshold = 80 threshold_type = "PERCENTAGE" notification_type = "ACTUAL" subscriber_email_addresses = ["team@example.com"] } } ``` ### Cost Anomaly Detection - AWS Cost Anomaly Detection - Azure Cost Management alerts - GCP Budget alerts ## Architecture Patterns ### Pattern 1: Serverless First - Use Lambda/Functions for event-driven - Pay only for execution time - Auto-scaling included - No idle costs ### Pattern 2: Right-Sized Databases ``` Development: t3.small RDS Staging: t3.large RDS Production: r6g.2xlarge RDS with read replicas ``` ### Pattern 3: Multi-Tier Storage ``` Hot data: S3 Standard Warm data: S3 Standard-IA (30 days) Cold data: S3 Glacier (90 days) Archive: S3 Deep Archive (365 days) ``` ### Pattern 4: Auto-Scaling ```hcl resource "aws_autoscaling_policy" "scale_up" { name = "scale-up" scaling_adjustment = 2 adjustment_type = "ChangeInCapacity" cooldown = 300 autoscaling_group_name = aws_autoscaling_group.main.name } resource "aws_cloudwatch_metric_alarm" "cpu_high" { alarm_name = "cpu-high" comparison_operator = "GreaterThanThreshold" evaluation_periods = "2" metric_name = "CPUUtilization" namespace = "AWS/EC2" period = "60" statistic = "Average" threshold = "80" alarm_actions = [aws_autoscaling_policy.scale_up.arn] } ``` ## Cost Optimization Checklist - [ ] Implement cost allocation tags - [ ] Delete unused resources (EBS, EIPs, snapshots) - [ ] Right-size instances based on utilization - [ ] Use reserved capacity for steady workloads - [ ] Implement auto-scaling - [ ] Optimize storage classes - [ ] Use lifecycle policies - [ ] Enable cost anomaly detection - [ ] Set budget alerts - [ ] Review costs weekly - [ ] Use spot/preemptible instances - [ ] Optimize data transfer costs - [ ] Implement caching layers - [ ] Use managed services - [ ] Monitor and optimize continuously ## Tools - **AWS:** Cost Explorer, Cost Anomaly Detection, Compute Optimizer - **Azure:** Cost Management, Advisor - **GCP:** Cost Management, Recommender - **Multi-cloud:** CloudHealth, Cloudability, Kubecost ## Reference Files - `references/tagging-standards.md` - Tagging conventions - `assets/cost-analysis-template.xlsx` - Cost analysis spreadsheet ## Related Skills - `terraform-module-library` - For resource provisioning - `multi-cloud-architecture` - For cloud selection
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

hybrid-cloud-networking

Configure secure, high-performance connectivity between on-premises

architecture
⭐1
# Hybrid Cloud Networking Configure secure, high-performance connectivity between on-premises and cloud environments using VPN, Direct Connect, and ExpressRoute. ## Purpose Establish secure, reliable network connectivity between on-premises data centers and cloud providers (AWS, Azure, GCP). ## When to Use - Connect on-premises to cloud - Extend datacenter to cloud - Implement hybrid active-active setups - Meet compliance requirements - Migrate to cloud gradually ## Connection Options ### AWS Connectivity #### 1. Site-to-Site VPN - IPSec VPN over internet - Up to 1.25 Gbps per tunnel - Cost-effective for moderate bandwidth - Higher latency, internet-dependent ```hcl resource "aws_vpn_gateway" "main" { vpc_id = aws_vpc.main.id tags = { Name = "main-vpn-gateway" } } resource "aws_customer_gateway" "main" { bgp_asn = 65000 ip_address = "203.0.113.1" type = "ipsec.1" } resource "aws_vpn_connection" "main" { vpn_gateway_id = aws_vpn_gateway.main.id customer_gateway_id = aws_customer_gateway.main.id type = "ipsec.1" static_routes_only = false } ``` #### 2. AWS Direct Connect - Dedicated network connection - 1 Gbps to 100 Gbps - Lower latency, consistent bandwidth - More expensive, setup time required **Reference:** See `references/direct-connect.md` ### Azure Connectivity #### 1. Site-to-Site VPN ```hcl resource "azurerm_virtual_network_gateway" "vpn" { name = "vpn-gateway" location = azurerm_resource_group.main.location resource_group_name = azurerm_resource_group.main.name type = "Vpn" vpn_type = "RouteBased" sku = "VpnGw1" ip_configuration { name = "vnetGatewayConfig" public_ip_address_id = azurerm_public_ip.vpn.id private_ip_address_allocation = "Dynamic" subnet_id = azurerm_subnet.gateway.id } } ``` #### 2. Azure ExpressRoute - Private connection via connectivity provider - Up to 100 Gbps - Low latency, high reliability - Premium for global connectivity ### GCP Connectivity #### 1. Cloud VPN - IPSec VPN (Classic or HA VPN) - HA VPN: 99.99% SLA - Up to 3 Gbps per tunnel #### 2. Cloud Interconnect - Dedicated (10 Gbps, 100 Gbps) - Partner (50 Mbps to 50 Gbps) - Lower latency than VPN ## Hybrid Network Patterns ### Pattern 1: Hub-and-Spoke ``` On-Premises Datacenter ↓ VPN/Direct Connect ↓ Transit Gateway (AWS) / vWAN (Azure) ↓ ā”œā”€ Production VPC/VNet ā”œā”€ Staging VPC/VNet └─ Development VPC/VNet ``` ### Pattern 2: Multi-Region Hybrid ``` On-Premises ā”œā”€ Direct Connect → us-east-1 └─ Direct Connect → us-west-2 ↓ Cross-Region Peering ``` ### Pattern 3: Multi-Cloud Hybrid ``` On-Premises Datacenter ā”œā”€ Direct Connect → AWS ā”œā”€ ExpressRoute → Azure └─ Interconnect → GCP ``` ## Routing Configuration ### BGP Configuration ``` On-Premises Router: - AS Number: 65000 - Advertise: 10.0.0.0/8 Cloud Router: - AS Number: 64512 (AWS), 65515 (Azure) - Advertise: Cloud VPC/VNet CIDRs ``` ### Route Propagation - Enable route propagation on route tables - Use BGP for dynamic routing - Implement route filtering - Monitor route advertisements ## Security Best Practices 1. **Use private connectivity** (Direct Connect/ExpressRoute) 2. **Implement encryption** for VPN tunnels 3. **Use VPC endpoints** to avoid internet routing 4. **Configure network ACLs** and security groups 5. **Enable VPC Flow Logs** for monitoring 6. **Implement DDoS protection** 7. **Use PrivateLink/Private Endpoints** 8. **Monitor connections** with CloudWatch/Monitor 9. **Implement redundancy** (dual tunnels) 10. **Regular security audits** ## High Availability ### Dual VPN Tunnels ```hcl resource "aws_vpn_connection" "primary" { vpn_gateway_id = aws_vpn_gateway.main.id customer_gateway_id = aws_customer_gateway.primary.id type = "ipsec.1" } resource "aws_vpn_connection" "secondary" { vpn_gateway_id = aws_vpn_gateway.main.id customer_gateway_id = aws_customer_gateway.secondary.id type = "ipsec.1" } ``` ### Active-Active Configuration - Multiple connections from different locations - BGP for automatic failover - Equal-cost multi-path (ECMP) routing - Monitor health of all connections ## Monitoring and Troubleshooting ### Key Metrics - Tunnel status (up/down) - Bytes in/out - Packet loss - Latency - BGP session status ### Troubleshooting ```bash # AWS VPN aws ec2 describe-vpn-connections aws ec2 get-vpn-connection-telemetry # Azure VPN az network vpn-connection show az network vpn-connection show-device-config-script ``` ## Cost Optimization 1. **Right-size connections** based on traffic 2. **Use VPN for low-bandwidth** workloads 3. **Consolidate traffic** through fewer connections 4. **Minimize data transfer** costs 5. **Use Direct Connect** for high bandwidth 6. **Implement caching** to reduce traffic ## Reference Files - `references/vpn-setup.md` - VPN configuration guide - `references/direct-connect.md` - Direct Connect setup ## Related Skills - `multi-cloud-architecture` - For architecture decisions - `terraform-module-library` - For IaC implementation
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

multi-cloud-architecture

Design multi-cloud architectures using a decision framework to

architecture
⭐1
# Multi-Cloud Architecture Decision framework and patterns for architecting applications across AWS, Azure, and GCP. ## Purpose Design cloud-agnostic architectures and make informed decisions about service selection across cloud providers. ## When to Use - Design multi-cloud strategies - Migrate between cloud providers - Select cloud services for specific workloads - Implement cloud-agnostic architectures - Optimize costs across providers ## Cloud Service Comparison ### Compute Services | AWS | Azure | GCP | Use Case | | ------- | ------------------- | --------------- | ------------------ | | EC2 | Virtual Machines | Compute Engine | IaaS VMs | | ECS | Container Instances | Cloud Run | Containers | | EKS | AKS | GKE | Kubernetes | | Lambda | Functions | Cloud Functions | Serverless | | Fargate | Container Apps | Cloud Run | Managed containers | ### Storage Services | AWS | Azure | GCP | Use Case | | ------- | --------------- | --------------- | -------------- | | S3 | Blob Storage | Cloud Storage | Object storage | | EBS | Managed Disks | Persistent Disk | Block storage | | EFS | Azure Files | Filestore | File storage | | Glacier | Archive Storage | Archive Storage | Cold storage | ### Database Services | AWS | Azure | GCP | Use Case | | ----------- | ---------------- | ------------- | --------------- | | RDS | SQL Database | Cloud SQL | Managed SQL | | DynamoDB | Cosmos DB | Firestore | NoSQL | | Aurora | PostgreSQL/MySQL | Cloud Spanner | Distributed SQL | | ElastiCache | Cache for Redis | Memorystore | Caching | **Reference:** See `references/service-comparison.md` for complete comparison ## Multi-Cloud Patterns ### Pattern 1: Single Provider with DR - Primary workload in one cloud - Disaster recovery in another - Database replication across clouds - Automated failover ### Pattern 2: Best-of-Breed - Use best service from each provider - AI/ML on GCP - Enterprise apps on Azure - General compute on AWS ### Pattern 3: Geographic Distribution - Serve users from nearest cloud region - Data sovereignty compliance - Global load balancing - Regional failover ### Pattern 4: Cloud-Agnostic Abstraction - Kubernetes for compute - PostgreSQL for database - S3-compatible storage (MinIO) - Open source tools ## Cloud-Agnostic Architecture ### Use Cloud-Native Alternatives - **Compute:** Kubernetes (EKS/AKS/GKE) - **Database:** PostgreSQL/MySQL (RDS/SQL Database/Cloud SQL) - **Message Queue:** Apache Kafka (MSK/Event Hubs/Confluent) - **Cache:** Redis (ElastiCache/Azure Cache/Memorystore) - **Object Storage:** S3-compatible API - **Monitoring:** Prometheus/Grafana - **Service Mesh:** Istio/Linkerd ### Abstraction Layers ``` Application Layer ↓ Infrastructure Abstraction (Terraform) ↓ Cloud Provider APIs ↓ AWS / Azure / GCP ``` ## Cost Comparison ### Compute Pricing Factors - **AWS:** On-demand, Reserved, Spot, Savings Plans - **Azure:** Pay-as-you-go, Reserved, Spot - **GCP:** On-demand, Committed use, Preemptible ### Cost Optimization Strategies 1. Use reserved/committed capacity (30-70% savings) 2. Leverage spot/preemptible instances 3. Right-size resources 4. Use serverless for variable workloads 5. Optimize data transfer costs 6. Implement lifecycle policies 7. Use cost allocation tags 8. Monitor with cloud cost tools **Reference:** See `references/multi-cloud-patterns.md` ## Migration Strategy ### Phase 1: Assessment - Inventory current infrastructure - Identify dependencies - Assess cloud compatibility - Estimate costs ### Phase 2: Pilot - Select pilot workload - Implement in target cloud - Test thoroughly - Document learnings ### Phase 3: Migration - Migrate workloads incrementally - Maintain dual-run period - Monitor performance - Validate functionality ### Phase 4: Optimization - Right-size resources - Implement cloud-native services - Optimize costs - Enhance security ## Best Practices 1. **Use infrastructure as code** (Terraform/OpenTofu) 2. **Implement CI/CD pipelines** for deployments 3. **Design for failure** across clouds 4. **Use managed services** when possible 5. **Implement comprehensive monitoring** 6. **Automate cost optimization** 7. **Follow security best practices** 8. **Document cloud-specific configurations** 9. **Test disaster recovery** procedures 10. **Train teams** on multiple clouds ## Reference Files - `references/service-comparison.md` - Complete service comparison - `references/multi-cloud-patterns.md` - Architecture patterns ## Related Skills - `terraform-module-library` - For IaC implementation - `cost-optimization` - For cost management - `hybrid-cloud-networking` - For connectivity
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

data-quality-frameworks

Implement data quality validation with Great Expectations, dbt

data
⭐1
# Data Quality Frameworks Production patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines. ## When to Use This Skill - Implementing data quality checks in pipelines - Setting up Great Expectations validation - Building comprehensive dbt test suites - Establishing data contracts between teams - Monitoring data quality metrics - Automating data validation in CI/CD ## Core Concepts ### 1. Data Quality Dimensions | Dimension | Description | Example Check | | ---------------- | ------------------------ | -------------------------------------------------- | | **Completeness** | No missing values | `expect_column_values_to_not_be_null` | | **Uniqueness** | No duplicates | `expect_column_values_to_be_unique` | | **Validity** | Values in expected range | `expect_column_values_to_be_in_set` | | **Accuracy** | Data matches reality | Cross-reference validation | | **Consistency** | No contradictions | `expect_column_pair_values_A_to_be_greater_than_B` | | **Timeliness** | Data is recent | `expect_column_max_to_be_between` | ### 2. Testing Pyramid for Data ``` /\ / \ Integration Tests (cross-table) /────\ / \ Unit Tests (single column) /────────\ / \ Schema Tests (structure) /────────────\ ``` ## Quick Start ### Great Expectations Setup ```bash # Install pip install great_expectations # Initialize project great_expectations init # Create datasource great_expectations datasource new ``` ```python # great_expectations/checkpoints/daily_validation.yml import great_expectations as gx # Create context context = gx.get_context() # Create expectation suite suite = context.add_expectation_suite("orders_suite") # Add expectations suite.add_expectation( gx.expectations.ExpectColumnValuesToNotBeNull(column="order_id") ) suite.add_expectation( gx.expectations.ExpectColumnValuesToBeUnique(column="order_id") ) # Validate results = context.run_checkpoint(checkpoint_name="daily_orders") ``` ## Patterns ### Pattern 1: Great Expectations Suite ```python # expectations/orders_suite.py import great_expectations as gx from great_expectations.core import ExpectationSuite from great_expectations.core.expectation_configuration import ExpectationConfiguration def build_orders_suite() -> ExpectationSuite: """Build comprehensive orders expectation suite""" suite = ExpectationSuite(expectation_suite_name="orders_suite") # Schema expectations suite.add_expectation(ExpectationConfiguration( expectation_type="expect_table_columns_to_match_set", kwargs={ "column_set": ["order_id", "customer_id", "amount", "status", "created_at"], "exact_match": False # Allow additional columns } )) # Primary key suite.add_expectation(ExpectationConfiguration( expectation_type="expect_column_values_to_not_be_null", kwargs={"column": "order_id"} )) suite.add_expectation(ExpectationConfiguration( expectation_type="expect_column_values_to_be_unique", kwargs={"column": "order_id"} )) # Foreign key suite.add_expectation(ExpectationConfiguration( expectation_type="expect_column_values_to_not_be_null", kwargs={"column": "customer_id"} )) # Categorical values suite.add_expectation(ExpectationConfiguration( expectation_type="expect_column_values_to_be_in_set", kwargs={ "column": "status", "value_set": ["pending", "processing", "shipped", "delivered", "cancelled"] } )) # Numeric ranges suite.add_expectation(ExpectationConfiguration( expectation_type="expect_column_values_to_be_between", kwargs={ "column": "amount", "min_value": 0, "max_value": 100000, "strict_min": True # amount > 0 } )) # Date validity suite.add_expectation(ExpectationConfiguration( expectation_type="expect_column_values_to_be_dateutil_parseable", kwargs={"column": "created_at"} )) # Freshness - data should be recent suite.add_expectation(ExpectationConfiguration( expectation_type="expect_column_max_to_be_between", kwargs={ "column": "created_at", "min_value": {"$PARAMETER": "now - timedelta(days=1)"}, "max_value": {"$PARAMETER": "now"} } )) # Row count sanity suite.add_expectation(ExpectationConfiguration( expectation_type="expect_table_row_count_to_be_between", kwargs={ "min_value": 1000, # Expect at least 1000 rows "max_value": 10000000 } )) # Statistical expectations suite.add_expectation(ExpectationConfiguration( expectation_type="expect_column_mean_to_be_between", kwargs={ "column": "amount", "min_value": 50, "max_value": 500 } )) return suite ``` ### Pattern 2: Great Expectations Checkpoint ```yaml # great_expectations/checkpoints/orders_checkpoint.yml name: orders_checkpoint config_version: 1.0 class_name: Checkpoint run_name_template: "%Y%m%d-%H%M%S-orders-validation" validations: - batch_request: datasource_name: warehouse data_connector_name: default_inferred_data_connector_name data_asset_name: orders data_connector_query: index: -1 # Latest batch expectation_suite_name: orders_suite action_list: - name: store_validation_result action: class_name: StoreValidationResultAction - name: store_evaluation_parameters action: class_name: StoreEvaluationParametersAction - name: update_data_docs action: class_name: UpdateDataDocsAction # Slack notification on failure - name: send_slack_notification action: class_name: SlackNotificationAction slack_webhook: ${SLACK_WEBHOOK} notify_on: failure renderer: module_name: great_expectations.render.renderer.slack_renderer class_name: SlackRenderer ``` ```python # Run checkpoint import great_expectations as gx context = gx.get_context() result = context.run_checkpoint(checkpoint_name="orders_checkpoint") if not result.success: failed_expectations = [ r for r in result.run_results.values() if not r.success ] raise ValueError(f"Data quality check failed: {failed_expectations}") ``` ### Pattern 3: dbt Data Tests ```yaml # models/marts/core/_core__models.yml version: 2 models: - name: fct_orders description: Order fact table tests: # Table-level tests - dbt_utils.recency: datepart: day field: created_at interval: 1 - dbt_utils.at_least_one - dbt_utils.expression_is_true: expression: "total_amount >= 0" columns: - name: order_id description: Primary key tests: - unique - not_null - name: customer_id description: Foreign key to dim_customers tests: - not_null - relationships: to: ref('dim_customers') field: customer_id - name: order_status tests: - accepted_values: values: ["pending", "processing", "shipped", "delivered", "cancelled"] - name: total_amount tests: - not_null - dbt_utils.expression_is_true: expression: ">= 0" - name: created_at tests: - not_null - dbt_utils.expression_is_true: expression: "<= current_timestamp" - name: dim_customers columns: - name: customer_id tests: - unique - not_null - name: email tests: - unique - not_null # Custom regex test - dbt_utils.expression_is_true: expression: "email ~ '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}$'" ``` ### Pattern 4: Custom dbt Tests ```sql -- tests/generic/test_row_count_in_range.sql {% test row_count_in_range(model, min_count, max_count) %} with row_count as ( select count(*) as cnt from {{ model }} ) select cnt from row_count where cnt < {{ min_count }} or cnt > {{ max_count }} {% endtest %} -- Usage in schema.yml: -- tests: -- - row_count_in_range: -- min_count: 1000 -- max_count: 10000000 ``` ```sql -- tests/generic/test_sequential_values.sql {% test sequential_values(model, column_name, interval=1) %} with lagged as ( select {{ column_name }}, lag({{ column_name }}) over (order by {{ column_name }}) as prev_value from {{ model }} ) select * from lagged where {{ column_name }} - prev_value != {{ interval }} and prev_value is not null {% endtest %} ``` ```sql -- tests/singular/assert_orders_customers_match.sql -- Singular test: specific business rule with orders_customers as ( select distinct customer_id from {{ ref('fct_orders') }} ), dim_customers as ( select customer_id from {{ ref('dim_customers') }} ), orphaned_orders as ( select o.customer_id from orders_customers o left join dim_customers c using (customer_id) where c.customer_id is null ) select * from orphaned_orders -- Test passes if this returns 0 rows ``` ### Pattern 5: Data Contracts ```yaml # contracts/orders_contract.yaml apiVersion: datacontract.com/v1.0.0 kind: DataContract metadata: name: orders version: 1.0.0 owner: data-platform-team contact: data-team@company.com info: title: Orders Data Contract description: Contract for order event data from the ecommerce platform purpose: Analytics, reporting, and ML features servers: production: type: snowflake account: company.us-east-1 database: ANALYTICS schema: CORE terms: usage: Internal analytics only limitations: PII must not be exposed in downstream marts billing: Charged per query TB scanned schema: type: object properties: order_id: type: string format: uuid description: Unique order identifier required: true unique: true pii: false customer_id: type: string format: uuid description: Customer identifier required: true pii: true piiClassification: indirect total_amount: type: number minimum: 0 maximum: 100000 description: Order total in USD created_at: type: string format: date-time description: Order creation timestamp required: true status: type: string enum: [pending, processing, shipped, delivered, cancelled] description: Current order status quality: type: SodaCL specification: checks for orders: - row_count > 0 - missing_count(order_id) = 0 - duplicate_count(order_id) = 0 - invalid_count(status) = 0: valid values: [pending, processing, shipped, delivered, cancelled] - freshness(created_at) < 24h sla: availability: 99.9% freshness: 1 hour latency: 5 minutes ``` ### Pattern 6: Automated Quality Pipeline ```python # quality_pipeline.py from dataclasses import dataclass from typing import List, Dict, Any import great_expectations as gx from datetime import datetime @dataclass class QualityResult: table: str passed: bool total_expectations: int failed_expectations: int details: List[Dict[str, Any]] timestamp: datetime class DataQualityPipeline: """Orchestrate data quality checks across tables""" def __init__(self, context: gx.DataContext): self.context = context self.results: List[QualityResult] = [] def validate_table(self, table: str, suite: str) -> QualityResult: """Validate a single table against expectation suite""" checkpoint_config = { "name": f"{table}_validation", "config_version": 1.0, "class_name": "Checkpoint", "validations": [{ "batch_request": { "datasource_name": "warehouse", "data_asset_name": table, }, "expectation_suite_name": suite, }], } result = self.context.run_checkpoint(**checkpoint_config) # Parse results validation_result = list(result.run_results.values())[0] results = validation_result.results failed = [r for r in results if not r.success] return QualityResult( table=table, passed=result.success, total_expectations=len(results), failed_expectations=len(failed), details=[{ "expectation": r.expectation_config.expectation_type, "success": r.success, "observed_value": r.result.get("observed_value"), } for r in results], timestamp=datetime.now() ) def run_all(self, tables: Dict[str, str]) -> Dict[str, QualityResult]: """Run validation for all tables""" results = {} for table, suite in tables.items(): print(f"Validating {table}...") results[table] = self.validate_table(table, suite) return results def generate_report(self, results: Dict[str, QualityResult]) -> str: """Generate quality report""" report = ["# Data Quality Report", f"Generated: {datetime.now()}", ""] total_passed = sum(1 for r in results.values() if r.passed) total_tables = len(results) report.append(f"## Summary: {total_passed}/{total_tables} tables passed") report.append("") for table, result in results.items(): status = "āœ…" if result.passed else "āŒ" report.append(f"### {status} {table}") report.append(f"- Expectations: {result.total_expectations}") report.append(f"- Failed: {result.failed_expectations}") if not result.passed: report.append("- Failed checks:") for detail in result.details: if not detail["success"]: report.append(f" - {detail['expectation']}: {detail['observed_value']}") report.append("") return "\n".join(report) # Usage context = gx.get_context() pipeline = DataQualityPipeline(context) tables_to_validate = { "orders": "orders_suite", "customers": "customers_suite", "products": "products_suite", } results = pipeline.run_all(tables_to_validate) report = pipeline.generate_report(results) # Fail pipeline if any table failed if not all(r.passed for r in results.values()): print(report) raise ValueError("Data quality checks failed!") ``` ## Best Practices ### Do's - **Test early** - Validate source data before transformations - **Test incrementally** - Add tests as you find issues - **Document expectations** - Clear descriptions for each test - **Alert on failures** - Integrate with monitoring - **Version contracts** - Track schema changes ### Don'ts - **Don't test everything** - Focus on critical columns - **Don't ignore warnings** - They often precede failures - **Don't skip freshness** - Stale data is bad data - **Don't hardcode thresholds** - Use dynamic baselines - **Don't test in isolation** - Test relationships too ## Resources - [Great Expectations Documentation](https://docs.greatexpectations.io/) - [dbt Testing Documentation](https://docs.getdbt.com/docs/build/tests) - [Data Contract Specification](https://datacontract.com/) - [Soda Core](https://docs.soda.io/soda-core/overview.html)
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

monorepo-management

Master monorepo management with Turborepo, Nx, and pnpm workspaces

coding
⭐1
# Monorepo Management Build efficient, scalable monorepos that enable code sharing, consistent tooling, and atomic changes across multiple packages and applications. ## When to Use This Skill - Setting up new monorepo projects - Migrating from multi-repo to monorepo - Optimizing build and test performance - Managing shared dependencies - Implementing code sharing strategies - Setting up CI/CD for monorepos - Versioning and publishing packages - Debugging monorepo-specific issues ## Core Concepts ### 1. Why Monorepos? **Advantages:** - Shared code and dependencies - Atomic commits across projects - Consistent tooling and standards - Easier refactoring - Simplified dependency management - Better code visibility **Challenges:** - Build performance at scale - CI/CD complexity - Access control - Large Git repository ### 2. Monorepo Tools **Package Managers:** - pnpm workspaces (recommended) - npm workspaces - Yarn workspaces **Build Systems:** - Turborepo (recommended for most) - Nx (feature-rich, complex) - Lerna (older, maintenance mode) ## Turborepo Setup ### Initial Setup ```bash # Create new monorepo npx create-turbo@latest my-monorepo cd my-monorepo # Structure: # apps/ # web/ - Next.js app # docs/ - Documentation site # packages/ # ui/ - Shared UI components # config/ - Shared configurations # tsconfig/ - Shared TypeScript configs # turbo.json - Turborepo configuration # package.json - Root package.json ``` ### Configuration ```json // turbo.json { "$schema": "https://turbo.build/schema.json", "globalDependencies": ["**/.env.*local"], "pipeline": { "build": { "dependsOn": ["^build"], "outputs": ["dist/**", ".next/**", "!.next/cache/**"] }, "test": { "dependsOn": ["build"], "outputs": ["coverage/**"] }, "lint": { "outputs": [] }, "dev": { "cache": false, "persistent": true }, "type-check": { "dependsOn": ["^build"], "outputs": [] } } } ``` ```json // package.json (root) { "name": "my-monorepo", "private": true, "workspaces": ["apps/*", "packages/*"], "scripts": { "build": "turbo run build", "dev": "turbo run dev", "test": "turbo run test", "lint": "turbo run lint", "format": "prettier --write \"**/*.{ts,tsx,md}\"", "clean": "turbo run clean && rm -rf node_modules" }, "devDependencies": { "turbo": "^1.10.0", "prettier": "^3.0.0", "typescript": "^5.0.0" }, "packageManager": "pnpm@8.0.0" } ``` ### Package Structure ```json // packages/ui/package.json { "name": "@repo/ui", "version": "0.0.0", "private": true, "main": "./dist/index.js", "types": "./dist/index.d.ts", "exports": { ".": { "import": "./dist/index.js", "types": "./dist/index.d.ts" }, "./button": { "import": "./dist/button.js", "types": "./dist/button.d.ts" } }, "scripts": { "build": "tsup src/index.ts --format esm,cjs --dts", "dev": "tsup src/index.ts --format esm,cjs --dts --watch", "lint": "eslint src/", "type-check": "tsc --noEmit" }, "devDependencies": { "@repo/tsconfig": "workspace:*", "tsup": "^7.0.0", "typescript": "^5.0.0" }, "dependencies": { "react": "^18.2.0" } } ``` ## pnpm Workspaces ### Setup ```yaml # pnpm-workspace.yaml packages: - "apps/*" - "packages/*" - "tools/*" ``` ```json // .npmrc # Hoist shared dependencies shamefully-hoist=true # Strict peer dependencies auto-install-peers=true strict-peer-dependencies=true # Performance store-dir=~/.pnpm-store ``` ### Dependency Management ```bash # Install dependency in specific package pnpm add react --filter @repo/ui pnpm add -D typescript --filter @repo/ui # Install workspace dependency pnpm add @repo/ui --filter web # Install in all packages pnpm add -D eslint -w # Update all dependencies pnpm update -r # Remove dependency pnpm remove react --filter @repo/ui ``` ### Scripts ```bash # Run script in specific package pnpm --filter web dev pnpm --filter @repo/ui build # Run in all packages pnpm -r build pnpm -r test # Run in parallel pnpm -r --parallel dev # Filter by pattern pnpm --filter "@repo/*" build pnpm --filter "...web" build # Build web and dependencies ``` ## Nx Monorepo ### Setup ```bash # Create Nx monorepo npx create-nx-workspace@latest my-org # Generate applications nx generate @nx/react:app my-app nx generate @nx/next:app my-next-app # Generate libraries nx generate @nx/react:lib ui-components nx generate @nx/js:lib utils ``` ### Configuration ```json // nx.json { "extends": "nx/presets/npm.json", "$schema": "./node_modules/nx/schemas/nx-schema.json", "targetDefaults": { "build": { "dependsOn": ["^build"], "inputs": ["production", "^production"], "cache": true }, "test": { "inputs": ["default", "^production", "{workspaceRoot}/jest.preset.js"], "cache": true }, "lint": { "inputs": ["default", "{workspaceRoot}/.eslintrc.json"], "cache": true } }, "namedInputs": { "default": ["{projectRoot}/**/*", "sharedGlobals"], "production": [ "default", "!{projectRoot}/**/?(*.)+(spec|test).[jt]s?(x)?(.snap)", "!{projectRoot}/tsconfig.spec.json" ], "sharedGlobals": [] } } ``` ### Running Tasks ```bash # Run task for specific project nx build my-app nx test ui-components nx lint utils # Run for affected projects nx affected:build nx affected:test --base=main # Visualize dependencies nx graph # Run in parallel nx run-many --target=build --all --parallel=3 ``` ## Shared Configurations ### TypeScript Configuration ```json // packages/tsconfig/base.json { "compilerOptions": { "strict": true, "esModuleInterop": true, "skipLibCheck": true, "forceConsistentCasingInFileNames": true, "module": "ESNext", "moduleResolution": "bundler", "resolveJsonModule": true, "isolatedModules": true, "incremental": true, "declaration": true }, "exclude": ["node_modules"] } // packages/tsconfig/react.json { "extends": "./base.json", "compilerOptions": { "jsx": "react-jsx", "lib": ["ES2022", "DOM", "DOM.Iterable"] } } // apps/web/tsconfig.json { "extends": "@repo/tsconfig/react.json", "compilerOptions": { "outDir": "dist", "rootDir": "src" }, "include": ["src"], "exclude": ["node_modules", "dist"] } ``` ### ESLint Configuration ```javascript // packages/config/eslint-preset.js module.exports = { extends: [ "eslint:recommended", "plugin:@typescript-eslint/recommended", "plugin:react/recommended", "plugin:react-hooks/recommended", "prettier", ], plugins: ["@typescript-eslint", "react", "react-hooks"], parser: "@typescript-eslint/parser", parserOptions: { ecmaVersion: 2022, sourceType: "module", ecmaFeatures: { jsx: true, }, }, settings: { react: { version: "detect", }, }, rules: { "@typescript-eslint/no-unused-vars": "error", "react/react-in-jsx-scope": "off", }, }; // apps/web/.eslintrc.js module.exports = { extends: ["@repo/config/eslint-preset"], rules: { // App-specific rules }, }; ``` ## Code Sharing Patterns ### Pattern 1: Shared UI Components ```typescript // packages/ui/src/button.tsx import * as React from 'react'; export interface ButtonProps { variant?: 'primary' | 'secondary'; children: React.ReactNode; onClick?: () => void; } export function Button({ variant = 'primary', children, onClick }: ButtonProps) { return ( <button className={`btn btn-${variant}`} onClick={onClick} > {children} </button> ); } // packages/ui/src/index.ts export { Button, type ButtonProps } from './button'; export { Input, type InputProps } from './input'; // apps/web/src/app.tsx import { Button } from '@repo/ui'; export function App() { return <Button variant="primary">Click me</Button>; } ``` ### Pattern 2: Shared Utilities ```typescript // packages/utils/src/string.ts export function capitalize(str: string): string { return str.charAt(0).toUpperCase() + str.slice(1); } export function truncate(str: string, length: number): string { return str.length > length ? str.slice(0, length) + "..." : str; } // packages/utils/src/index.ts export * from "./string"; export * from "./array"; export * from "./date"; // Usage in apps import { capitalize, truncate } from "@repo/utils"; ``` ### Pattern 3: Shared Types ```typescript // packages/types/src/user.ts export interface User { id: string; email: string; name: string; role: "admin" | "user"; } export interface CreateUserInput { email: string; name: string; password: string; } // Used in both frontend and backend import type { User, CreateUserInput } from "@repo/types"; ``` ## Build Optimization ### Turborepo Caching ```json // turbo.json { "pipeline": { "build": { // Build depends on dependencies being built first "dependsOn": ["^build"], // Cache these outputs "outputs": ["dist/**", ".next/**"], // Cache based on these inputs (default: all files) "inputs": ["src/**/*.tsx", "src/**/*.ts", "package.json"] }, "test": { // Run tests in parallel, don't depend on build "cache": true, "outputs": ["coverage/**"] } } } ``` ### Remote Caching ```bash # Turborepo Remote Cache (Vercel) npx turbo login npx turbo link # Custom remote cache # turbo.json { "remoteCache": { "signature": true, "enabled": true } } ``` ## CI/CD for Monorepos ### GitHub Actions ```yaml # .github/workflows/ci.yml name: CI on: push: branches: [main] pull_request: branches: [main] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 with: fetch-depth: 0 # For Nx affected commands - uses: pnpm/action-setup@v2 with: version: 8 - uses: actions/setup-node@v3 with: node-version: 18 cache: "pnpm" - name: Install dependencies run: pnpm install --frozen-lockfile - name: Build run: pnpm turbo run build - name: Test run: pnpm turbo run test - name: Lint run: pnpm turbo run lint - name: Type check run: pnpm turbo run type-check ``` ### Deploy Affected Only ```yaml # Deploy only changed apps - name: Deploy affected apps run: | if pnpm nx affected:apps --base=origin/main --head=HEAD | grep -q "web"; then echo "Deploying web app" pnpm --filter web deploy fi ``` ## Best Practices 1. **Consistent Versioning**: Lock dependency versions across workspace 2. **Shared Configs**: Centralize ESLint, TypeScript, Prettier configs 3. **Dependency Graph**: Keep it acyclic, avoid circular dependencies 4. **Cache Effectively**: Configure inputs/outputs correctly 5. **Type Safety**: Share types between frontend/backend 6. **Testing Strategy**: Unit tests in packages, E2E in apps 7. **Documentation**: README in each package 8. **Release Strategy**: Use changesets for versioning ## Common Pitfalls - **Circular Dependencies**: A depends on B, B depends on A - **Phantom Dependencies**: Using deps not in package.json - **Incorrect Cache Inputs**: Missing files in Turborepo inputs - **Over-Sharing**: Sharing code that should be separate - **Under-Sharing**: Duplicating code across packages - **Large Monorepos**: Without proper tooling, builds slow down ## Publishing Packages ```bash # Using Changesets pnpm add -Dw @changesets/cli pnpm changeset init # Create changeset pnpm changeset # Version packages pnpm changeset version # Publish pnpm changeset publish ``` ```yaml # .github/workflows/release.yml - name: Create Release Pull Request or Publish uses: changesets/action@v1 with: publish: pnpm release env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} NPM_TOKEN: ${{ secrets.NPM_TOKEN }} ``` ## Resources - **references/turborepo-guide.md**: Comprehensive Turborepo documentation - **references/nx-guide.md**: Nx monorepo patterns - **references/pnpm-workspaces.md**: pnpm workspace features - **assets/monorepo-checklist.md**: Setup checklist - **assets/migration-guide.md**: Multi-repo to monorepo migration - **scripts/dependency-graph.ts**: Visualize package dependencies
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

architecture-decision-records

Write and maintain Architecture Decision Records (ADRs) following

coding
⭐1
# Architecture Decision Records Comprehensive patterns for creating, maintaining, and managing Architecture Decision Records (ADRs) that capture the context and rationale behind significant technical decisions. ## When to Use This Skill - Making significant architectural decisions - Documenting technology choices - Recording design trade-offs - Onboarding new team members - Reviewing historical decisions - Establishing decision-making processes ## Core Concepts ### 1. What is an ADR? An Architecture Decision Record captures: - **Context**: Why we needed to make a decision - **Decision**: What we decided - **Consequences**: What happens as a result ### 2. When to Write an ADR | Write ADR | Skip ADR | | -------------------------- | ---------------------- | | New framework adoption | Minor version upgrades | | Database technology choice | Bug fixes | | API design patterns | Implementation details | | Security architecture | Routine maintenance | | Integration patterns | Configuration changes | ### 3. ADR Lifecycle ``` Proposed → Accepted → Deprecated → Superseded ↓ Rejected ``` ## Templates ### Template 1: Standard ADR (MADR Format) ```markdown # ADR-0001: Use PostgreSQL as Primary Database ## Status Accepted ## Context We need to select a primary database for our new e-commerce platform. The system will handle: - ~10,000 concurrent users - Complex product catalog with hierarchical categories - Transaction processing for orders and payments - Full-text search for products - Geospatial queries for store locator The team has experience with MySQL, PostgreSQL, and MongoDB. We need ACID compliance for financial transactions. ## Decision Drivers - **Must have ACID compliance** for payment processing - **Must support complex queries** for reporting - **Should support full-text search** to reduce infrastructure complexity - **Should have good JSON support** for flexible product attributes - **Team familiarity** reduces onboarding time ## Considered Options ### Option 1: PostgreSQL - **Pros**: ACID compliant, excellent JSON support (JSONB), built-in full-text search, PostGIS for geospatial, team has experience - **Cons**: Slightly more complex replication setup than MySQL ### Option 2: MySQL - **Pros**: Very familiar to team, simple replication, large community - **Cons**: Weaker JSON support, no built-in full-text search (need Elasticsearch), no geospatial without extensions ### Option 3: MongoDB - **Pros**: Flexible schema, native JSON, horizontal scaling - **Cons**: No ACID for multi-document transactions (at decision time), team has limited experience, requires schema design discipline ## Decision We will use **PostgreSQL 15** as our primary database. ## Rationale PostgreSQL provides the best balance of: 1. **ACID compliance** essential for e-commerce transactions 2. **Built-in capabilities** (full-text search, JSONB, PostGIS) reduce infrastructure complexity 3. **Team familiarity** with SQL databases reduces learning curve 4. **Mature ecosystem** with excellent tooling and community support The slight complexity in replication is outweighed by the reduction in additional services (no separate Elasticsearch needed). ## Consequences ### Positive - Single database handles transactions, search, and geospatial queries - Reduced operational complexity (fewer services to manage) - Strong consistency guarantees for financial data - Team can leverage existing SQL expertise ### Negative - Need to learn PostgreSQL-specific features (JSONB, full-text search syntax) - Vertical scaling limits may require read replicas sooner - Some team members need PostgreSQL-specific training ### Risks - Full-text search may not scale as well as dedicated search engines - Mitigation: Design for potential Elasticsearch addition if needed ## Implementation Notes - Use JSONB for flexible product attributes - Implement connection pooling with PgBouncer - Set up streaming replication for read replicas - Use pg_trgm extension for fuzzy search ## Related Decisions - ADR-0002: Caching Strategy (Redis) - complements database choice - ADR-0005: Search Architecture - may supersede if Elasticsearch needed ## References - [PostgreSQL JSON Documentation](https://www.postgresql.org/docs/current/datatype-json.html) - [PostgreSQL Full Text Search](https://www.postgresql.org/docs/current/textsearch.html) - Internal: Performance benchmarks in `/docs/benchmarks/database-comparison.md` ``` ### Template 2: Lightweight ADR ```markdown # ADR-0012: Adopt TypeScript for Frontend Development **Status**: Accepted **Date**: 2024-01-15 **Deciders**: @alice, @bob, @charlie ## Context Our React codebase has grown to 50+ components with increasing bug reports related to prop type mismatches and undefined errors. PropTypes provide runtime-only checking. ## Decision Adopt TypeScript for all new frontend code. Migrate existing code incrementally. ## Consequences **Good**: Catch type errors at compile time, better IDE support, self-documenting code. **Bad**: Learning curve for team, initial slowdown, build complexity increase. **Mitigations**: TypeScript training sessions, allow gradual adoption with `allowJs: true`. ``` ### Template 3: Y-Statement Format ```markdown # ADR-0015: API Gateway Selection In the context of **building a microservices architecture**, facing **the need for centralized API management, authentication, and rate limiting**, we decided for **Kong Gateway** and against **AWS API Gateway and custom Nginx solution**, to achieve **vendor independence, plugin extensibility, and team familiarity with Lua**, accepting that **we need to manage Kong infrastructure ourselves**. ``` ### Template 4: ADR for Deprecation ```markdown # ADR-0020: Deprecate MongoDB in Favor of PostgreSQL ## Status Accepted (Supersedes ADR-0003) ## Context ADR-0003 (2021) chose MongoDB for user profile storage due to schema flexibility needs. Since then: - MongoDB's multi-document transactions remain problematic for our use case - Our schema has stabilized and rarely changes - We now have PostgreSQL expertise from other services - Maintaining two databases increases operational burden ## Decision Deprecate MongoDB and migrate user profiles to PostgreSQL. ## Migration Plan 1. **Phase 1** (Week 1-2): Create PostgreSQL schema, dual-write enabled 2. **Phase 2** (Week 3-4): Backfill historical data, validate consistency 3. **Phase 3** (Week 5): Switch reads to PostgreSQL, monitor 4. **Phase 4** (Week 6): Remove MongoDB writes, decommission ## Consequences ### Positive - Single database technology reduces operational complexity - ACID transactions for user data - Team can focus PostgreSQL expertise ### Negative - Migration effort (~4 weeks) - Risk of data issues during migration - Lose some schema flexibility ## Lessons Learned Document from ADR-0003 experience: - Schema flexibility benefits were overestimated - Operational cost of multiple databases was underestimated - Consider long-term maintenance in technology decisions ``` ### Template 5: Request for Comments (RFC) Style ```markdown # RFC-0025: Adopt Event Sourcing for Order Management ## Summary Propose adopting event sourcing pattern for the order management domain to improve auditability, enable temporal queries, and support business analytics. ## Motivation Current challenges: 1. Audit requirements need complete order history 2. "What was the order state at time X?" queries are impossible 3. Analytics team needs event stream for real-time dashboards 4. Order state reconstruction for customer support is manual ## Detailed Design ### Event Store ``` OrderCreated { orderId, customerId, items[], timestamp } OrderItemAdded { orderId, item, timestamp } OrderItemRemoved { orderId, itemId, timestamp } PaymentReceived { orderId, amount, paymentId, timestamp } OrderShipped { orderId, trackingNumber, timestamp } ``` ### Projections - **CurrentOrderState**: Materialized view for queries - **OrderHistory**: Complete timeline for audit - **DailyOrderMetrics**: Analytics aggregation ### Technology - Event Store: EventStoreDB (purpose-built, handles projections) - Alternative considered: Kafka + custom projection service ## Drawbacks - Learning curve for team - Increased complexity vs. CRUD - Need to design events carefully (immutable once stored) - Storage growth (events never deleted) ## Alternatives 1. **Audit tables**: Simpler but doesn't enable temporal queries 2. **CDC from existing DB**: Complex, doesn't change data model 3. **Hybrid**: Event source only for order state changes ## Unresolved Questions - [ ] Event schema versioning strategy - [ ] Retention policy for events - [ ] Snapshot frequency for performance ## Implementation Plan 1. Prototype with single order type (2 weeks) 2. Team training on event sourcing (1 week) 3. Full implementation and migration (4 weeks) 4. Monitoring and optimization (ongoing) ## References - [Event Sourcing by Martin Fowler](https://martinfowler.com/eaaDev/EventSourcing.html) - [EventStoreDB Documentation](https://www.eventstore.com/docs) ``` ## ADR Management ### Directory Structure ``` docs/ ā”œā”€ā”€ adr/ │ ā”œā”€ā”€ README.md # Index and guidelines │ ā”œā”€ā”€ template.md # Team's ADR template │ ā”œā”€ā”€ 0001-use-postgresql.md │ ā”œā”€ā”€ 0002-caching-strategy.md │ ā”œā”€ā”€ 0003-mongodb-user-profiles.md # [DEPRECATED] │ └── 0020-deprecate-mongodb.md # Supersedes 0003 ``` ### ADR Index (README.md) ```markdown # Architecture Decision Records This directory contains Architecture Decision Records (ADRs) for [Project Name]. ## Index | ADR | Title | Status | Date | | ------------------------------------- | ---------------------------------- | ---------- | ---------- | | [0001](0001-use-postgresql.md) | Use PostgreSQL as Primary Database | Accepted | 2024-01-10 | | [0002](0002-caching-strategy.md) | Caching Strategy with Redis | Accepted | 2024-01-12 | | [0003](0003-mongodb-user-profiles.md) | MongoDB for User Profiles | Deprecated | 2023-06-15 | | [0020](0020-deprecate-mongodb.md) | Deprecate MongoDB | Accepted | 2024-01-15 | ## Creating a New ADR 1. Copy `template.md` to `NNNN-title-with-dashes.md` 2. Fill in the template 3. Submit PR for review 4. Update this index after approval ## ADR Status - **Proposed**: Under discussion - **Accepted**: Decision made, implementing - **Deprecated**: No longer relevant - **Superseded**: Replaced by another ADR - **Rejected**: Considered but not adopted ``` ### Automation (adr-tools) ```bash # Install adr-tools brew install adr-tools # Initialize ADR directory adr init docs/adr # Create new ADR adr new "Use PostgreSQL as Primary Database" # Supersede an ADR adr new -s 3 "Deprecate MongoDB in Favor of PostgreSQL" # Generate table of contents adr generate toc > docs/adr/README.md # Link related ADRs adr link 2 "Complements" 1 "Is complemented by" ``` ## Review Process ```markdown ## ADR Review Checklist ### Before Submission - [ ] Context clearly explains the problem - [ ] All viable options considered - [ ] Pros/cons balanced and honest - [ ] Consequences (positive and negative) documented - [ ] Related ADRs linked ### During Review - [ ] At least 2 senior engineers reviewed - [ ] Affected teams consulted - [ ] Security implications considered - [ ] Cost implications documented - [ ] Reversibility assessed ### After Acceptance - [ ] ADR index updated - [ ] Team notified - [ ] Implementation tickets created - [ ] Related documentation updated ``` ## Best Practices ### Do's - **Write ADRs early** - Before implementation starts - **Keep them short** - 1-2 pages maximum - **Be honest about trade-offs** - Include real cons - **Link related decisions** - Build decision graph - **Update status** - Deprecate when superseded ### Don'ts - **Don't change accepted ADRs** - Write new ones to supersede - **Don't skip context** - Future readers need background - **Don't hide failures** - Rejected decisions are valuable - **Don't be vague** - Specific decisions, specific consequences - **Don't forget implementation** - ADR without action is waste ## Resources - [Documenting Architecture Decisions (Michael Nygard)](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions) - [MADR Template](https://adr.github.io/madr/) - [ADR GitHub Organization](https://adr.github.io/) - [adr-tools](https://github.com/npryce/adr-tools)
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

changelog-automation

Automate changelog generation from commits, PRs, and releases

coding
⭐1
# Changelog Automation Patterns and tools for automating changelog generation, release notes, and version management following industry standards. ## When to Use This Skill - Setting up automated changelog generation - Implementing Conventional Commits - Creating release note workflows - Standardizing commit message formats - Generating GitHub/GitLab release notes - Managing semantic versioning ## Core Concepts ### 1. Keep a Changelog Format ```markdown # Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] ### Added - New feature X ## [1.2.0] - 2024-01-15 ### Added - User profile avatars - Dark mode support ### Changed - Improved loading performance by 40% ### Deprecated - Old authentication API (use v2) ### Removed - Legacy payment gateway ### Fixed - Login timeout issue (#123) ### Security - Updated dependencies for CVE-2024-1234 [Unreleased]: https://github.com/user/repo/compare/v1.2.0...HEAD [1.2.0]: https://github.com/user/repo/compare/v1.1.0...v1.2.0 ``` ### 2. Conventional Commits ``` <type>[optional scope]: <description> [optional body] [optional footer(s)] ``` | Type | Description | Changelog Section | | ---------- | ---------------- | ------------------ | | `feat` | New feature | Added | | `fix` | Bug fix | Fixed | | `docs` | Documentation | (usually excluded) | | `style` | Formatting | (usually excluded) | | `refactor` | Code restructure | Changed | | `perf` | Performance | Changed | | `test` | Tests | (usually excluded) | | `chore` | Maintenance | (usually excluded) | | `ci` | CI changes | (usually excluded) | | `build` | Build system | (usually excluded) | | `revert` | Revert commit | Removed | ### 3. Semantic Versioning ``` MAJOR.MINOR.PATCH MAJOR: Breaking changes (feat! or BREAKING CHANGE) MINOR: New features (feat) PATCH: Bug fixes (fix) ``` ## Implementation ### Method 1: Conventional Changelog (Node.js) ```bash # Install tools npm install -D @commitlint/cli @commitlint/config-conventional npm install -D husky npm install -D standard-version # or npm install -D semantic-release # Setup commitlint cat > commitlint.config.js << 'EOF' module.exports = { extends: ['@commitlint/config-conventional'], rules: { 'type-enum': [ 2, 'always', [ 'feat', 'fix', 'docs', 'style', 'refactor', 'perf', 'test', 'chore', 'ci', 'build', 'revert', ], ], 'subject-case': [2, 'never', ['start-case', 'pascal-case', 'upper-case']], 'subject-max-length': [2, 'always', 72], }, }; EOF # Setup husky npx husky init echo "npx --no -- commitlint --edit \$1" > .husky/commit-msg ``` ### Method 2: standard-version Configuration ```javascript // .versionrc.js module.exports = { types: [ { type: "feat", section: "Features" }, { type: "fix", section: "Bug Fixes" }, { type: "perf", section: "Performance Improvements" }, { type: "revert", section: "Reverts" }, { type: "docs", section: "Documentation", hidden: true }, { type: "style", section: "Styles", hidden: true }, { type: "chore", section: "Miscellaneous", hidden: true }, { type: "refactor", section: "Code Refactoring", hidden: true }, { type: "test", section: "Tests", hidden: true }, { type: "build", section: "Build System", hidden: true }, { type: "ci", section: "CI/CD", hidden: true }, ], commitUrlFormat: "{{host}}/{{owner}}/{{repository}}/commit/{{hash}}", compareUrlFormat: "{{host}}/{{owner}}/{{repository}}/compare/{{previousTag}}...{{currentTag}}", issueUrlFormat: "{{host}}/{{owner}}/{{repository}}/issues/{{id}}", userUrlFormat: "{{host}}/{{user}}", releaseCommitMessageFormat: "chore(release): {{currentTag}}", scripts: { prebump: 'echo "Running prebump"', postbump: 'echo "Running postbump"', prechangelog: 'echo "Running prechangelog"', postchangelog: 'echo "Running postchangelog"', }, }; ``` ```json // package.json scripts { "scripts": { "release": "standard-version", "release:minor": "standard-version --release-as minor", "release:major": "standard-version --release-as major", "release:patch": "standard-version --release-as patch", "release:dry": "standard-version --dry-run" } } ``` ### Method 3: semantic-release (Full Automation) ```javascript // release.config.js module.exports = { branches: [ "main", { name: "beta", prerelease: true }, { name: "alpha", prerelease: true }, ], plugins: [ "@semantic-release/commit-analyzer", "@semantic-release/release-notes-generator", [ "@semantic-release/changelog", { changelogFile: "CHANGELOG.md", }, ], [ "@semantic-release/npm", { npmPublish: true, }, ], [ "@semantic-release/github", { assets: ["dist/**/*.js", "dist/**/*.css"], }, ], [ "@semantic-release/git", { assets: ["CHANGELOG.md", "package.json"], message: "chore(release): ${nextRelease.version} [skip ci]\n\n${nextRelease.notes}", }, ], ], }; ``` ### Method 4: GitHub Actions Workflow ```yaml # .github/workflows/release.yml name: Release on: push: branches: [main] workflow_dispatch: inputs: release_type: description: "Release type" required: true default: "patch" type: choice options: - patch - minor - major permissions: contents: write pull-requests: write jobs: release: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 token: ${{ secrets.GITHUB_TOKEN }} - uses: actions/setup-node@v4 with: node-version: "20" cache: "npm" - run: npm ci - name: Configure Git run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" - name: Run semantic-release env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} NPM_TOKEN: ${{ secrets.NPM_TOKEN }} run: npx semantic-release # Alternative: manual release with standard-version manual-release: if: github.event_name == 'workflow_dispatch' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: actions/setup-node@v4 with: node-version: "20" - run: npm ci - name: Configure Git run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" - name: Bump version and generate changelog run: npx standard-version --release-as ${{ inputs.release_type }} - name: Push changes run: git push --follow-tags origin main - name: Create GitHub Release uses: softprops/action-gh-release@v1 with: tag_name: ${{ steps.version.outputs.tag }} body_path: RELEASE_NOTES.md generate_release_notes: true ``` ### Method 5: git-cliff (Rust-based, Fast) ```toml # cliff.toml [changelog] header = """ # Changelog All notable changes to this project will be documented in this file. """ body = """ {% if version %}\ ## [{{ version | trim_start_matches(pat="v") }}] - {{ timestamp | date(format="%Y-%m-%d") }} {% else %}\ ## [Unreleased] {% endif %}\ {% for group, commits in commits | group_by(attribute="group") %} ### {{ group | upper_first }} {% for commit in commits %} - {% if commit.scope %}**{{ commit.scope }}:** {% endif %}\ {{ commit.message | upper_first }}\ {% if commit.github.pr_number %} ([#{{ commit.github.pr_number }}](https://github.com/owner/repo/pull/{{ commit.github.pr_number }})){% endif %}\ {% endfor %} {% endfor %} """ footer = """ {% for release in releases -%} {% if release.version -%} {% if release.previous.version -%} [{{ release.version | trim_start_matches(pat="v") }}]: \ https://github.com/owner/repo/compare/{{ release.previous.version }}...{{ release.version }} {% endif -%} {% else -%} [unreleased]: https://github.com/owner/repo/compare/{{ release.previous.version }}...HEAD {% endif -%} {% endfor %} """ trim = true [git] conventional_commits = true filter_unconventional = true split_commits = false commit_parsers = [ { message = "^feat", group = "Features" }, { message = "^fix", group = "Bug Fixes" }, { message = "^doc", group = "Documentation" }, { message = "^perf", group = "Performance" }, { message = "^refactor", group = "Refactoring" }, { message = "^style", group = "Styling" }, { message = "^test", group = "Testing" }, { message = "^chore\\(release\\)", skip = true }, { message = "^chore", group = "Miscellaneous" }, ] filter_commits = false tag_pattern = "v[0-9]*" skip_tags = "" ignore_tags = "" topo_order = false sort_commits = "oldest" [github] owner = "owner" repo = "repo" ``` ```bash # Generate changelog git cliff -o CHANGELOG.md # Generate for specific range git cliff v1.0.0..v2.0.0 -o RELEASE_NOTES.md # Preview without writing git cliff --unreleased --dry-run ``` ### Method 6: Python (commitizen) ```toml # pyproject.toml [tool.commitizen] name = "cz_conventional_commits" version = "1.0.0" version_files = [ "pyproject.toml:version", "src/__init__.py:__version__", ] tag_format = "v$version" update_changelog_on_bump = true changelog_incremental = true changelog_start_rev = "v0.1.0" [tool.commitizen.customize] message_template = "{{change_type}}{% if scope %}({{scope}}){% endif %}: {{message}}" schema = "<type>(<scope>): <subject>" schema_pattern = "^(feat|fix|docs|style|refactor|perf|test|chore)(\\(\\w+\\))?:\\s.*" bump_pattern = "^(feat|fix|perf|refactor)" bump_map = {"feat" = "MINOR", "fix" = "PATCH", "perf" = "PATCH", "refactor" = "PATCH"} ``` ```bash # Install pip install commitizen # Create commit interactively cz commit # Bump version and update changelog cz bump --changelog # Check commits cz check --rev-range HEAD~5..HEAD ``` ## Release Notes Templates ### GitHub Release Template ```markdown ## What's Changed ### šŸš€ Features {{ range .Features }} - {{ .Title }} by @{{ .Author }} in #{{ .PR }} {{ end }} ### šŸ› Bug Fixes {{ range .Fixes }} - {{ .Title }} by @{{ .Author }} in #{{ .PR }} {{ end }} ### šŸ“š Documentation {{ range .Docs }} - {{ .Title }} by @{{ .Author }} in #{{ .PR }} {{ end }} ### šŸ”§ Maintenance {{ range .Chores }} - {{ .Title }} by @{{ .Author }} in #{{ .PR }} {{ end }} ## New Contributors {{ range .NewContributors }} - @{{ .Username }} made their first contribution in #{{ .PR }} {{ end }} **Full Changelog**: https://github.com/owner/repo/compare/v{{ .Previous }}...v{{ .Current }} ``` ### Internal Release Notes ```markdown # Release v2.1.0 - January 15, 2024 ## Summary This release introduces dark mode support and improves checkout performance by 40%. It also includes important security updates. ## Highlights ### šŸŒ™ Dark Mode Users can now switch to dark mode from settings. The preference is automatically saved and synced across devices. ### ⚔ Performance - Checkout flow is 40% faster - Reduced bundle size by 15% ## Breaking Changes None in this release. ## Upgrade Guide No special steps required. Standard deployment process applies. ## Known Issues - Dark mode may flicker on initial load (fix scheduled for v2.1.1) ## Dependencies Updated | Package | From | To | Reason | | ------- | ------- | ------- | ------------------------ | | react | 18.2.0 | 18.3.0 | Performance improvements | | lodash | 4.17.20 | 4.17.21 | Security patch | ``` ## Commit Message Examples ```bash # Feature with scope feat(auth): add OAuth2 support for Google login # Bug fix with issue reference fix(checkout): resolve race condition in payment processing Closes #123 # Breaking change feat(api)!: change user endpoint response format BREAKING CHANGE: The user endpoint now returns `userId` instead of `id`. Migration guide: Update all API consumers to use the new field name. # Multiple paragraphs fix(database): handle connection timeouts gracefully Previously, connection timeouts would cause the entire request to fail without retry. This change implements exponential backoff with up to 3 retries before failing. The timeout threshold has been increased from 5s to 10s based on p99 latency analysis. Fixes #456 Reviewed-by: @alice ``` ## Best Practices ### Do's - **Follow Conventional Commits** - Enables automation - **Write clear messages** - Future you will thank you - **Reference issues** - Link commits to tickets - **Use scopes consistently** - Define team conventions - **Automate releases** - Reduce manual errors ### Don'ts - **Don't mix changes** - One logical change per commit - **Don't skip validation** - Use commitlint - **Don't manual edit** - Generated changelogs only - **Don't forget breaking changes** - Mark with `!` or footer - **Don't ignore CI** - Validate commits in pipeline ## Resources - [Keep a Changelog](https://keepachangelog.com/) - [Conventional Commits](https://www.conventionalcommits.org/) - [Semantic Versioning](https://semver.org/) - [semantic-release](https://semantic-release.gitbook.io/) - [git-cliff](https://git-cliff.org/)
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

dependency-upgrade

Manage major dependency version upgrades with compatibility

coding
⭐1
# Dependency Upgrade Master major dependency version upgrades, compatibility analysis, staged upgrade strategies, and comprehensive testing approaches. ## When to Use This Skill - Upgrading major framework versions - Updating security-vulnerable dependencies - Modernizing legacy dependencies - Resolving dependency conflicts - Planning incremental upgrade paths - Testing compatibility matrices - Automating dependency updates ## Semantic Versioning Review ``` MAJOR.MINOR.PATCH (e.g., 2.3.1) MAJOR: Breaking changes MINOR: New features, backward compatible PATCH: Bug fixes, backward compatible ^2.3.1 = >=2.3.1 <3.0.0 (minor updates) ~2.3.1 = >=2.3.1 <2.4.0 (patch updates) 2.3.1 = exact version ``` ## Dependency Analysis ### Audit Dependencies ```bash # npm npm outdated npm audit npm audit fix # yarn yarn outdated yarn audit # Check for major updates npx npm-check-updates npx npm-check-updates -u # Update package.json ``` ### Analyze Dependency Tree ```bash # See why a package is installed npm ls package-name yarn why package-name # Find duplicate packages npm dedupe yarn dedupe # Visualize dependencies npx madge --image graph.png src/ ``` ## Compatibility Matrix ```javascript // compatibility-matrix.js const compatibilityMatrix = { react: { "16.x": { "react-dom": "^16.0.0", "react-router-dom": "^5.0.0", "@testing-library/react": "^11.0.0", }, "17.x": { "react-dom": "^17.0.0", "react-router-dom": "^5.0.0 || ^6.0.0", "@testing-library/react": "^12.0.0", }, "18.x": { "react-dom": "^18.0.0", "react-router-dom": "^6.0.0", "@testing-library/react": "^13.0.0", }, }, }; function checkCompatibility(packages) { // Validate package versions against matrix } ``` ## Staged Upgrade Strategy ### Phase 1: Planning ```bash # 1. Identify current versions npm list --depth=0 # 2. Check for breaking changes # Read CHANGELOG.md and MIGRATION.md # 3. Create upgrade plan echo "Upgrade order: 1. TypeScript 2. React 3. React Router 4. Testing libraries 5. Build tools" > UPGRADE_PLAN.md ``` ### Phase 2: Incremental Updates ```bash # Don't upgrade everything at once! # Step 1: Update TypeScript npm install typescript@latest # Test npm run test npm run build # Step 2: Update React (one major version at a time) npm install react@17 react-dom@17 # Test again npm run test # Step 3: Continue with other packages npm install react-router-dom@6 # And so on... ``` ### Phase 3: Validation ```javascript // tests/compatibility.test.js describe("Dependency Compatibility", () => { it("should have compatible React versions", () => { const reactVersion = require("react/package.json").version; const reactDomVersion = require("react-dom/package.json").version; expect(reactVersion).toBe(reactDomVersion); }); it("should not have peer dependency warnings", () => { // Run npm ls and check for warnings }); }); ``` ## Breaking Change Handling ### Identifying Breaking Changes ```bash # Check the changelog directly curl https://raw.githubusercontent.com/facebook/react/master/CHANGELOG.md ``` ### Codemod for Automated Fixes ```bash # Run jscodeshift with transform URL npx jscodeshift -t <transform-url> <path> # Example: Rename unsafe lifecycle methods npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js src/ # For TypeScript files npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js --parser=tsx src/ # Dry run to preview changes npx jscodeshift -t https://raw.githubusercontent.com/reactjs/react-codemod/master/transforms/rename-unsafe-lifecycles.js --dry src/ ``` ### Custom Migration Script ```javascript // migration-script.js const fs = require("fs"); const glob = require("glob"); glob("src/**/*.tsx", (err, files) => { files.forEach((file) => { let content = fs.readFileSync(file, "utf8"); // Replace old API with new API content = content.replace( /componentWillMount/g, "UNSAFE_componentWillMount", ); // Update imports content = content.replace( /import { Component } from 'react'/g, "import React, { Component } from 'react'", ); fs.writeFileSync(file, content); }); }); ``` ## Testing Strategy ### Unit Tests ```javascript // Ensure tests pass before and after upgrade npm run test // Update test utilities if needed npm install @testing-library/react@latest ``` ### Integration Tests ```javascript // tests/integration/app.test.js describe("App Integration", () => { it("should render without crashing", () => { render(<App />); }); it("should handle navigation", () => { const { getByText } = render(<App />); fireEvent.click(getByText("Navigate")); expect(screen.getByText("New Page")).toBeInTheDocument(); }); }); ``` ### Visual Regression Tests ```javascript // visual-regression.test.js describe("Visual Regression", () => { it("should match snapshot", () => { const { container } = render(<App />); expect(container.firstChild).toMatchSnapshot(); }); }); ``` ### E2E Tests ```javascript // cypress/e2e/app.cy.js describe("E2E Tests", () => { it("should complete user flow", () => { cy.visit("/"); cy.get('[data-testid="login"]').click(); cy.get('input[name="email"]').type("user@example.com"); cy.get('button[type="submit"]').click(); cy.url().should("include", "/dashboard"); }); }); ``` ## Automated Dependency Updates ### Renovate Configuration ```json // renovate.json { "extends": ["config:base"], "packageRules": [ { "matchUpdateTypes": ["minor", "patch"], "automerge": true }, { "matchUpdateTypes": ["major"], "automerge": false, "labels": ["major-update"] } ], "schedule": ["before 3am on Monday"], "timezone": "America/New_York" } ``` ### Dependabot Configuration ```yaml # .github/dependabot.yml version: 2 updates: - package-ecosystem: "npm" directory: "/" schedule: interval: "weekly" open-pull-requests-limit: 5 reviewers: - "team-leads" commit-message: prefix: "chore" include: "scope" ``` ## Rollback Plan ```javascript // rollback.sh #!/bin/bash # Save current state git stash git checkout -b upgrade-branch # Attempt upgrade npm install package@latest # Run tests if npm run test; then echo "Upgrade successful" git add package.json package-lock.json git commit -m "chore: upgrade package" else echo "Upgrade failed, rolling back" git checkout main git branch -D upgrade-branch npm install # Restore from package-lock.json fi ``` ## Common Upgrade Patterns ### Lock File Management ```bash # npm npm install --package-lock-only # Update lock file only npm ci # Clean install from lock file # yarn yarn install --frozen-lockfile # CI mode yarn upgrade-interactive # Interactive upgrades ``` ### Peer Dependency Resolution ```bash # npm 7+: strict peer dependencies npm install --legacy-peer-deps # Ignore peer deps # npm 8+: override peer dependencies npm install --force ``` ### Workspace Upgrades ```bash # Update all workspace packages npm install --workspaces # Update specific workspace npm install package@latest --workspace=packages/app ``` ## Resources - **references/semver.md**: Semantic versioning guide - **references/compatibility-matrix.md**: Common compatibility issues - **references/staged-upgrades.md**: Incremental upgrade strategies - **references/testing-strategy.md**: Comprehensive testing approaches - **assets/upgrade-checklist.md**: Step-by-step checklist - **assets/compatibility-matrix.csv**: Version compatibility table - **scripts/audit-dependencies.sh**: Dependency audit script ## Best Practices 1. **Read Changelogs**: Understand what changed 2. **Upgrade Incrementally**: One major version at a time 3. **Test Thoroughly**: Unit, integration, E2E tests 4. **Check Peer Dependencies**: Resolve conflicts early 5. **Use Lock Files**: Ensure reproducible installs 6. **Automate Updates**: Use Renovate or Dependabot 7. **Monitor**: Watch for runtime errors post-upgrade 8. **Document**: Keep upgrade notes ## Upgrade Checklist ```markdown Pre-Upgrade: - [ ] Review current dependency versions - [ ] Read changelogs for breaking changes - [ ] Create feature branch - [ ] Backup current state (git tag) - [ ] Run full test suite (baseline) During Upgrade: - [ ] Upgrade one dependency at a time - [ ] Update peer dependencies - [ ] Fix TypeScript errors - [ ] Update tests if needed - [ ] Run test suite after each upgrade - [ ] Check bundle size impact Post-Upgrade: - [ ] Full regression testing - [ ] Performance testing - [ ] Update documentation - [ ] Deploy to staging - [ ] Monitor for errors - [ ] Deploy to production ``` ## Common Pitfalls - Upgrading all dependencies at once - Not testing after each upgrade - Ignoring peer dependency warnings - Forgetting to update lock file - Not reading breaking change notes - Skipping major versions - Not having rollback plan
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

unity-ecs-patterns

Master Unity ECS (Entity Component System) with DOTS, Jobs, and

coding
⭐1
# Unity ECS Patterns Production patterns for Unity's Data-Oriented Technology Stack (DOTS) including Entity Component System, Job System, and Burst Compiler. ## When to Use This Skill - Building high-performance Unity games - Managing thousands of entities efficiently - Implementing data-oriented game systems - Optimizing CPU-bound game logic - Converting OOP game code to ECS - Using Jobs and Burst for parallelization ## Core Concepts ### 1. ECS vs OOP | Aspect | Traditional OOP | ECS/DOTS | | ----------- | ----------------- | --------------- | | Data layout | Object-oriented | Data-oriented | | Memory | Scattered | Contiguous | | Processing | Per-object | Batched | | Scaling | Poor with count | Linear scaling | | Best for | Complex behaviors | Mass simulation | ### 2. DOTS Components ``` Entity: Lightweight ID (no data) Component: Pure data (no behavior) System: Logic that processes components World: Container for entities Archetype: Unique combination of components Chunk: Memory block for same-archetype entities ``` ## Patterns ### Pattern 1: Basic ECS Setup ```csharp using Unity.Entities; using Unity.Mathematics; using Unity.Transforms; using Unity.Burst; using Unity.Collections; // Component: Pure data, no methods public struct Speed : IComponentData { public float Value; } public struct Health : IComponentData { public float Current; public float Max; } public struct Target : IComponentData { public Entity Value; } // Tag component (zero-size marker) public struct EnemyTag : IComponentData { } public struct PlayerTag : IComponentData { } // Buffer component (variable-size array) [InternalBufferCapacity(8)] public struct InventoryItem : IBufferElementData { public int ItemId; public int Quantity; } // Shared component (grouped entities) public struct TeamId : ISharedComponentData { public int Value; } ``` ### Pattern 2: Systems with ISystem (Recommended) ```csharp using Unity.Entities; using Unity.Transforms; using Unity.Mathematics; using Unity.Burst; // ISystem: Unmanaged, Burst-compatible, highest performance [BurstCompile] public partial struct MovementSystem : ISystem { [BurstCompile] public void OnCreate(ref SystemState state) { // Require components before system runs state.RequireForUpdate<Speed>(); } [BurstCompile] public void OnUpdate(ref SystemState state) { float deltaTime = SystemAPI.Time.DeltaTime; // Simple foreach - auto-generates job foreach (var (transform, speed) in SystemAPI.Query<RefRW<LocalTransform>, RefRO<Speed>>()) { transform.ValueRW.Position += new float3(0, 0, speed.ValueRO.Value * deltaTime); } } [BurstCompile] public void OnDestroy(ref SystemState state) { } } // With explicit job for more control [BurstCompile] public partial struct MovementJobSystem : ISystem { [BurstCompile] public void OnUpdate(ref SystemState state) { var job = new MoveJob { DeltaTime = SystemAPI.Time.DeltaTime }; state.Dependency = job.ScheduleParallel(state.Dependency); } } [BurstCompile] public partial struct MoveJob : IJobEntity { public float DeltaTime; void Execute(ref LocalTransform transform, in Speed speed) { transform.Position += new float3(0, 0, speed.Value * DeltaTime); } } ``` ### Pattern 3: Entity Queries ```csharp [BurstCompile] public partial struct QueryExamplesSystem : ISystem { private EntityQuery _enemyQuery; public void OnCreate(ref SystemState state) { // Build query manually for complex cases _enemyQuery = new EntityQueryBuilder(Allocator.Temp) .WithAll<EnemyTag, Health, LocalTransform>() .WithNone<Dead>() .WithOptions(EntityQueryOptions.FilterWriteGroup) .Build(ref state); } [BurstCompile] public void OnUpdate(ref SystemState state) { // SystemAPI.Query - simplest approach foreach (var (health, entity) in SystemAPI.Query<RefRW<Health>>() .WithAll<EnemyTag>() .WithEntityAccess()) { if (health.ValueRO.Current <= 0) { // Mark for destruction SystemAPI.GetSingleton<EndSimulationEntityCommandBufferSystem.Singleton>() .CreateCommandBuffer(state.WorldUnmanaged) .DestroyEntity(entity); } } // Get count int enemyCount = _enemyQuery.CalculateEntityCount(); // Get all entities var enemies = _enemyQuery.ToEntityArray(Allocator.Temp); // Get component arrays var healths = _enemyQuery.ToComponentDataArray<Health>(Allocator.Temp); } } ``` ### Pattern 4: Entity Command Buffers (Structural Changes) ```csharp // Structural changes (create/destroy/add/remove) require command buffers [BurstCompile] [UpdateInGroup(typeof(SimulationSystemGroup))] public partial struct SpawnSystem : ISystem { [BurstCompile] public void OnUpdate(ref SystemState state) { var ecbSingleton = SystemAPI.GetSingleton<BeginSimulationEntityCommandBufferSystem.Singleton>(); var ecb = ecbSingleton.CreateCommandBuffer(state.WorldUnmanaged); foreach (var (spawner, transform) in SystemAPI.Query<RefRW<Spawner>, RefRO<LocalTransform>>()) { spawner.ValueRW.Timer -= SystemAPI.Time.DeltaTime; if (spawner.ValueRO.Timer <= 0) { spawner.ValueRW.Timer = spawner.ValueRO.Interval; // Create entity (deferred until sync point) Entity newEntity = ecb.Instantiate(spawner.ValueRO.Prefab); // Set component values ecb.SetComponent(newEntity, new LocalTransform { Position = transform.ValueRO.Position, Rotation = quaternion.identity, Scale = 1f }); // Add component ecb.AddComponent(newEntity, new Speed { Value = 5f }); } } } } // Parallel ECB usage [BurstCompile] public partial struct ParallelSpawnJob : IJobEntity { public EntityCommandBuffer.ParallelWriter ECB; void Execute([EntityIndexInQuery] int index, in Spawner spawner) { Entity e = ECB.Instantiate(index, spawner.Prefab); ECB.AddComponent(index, e, new Speed { Value = 5f }); } } ``` ### Pattern 5: Aspect (Grouping Components) ```csharp using Unity.Entities; using Unity.Transforms; using Unity.Mathematics; // Aspect: Groups related components for cleaner code public readonly partial struct CharacterAspect : IAspect { public readonly Entity Entity; private readonly RefRW<LocalTransform> _transform; private readonly RefRO<Speed> _speed; private readonly RefRW<Health> _health; // Optional component [Optional] private readonly RefRO<Shield> _shield; // Buffer private readonly DynamicBuffer<InventoryItem> _inventory; public float3 Position { get => _transform.ValueRO.Position; set => _transform.ValueRW.Position = value; } public float CurrentHealth => _health.ValueRO.Current; public float MaxHealth => _health.ValueRO.Max; public float MoveSpeed => _speed.ValueRO.Value; public bool HasShield => _shield.IsValid; public float ShieldAmount => HasShield ? _shield.ValueRO.Amount : 0f; public void TakeDamage(float amount) { float remaining = amount; if (HasShield && _shield.ValueRO.Amount > 0) { // Shield absorbs damage first remaining = math.max(0, amount - _shield.ValueRO.Amount); } _health.ValueRW.Current = math.max(0, _health.ValueRO.Current - remaining); } public void Move(float3 direction, float deltaTime) { _transform.ValueRW.Position += direction * _speed.ValueRO.Value * deltaTime; } public void AddItem(int itemId, int quantity) { _inventory.Add(new InventoryItem { ItemId = itemId, Quantity = quantity }); } } // Using aspect in system [BurstCompile] public partial struct CharacterSystem : ISystem { [BurstCompile] public void OnUpdate(ref SystemState state) { float dt = SystemAPI.Time.DeltaTime; foreach (var character in SystemAPI.Query<CharacterAspect>()) { character.Move(new float3(1, 0, 0), dt); if (character.CurrentHealth < character.MaxHealth * 0.5f) { // Low health logic } } } } ``` ### Pattern 6: Singleton Components ```csharp // Singleton: Exactly one entity with this component public struct GameConfig : IComponentData { public float DifficultyMultiplier; public int MaxEnemies; public float SpawnRate; } public struct GameState : IComponentData { public int Score; public int Wave; public float TimeRemaining; } // Create singleton on world creation public partial struct GameInitSystem : ISystem { public void OnCreate(ref SystemState state) { var entity = state.EntityManager.CreateEntity(); state.EntityManager.AddComponentData(entity, new GameConfig { DifficultyMultiplier = 1.0f, MaxEnemies = 100, SpawnRate = 2.0f }); state.EntityManager.AddComponentData(entity, new GameState { Score = 0, Wave = 1, TimeRemaining = 120f }); } } // Access singleton in system [BurstCompile] public partial struct ScoreSystem : ISystem { [BurstCompile] public void OnUpdate(ref SystemState state) { // Read singleton var config = SystemAPI.GetSingleton<GameConfig>(); // Write singleton ref var gameState = ref SystemAPI.GetSingletonRW<GameState>().ValueRW; gameState.TimeRemaining -= SystemAPI.Time.DeltaTime; // Check exists if (SystemAPI.HasSingleton<GameConfig>()) { // ... } } } ``` ### Pattern 7: Baking (Converting GameObjects) ```csharp using Unity.Entities; using UnityEngine; // Authoring component (MonoBehaviour in Editor) public class EnemyAuthoring : MonoBehaviour { public float Speed = 5f; public float Health = 100f; public GameObject ProjectilePrefab; class Baker : Baker<EnemyAuthoring> { public override void Bake(EnemyAuthoring authoring) { var entity = GetEntity(TransformUsageFlags.Dynamic); AddComponent(entity, new Speed { Value = authoring.Speed }); AddComponent(entity, new Health { Current = authoring.Health, Max = authoring.Health }); AddComponent(entity, new EnemyTag()); if (authoring.ProjectilePrefab != null) { AddComponent(entity, new ProjectilePrefab { Value = GetEntity(authoring.ProjectilePrefab, TransformUsageFlags.Dynamic) }); } } } } // Complex baking with dependencies public class SpawnerAuthoring : MonoBehaviour { public GameObject[] Prefabs; public float Interval = 1f; class Baker : Baker<SpawnerAuthoring> { public override void Bake(SpawnerAuthoring authoring) { var entity = GetEntity(TransformUsageFlags.Dynamic); AddComponent(entity, new Spawner { Interval = authoring.Interval, Timer = 0f }); // Bake buffer of prefabs var buffer = AddBuffer<SpawnPrefabElement>(entity); foreach (var prefab in authoring.Prefabs) { buffer.Add(new SpawnPrefabElement { Prefab = GetEntity(prefab, TransformUsageFlags.Dynamic) }); } // Declare dependencies DependsOn(authoring.Prefabs); } } } ``` ### Pattern 8: Jobs with Native Collections ```csharp using Unity.Jobs; using Unity.Collections; using Unity.Burst; using Unity.Mathematics; [BurstCompile] public struct SpatialHashJob : IJobParallelFor { [ReadOnly] public NativeArray<float3> Positions; // Thread-safe write to hash map public NativeParallelMultiHashMap<int, int>.ParallelWriter HashMap; public float CellSize; public void Execute(int index) { float3 pos = Positions[index]; int hash = GetHash(pos); HashMap.Add(hash, index); } int GetHash(float3 pos) { int x = (int)math.floor(pos.x / CellSize); int y = (int)math.floor(pos.y / CellSize); int z = (int)math.floor(pos.z / CellSize); return x * 73856093 ^ y * 19349663 ^ z * 83492791; } } [BurstCompile] public partial struct SpatialHashSystem : ISystem { private NativeParallelMultiHashMap<int, int> _hashMap; public void OnCreate(ref SystemState state) { _hashMap = new NativeParallelMultiHashMap<int, int>(10000, Allocator.Persistent); } public void OnDestroy(ref SystemState state) { _hashMap.Dispose(); } [BurstCompile] public void OnUpdate(ref SystemState state) { var query = SystemAPI.QueryBuilder() .WithAll<LocalTransform>() .Build(); int count = query.CalculateEntityCount(); // Resize if needed if (_hashMap.Capacity < count) { _hashMap.Capacity = count * 2; } _hashMap.Clear(); // Get positions var positions = query.ToComponentDataArray<LocalTransform>(Allocator.TempJob); var posFloat3 = new NativeArray<float3>(count, Allocator.TempJob); for (int i = 0; i < count; i++) { posFloat3[i] = positions[i].Position; } // Build hash map var hashJob = new SpatialHashJob { Positions = posFloat3, HashMap = _hashMap.AsParallelWriter(), CellSize = 10f }; state.Dependency = hashJob.Schedule(count, 64, state.Dependency); // Cleanup positions.Dispose(state.Dependency); posFloat3.Dispose(state.Dependency); } } ``` ## Performance Tips ```csharp // 1. Use Burst everywhere [BurstCompile] public partial struct MySystem : ISystem { } // 2. Prefer IJobEntity over manual iteration [BurstCompile] partial struct OptimizedJob : IJobEntity { void Execute(ref LocalTransform transform) { } } // 3. Schedule parallel when possible state.Dependency = job.ScheduleParallel(state.Dependency); // 4. Use ScheduleParallel with chunk iteration [BurstCompile] partial struct ChunkJob : IJobChunk { public ComponentTypeHandle<Health> HealthHandle; public void Execute(in ArchetypeChunk chunk, int unfilteredChunkIndex, bool useEnabledMask, in v128 chunkEnabledMask) { var healths = chunk.GetNativeArray(ref HealthHandle); for (int i = 0; i < chunk.Count; i++) { // Process } } } // 5. Avoid structural changes in hot paths // Use enableable components instead of add/remove public struct Disabled : IComponentData, IEnableableComponent { } ``` ## Best Practices ### Do's - **Use ISystem over SystemBase** - Better performance - **Burst compile everything** - Massive speedup - **Batch structural changes** - Use ECB - **Profile with Profiler** - Identify bottlenecks - **Use Aspects** - Clean component grouping ### Don'ts - **Don't use managed types** - Breaks Burst - **Don't structural change in jobs** - Use ECB - **Don't over-architect** - Start simple - **Don't ignore chunk utilization** - Group similar entities - **Don't forget disposal** - Native collections leak ## Resources - [Unity DOTS Documentation](https://docs.unity3d.com/Packages/com.unity.entities@latest) - [Unity DOTS Samples](https://github.com/Unity-Technologies/EntityComponentSystemSamples) - [Burst User Guide](https://docs.unity3d.com/Packages/com.unity.burst@latest)
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

helm-chart-scaffolding

Design, organize, and manage Helm charts for templating and

architecture
⭐1
# Helm Chart Scaffolding Comprehensive guidance for creating, organizing, and managing Helm charts for packaging and deploying Kubernetes applications. ## Purpose This skill provides step-by-step instructions for building production-ready Helm charts, including chart structure, templating patterns, values management, and validation strategies. ## When to Use This Skill Use this skill when you need to: - Create new Helm charts from scratch - Package Kubernetes applications for distribution - Manage multi-environment deployments with Helm - Implement templating for reusable Kubernetes manifests - Set up Helm chart repositories - Follow Helm best practices and conventions ## Helm Overview **Helm** is the package manager for Kubernetes that: - Templates Kubernetes manifests for reusability - Manages application releases and rollbacks - Handles dependencies between charts - Provides version control for deployments - Simplifies configuration management across environments ## Step-by-Step Workflow ### 1. Initialize Chart Structure **Create new chart:** ```bash helm create my-app ``` **Standard chart structure:** ``` my-app/ ā”œā”€ā”€ Chart.yaml # Chart metadata ā”œā”€ā”€ values.yaml # Default configuration values ā”œā”€ā”€ charts/ # Chart dependencies ā”œā”€ā”€ templates/ # Kubernetes manifest templates │ ā”œā”€ā”€ NOTES.txt # Post-install notes │ ā”œā”€ā”€ _helpers.tpl # Template helpers │ ā”œā”€ā”€ deployment.yaml │ ā”œā”€ā”€ service.yaml │ ā”œā”€ā”€ ingress.yaml │ ā”œā”€ā”€ serviceaccount.yaml │ ā”œā”€ā”€ hpa.yaml │ └── tests/ │ └── test-connection.yaml └── .helmignore # Files to ignore ``` ### 2. Configure Chart.yaml **Chart metadata defines the package:** ```yaml apiVersion: v2 name: my-app description: A Helm chart for My Application type: application version: 1.0.0 # Chart version appVersion: "2.1.0" # Application version # Keywords for chart discovery keywords: - web - api - backend # Maintainer information maintainers: - name: DevOps Team email: devops@example.com url: https://github.com/example/my-app # Source code repository sources: - https://github.com/example/my-app # Homepage home: https://example.com # Chart icon icon: https://example.com/icon.png # Dependencies dependencies: - name: postgresql version: "12.0.0" repository: "https://charts.bitnami.com/bitnami" condition: postgresql.enabled - name: redis version: "17.0.0" repository: "https://charts.bitnami.com/bitnami" condition: redis.enabled ``` **Reference:** See `assets/Chart.yaml.template` for complete example ### 3. Design values.yaml Structure **Organize values hierarchically:** ```yaml # Image configuration image: repository: myapp tag: "1.0.0" pullPolicy: IfNotPresent # Number of replicas replicaCount: 3 # Service configuration service: type: ClusterIP port: 80 targetPort: 8080 # Ingress configuration ingress: enabled: false className: nginx hosts: - host: app.example.com paths: - path: / pathType: Prefix # Resources resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "512Mi" cpu: "500m" # Autoscaling autoscaling: enabled: false minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 80 # Environment variables env: - name: LOG_LEVEL value: "info" # ConfigMap data configMap: data: APP_MODE: production # Dependencies postgresql: enabled: true auth: database: myapp username: myapp redis: enabled: false ``` **Reference:** See `assets/values.yaml.template` for complete structure ### 4. Create Template Files **Use Go templating with Helm functions:** **templates/deployment.yaml:** ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: {{ include "my-app.fullname" . }} labels: {{- include "my-app.labels" . | nindent 4 }} spec: {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} selector: matchLabels: {{- include "my-app.selectorLabels" . | nindent 6 }} template: metadata: labels: {{- include "my-app.selectorLabels" . | nindent 8 }} spec: containers: - name: {{ .Chart.Name }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" imagePullPolicy: {{ .Values.image.pullPolicy }} ports: - name: http containerPort: {{ .Values.service.targetPort }} resources: {{- toYaml .Values.resources | nindent 12 }} env: {{- toYaml .Values.env | nindent 12 }} ``` ### 5. Create Template Helpers **templates/\_helpers.tpl:** ```yaml {{/* Expand the name of the chart. */}} {{- define "my-app.name" -}} {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} {{- end }} {{/* Create a default fully qualified app name. */}} {{- define "my-app.fullname" -}} {{- if .Values.fullnameOverride }} {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} {{- else }} {{- $name := default .Chart.Name .Values.nameOverride }} {{- if contains $name .Release.Name }} {{- .Release.Name | trunc 63 | trimSuffix "-" }} {{- else }} {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} {{- end }} {{- end }} {{- end }} {{/* Common labels */}} {{- define "my-app.labels" -}} helm.sh/chart: {{ include "my-app.chart" . }} {{ include "my-app.selectorLabels" . }} {{- if .Chart.AppVersion }} app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} {{- end }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end }} {{/* Selector labels */}} {{- define "my-app.selectorLabels" -}} app.kubernetes.io/name: {{ include "my-app.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end }} ``` ### 6. Manage Dependencies **Add dependencies in Chart.yaml:** ```yaml dependencies: - name: postgresql version: "12.0.0" repository: "https://charts.bitnami.com/bitnami" condition: postgresql.enabled ``` **Update dependencies:** ```bash helm dependency update helm dependency build ``` **Override dependency values:** ```yaml # values.yaml postgresql: enabled: true auth: database: myapp username: myapp password: changeme primary: persistence: enabled: true size: 10Gi ``` ### 7. Test and Validate **Validation commands:** ```bash # Lint the chart helm lint my-app/ # Dry-run installation helm install my-app ./my-app --dry-run --debug # Template rendering helm template my-app ./my-app # Template with values helm template my-app ./my-app -f values-prod.yaml # Show computed values helm show values ./my-app ``` **Validation script:** ```bash #!/bin/bash set -e echo "Linting chart..." helm lint . echo "Testing template rendering..." helm template test-release . --dry-run echo "Checking for required values..." helm template test-release . --validate echo "All validations passed!" ``` **Reference:** See `scripts/validate-chart.sh` ### 8. Package and Distribute **Package the chart:** ```bash helm package my-app/ # Creates: my-app-1.0.0.tgz ``` **Create chart repository:** ```bash # Create index helm repo index . # Upload to repository # AWS S3 example aws s3 sync . s3://my-helm-charts/ --exclude "*" --include "*.tgz" --include "index.yaml" ``` **Use the chart:** ```bash helm repo add my-repo https://charts.example.com helm repo update helm install my-app my-repo/my-app ``` ### 9. Multi-Environment Configuration **Environment-specific values files:** ``` my-app/ ā”œā”€ā”€ values.yaml # Defaults ā”œā”€ā”€ values-dev.yaml # Development ā”œā”€ā”€ values-staging.yaml # Staging └── values-prod.yaml # Production ``` **values-prod.yaml:** ```yaml replicaCount: 5 image: tag: "2.1.0" resources: requests: memory: "512Mi" cpu: "500m" limits: memory: "1Gi" cpu: "1000m" autoscaling: enabled: true minReplicas: 3 maxReplicas: 20 ingress: enabled: true hosts: - host: app.example.com paths: - path: / pathType: Prefix postgresql: enabled: true primary: persistence: size: 100Gi ``` **Install with environment:** ```bash helm install my-app ./my-app -f values-prod.yaml --namespace production ``` ### 10. Implement Hooks and Tests **Pre-install hook:** ```yaml # templates/pre-install-job.yaml apiVersion: batch/v1 kind: Job metadata: name: {{ include "my-app.fullname" . }}-db-setup annotations: "helm.sh/hook": pre-install "helm.sh/hook-weight": "-5" "helm.sh/hook-delete-policy": hook-succeeded spec: template: spec: containers: - name: db-setup image: postgres:15 command: ["psql", "-c", "CREATE DATABASE myapp"] restartPolicy: Never ``` **Test connection:** ```yaml # templates/tests/test-connection.yaml apiVersion: v1 kind: Pod metadata: name: "{{ include "my-app.fullname" . }}-test-connection" annotations: "helm.sh/hook": test spec: containers: - name: wget image: busybox command: ['wget'] args: ['{{ include "my-app.fullname" . }}:{{ .Values.service.port }}'] restartPolicy: Never ``` **Run tests:** ```bash helm test my-app ``` ## Common Patterns ### Pattern 1: Conditional Resources ```yaml {{- if .Values.ingress.enabled }} apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: {{ include "my-app.fullname" . }} spec: # ... {{- end }} ``` ### Pattern 2: Iterating Over Lists ```yaml env: {{- range .Values.env }} - name: {{ .name }} value: {{ .value | quote }} {{- end }} ``` ### Pattern 3: Including Files ```yaml data: config.yaml: | {{- .Files.Get "config/application.yaml" | nindent 4 }} ``` ### Pattern 4: Global Values ```yaml global: imageRegistry: docker.io imagePullSecrets: - name: regcred # Use in templates: image: {{ .Values.global.imageRegistry }}/{{ .Values.image.repository }} ``` ## Best Practices 1. **Use semantic versioning** for chart and app versions 2. **Document all values** in values.yaml with comments 3. **Use template helpers** for repeated logic 4. **Validate charts** before packaging 5. **Pin dependency versions** explicitly 6. **Use conditions** for optional resources 7. **Follow naming conventions** (lowercase, hyphens) 8. **Include NOTES.txt** with usage instructions 9. **Add labels** consistently using helpers 10. **Test installations** in all environments ## Troubleshooting **Template rendering errors:** ```bash helm template my-app ./my-app --debug ``` **Dependency issues:** ```bash helm dependency update helm dependency list ``` **Installation failures:** ```bash helm install my-app ./my-app --dry-run --debug kubectl get events --sort-by='.lastTimestamp' ``` ## Reference Files - `assets/Chart.yaml.template` - Chart metadata template - `assets/values.yaml.template` - Values structure template - `scripts/validate-chart.sh` - Validation script - `references/chart-structure.md` - Detailed chart organization ## Related Skills - `k8s-manifest-generator` - For creating base Kubernetes manifests - `gitops-workflow` - For automated Helm chart deployments
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

python-project-structure

Python project organization, module architecture, and public API

coding
⭐1
# Python Project Structure & Module Architecture Design well-organized Python projects with clear module boundaries, explicit public interfaces, and maintainable directory structures. Good organization makes code discoverable and changes predictable. ## When to Use This Skill - Starting a new Python project from scratch - Reorganizing an existing codebase for clarity - Defining module public APIs with `__all__` - Deciding between flat and nested directory structures - Determining test file placement strategies - Creating reusable library packages ## Core Concepts ### 1. Module Cohesion Group related code that changes together. A module should have a single, clear purpose. ### 2. Explicit Interfaces Define what's public with `__all__`. Everything not listed is an internal implementation detail. ### 3. Flat Hierarchies Prefer shallow directory structures. Add depth only for genuine sub-domains. ### 4. Consistent Conventions Apply naming and organization patterns uniformly across the project. ## Quick Start ``` myproject/ ā”œā”€ā”€ src/ │ └── myproject/ │ ā”œā”€ā”€ __init__.py │ ā”œā”€ā”€ services/ │ ā”œā”€ā”€ models/ │ └── api/ ā”œā”€ā”€ tests/ ā”œā”€ā”€ pyproject.toml └── README.md ``` ## Fundamental Patterns ### Pattern 1: One Concept Per File Each file should focus on a single concept or closely related set of functions. Consider splitting when a file: - Handles multiple unrelated responsibilities - Grows beyond 300-500 lines (varies by complexity) - Contains classes that change for different reasons ```python # Good: Focused files # user_service.py - User business logic # user_repository.py - User data access # user_models.py - User data structures # Avoid: Kitchen sink files # user.py - Contains service, repository, models, utilities... ``` ### Pattern 2: Explicit Public APIs with `__all__` Define the public interface for every module. Unlisted members are internal implementation details. ```python # mypackage/services/__init__.py from .user_service import UserService from .order_service import OrderService from .exceptions import ServiceError, ValidationError __all__ = [ "UserService", "OrderService", "ServiceError", "ValidationError", ] # Internal helpers remain private by omission # from .internal_helpers import _validate_input # Not exported ``` ### Pattern 3: Flat Directory Structure Prefer minimal nesting. Deep hierarchies make imports verbose and navigation difficult. ``` # Preferred: Flat structure project/ ā”œā”€ā”€ api/ │ ā”œā”€ā”€ routes.py │ └── middleware.py ā”œā”€ā”€ services/ │ ā”œā”€ā”€ user_service.py │ └── order_service.py ā”œā”€ā”€ models/ │ ā”œā”€ā”€ user.py │ └── order.py └── utils/ └── validation.py # Avoid: Deep nesting project/core/internal/services/impl/user/ ``` Add sub-packages only when there's a genuine sub-domain requiring isolation. ### Pattern 4: Test File Organization Choose one approach and apply it consistently throughout the project. **Option A: Colocated Tests** ``` src/ ā”œā”€ā”€ user_service.py ā”œā”€ā”€ test_user_service.py ā”œā”€ā”€ order_service.py └── test_order_service.py ``` Benefits: Tests live next to the code they verify. Easy to see coverage gaps. **Option B: Parallel Test Directory** ``` src/ ā”œā”€ā”€ services/ │ ā”œā”€ā”€ user_service.py │ └── order_service.py tests/ ā”œā”€ā”€ services/ │ ā”œā”€ā”€ test_user_service.py │ └── test_order_service.py ``` Benefits: Clean separation between production and test code. Standard for larger projects. ## Advanced Patterns ### Pattern 5: Package Initialization Use `__init__.py` to provide a clean public interface for package consumers. ```python # mypackage/__init__.py """MyPackage - A library for doing useful things.""" from .core import MainClass, HelperClass from .exceptions import PackageError, ConfigError from .config import Settings __all__ = [ "MainClass", "HelperClass", "PackageError", "ConfigError", "Settings", ] __version__ = "1.0.0" ``` Consumers can then import directly from the package: ```python from mypackage import MainClass, Settings ``` ### Pattern 6: Layered Architecture Organize code by architectural layer for clear separation of concerns. ``` myapp/ ā”œā”€ā”€ api/ # HTTP handlers, request/response │ ā”œā”€ā”€ routes/ │ └── middleware/ ā”œā”€ā”€ services/ # Business logic ā”œā”€ā”€ repositories/ # Data access ā”œā”€ā”€ models/ # Domain entities ā”œā”€ā”€ schemas/ # API schemas (Pydantic) └── config/ # Configuration ``` Each layer should only depend on layers below it, never above. ### Pattern 7: Domain-Driven Structure For complex applications, organize by business domain rather than technical layer. ``` ecommerce/ ā”œā”€ā”€ users/ │ ā”œā”€ā”€ models.py │ ā”œā”€ā”€ services.py │ ā”œā”€ā”€ repository.py │ └── api.py ā”œā”€ā”€ orders/ │ ā”œā”€ā”€ models.py │ ā”œā”€ā”€ services.py │ ā”œā”€ā”€ repository.py │ └── api.py └── shared/ ā”œā”€ā”€ database.py └── exceptions.py ``` ## File and Module Naming ### Conventions - Use `snake_case` for all file and module names: `user_repository.py` - Avoid abbreviations that obscure meaning: `user_repository.py` not `usr_repo.py` - Match class names to file names: `UserService` in `user_service.py` ### Import Style Use absolute imports for clarity and reliability: ```python # Preferred: Absolute imports from myproject.services import UserService from myproject.models import User # Avoid: Relative imports from ..services import UserService from . import models ``` Relative imports can break when modules are moved or reorganized. ## Best Practices Summary 1. **Keep files focused** - One concept per file, consider splitting at 300-500 lines (varies by complexity) 2. **Define `__all__` explicitly** - Make public interfaces clear 3. **Prefer flat structures** - Add depth only for genuine sub-domains 4. **Use absolute imports** - More reliable and clearer 5. **Be consistent** - Apply patterns uniformly across the project 6. **Match names to content** - File names should describe their purpose 7. **Separate concerns** - Keep layers distinct and dependencies flowing one direction 8. **Document your structure** - Include a README explaining the organization
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

uv-package-manager

Master the uv package manager for fast Python dependency

coding
⭐1
# UV Package Manager Comprehensive guide to using uv, an extremely fast Python package installer and resolver written in Rust, for modern Python project management and dependency workflows. ## When to Use This Skill - Setting up new Python projects quickly - Managing Python dependencies faster than pip - Creating and managing virtual environments - Installing Python interpreters - Resolving dependency conflicts efficiently - Migrating from pip/pip-tools/poetry - Speeding up CI/CD pipelines - Managing monorepo Python projects - Working with lockfiles for reproducible builds - Optimizing Docker builds with Python dependencies ## Core Concepts ### 1. What is uv? - **Ultra-fast package installer**: 10-100x faster than pip - **Written in Rust**: Leverages Rust's performance - **Drop-in pip replacement**: Compatible with pip workflows - **Virtual environment manager**: Create and manage venvs - **Python installer**: Download and manage Python versions - **Resolver**: Advanced dependency resolution - **Lockfile support**: Reproducible installations ### 2. Key Features - Blazing fast installation speeds - Disk space efficient with global cache - Compatible with pip, pip-tools, poetry - Comprehensive dependency resolution - Cross-platform support (Linux, macOS, Windows) - No Python required for installation - Built-in virtual environment support ### 3. UV vs Traditional Tools - **vs pip**: 10-100x faster, better resolver - **vs pip-tools**: Faster, simpler, better UX - **vs poetry**: Faster, less opinionated, lighter - **vs conda**: Faster, Python-focused ## Installation ### Quick Install ```bash # macOS/Linux curl -LsSf https://astral.sh/uv/install.sh | sh # Windows (PowerShell) powershell -c "irm https://astral.sh/uv/install.ps1 | iex" # Using pip (if you already have Python) pip install uv # Using Homebrew (macOS) brew install uv # Using cargo (if you have Rust) cargo install --git https://github.com/astral-sh/uv uv ``` ### Verify Installation ```bash uv --version # uv 0.x.x ``` ## Quick Start ### Create a New Project ```bash # Create new project with virtual environment uv init my-project cd my-project # Or create in current directory uv init . # Initialize creates: # - .python-version (Python version) # - pyproject.toml (project config) # - README.md # - .gitignore ``` ### Install Dependencies ```bash # Install packages (creates venv if needed) uv add requests pandas # Install dev dependencies uv add --dev pytest black ruff # Install from requirements.txt uv pip install -r requirements.txt # Install from pyproject.toml uv sync ``` ## Virtual Environment Management ### Pattern 1: Creating Virtual Environments ```bash # Create virtual environment with uv uv venv # Create with specific Python version uv venv --python 3.12 # Create with custom name uv venv my-env # Create with system site packages uv venv --system-site-packages # Specify location uv venv /path/to/venv ``` ### Pattern 2: Activating Virtual Environments ```bash # Linux/macOS source .venv/bin/activate # Windows (Command Prompt) .venv\Scripts\activate.bat # Windows (PowerShell) .venv\Scripts\Activate.ps1 # Or use uv run (no activation needed) uv run python script.py uv run pytest ``` ### Pattern 3: Using uv run ```bash # Run Python script (auto-activates venv) uv run python app.py # Run installed CLI tool uv run black . uv run pytest # Run with specific Python version uv run --python 3.11 python script.py # Pass arguments uv run python script.py --arg value ``` ## Package Management ### Pattern 4: Adding Dependencies ```bash # Add package (adds to pyproject.toml) uv add requests # Add with version constraint uv add "django>=4.0,<5.0" # Add multiple packages uv add numpy pandas matplotlib # Add dev dependency uv add --dev pytest pytest-cov # Add optional dependency group uv add --optional docs sphinx # Add from git uv add git+https://github.com/user/repo.git # Add from git with specific ref uv add git+https://github.com/user/repo.git@v1.0.0 # Add from local path uv add ./local-package # Add editable local package uv add -e ./local-package ``` ### Pattern 5: Removing Dependencies ```bash # Remove package uv remove requests # Remove dev dependency uv remove --dev pytest # Remove multiple packages uv remove numpy pandas matplotlib ``` ### Pattern 6: Upgrading Dependencies ```bash # Upgrade specific package uv add --upgrade requests # Upgrade all packages uv sync --upgrade # Upgrade package to latest uv add --upgrade requests # Show what would be upgraded uv tree --outdated ``` ### Pattern 7: Locking Dependencies ```bash # Generate uv.lock file uv lock # Update lock file uv lock --upgrade # Lock without installing uv lock --no-install # Lock specific package uv lock --upgrade-package requests ``` ## Python Version Management ### Pattern 8: Installing Python Versions ```bash # Install Python version uv python install 3.12 # Install multiple versions uv python install 3.11 3.12 3.13 # Install latest version uv python install # List installed versions uv python list # Find available versions uv python list --all-versions ``` ### Pattern 9: Setting Python Version ```bash # Set Python version for project uv python pin 3.12 # This creates/updates .python-version file # Use specific Python version for command uv --python 3.11 run python script.py # Create venv with specific version uv venv --python 3.12 ``` ## Project Configuration ### Pattern 10: pyproject.toml with uv ```toml [project] name = "my-project" version = "0.1.0" description = "My awesome project" readme = "README.md" requires-python = ">=3.8" dependencies = [ "requests>=2.31.0", "pydantic>=2.0.0", "click>=8.1.0", ] [project.optional-dependencies] dev = [ "pytest>=7.4.0", "pytest-cov>=4.1.0", "black>=23.0.0", "ruff>=0.1.0", "mypy>=1.5.0", ] docs = [ "sphinx>=7.0.0", "sphinx-rtd-theme>=1.3.0", ] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.uv] dev-dependencies = [ # Additional dev dependencies managed by uv ] [tool.uv.sources] # Custom package sources my-package = { git = "https://github.com/user/repo.git" } ``` ### Pattern 11: Using uv with Existing Projects ```bash # Migrate from requirements.txt uv add -r requirements.txt # Migrate from poetry # Already have pyproject.toml, just use: uv sync # Export to requirements.txt uv pip freeze > requirements.txt # Export with hashes uv pip freeze --require-hashes > requirements.txt ``` ## Advanced Workflows ### Pattern 12: Monorepo Support ```bash # Project structure # monorepo/ # packages/ # package-a/ # pyproject.toml # package-b/ # pyproject.toml # pyproject.toml (root) # Root pyproject.toml [tool.uv.workspace] members = ["packages/*"] # Install all workspace packages uv sync # Add workspace dependency uv add --path ./packages/package-a ``` ### Pattern 13: CI/CD Integration ```yaml # .github/workflows/test.yml name: Tests on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Install uv uses: astral-sh/setup-uv@v2 with: enable-cache: true - name: Set up Python run: uv python install 3.12 - name: Install dependencies run: uv sync --all-extras --dev - name: Run tests run: uv run pytest - name: Run linting run: | uv run ruff check . uv run black --check . ``` ### Pattern 14: Docker Integration ```dockerfile # Dockerfile FROM python:3.12-slim # Install uv COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv # Set working directory WORKDIR /app # Copy dependency files COPY pyproject.toml uv.lock ./ # Install dependencies RUN uv sync --frozen --no-dev # Copy application code COPY . . # Run application CMD ["uv", "run", "python", "app.py"] ``` **Optimized multi-stage build:** ```dockerfile # Multi-stage Dockerfile FROM python:3.12-slim AS builder # Install uv COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv WORKDIR /app # Install dependencies to venv COPY pyproject.toml uv.lock ./ RUN uv sync --frozen --no-dev --no-editable # Runtime stage FROM python:3.12-slim WORKDIR /app # Copy venv from builder COPY --from=builder /app/.venv .venv COPY . . # Use venv ENV PATH="/app/.venv/bin:$PATH" CMD ["python", "app.py"] ``` ### Pattern 15: Lockfile Workflows ```bash # Create lockfile (uv.lock) uv lock # Install from lockfile (exact versions) uv sync --frozen # Update lockfile without installing uv lock --no-install # Upgrade specific package in lock uv lock --upgrade-package requests # Check if lockfile is up to date uv lock --check # Export lockfile to requirements.txt uv export --format requirements-txt > requirements.txt # Export with hashes for security uv export --format requirements-txt --hash > requirements.txt ``` ## Performance Optimization ### Pattern 16: Using Global Cache ```bash # UV automatically uses global cache at: # Linux: ~/.cache/uv # macOS: ~/Library/Caches/uv # Windows: %LOCALAPPDATA%\uv\cache # Clear cache uv cache clean # Check cache size uv cache dir ``` ### Pattern 17: Parallel Installation ```bash # UV installs packages in parallel by default # Control parallelism uv pip install --jobs 4 package1 package2 # No parallel (sequential) uv pip install --jobs 1 package ``` ### Pattern 18: Offline Mode ```bash # Install from cache only (no network) uv pip install --offline package # Sync from lockfile offline uv sync --frozen --offline ``` ## Comparison with Other Tools ### uv vs pip ```bash # pip python -m venv .venv source .venv/bin/activate pip install requests pandas numpy # ~30 seconds # uv uv venv uv add requests pandas numpy # ~2 seconds (10-15x faster) ``` ### uv vs poetry ```bash # poetry poetry init poetry add requests pandas poetry install # ~20 seconds # uv uv init uv add requests pandas uv sync # ~3 seconds (6-7x faster) ``` ### uv vs pip-tools ```bash # pip-tools pip-compile requirements.in pip-sync requirements.txt # ~15 seconds # uv uv lock uv sync --frozen # ~2 seconds (7-8x faster) ``` ## Common Workflows ### Pattern 19: Starting a New Project ```bash # Complete workflow uv init my-project cd my-project # Set Python version uv python pin 3.12 # Add dependencies uv add fastapi uvicorn pydantic # Add dev dependencies uv add --dev pytest black ruff mypy # Create structure mkdir -p src/my_project tests # Run tests uv run pytest # Format code uv run black . uv run ruff check . ``` ### Pattern 20: Maintaining Existing Project ```bash # Clone repository git clone https://github.com/user/project.git cd project # Install dependencies (creates venv automatically) uv sync # Install with dev dependencies uv sync --all-extras # Update dependencies uv lock --upgrade # Run application uv run python app.py # Run tests uv run pytest # Add new dependency uv add new-package # Commit updated files git add pyproject.toml uv.lock git commit -m "Add new-package dependency" ``` ## Tool Integration ### Pattern 21: Pre-commit Hooks ```yaml # .pre-commit-config.yaml repos: - repo: local hooks: - id: uv-lock name: uv lock entry: uv lock language: system pass_filenames: false - id: ruff name: ruff entry: uv run ruff check --fix language: system types: [python] - id: black name: black entry: uv run black language: system types: [python] ``` ### Pattern 22: VS Code Integration ```json // .vscode/settings.json { "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python", "python.terminal.activateEnvironment": true, "python.testing.pytestEnabled": true, "python.testing.pytestArgs": ["-v"], "python.linting.enabled": true, "python.formatting.provider": "black", "[python]": { "editor.defaultFormatter": "ms-python.black-formatter", "editor.formatOnSave": true } } ``` ## Troubleshooting ### Common Issues ```bash # Issue: uv not found # Solution: Add to PATH or reinstall echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.bashrc # Issue: Wrong Python version # Solution: Pin version explicitly uv python pin 3.12 uv venv --python 3.12 # Issue: Dependency conflict # Solution: Check resolution uv lock --verbose # Issue: Cache issues # Solution: Clear cache uv cache clean # Issue: Lockfile out of sync # Solution: Regenerate uv lock --upgrade ``` ## Best Practices ### Project Setup 1. **Always use lockfiles** for reproducibility 2. **Pin Python version** with .python-version 3. **Separate dev dependencies** from production 4. **Use uv run** instead of activating venv 5. **Commit uv.lock** to version control 6. **Use --frozen in CI** for consistent builds 7. **Leverage global cache** for speed 8. **Use workspace** for monorepos 9. **Export requirements.txt** for compatibility 10. **Keep uv updated** for latest features ### Performance Tips ```bash # Use frozen installs in CI uv sync --frozen # Use offline mode when possible uv sync --offline # Parallel operations (automatic) # uv does this by default # Reuse cache across environments # uv shares cache globally # Use lockfiles to skip resolution uv sync --frozen # skips resolution ``` ## Migration Guide ### From pip + requirements.txt ```bash # Before python -m venv .venv source .venv/bin/activate pip install -r requirements.txt # After uv venv uv pip install -r requirements.txt # Or better: uv init uv add -r requirements.txt ``` ### From Poetry ```bash # Before poetry install poetry add requests # After uv sync uv add requests # Keep existing pyproject.toml # uv reads [project] and [tool.poetry] sections ``` ### From pip-tools ```bash # Before pip-compile requirements.in pip-sync requirements.txt # After uv lock uv sync --frozen ``` ## Command Reference ### Essential Commands ```bash # Project management uv init [PATH] # Initialize project uv add PACKAGE # Add dependency uv remove PACKAGE # Remove dependency uv sync # Install dependencies uv lock # Create/update lockfile # Virtual environments uv venv [PATH] # Create venv uv run COMMAND # Run in venv # Python management uv python install VERSION # Install Python uv python list # List installed Pythons uv python pin VERSION # Pin Python version # Package installation (pip-compatible) uv pip install PACKAGE # Install package uv pip uninstall PACKAGE # Uninstall package uv pip freeze # List installed uv pip list # List packages # Utility uv cache clean # Clear cache uv cache dir # Show cache location uv --version # Show version ``` ## Resources - **Official documentation**: https://docs.astral.sh/uv/ - **GitHub repository**: https://github.com/astral-sh/uv - **Astral blog**: https://astral.sh/blog - **Migration guides**: https://docs.astral.sh/uv/guides/ - **Comparison with other tools**: https://docs.astral.sh/uv/pip/compatibility/ ## Best Practices Summary 1. **Use uv for all new projects** - Start with `uv init` 2. **Commit lockfiles** - Ensure reproducible builds 3. **Pin Python versions** - Use .python-version 4. **Use uv run** - Avoid manual venv activation 5. **Leverage caching** - Let uv manage global cache 6. **Use --frozen in CI** - Exact reproduction 7. **Keep uv updated** - Fast-moving project 8. **Use workspaces** - For monorepo projects 9. **Export for compatibility** - Generate requirements.txt when needed 10. **Read the docs** - uv is feature-rich and evolving
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

anti-reversing-techniques

Understand anti-reversing, obfuscation, and protection techniques

security
⭐1
> **AUTHORIZED USE ONLY**: This skill contains dual-use security techniques. Before proceeding with any bypass or analysis: > > 1. **Verify authorization**: Confirm you have explicit written permission from the software owner, or are operating within a legitimate security context (CTF, authorized pentest, malware analysis, security research) > 2. **Document scope**: Ensure your activities fall within the defined scope of your authorization > 3. **Legal compliance**: Understand that unauthorized bypassing of software protection may violate laws (CFAA, DMCA anti-circumvention, etc.) > > **Legitimate use cases**: Malware analysis, authorized penetration testing, CTF competitions, academic security research, analyzing software you own/have rights to # Anti-Reversing Techniques Understanding protection mechanisms encountered during authorized software analysis, security research, and malware analysis. This knowledge helps analysts bypass protections to complete legitimate analysis tasks. ## Anti-Debugging Techniques ### Windows Anti-Debugging #### API-Based Detection ```c // IsDebuggerPresent if (IsDebuggerPresent()) { exit(1); } // CheckRemoteDebuggerPresent BOOL debugged = FALSE; CheckRemoteDebuggerPresent(GetCurrentProcess(), &debugged); if (debugged) exit(1); // NtQueryInformationProcess typedef NTSTATUS (NTAPI *pNtQueryInformationProcess)( HANDLE, PROCESSINFOCLASS, PVOID, ULONG, PULONG); DWORD debugPort = 0; NtQueryInformationProcess( GetCurrentProcess(), ProcessDebugPort, // 7 &debugPort, sizeof(debugPort), NULL ); if (debugPort != 0) exit(1); // Debug flags DWORD debugFlags = 0; NtQueryInformationProcess( GetCurrentProcess(), ProcessDebugFlags, // 0x1F &debugFlags, sizeof(debugFlags), NULL ); if (debugFlags == 0) exit(1); // 0 means being debugged ``` **Bypass Approaches:** ```python # x64dbg: ScyllaHide plugin # Patches common anti-debug checks # Manual patching in debugger: # - Set IsDebuggerPresent return to 0 # - Patch PEB.BeingDebugged to 0 # - Hook NtQueryInformationProcess # IDAPython: Patch checks ida_bytes.patch_byte(check_addr, 0x90) # NOP ``` #### PEB-Based Detection ```c // Direct PEB access #ifdef _WIN64 PPEB peb = (PPEB)__readgsqword(0x60); #else PPEB peb = (PPEB)__readfsdword(0x30); #endif // BeingDebugged flag if (peb->BeingDebugged) exit(1); // NtGlobalFlag // Debugged: 0x70 (FLG_HEAP_ENABLE_TAIL_CHECK | // FLG_HEAP_ENABLE_FREE_CHECK | // FLG_HEAP_VALIDATE_PARAMETERS) if (peb->NtGlobalFlag & 0x70) exit(1); // Heap flags PDWORD heapFlags = (PDWORD)((PBYTE)peb->ProcessHeap + 0x70); if (*heapFlags & 0x50000062) exit(1); ``` **Bypass Approaches:** ```assembly ; In debugger, modify PEB directly ; x64dbg: dump at gs:[60] (x64) or fs:[30] (x86) ; Set BeingDebugged (offset 2) to 0 ; Clear NtGlobalFlag (offset 0xBC for x64) ``` #### Timing-Based Detection ```c // RDTSC timing uint64_t start = __rdtsc(); // ... some code ... uint64_t end = __rdtsc(); if ((end - start) > THRESHOLD) exit(1); // QueryPerformanceCounter LARGE_INTEGER start, end, freq; QueryPerformanceFrequency(&freq); QueryPerformanceCounter(&start); // ... code ... QueryPerformanceCounter(&end); double elapsed = (double)(end.QuadPart - start.QuadPart) / freq.QuadPart; if (elapsed > 0.1) exit(1); // Too slow = debugger // GetTickCount DWORD start = GetTickCount(); // ... code ... if (GetTickCount() - start > 1000) exit(1); ``` **Bypass Approaches:** ``` - Use hardware breakpoints instead of software - Patch timing checks - Use VM with controlled time - Hook timing APIs to return consistent values ``` #### Exception-Based Detection ```c // SEH-based detection __try { __asm { int 3 } // Software breakpoint } __except(EXCEPTION_EXECUTE_HANDLER) { // Normal execution: exception caught return; } // Debugger ate the exception exit(1); // VEH-based detection LONG CALLBACK VectoredHandler(PEXCEPTION_POINTERS ep) { if (ep->ExceptionRecord->ExceptionCode == EXCEPTION_BREAKPOINT) { ep->ContextRecord->Rip++; // Skip INT3 return EXCEPTION_CONTINUE_EXECUTION; } return EXCEPTION_CONTINUE_SEARCH; } ``` ### Linux Anti-Debugging ```c // ptrace self-trace if (ptrace(PTRACE_TRACEME, 0, NULL, NULL) == -1) { // Already being traced exit(1); } // /proc/self/status FILE *f = fopen("/proc/self/status", "r"); char line[256]; while (fgets(line, sizeof(line), f)) { if (strncmp(line, "TracerPid:", 10) == 0) { int tracer_pid = atoi(line + 10); if (tracer_pid != 0) exit(1); } } // Parent process check if (getppid() != 1 && strcmp(get_process_name(getppid()), "bash") != 0) { // Unusual parent (might be debugger) } ``` **Bypass Approaches:** ```bash # LD_PRELOAD to hook ptrace # Compile: gcc -shared -fPIC -o hook.so hook.c long ptrace(int request, ...) { return 0; // Always succeed } # Usage LD_PRELOAD=./hook.so ./target ``` ## Anti-VM Detection ### Hardware Fingerprinting ```c // CPUID-based detection int cpuid_info[4]; __cpuid(cpuid_info, 1); // Check hypervisor bit (bit 31 of ECX) if (cpuid_info[2] & (1 << 31)) { // Running in hypervisor } // CPUID brand string __cpuid(cpuid_info, 0x40000000); char vendor[13] = {0}; memcpy(vendor, &cpuid_info[1], 12); // "VMwareVMware", "Microsoft Hv", "KVMKVMKVM", "VBoxVBoxVBox" // MAC address prefix // VMware: 00:0C:29, 00:50:56 // VirtualBox: 08:00:27 // Hyper-V: 00:15:5D ``` ### Registry/File Detection ```c // Windows registry keys // HKLM\SOFTWARE\VMware, Inc.\VMware Tools // HKLM\SOFTWARE\Oracle\VirtualBox Guest Additions // HKLM\HARDWARE\ACPI\DSDT\VBOX__ // Files // C:\Windows\System32\drivers\vmmouse.sys // C:\Windows\System32\drivers\vmhgfs.sys // C:\Windows\System32\drivers\VBoxMouse.sys // Processes // vmtoolsd.exe, vmwaretray.exe // VBoxService.exe, VBoxTray.exe ``` ### Timing-Based VM Detection ```c // VM exits cause timing anomalies uint64_t start = __rdtsc(); __cpuid(cpuid_info, 0); // Causes VM exit uint64_t end = __rdtsc(); if ((end - start) > 500) { // Likely in VM (CPUID takes longer) } ``` **Bypass Approaches:** ``` - Use bare-metal analysis environment - Harden VM (remove guest tools, change MAC) - Patch detection code - Use specialized analysis VMs (FLARE-VM) ``` ## Code Obfuscation ### Control Flow Obfuscation #### Control Flow Flattening ```c // Original if (cond) { func_a(); } else { func_b(); } func_c(); // Flattened int state = 0; while (1) { switch (state) { case 0: state = cond ? 1 : 2; break; case 1: func_a(); state = 3; break; case 2: func_b(); state = 3; break; case 3: func_c(); return; } } ``` **Analysis Approach:** - Identify state variable - Map state transitions - Reconstruct original flow - Tools: D-810 (IDA), SATURN #### Opaque Predicates ```c // Always true, but complex to analyze int x = rand(); if ((x * x) >= 0) { // Always true real_code(); } else { junk_code(); // Dead code } // Always false if ((x * (x + 1)) % 2 == 1) { // Product of consecutive = even junk_code(); } ``` **Analysis Approach:** - Identify constant expressions - Symbolic execution to prove predicates - Pattern matching for known opaque predicates ### Data Obfuscation #### String Encryption ```c // XOR encryption char decrypt_string(char *enc, int len, char key) { char *dec = malloc(len + 1); for (int i = 0; i < len; i++) { dec[i] = enc[i] ^ key; } dec[len] = 0; return dec; } // Stack strings char url[20]; url[0] = 'h'; url[1] = 't'; url[2] = 't'; url[3] = 'p'; url[4] = ':'; url[5] = '/'; url[6] = '/'; // ... ``` **Analysis Approach:** ```python # FLOSS for automatic string deobfuscation floss malware.exe # IDAPython string decryption def decrypt_xor(ea, length, key): result = "" for i in range(length): byte = ida_bytes.get_byte(ea + i) result += chr(byte ^ key) return result ``` #### API Obfuscation ```c // Dynamic API resolution typedef HANDLE (WINAPI *pCreateFileW)(LPCWSTR, DWORD, DWORD, LPSECURITY_ATTRIBUTES, DWORD, DWORD, HANDLE); HMODULE kernel32 = LoadLibraryA("kernel32.dll"); pCreateFileW myCreateFile = (pCreateFileW)GetProcAddress( kernel32, "CreateFileW"); // API hashing DWORD hash_api(char *name) { DWORD hash = 0; while (*name) { hash = ((hash >> 13) | (hash << 19)) + *name++; } return hash; } // Resolve by hash comparison instead of string ``` **Analysis Approach:** - Identify hash algorithm - Build hash database of known APIs - Use HashDB plugin for IDA - Dynamic analysis to resolve at runtime ### Instruction-Level Obfuscation #### Dead Code Insertion ```asm ; Original mov eax, 1 ; With dead code push ebx ; Dead mov eax, 1 pop ebx ; Dead xor ecx, ecx ; Dead add ecx, ecx ; Dead ``` #### Instruction Substitution ```asm ; Original: xor eax, eax (set to 0) ; Substitutions: sub eax, eax mov eax, 0 and eax, 0 lea eax, [0] ; Original: mov eax, 1 ; Substitutions: xor eax, eax inc eax push 1 pop eax ``` ## Packing and Encryption ### Common Packers ``` UPX - Open source, easy to unpack Themida - Commercial, VM-based protection VMProtect - Commercial, code virtualization ASPack - Compression packer PECompact - Compression packer Enigma - Commercial protector ``` ### Unpacking Methodology ``` 1. Identify packer (DIE, Exeinfo PE, PEiD) 2. Static unpacking (if known packer): - UPX: upx -d packed.exe - Use existing unpackers 3. Dynamic unpacking: a. Find Original Entry Point (OEP) b. Set breakpoint on OEP c. Dump memory when OEP reached d. Fix import table (Scylla, ImpREC) 4. OEP finding techniques: - Hardware breakpoint on stack (ESP trick) - Break on common API calls (GetCommandLineA) - Trace and look for typical entry patterns ``` ### Manual Unpacking Example ``` 1. Load packed binary in x64dbg 2. Note entry point (packer stub) 3. Use ESP trick: - Run to entry - Set hardware breakpoint on [ESP] - Run until breakpoint hits (after PUSHAD/POPAD) 4. Look for JMP to OEP 5. At OEP, use Scylla to: - Dump process - Find imports (IAT autosearch) - Fix dump ``` ## Virtualization-Based Protection ### Code Virtualization ``` Original x86 code is converted to custom bytecode interpreted by embedded VM at runtime. Original: VM Protected: mov eax, 1 push vm_context add eax, 2 call vm_entry ; VM interprets bytecode ; equivalent to original ``` ### Analysis Approaches ``` 1. Identify VM components: - VM entry (dispatcher) - Handler table - Bytecode location - Virtual registers/stack 2. Trace execution: - Log handler calls - Map bytecode to operations - Understand instruction set 3. Lifting/devirtualization: - Map VM instructions back to native - Tools: VMAttack, SATURN, NoVmp 4. Symbolic execution: - Analyze VM semantically - angr, Triton ``` ## Bypass Strategies Summary ### General Principles 1. **Understand the protection**: Identify what technique is used 2. **Find the check**: Locate protection code in binary 3. **Patch or hook**: Modify check to always pass 4. **Use appropriate tools**: ScyllaHide, x64dbg plugins 5. **Document findings**: Keep notes on bypassed protections ### Tool Recommendations ``` Anti-debug bypass: ScyllaHide, TitanHide Unpacking: x64dbg + Scylla, OllyDumpEx Deobfuscation: D-810, SATURN, miasm VM analysis: VMAttack, NoVmp, manual tracing String decryption: FLOSS, custom scripts Symbolic execution: angr, Triton ``` ### Ethical Considerations This knowledge should only be used for: - Authorized security research - Malware analysis (defensive) - CTF competitions - Understanding protections for legitimate purposes - Educational purposes Never use to bypass protections for: - Software piracy - Unauthorized access - Malicious purposes
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered