Skip to main content
EVOKORE// BROWSE
>

./browse/prompts

25 NODES
šŸ“text•6 months ago

ARCH-AEP Tiered Remediation Cycle

Run a scope-locked remediation cycle that normalizes findings, revalidates them in parallel, and clears tiers with verification evidence.

architecture
⭐1
# ARCH-AEP Tiered Remediation Cycle Imported from curated first-party documentation sources. ## What this covers Use this workflow when you need an orchestrated engineering review cycle with strict scope lock, tiered execution, and audit-grade evidence. ## Use this when - Large remediation efforts spanning multiple PRs - Parallel specialist review with shared backlog authority - Tracking verification evidence through closure ## Expected outcomes - Scope changes are controlled instead of creeping mid-cycle - Severity tiers drive remediation order predictably - Verification evidence is treated as part of done ## Source synthesis - CHELATEDAI/docs/ARCH AGENTIC ENGINEERING AND PLANNING/workflow.md (https://github.com/mattmre/CHELATEDAI/blob/main/docs/ARCH%20AGENTIC%20ENGINEERING%20AND%20PLANNING/workflow.md) - CHELATEDAI/docs/ARCH AGENTIC ENGINEERING AND PLANNING/orchestrator-briefing.md (https://github.com/mattmre/CHELATEDAI/blob/main/docs/ARCH%20AGENTIC%20ENGINEERING%20AND%20PLANNING/orchestrator-briefing.md) ## Dedupe notes Condenses the core ARCH-AEP workflow and briefing docs into a single remediation-cycle entry instead of importing both separately. ## Source excerpts ### CHELATEDAI/docs/ARCH AGENTIC ENGINEERING AND PLANNING/workflow.md ## Workflow (ARCH phase) 1. Scope lock - Define PR range and time window. - Freeze inputs (refinement report + framework + PR list). 2. Discovery + normalization - Orchestrator ingests `agentic-review-framework.md` and the latest refinement report. - Normalize all findings into a single backlog with unique IDs. - De-duplicate, merge overlaps, and assign provisional severity. - Record backlog in `docs/ARCH AGENTIC ENGINEERING AND PLANNING/backlog-YYYY-MM-DD.md`. 3. Parallel re-validation - Spawn specialist agents to re-validate findings against current main. - Each agent must attach exact file paths and line ranges. - Propose the smallest safe fix, acceptance criteria, and effort sizing (S/M/L). 4. Architecture and planning synthesis - Orchestrator merges validated findings into a master backlog. - Re-score and re-rank using the sorting model. - Identify dependency chains and blockers. ### CHELATEDAI/docs/ARCH AGENTIC ENGINEERING AND PLANNING/orchestrator-briefing.md Purpose: Quick reference for the ARCH-AEP documentation set, plus the narrative to start a new session. Full file index: `docs/INDEX.md` ## What Exists In This Folder - `README.md`: narrative overview and intent for ARCH-AEP. - `workflow.md`: end-to-end workflow specification with phases, guardrails, and enhancements. - `templates.md`: ID and branch conventions, tracker table format, and status log. - `backlog-template.md`: template for the master backlog file. - `backlog-index.md`: index of backlog files. - `next-session.md`: short handoff checklist for resuming work. - `phase-planning.md`: long-running planning record for the current cycle. - `schedule-and-tracking.md`: cadence, gates, and milestone tracking. - `agent-learning.md`: cross-session learnings and reusable patterns. - `change-log.md`: scope/defer decisions with audit context. - `risk-memo-template.md`: Critical/High risk memo template. - `risk-memos/README.md`: storage location for risk memos. - `phase-summary-template.md`: template for per-PR or per-phase summaries. - `phase-summaries/README.md`: storage location for phase summaries. - `verification-log.md`: test/build evidence mirror for PRs. - `tracker-pointer.md`: pointer to the active tracker file. - `tracker-index.md`: index of tracker files. - `scope-lock-t ...
šŸ‘0
šŸ‘ļø0
docs
šŸ“text•6 months ago

Submodule Hygiene Workflow

Add, update, validate, document, and review git submodule changes without losing track of dependency state.

productivity
⭐1
# Submodule Hygiene Workflow Imported from curated first-party documentation sources. ## What this covers Use this workflow when repositories rely on submodules and you need a disciplined way to update them safely. ## Use this when - Adding or updating repository submodules - Documenting submodule changes alongside code changes - Avoiding broken references during review ## Expected outcomes - Submodule changes are treated as a documented workflow - Validation happens before the review handoff - Dependency movement stays visible to the team ## Source synthesis - EVOKORE-MCP/docs/SUBMODULE_WORKFLOW.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/SUBMODULE_WORKFLOW.md) ## Dedupe notes Uses the dedicated submodule workflow doc because the guidance is operationally specific and not already represented on the site. ## Source excerpts ### EVOKORE-MCP/docs/SUBMODULE_WORKFLOW.md This repository may consume external content through git submodules. Use this workflow to keep documentation updates reviewable and predictable. ## 1) Add or Register a Submodule ```bash git submodule add <repo-url> <target-path> git submodule update --init --recursive ``` Commit both: - `.gitmodules` - the submodule pointer change at `<target-path>` ## 2) Pull Latest Submodule Content ```bash git submodule update --remote --merge git submodule update --init --recursive ``` Then inspect: ```bash git status git diff --submodule ``` ## 3) Validate Submodule Cleanliness (Local + CI) Run the same guard used in CI before opening your PR: ```bash git submodule status --recursive node scripts/validate-submodule-cleanliness.js ``` Cleanliness semantics: - `-` => uninitialized submodule - `+` => submodule commit mismatch (worktree vs gitlink pointer) - `U` => submodule merge conflict - dirty submodule worktree => non-empty `git -C <submodule> status --porcelain` Submodule paths can include spaces (for example `SKILLS/ANTHROPIC COOKBOOK`), so always quote path arguments when running manual `git -C` commands. ## 4) Update Docs in This Repo When a submodule changes behavior, update: - `docs/README.md` if canonical links changed - `docs/USAGE.md` / `docs/TROUBLESHOOTING.md` if runtime guidance changed - `README.md` / `CONTRIBUTING.md` if contributor workflow changed ## 5) PR Expectations For submodule-related PRs: 1. Commit inside the submodule first. 2. Return to the parent repo and verify `git submodule status` has no unexpected `-dirty` entries. 3. Commit the updated submodule pointer in the parent repository. 4. Include any matching docs updates in this repository. 5. Mention the upstream submodule commit SHA in the PR description.
šŸ‘0
šŸ‘ļø0
docs
šŸ“text•6 months ago

PR Merge Boundary Validation Runbook

Validate chained pull requests at each merge boundary with explicit checks, stale-approval handling, and rollback awareness.

devops
⭐1
# PR Merge Boundary Validation Runbook Imported from curated first-party documentation sources. ## What this covers Use this runbook when a merge sequence spans dependent pull requests and each boundary needs fresh validation. ## Use this when - Managing dependency-chained pull requests - Refreshing stale approvals near merge time - Reducing merge surprises through checkpoint validation ## Expected outcomes - Each merge boundary has a concrete verification step - Review freshness stays visible as branches evolve - Rollback planning is captured before the merge happens ## Source synthesis - EVOKORE-MCP/docs/PR_MERGE_RUNBOOK.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/PR_MERGE_RUNBOOK.md) ## Dedupe notes Keeps the merge-boundary validation concept separate from more general workflow or orchestration docs. ## Source excerpts ### EVOKORE-MCP/docs/PR_MERGE_RUNBOOK.md Operator runbook for reliable merges and context-rot prevention. ## Pre-merge Checklist - [ ] PR scope matches approved plan - [ ] PR description is filled using `.github/PULL_REQUEST_TEMPLATE.md` - [ ] PR metadata automation check (`scripts/validate-pr-metadata.js`) is passing for pull_request CI runs - [ ] Required tests pass locally/CI - [ ] Docs updated for user-facing behavior changes - [ ] Release-impacting changes called out - [ ] Follow-up issues captured (if any) ## Required Checks by Change Type Use this as the minimum check set before approval and merge: | Change type | Required checks | | --- | --- | | Docs-only changes | `node test-docs-canonical-links.js` | | Ops/docs process changes (`docs/PR_MERGE_RUNBOOK.md`, `next-session.md`, orchestration docs) | `node test-ops-docs-validation.js` and `node test-docs-canonical-links.js` | | Source/tooling/config changes (`src/`, `scripts/`, workflow/config files) | Relevant targeted tests for touched area plus CI-required suite | | Release-flow changes | `node test-npm-release-flow-validation.js` plus docs/link checks | If a PR spans multiple change types, run the union of required checks. ## Reviewer Responsibilities - Confirm PR scope and dependency assumptions are explicit in description. - Confirm PR metadata fields from `.github/PULL_REQUEST_TEMPLATE.md` are complete. - Verify required checks for each change type are attached in PR evidence. - Block approval if dependency base PR is not merged or branch is stale. - Approve only the current chain head; do not pre-approve non-head dependent PRs. - Confirm all review conversations are resolved before final approval. - Ensure merge strategy and rollback notes are documented for risky changes. ## Merge Steps 1. Rebase or update branch with latest target branch. 2. Re-run required validations. 3. Confirm reviewer approvals and resolved conversations. 4. Merge via approved strategy. 5. Record merge commit/PR number in tracker. ## Merge-boundary Checkpoints At every dependency merge boundary (`base -> dependent`): 1. Rebase dependent PR branch on latest `main` immediately after parent merge. 2. Re-run required checks and attach updated evidence in PR metadata. 3. Revalidate approvals for the new head state (stale approvals must be refreshed). 4. Confirm merge-boundary revalidation notes are updated before merge. ## Merge-order Controls (Dependency Chain) 1. Define merge order explicitly in PR descriptions (`base -> dependent`). 2. Merge only the current chain head; hold dependents until parent merge is complete. 3. After each merge, rebase ...
šŸ‘0
šŸ‘ļø0
docs
šŸ¤–system prompt•6 months ago

Dynamic Tool Discovery and Session Activation

Expose only native tools at session start, then activate proxied tools on demand through a searchable discovery layer.

architecture
⭐1
# Dynamic Tool Discovery and Session Activation Imported from curated first-party documentation sources. ## What this covers Use this skill when you need smaller tool payloads, exact-name compatibility, and session-scoped tool activation. ## Use this when - Reducing tool-list bloat for focused sessions - Preserving exact-name tool calls while hiding noise - Activating proxied tools only when a task requires them ## Expected outcomes - Discovery changes listing visibility without breaking routing - Session-scoped activation keeps tools relevant to the task - Operators can trade compatibility against payload size intentionally ## Source synthesis - EVOKORE-MCP/docs/TOOLS_AND_DISCOVERY.md (https://github.com/mattmre/EVOKORE-MCP/blob/main/docs/TOOLS_AND_DISCOVERY.md) ## Dedupe notes Uses the canonical EVOKORE-MCP discovery doc instead of duplicating the same concept from walkthrough and migration docs. ## Source excerpts ### EVOKORE-MCP/docs/TOOLS_AND_DISCOVERY.md This page explains how EVOKORE presents tools, how proxy names are built, and how `discover_tools` changes the visible tool surface. ## Two tool populations ### Native EVOKORE tools These tools are defined by EVOKORE itself: - `docs_architect` - `skill_creator` - `resolve_workflow` - `search_skills` - `get_skill_help` - `discover_tools` Properties: - always available - always visible - not subject to proxy prefixing ### Proxied child-server tools These come from child servers in `mcp.config.json`. Current configured sources: - `github` - `fs` - optional `elevenlabs` Properties: - fetched from child servers at startup - renamed with server prefixes - governed by `permissions.yml` - routed through `ProxyManager` ## Prefixing and compatibility EVOKORE rewrites proxied tool names to: ```text ${serverId}_${tool.name} ``` Why this exists: - prevents tool-name collisions across child servers - makes origin obvious during execution and review - keeps exact-name routing deterministic Examples: | Upstream tool | EVOKORE-exposed tool | |---|---| | `read_file` from `fs` | `fs_read_file` | | `create_issue` from `github` | `github_create_issue` | ### Duplicate-prefixed name policy If two child registrations would create the same final prefixed name: - the first registration wins - later duplicates are skipped - EVOKORE logs a warning and duplicate summary ## Discovery modes | Mode | What `tools/list` returns | Best for | |---|---|---| | `legacy` | all native + proxied tools | maximum compatibility | | `dynamic` | native tools + session-activated proxied tools | smaller initial tool payloads | Environment toggle: ```bash EVOKORE_TOOL_DISCOVERY_MODE=legacy EVOKORE_TOOL_DISCOVERY_MODE=dynamic ``` ## Dynamic discovery lifecycle In `dynamic` mode, EVOKORE uses a session-scoped activation set. Lifecycle: 1. session starts with only native tools visible 2. user/model calls `discover_tools` 3. `ToolCatalogIndex` searches the merged native + proxied catalog 4. matching proxied tools are activated for that session 5. EVOKORE emits `sendToolListChanged()` best-effort 6. client re-runs `tools/list` or auto-refreshes ```mermaid flowchart TD A[Session starts in dynamic mode] --> B[tools/list shows native tools] B --> C[discover_tools query] C --> D[ToolCatalogIndex searches merged catalog] D --> E[Matching proxied tools added to session activation set] E --> F[sendToolListChanged best-effort] F --> G[Client refreshes tools/list] G --> H[Activated proxied tools now visible] D --> I[Exact-name proxied call still works eve ...
šŸ‘0
šŸ‘ļø0
docs
šŸ“text•6 months ago

Improvement Cycle Review Wizard

Run a live improvement-cycle workflow that links review artifacts, approvals, tool requests, and operator checkpoints.

productivity
⭐1
# Improvement Cycle Review Wizard Imported from curated first-party documentation sources. ## What this covers Use this workflow when an operator needs a guided improvement cycle with live status, explicit approvals, and linked artifacts. ## Use this when - Operational review sessions with live progress - Approval-heavy improvement cycles - Coordinating review artifacts with execution steps ## Expected outcomes - Live workflow execution remains visible to operators - Artifacts, approvals, and tool requests stay connected - Improvement loops become easier to audit and rerun ## Source synthesis - AGENT33/docs/phase25-26-live-review-walkthrough.md (https://github.com/mattmre/AGENT33/blob/main/docs/phase25-26-live-review-walkthrough.md) - AGENT33/docs/operator-improvement-cycle-and-jupyter.md (https://github.com/mattmre/AGENT33/blob/main/docs/operator-improvement-cycle-and-jupyter.md) ## Dedupe notes Merges AGENT33 live review walkthrough details with the shorter operator-focused improvement-cycle guide. ## Source excerpts ### AGENT33/docs/phase25-26-live-review-walkthrough.md This guide documents the operator flow introduced by the Phase 25 live workflow transport and the Phase 26/27 improvement-cycle review wizard stack. ## Scope - Live workflow execution with run-scoped graph refresh - WebSocket-first status streaming with authenticated SSE fallback - Improvement-cycle preset creation - Explanation artifact generation for `plan_review` and `diff_review` - Linked review creation, risk assessment, L1/L2 signoff, and final approval - Pending tool-approval triage inside the same workflow domain > Note > This walkthrough reflects the review stack built on `codex/session58-phase26-wizard` and validated in `codex/session60-phase22-docs-validation`. If the related PRs are still open, `main` may not yet expose every surface described here. ## Operator Flow ```mermaid sequenceDiagram participant UI as "Frontend Control Plane" participant WF as "Workflow APIs" participant VIZ as "Visualization APIs" participant EXP as "Explanation APIs" participant REV as "Review APIs" participant HITL as "Tool Approval APIs" UI->>WF: "POST /v1/workflows/{name}/execute (single mode, caller run_id)" UI->>VIZ: "GET /v1/visualizations/workflows/{workflow_id}/graph?run_id=..." UI->>WF: "WS /v1/workflows/{run_id}/ws" alt "WebSocket unav ... ### AGENT33/docs/operator-improvement-cycle-and-jupyter.md This guide covers the merged operator surfaces for: - the Phase 26 improvement-cycle review wizard - the Phase 27 canonical workflow presets - the Phase 38 Docker-backed Jupyter kernel workflow Use it when you want the shortest current path from UI entry point to a real workflow run. ## 1. Improvement-Cycle Wizard The wizard is mounted inside the frontend control plane under the `Workflows` domain. ### Entry path 1. Open the frontend at `http://localhost:3000` 2. Authenticate with a bearer token or API key 3. Open `Advanced Settings` 4. Select the `Workflows` domain 5. Use the `Improvement Cycle Wizard` panel at the top of the page ### What the wizard does The wizard stitches together the backend surfaces that previously had to be called manually: - plan review / diff review generation - review creation and risk assessment - L1 and L2 review submission - tool approval request review and decision capture Reference implementation: - frontend: `frontend/src/features/improvement-cycle/ImprovementCycleWizard.tsx` - tests: `frontend/src/features/improvement-cycle/ImprovementCycleWizard.test.tsx` For the detailed review flow, keep using: - [`phase25-26-live-review-walkthrough.md`](phase25-26-live-review-walkthrough.md) ## 2. Canonical Workflow Presets The `Workflows` doma ...
šŸ‘0
šŸ‘ļø0
docs
šŸ“text•6 months ago

Release Command Center Workflow

Coordinate release freezes, candidate promotion, validation, publication, and rollback handling through a defined state model.

devops
⭐1
# Release Command Center Workflow Imported from curated first-party documentation sources. ## What this covers Use this workflow to move a release from planning to publication with explicit validation checkpoints and rollback branches. ## Use this when - Managing release freezes and candidate promotion - Tracking validation state before publication - Documenting rollback-friendly release operations ## Expected outcomes - Release state changes are visible and repeatable - Rollback paths are treated as first-class outcomes - Validation discipline is preserved between phases ## Source synthesis - AGENT33/docs/functionality-and-workflows.md (https://github.com/mattmre/AGENT33/blob/main/docs/functionality-and-workflows.md) - AGENT33/docs/use-cases.md (https://github.com/mattmre/AGENT33/blob/main/docs/use-cases.md) ## Dedupe notes Merges AGENT33 release lifecycle details with release-oriented use-case guidance. ## Source excerpts ### AGENT33/docs/functionality-and-workflows.md ### 4.2 Release Lifecycle States: - `planned -> frozen -> rc -> validating -> released` - Failure/rollback branches: `failed`, `rolled_back` Main APIs: - `/v1/releases/{id}/freeze` - `/v1/releases/{id}/rc` - `/v1/releases/{id}/validate` - `/v1/releases/{id}/publish` - `/v1/releases/{id}/rollback` ### AGENT33/docs/use-cases.md ## 2. Release Command Center Goal: - Move releases through frozen -> RC -> validate -> publish with checklist controls. Use these modules: - `api/routes/releases.py` - `release/service.py` - `release/checklist.py` - `release/sync.py` - `release/rollback.py` Typical flow: 1. Create release (`POST /v1/releases`). 2. Freeze, cut RC, validate. 3. Update/evaluate checklist items. 4. Publish when checks pass. 5. Run sync dry-runs or real syncs. 6. Initiate rollback if needed. Best fit: - Teams with repeatable release compliance requirements.
šŸ‘0
šŸ‘ļø0
docs
šŸ“text•6 months ago

Guardrailed Code Review Pipeline

Run a staged code review workflow with explicit risk assessment, reviewer handoffs, and merge gates.

coding
⭐1
# Guardrailed Code Review Pipeline Imported from curated first-party documentation sources. ## What this covers Use this workflow when a change needs structured review evidence, clear approval stages, and a rollback-aware path to merge. ## Use this when - Multi-step L1 and L2 review signoff - High-risk changes that need audit trails - Standardizing reviewer decisions before merge ## Expected outcomes - Review states and approval checkpoints stay explicit - Risk assessment and evidence are captured before merge - The workflow remains small enough to reuse across repositories ## Source synthesis - AGENT33/docs/functionality-and-workflows.md (https://github.com/mattmre/AGENT33/blob/main/docs/functionality-and-workflows.md) - AGENT33/docs/walkthroughs.md (https://github.com/mattmre/AGENT33/blob/main/docs/walkthroughs.md) ## Dedupe notes Combines AGENT33 lifecycle mapping and operator walkthrough material into one review-oriented site entry. ## Source excerpts ### AGENT33/docs/functionality-and-workflows.md ### 4.1 Review Lifecycle States: - `draft -> ready -> l1-review -> l1-approved -> (optional l2-review -> l2-approved) -> approved -> merged` Main APIs: - `/v1/reviews/{id}/assess` - `/v1/reviews/{id}/assign-l1` - `/v1/reviews/{id}/l1` - `/v1/reviews/{id}/assign-l2` - `/v1/reviews/{id}/l2` - `/v1/reviews/{id}/approve` - `/v1/reviews/{id}/merge` ### AGENT33/docs/walkthroughs.md ## 4. Review Lifecycle (Two-Layer Signoff) Create review: ```bash curl -X POST http://localhost:8000/v1/reviews/ \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"task_id":"TASK-101","branch":"feat/docs-refresh","pr_number":12}' ``` Assess risk: ```bash curl -X POST http://localhost:8000/v1/reviews/<review_id>/assess \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"triggers":["api-public","security"]}' ``` Move to ready and assign L1: ```bash curl -X POST http://localhost:8000/v1/reviews/<review_id>/ready -H "Authorization: Bearer $TOKEN" curl -X POST http://localhost:8000/v1/reviews/<review_id>/assign-l1 -H "Authorization: Bearer $TOKEN" ``` Submit L1 decision: ```bash curl -X POST http://localhost:8000/v1/reviews/<review_id>/l1 \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"decision":"approved","issues":[],"comments":"L1 pass"}' ``` If L2 required, continue: ```bash curl -X POST http://localhost:8000/v1/reviews/<review_id>/assign-l2 -H "Authorization: Bearer $TOKEN" curl -X POST http://localhost:8000/v1/reviews/<review_id>/l2 \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -d '{"decision":"approved","issues":[],"commen ...
šŸ‘0
šŸ‘ļø0
docs
šŸ¤–system prompt•7 months ago

spark-optimization

Optimize Apache Spark jobs with partitioning, caching, shuffle

data
⭐1
# Apache Spark Optimization Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning. ## When to Use This Skill - Optimizing slow Spark jobs - Tuning memory and executor configuration - Implementing efficient partitioning strategies - Debugging Spark performance issues - Scaling Spark pipelines for large datasets - Reducing shuffle and data skew ## Core Concepts ### 1. Spark Execution Model ``` Driver Program ↓ Job (triggered by action) ↓ Stages (separated by shuffles) ↓ Tasks (one per partition) ``` ### 2. Key Performance Factors | Factor | Impact | Solution | | ----------------- | --------------------- | ----------------------------- | | **Shuffle** | Network I/O, disk I/O | Minimize wide transformations | | **Data Skew** | Uneven task duration | Salting, broadcast joins | | **Serialization** | CPU overhead | Use Kryo, columnar formats | | **Memory** | GC pressure, spills | Tune executor memory | | **Partitions** | Parallelism | Right-size partitions | ## Quick Start ```python from pyspark.sql import SparkSession from pyspark.sql import functions as F # Create optimized Spark session spark = (SparkSession.builder .appName("OptimizedJob") .config("spark.sql.adaptive.enabled", "true") .config("spark.sql.adaptive.coalescePartitions.enabled", "true") .config("spark.sql.adaptive.skewJoin.enabled", "true") .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") .config("spark.sql.shuffle.partitions", "200") .getOrCreate()) # Read with optimized settings df = (spark.read .format("parquet") .option("mergeSchema", "false") .load("s3://bucket/data/")) # Efficient transformations result = (df .filter(F.col("date") >= "2024-01-01") .select("id", "amount", "category") .groupBy("category") .agg(F.sum("amount").alias("total"))) result.write.mode("overwrite").parquet("s3://bucket/output/") ``` ## Patterns ### Pattern 1: Optimal Partitioning ```python # Calculate optimal partition count def calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int: """ Optimal partition size: 128MB - 256MB Too few: Under-utilization, memory pressure Too many: Task scheduling overhead """ return max(int(data_size_gb * 1024 / partition_size_mb), 1) # Repartition for even distribution df_repartitioned = df.repartition(200, "partition_key") # Coalesce to reduce partitions (no shuffle) df_coalesced = df.coalesce(100) # Partition pruning with predicate pushdown df = (spark.read.parquet("s3://bucket/data/") .filter(F.col("date") == "2024-01-01")) # Spark pushes this down # Write with partitioning for future queries (df.write .partitionBy("year", "month", "day") .mode("overwrite") .parquet("s3://bucket/partitioned_output/")) ``` ### Pattern 2: Join Optimization ```python from pyspark.sql import functions as F from pyspark.sql.types import * # 1. Broadcast Join - Small table joins # Best when: One side < 10MB (configurable) small_df = spark.read.parquet("s3://bucket/small_table/") # < 10MB large_df = spark.read.parquet("s3://bucket/large_table/") # TBs # Explicit broadcast hint result = large_df.join( F.broadcast(small_df), on="key", how="left" ) # 2. Sort-Merge Join - Default for large tables # Requires shuffle, but handles any size result = large_df1.join(large_df2, on="key", how="inner") # 3. Bucket Join - Pre-sorted, no shuffle at join time # Write bucketed tables (df.write .bucketBy(200, "customer_id") .sortBy("customer_id") .mode("overwrite") .saveAsTable("bucketed_orders")) # Join bucketed tables (no shuffle!) orders = spark.table("bucketed_orders") customers = spark.table("bucketed_customers") # Same bucket count result = orders.join(customers, on="customer_id") # 4. Skew Join Handling # Enable AQE skew join optimization spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true") spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", "5") spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "256MB") # Manual salting for severe skew def salt_join(df_skewed, df_other, key_col, num_salts=10): """Add salt to distribute skewed keys""" # Add salt to skewed side df_salted = df_skewed.withColumn( "salt", (F.rand() * num_salts).cast("int") ).withColumn( "salted_key", F.concat(F.col(key_col), F.lit("_"), F.col("salt")) ) # Explode other side with all salts df_exploded = df_other.crossJoin( spark.range(num_salts).withColumnRenamed("id", "salt") ).withColumn( "salted_key", F.concat(F.col(key_col), F.lit("_"), F.col("salt")) ) # Join on salted key return df_salted.join(df_exploded, on="salted_key", how="inner") ``` ### Pattern 3: Caching and Persistence ```python from pyspark import StorageLevel # Cache when reusing DataFrame multiple times df = spark.read.parquet("s3://bucket/data/") df_filtered = df.filter(F.col("status") == "active") # Cache in memory (MEMORY_AND_DISK is default) df_filtered.cache() # Or with specific storage level df_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER) # Force materialization df_filtered.count() # Use in multiple actions agg1 = df_filtered.groupBy("category").count() agg2 = df_filtered.groupBy("region").sum("amount") # Unpersist when done df_filtered.unpersist() # Storage levels explained: # MEMORY_ONLY - Fast, but may not fit # MEMORY_AND_DISK - Spills to disk if needed (recommended) # MEMORY_ONLY_SER - Serialized, less memory, more CPU # DISK_ONLY - When memory is tight # OFF_HEAP - Tungsten off-heap memory # Checkpoint for complex lineage spark.sparkContext.setCheckpointDir("s3://bucket/checkpoints/") df_complex = (df .join(other_df, "key") .groupBy("category") .agg(F.sum("amount"))) df_complex.checkpoint() # Breaks lineage, materializes ``` ### Pattern 4: Memory Tuning ```python # Executor memory configuration # spark-submit --executor-memory 8g --executor-cores 4 # Memory breakdown (8GB executor): # - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage) # - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache) # - Remaining 2.4GB for execution (shuffles, joins, sorts) # - 40% = 3.2GB for user data structures and internal metadata spark = (SparkSession.builder .config("spark.executor.memory", "8g") .config("spark.executor.memoryOverhead", "2g") # For non-JVM memory .config("spark.memory.fraction", "0.6") .config("spark.memory.storageFraction", "0.5") .config("spark.sql.shuffle.partitions", "200") # For memory-intensive operations .config("spark.sql.autoBroadcastJoinThreshold", "50MB") # Prevent OOM on large shuffles .config("spark.sql.files.maxPartitionBytes", "128MB") .getOrCreate()) # Monitor memory usage def print_memory_usage(spark): """Print current memory usage""" sc = spark.sparkContext for executor in sc._jsc.sc().getExecutorMemoryStatus().keySet().toArray(): mem_status = sc._jsc.sc().getExecutorMemoryStatus().get(executor) total = mem_status._1() / (1024**3) free = mem_status._2() / (1024**3) print(f"{executor}: {total:.2f}GB total, {free:.2f}GB free") ``` ### Pattern 5: Shuffle Optimization ```python # Reduce shuffle data size spark.conf.set("spark.sql.shuffle.partitions", "auto") # With AQE spark.conf.set("spark.shuffle.compress", "true") spark.conf.set("spark.shuffle.spill.compress", "true") # Pre-aggregate before shuffle df_optimized = (df # Local aggregation first (combiner) .groupBy("key", "partition_col") .agg(F.sum("value").alias("partial_sum")) # Then global aggregation .groupBy("key") .agg(F.sum("partial_sum").alias("total"))) # Avoid shuffle with map-side operations # BAD: Shuffle for each distinct distinct_count = df.select("category").distinct().count() # GOOD: Approximate distinct (no shuffle) approx_count = df.select(F.approx_count_distinct("category")).collect()[0][0] # Use coalesce instead of repartition when reducing partitions df_reduced = df.coalesce(10) # No shuffle # Optimize shuffle with compression spark.conf.set("spark.io.compression.codec", "lz4") # Fast compression ``` ### Pattern 6: Data Format Optimization ```python # Parquet optimizations (df.write .option("compression", "snappy") # Fast compression .option("parquet.block.size", 128 * 1024 * 1024) # 128MB row groups .parquet("s3://bucket/output/")) # Column pruning - only read needed columns df = (spark.read.parquet("s3://bucket/data/") .select("id", "amount", "date")) # Spark only reads these columns # Predicate pushdown - filter at storage level df = (spark.read.parquet("s3://bucket/partitioned/year=2024/") .filter(F.col("status") == "active")) # Pushed to Parquet reader # Delta Lake optimizations (df.write .format("delta") .option("optimizeWrite", "true") # Bin-packing .option("autoCompact", "true") # Compact small files .mode("overwrite") .save("s3://bucket/delta_table/")) # Z-ordering for multi-dimensional queries spark.sql(""" OPTIMIZE delta.`s3://bucket/delta_table/` ZORDER BY (customer_id, date) """) ``` ### Pattern 7: Monitoring and Debugging ```python # Enable detailed metrics spark.conf.set("spark.sql.codegen.wholeStage", "true") spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true") # Explain query plan df.explain(mode="extended") # Modes: simple, extended, codegen, cost, formatted # Get physical plan statistics df.explain(mode="cost") # Monitor task metrics def analyze_stage_metrics(spark): """Analyze recent stage metrics""" status_tracker = spark.sparkContext.statusTracker() for stage_id in status_tracker.getActiveStageIds(): stage_info = status_tracker.getStageInfo(stage_id) print(f"Stage {stage_id}:") print(f" Tasks: {stage_info.numTasks}") print(f" Completed: {stage_info.numCompletedTasks}") print(f" Failed: {stage_info.numFailedTasks}") # Identify data skew def check_partition_skew(df): """Check for partition skew""" partition_counts = (df .withColumn("partition_id", F.spark_partition_id()) .groupBy("partition_id") .count() .orderBy(F.desc("count"))) partition_counts.show(20) stats = partition_counts.select( F.min("count").alias("min"), F.max("count").alias("max"), F.avg("count").alias("avg"), F.stddev("count").alias("stddev") ).collect()[0] skew_ratio = stats["max"] / stats["avg"] print(f"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)") ``` ## Configuration Cheat Sheet ```python # Production configuration template spark_configs = { # Adaptive Query Execution (AQE) "spark.sql.adaptive.enabled": "true", "spark.sql.adaptive.coalescePartitions.enabled": "true", "spark.sql.adaptive.skewJoin.enabled": "true", # Memory "spark.executor.memory": "8g", "spark.executor.memoryOverhead": "2g", "spark.memory.fraction": "0.6", "spark.memory.storageFraction": "0.5", # Parallelism "spark.sql.shuffle.partitions": "200", "spark.default.parallelism": "200", # Serialization "spark.serializer": "org.apache.spark.serializer.KryoSerializer", "spark.sql.execution.arrow.pyspark.enabled": "true", # Compression "spark.io.compression.codec": "lz4", "spark.shuffle.compress": "true", # Broadcast "spark.sql.autoBroadcastJoinThreshold": "50MB", # File handling "spark.sql.files.maxPartitionBytes": "128MB", "spark.sql.files.openCostInBytes": "4MB", } ``` ## Best Practices ### Do's - **Enable AQE** - Adaptive query execution handles many issues - **Use Parquet/Delta** - Columnar formats with compression - **Broadcast small tables** - Avoid shuffle for small joins - **Monitor Spark UI** - Check for skew, spills, GC - **Right-size partitions** - 128MB - 256MB per partition ### Don'ts - **Don't collect large data** - Keep data distributed - **Don't use UDFs unnecessarily** - Use built-in functions - **Don't over-cache** - Memory is limited - **Don't ignore data skew** - It dominates job time - **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()` ## Resources - [Spark Performance Tuning](https://spark.apache.org/docs/latest/sql-performance-tuning.html) - [Spark Configuration](https://spark.apache.org/docs/latest/configuration.html) - [Databricks Optimization Guide](https://docs.databricks.com/en/optimizations/index.html)
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

postgresql-table-design

Design a PostgreSQL-specific schema. Covers best-practices, data

data
⭐1
# PostgreSQL Table Design ## Core Rules - Define a **PRIMARY KEY** for reference tables (users, orders, etc.). Not always needed for time-series/event/log data. When used, prefer `BIGINT GENERATED ALWAYS AS IDENTITY`; use `UUID` only when global uniqueness/opacity is needed. - **Normalize first (to 3NF)** to eliminate data redundancy and update anomalies; denormalize **only** for measured, high-ROI reads where join performance is proven problematic. Premature denormalization creates maintenance burden. - Add **NOT NULL** everywhere it’s semantically required; use **DEFAULT**s for common values. - Create **indexes for access paths you actually query**: PK/unique (auto), **FK columns (manual!)**, frequent filters/sorts, and join keys. - Prefer **TIMESTAMPTZ** for event time; **NUMERIC** for money; **TEXT** for strings; **BIGINT** for integer values, **DOUBLE PRECISION** for floats (or `NUMERIC` for exact decimal arithmetic). ## PostgreSQL ā€œGotchasā€ - **Identifiers**: unquoted → lowercased. Avoid quoted/mixed-case names. Convention: use `snake_case` for table/column names. - **Unique + NULLs**: UNIQUE allows multiple NULLs. Use `UNIQUE (...) NULLS NOT DISTINCT` (PG15+) to restrict to one NULL. - **FK indexes**: PostgreSQL **does not** auto-index FK columns. Add them. - **No silent coercions**: length/precision overflows error out (no truncation). Example: inserting 999 into `NUMERIC(2,0)` fails with error, unlike some databases that silently truncate or round. - **Sequences/identity have gaps** (normal; don't "fix"). Rollbacks, crashes, and concurrent transactions create gaps in ID sequences (1, 2, 5, 6...). This is expected behavior—don't try to make IDs consecutive. - **Heap storage**: no clustered PK by default (unlike SQL Server/MySQL InnoDB); `CLUSTER` is one-off reorganization, not maintained on subsequent inserts. Row order on disk is insertion order unless explicitly clustered. - **MVCC**: updates/deletes leave dead tuples; vacuum handles them—design to avoid hot wide-row churn. ## Data Types - **IDs**: `BIGINT GENERATED ALWAYS AS IDENTITY` preferred (`GENERATED BY DEFAULT` also fine); `UUID` when merging/federating/used in a distributed system or for opaque IDs. Generate with `uuidv7()` (preferred if using PG18+) or `gen_random_uuid()` (if using an older PG version). - **Integers**: prefer `BIGINT` unless storage space is critical; `INTEGER` for smaller ranges; avoid `SMALLINT` unless constrained. - **Floats**: prefer `DOUBLE PRECISION` over `REAL` unless storage space is critical. Use `NUMERIC` for exact decimal arithmetic. - **Strings**: prefer `TEXT`; if length limits needed, use `CHECK (LENGTH(col) <= n)` instead of `VARCHAR(n)`; avoid `CHAR(n)`. Use `BYTEA` for binary data. Large strings/binary (>2KB default threshold) automatically stored in TOAST with compression. TOAST storage: `PLAIN` (no TOAST), `EXTENDED` (compress + out-of-line), `EXTERNAL` (out-of-line, no compress), `MAIN` (compress, keep in-line if possible). Default `EXTENDED` usually optimal. Control with `ALTER TABLE tbl ALTER COLUMN col SET STORAGE strategy` and `ALTER TABLE tbl SET (toast_tuple_target = 4096)` for threshold. Case-insensitive: for locale/accent handling use non-deterministic collations; for plain ASCII use expression indexes on `LOWER(col)` (preferred unless column needs case-insensitive PK/FK/UNIQUE) or `CITEXT`. - **Money**: `NUMERIC(p,s)` (never float). - **Time**: `TIMESTAMPTZ` for timestamps; `DATE` for date-only; `INTERVAL` for durations. Avoid `TIMESTAMP` (without timezone). Use `now()` for transaction start time, `clock_timestamp()` for current wall-clock time. - **Booleans**: `BOOLEAN` with `NOT NULL` constraint unless tri-state values are required. - **Enums**: `CREATE TYPE ... AS ENUM` for small, stable sets (e.g. US states, days of week). For business-logic-driven and evolving values (e.g. order statuses) → use TEXT (or INT) + CHECK or lookup table. - **Arrays**: `TEXT[]`, `INTEGER[]`, etc. Use for ordered lists where you query elements. Index with **GIN** for containment (`@>`, `<@`) and overlap (`&&`) queries. Access: `arr[1]` (1-indexed), `arr[1:3]` (slicing). Good for tags, categories; avoid for relations—use junction tables instead. Literal syntax: `'{val1,val2}'` or `ARRAY[val1,val2]`. - **Range types**: `daterange`, `numrange`, `tstzrange` for intervals. Support overlap (`&&`), containment (`@>`), operators. Index with **GiST**. Good for scheduling, versioning, numeric ranges. Pick a bounds scheme and use it consistently; prefer `[)` (inclusive/exclusive) by default. - **Network types**: `INET` for IP addresses, `CIDR` for network ranges, `MACADDR` for MAC addresses. Support network operators (`<<`, `>>`, `&&`). - **Geometric types**: `POINT`, `LINE`, `POLYGON`, `CIRCLE` for 2D spatial data. Index with **GiST**. Consider **PostGIS** for advanced spatial features. - **Text search**: `TSVECTOR` for full-text search documents, `TSQUERY` for search queries. Index `tsvector` with **GIN**. Always specify language: `to_tsvector('english', col)` and `to_tsquery('english', 'query')`. Never use single-argument versions. This applies to both index expressions and queries. - **Domain types**: `CREATE DOMAIN email AS TEXT CHECK (VALUE ~ '^[^@]+@[^@]+$')` for reusable custom types with validation. Enforces constraints across tables. - **Composite types**: `CREATE TYPE address AS (street TEXT, city TEXT, zip TEXT)` for structured data within columns. Access with `(col).field` syntax. - **JSONB**: preferred over JSON; index with **GIN**. Use only for optional/semi-structured attrs. ONLY use JSON if the original ordering of the contents MUST be preserved. - **Vector types**: `vector` type by `pgvector` for vector similarity search for embeddings. ### Do not use the following data types - DO NOT use `timestamp` (without time zone); DO use `timestamptz` instead. - DO NOT use `char(n)` or `varchar(n)`; DO use `text` instead. - DO NOT use `money` type; DO use `numeric` instead. - DO NOT use `timetz` type; DO use `timestamptz` instead. - DO NOT use `timestamptz(0)` or any other precision specification; DO use `timestamptz` instead - DO NOT use `serial` type; DO use `generated always as identity` instead. ## Table Types - **Regular**: default; fully durable, logged. - **TEMPORARY**: session-scoped, auto-dropped, not logged. Faster for scratch work. - **UNLOGGED**: persistent but not crash-safe. Faster writes; good for caches/staging. ## Row-Level Security Enable with `ALTER TABLE tbl ENABLE ROW LEVEL SECURITY`. Create policies: `CREATE POLICY user_access ON orders FOR SELECT TO app_users USING (user_id = current_user_id())`. Built-in user-based access control at the row level. ## Constraints - **PK**: implicit UNIQUE + NOT NULL; creates a B-tree index. - **FK**: specify `ON DELETE/UPDATE` action (`CASCADE`, `RESTRICT`, `SET NULL`, `SET DEFAULT`). Add explicit index on referencing column—speeds up joins and prevents locking issues on parent deletes/updates. Use `DEFERRABLE INITIALLY DEFERRED` for circular FK dependencies checked at transaction end. - **UNIQUE**: creates a B-tree index; allows multiple NULLs unless `NULLS NOT DISTINCT` (PG15+). Standard behavior: `(1, NULL)` and `(1, NULL)` are allowed. With `NULLS NOT DISTINCT`: only one `(1, NULL)` allowed. Prefer `NULLS NOT DISTINCT` unless you specifically need duplicate NULLs. - **CHECK**: row-local constraints; NULL values pass the check (three-valued logic). Example: `CHECK (price > 0)` allows NULL prices. Combine with `NOT NULL` to enforce: `price NUMERIC NOT NULL CHECK (price > 0)`. - **EXCLUDE**: prevents overlapping values using operators. `EXCLUDE USING gist (room_id WITH =, booking_period WITH &&)` prevents double-booking rooms. Requires appropriate index type (often GiST). ## Indexing - **B-tree**: default for equality/range queries (`=`, `<`, `>`, `BETWEEN`, `ORDER BY`) - **Composite**: order matters—index used if equality on leftmost prefix (`WHERE a = ? AND b > ?` uses index on `(a,b)`, but `WHERE b = ?` does not). Put most selective/frequently filtered columns first. - **Covering**: `CREATE INDEX ON tbl (id) INCLUDE (name, email)` - includes non-key columns for index-only scans without visiting table. - **Partial**: for hot subsets (`WHERE status = 'active'` → `CREATE INDEX ON tbl (user_id) WHERE status = 'active'`). Any query with `status = 'active'` can use this index. - **Expression**: for computed search keys (`CREATE INDEX ON tbl (LOWER(email))`). Expression must match exactly in WHERE clause: `WHERE LOWER(email) = 'user@example.com'`. - **GIN**: JSONB containment/existence, arrays (`@>`, `?`), full-text search (`@@`) - **GiST**: ranges, geometry, exclusion constraints - **BRIN**: very large, naturally ordered data (time-series)—minimal storage overhead. Effective when row order on disk correlates with indexed column (insertion order or after `CLUSTER`). ## Partitioning - Use for very large tables (>100M rows) where queries consistently filter on partition key (often time/date). - Alternate use: use for tables where data maintenance tasks dictates e.g. data pruned or bulk replaced periodically - **RANGE**: common for time-series (`PARTITION BY RANGE (created_at)`). Create partitions: `CREATE TABLE logs_2024_01 PARTITION OF logs FOR VALUES FROM ('2024-01-01') TO ('2024-02-01')`. **TimescaleDB** automates time-based or ID-based partitioning with retention policies and compression. - **LIST**: for discrete values (`PARTITION BY LIST (region)`). Example: `FOR VALUES IN ('us-east', 'us-west')`. - **HASH**: for even distribution when no natural key (`PARTITION BY HASH (user_id)`). Creates N partitions with modulus. - **Constraint exclusion**: requires `CHECK` constraints on partitions for query planner to prune. Auto-created for declarative partitioning (PG10+). - Prefer declarative partitioning or hypertables. Do NOT use table inheritance. - **Limitations**: no global UNIQUE constraints—include partition key in PK/UNIQUE. FKs from partitioned tables not supported; use triggers. ## Special Considerations ### Update-Heavy Tables - **Separate hot/cold columns**—put frequently updated columns in separate table to minimize bloat. - **Use `fillfactor=90`** to leave space for HOT updates that avoid index maintenance. - **Avoid updating indexed columns**—prevents beneficial HOT updates. - **Partition by update patterns**—separate frequently updated rows in a different partition from stable data. ### Insert-Heavy Workloads - **Minimize indexes**—only create what you query; every index slows inserts. - **Use `COPY` or multi-row `INSERT`** instead of single-row inserts. - **UNLOGGED tables** for rebuildable staging data—much faster writes. - **Defer index creation** for bulk loads—>drop index, load data, recreate indexes. - **Partition by time/hash** to distribute load. **TimescaleDB** automates partitioning and compression of insert-heavy data. - **Use a natural key for primary key** such as a (timestamp, device_id) if enforcing global uniqueness is important many insert-heavy tables don't need a primary key at all. - If you do need a surrogate key, **Prefer `BIGINT GENERATED ALWAYS AS IDENTITY` over `UUID`**. ### Upsert-Friendly Design - **Requires UNIQUE index** on conflict target columns—`ON CONFLICT (col1, col2)` needs exact matching unique index (partial indexes don't work). - **Use `EXCLUDED.column`** to reference would-be-inserted values; only update columns that actually changed to reduce write overhead. - **`DO NOTHING` faster** than `DO UPDATE` when no actual update needed. ### Safe Schema Evolution - **Transactional DDL**: most DDL operations can run in transactions and be rolled back—`BEGIN; ALTER TABLE...; ROLLBACK;` for safe testing. - **Concurrent index creation**: `CREATE INDEX CONCURRENTLY` avoids blocking writes but can't run in transactions. - **Volatile defaults cause rewrites**: adding `NOT NULL` columns with volatile defaults (e.g., `now()`, `gen_random_uuid()`) rewrites entire table. Non-volatile defaults are fast. - **Drop constraints before columns**: `ALTER TABLE DROP CONSTRAINT` then `DROP COLUMN` to avoid dependency issues. - **Function signature changes**: `CREATE OR REPLACE` with different arguments creates overloads, not replacements. DROP old version if no overload desired. ## Generated Columns - `... GENERATED ALWAYS AS (<expr>) STORED` for computed, indexable fields. PG18+ adds `VIRTUAL` columns (computed on read, not stored). ## Extensions - **`pgcrypto`**: `crypt()` for password hashing. - **`uuid-ossp`**: alternative UUID functions; prefer `pgcrypto` for new projects. - **`pg_trgm`**: fuzzy text search with `%` operator, `similarity()` function. Index with GIN for `LIKE '%pattern%'` acceleration. - **`citext`**: case-insensitive text type. Prefer expression indexes on `LOWER(col)` unless you need case-insensitive constraints. - **`btree_gin`/`btree_gist`**: enable mixed-type indexes (e.g., GIN index on both JSONB and text columns). - **`hstore`**: key-value pairs; mostly superseded by JSONB but useful for simple string mappings. - **`timescaledb`**: essential for time-series—automated partitioning, retention, compression, continuous aggregates. - **`postgis`**: comprehensive geospatial support beyond basic geometric types—essential for location-based applications. - **`pgvector`**: vector similarity search for embeddings. - **`pgaudit`**: audit logging for all database activity. ## JSONB Guidance - Prefer `JSONB` with **GIN** index. - Default: `CREATE INDEX ON tbl USING GIN (jsonb_col);` → accelerates: - **Containment** `jsonb_col @> '{"k":"v"}'` - **Key existence** `jsonb_col ? 'k'`, **any/all keys** `?\|`, `?&` - **Path containment** on nested docs - **Disjunction** `jsonb_col @> ANY(ARRAY['{"status":"active"}', '{"status":"pending"}'])` - Heavy `@>` workloads: consider opclass `jsonb_path_ops` for smaller/faster containment-only indexes: - `CREATE INDEX ON tbl USING GIN (jsonb_col jsonb_path_ops);` - **Trade-off**: loses support for key existence (`?`, `?|`, `?&`) queries—only supports containment (`@>`) - Equality/range on a specific scalar field: extract and index with B-tree (generated column or expression): - `ALTER TABLE tbl ADD COLUMN price INT GENERATED ALWAYS AS ((jsonb_col->>'price')::INT) STORED;` - `CREATE INDEX ON tbl (price);` - Prefer queries like `WHERE price BETWEEN 100 AND 500` (uses B-tree) over `WHERE (jsonb_col->>'price')::INT BETWEEN 100 AND 500` without index. - Arrays inside JSONB: use GIN + `@>` for containment (e.g., tags). Consider `jsonb_path_ops` if only doing containment. - Keep core relations in tables; use JSONB for optional/variable attributes. - Use constraints to limit allowed JSONB values in a column e.g. `config JSONB NOT NULL CHECK(jsonb_typeof(config) = 'object')` ## Examples ### Users ```sql CREATE TABLE users ( user_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, email TEXT NOT NULL UNIQUE, name TEXT NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE UNIQUE INDEX ON users (LOWER(email)); CREATE INDEX ON users (created_at); ``` ### Orders ```sql CREATE TABLE orders ( order_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, user_id BIGINT NOT NULL REFERENCES users(user_id), status TEXT NOT NULL DEFAULT 'PENDING' CHECK (status IN ('PENDING','PAID','CANCELED')), total NUMERIC(10,2) NOT NULL CHECK (total > 0), created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); CREATE INDEX ON orders (user_id); CREATE INDEX ON orders (created_at); ``` ### JSONB ```sql CREATE TABLE profiles ( user_id BIGINT PRIMARY KEY REFERENCES users(user_id), attrs JSONB NOT NULL DEFAULT '{}', theme TEXT GENERATED ALWAYS AS (attrs->>'theme') STORED ); CREATE INDEX profiles_attrs_gin ON profiles USING GIN (attrs); ```
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

git-advanced-workflows

Master advanced Git workflows including rebasing, cherry-picking,

coding
⭐1
# Git Advanced Workflows Master advanced Git techniques to maintain clean history, collaborate effectively, and recover from any situation with confidence. ## When to Use This Skill - Cleaning up commit history before merging - Applying specific commits across branches - Finding commits that introduced bugs - Working on multiple features simultaneously - Recovering from Git mistakes or lost commits - Managing complex branch workflows - Preparing clean PRs for review - Synchronizing diverged branches ## Core Concepts ### 1. Interactive Rebase Interactive rebase is the Swiss Army knife of Git history editing. **Common Operations:** - `pick`: Keep commit as-is - `reword`: Change commit message - `edit`: Amend commit content - `squash`: Combine with previous commit - `fixup`: Like squash but discard message - `drop`: Remove commit entirely **Basic Usage:** ```bash # Rebase last 5 commits git rebase -i HEAD~5 # Rebase all commits on current branch git rebase -i $(git merge-base HEAD main) # Rebase onto specific commit git rebase -i abc123 ``` ### 2. Cherry-Picking Apply specific commits from one branch to another without merging entire branches. ```bash # Cherry-pick single commit git cherry-pick abc123 # Cherry-pick range of commits (exclusive start) git cherry-pick abc123..def456 # Cherry-pick without committing (stage changes only) git cherry-pick -n abc123 # Cherry-pick and edit commit message git cherry-pick -e abc123 ``` ### 3. Git Bisect Binary search through commit history to find the commit that introduced a bug. ```bash # Start bisect git bisect start # Mark current commit as bad git bisect bad # Mark known good commit git bisect good v1.0.0 # Git will checkout middle commit - test it # Then mark as good or bad git bisect good # or: git bisect bad # Continue until bug found # When done git bisect reset ``` **Automated Bisect:** ```bash # Use script to test automatically git bisect start HEAD v1.0.0 git bisect run ./test.sh # test.sh should exit 0 for good, 1-127 (except 125) for bad ``` ### 4. Worktrees Work on multiple branches simultaneously without stashing or switching. ```bash # List existing worktrees git worktree list # Add new worktree for feature branch git worktree add ../project-feature feature/new-feature # Add worktree and create new branch git worktree add -b bugfix/urgent ../project-hotfix main # Remove worktree git worktree remove ../project-feature # Prune stale worktrees git worktree prune ``` ### 5. Reflog Your safety net - tracks all ref movements, even deleted commits. ```bash # View reflog git reflog # View reflog for specific branch git reflog show feature/branch # Restore deleted commit git reflog # Find commit hash git checkout abc123 git branch recovered-branch # Restore deleted branch git reflog git branch deleted-branch abc123 ``` ## Practical Workflows ### Workflow 1: Clean Up Feature Branch Before PR ```bash # Start with feature branch git checkout feature/user-auth # Interactive rebase to clean history git rebase -i main # Example rebase operations: # - Squash "fix typo" commits # - Reword commit messages for clarity # - Reorder commits logically # - Drop unnecessary commits # Force push cleaned branch (safe if no one else is using it) git push --force-with-lease origin feature/user-auth ``` ### Workflow 2: Apply Hotfix to Multiple Releases ```bash # Create fix on main git checkout main git commit -m "fix: critical security patch" # Apply to release branches git checkout release/2.0 git cherry-pick abc123 git checkout release/1.9 git cherry-pick abc123 # Handle conflicts if they arise git cherry-pick --continue # or git cherry-pick --abort ``` ### Workflow 3: Find Bug Introduction ```bash # Start bisect git bisect start git bisect bad HEAD git bisect good v2.1.0 # Git checks out middle commit - run tests npm test # If tests fail git bisect bad # If tests pass git bisect good # Git will automatically checkout next commit to test # Repeat until bug found # Automated version git bisect start HEAD v2.1.0 git bisect run npm test ``` ### Workflow 4: Multi-Branch Development ```bash # Main project directory cd ~/projects/myapp # Create worktree for urgent bugfix git worktree add ../myapp-hotfix hotfix/critical-bug # Work on hotfix in separate directory cd ../myapp-hotfix # Make changes, commit git commit -m "fix: resolve critical bug" git push origin hotfix/critical-bug # Return to main work without interruption cd ~/projects/myapp git fetch origin git cherry-pick hotfix/critical-bug # Clean up when done git worktree remove ../myapp-hotfix ``` ### Workflow 5: Recover from Mistakes ```bash # Accidentally reset to wrong commit git reset --hard HEAD~5 # Oh no! # Use reflog to find lost commits git reflog # Output shows: # abc123 HEAD@{0}: reset: moving to HEAD~5 # def456 HEAD@{1}: commit: my important changes # Recover lost commits git reset --hard def456 # Or create branch from lost commit git branch recovery def456 ``` ## Advanced Techniques ### Rebase vs Merge Strategy **When to Rebase:** - Cleaning up local commits before pushing - Keeping feature branch up-to-date with main - Creating linear history for easier review **When to Merge:** - Integrating completed features into main - Preserving exact history of collaboration - Public branches used by others ```bash # Update feature branch with main changes (rebase) git checkout feature/my-feature git fetch origin git rebase origin/main # Handle conflicts git status # Fix conflicts in files git add . git rebase --continue # Or merge instead git merge origin/main ``` ### Autosquash Workflow Automatically squash fixup commits during rebase. ```bash # Make initial commit git commit -m "feat: add user authentication" # Later, fix something in that commit # Stage changes git commit --fixup HEAD # or specify commit hash # Make more changes git commit --fixup abc123 # Rebase with autosquash git rebase -i --autosquash main # Git automatically marks fixup commits ``` ### Split Commit Break one commit into multiple logical commits. ```bash # Start interactive rebase git rebase -i HEAD~3 # Mark commit to split with 'edit' # Git will stop at that commit # Reset commit but keep changes git reset HEAD^ # Stage and commit in logical chunks git add file1.py git commit -m "feat: add validation" git add file2.py git commit -m "feat: add error handling" # Continue rebase git rebase --continue ``` ### Partial Cherry-Pick Cherry-pick only specific files from a commit. ```bash # Show files in commit git show --name-only abc123 # Checkout specific files from commit git checkout abc123 -- path/to/file1.py path/to/file2.py # Stage and commit git commit -m "cherry-pick: apply specific changes from abc123" ``` ## Best Practices 1. **Always Use --force-with-lease**: Safer than --force, prevents overwriting others' work 2. **Rebase Only Local Commits**: Don't rebase commits that have been pushed and shared 3. **Descriptive Commit Messages**: Future you will thank present you 4. **Atomic Commits**: Each commit should be a single logical change 5. **Test Before Force Push**: Ensure history rewrite didn't break anything 6. **Keep Reflog Aware**: Remember reflog is your safety net for 90 days 7. **Branch Before Risky Operations**: Create backup branch before complex rebases ```bash # Safe force push git push --force-with-lease origin feature/branch # Create backup before risky operation git branch backup-branch git rebase -i main # If something goes wrong git reset --hard backup-branch ``` ## Common Pitfalls - **Rebasing Public Branches**: Causes history conflicts for collaborators - **Force Pushing Without Lease**: Can overwrite teammate's work - **Losing Work in Rebase**: Resolve conflicts carefully, test after rebase - **Forgetting Worktree Cleanup**: Orphaned worktrees consume disk space - **Not Backing Up Before Experiment**: Always create safety branch - **Bisect on Dirty Working Directory**: Commit or stash before bisecting ## Recovery Commands ```bash # Abort operations in progress git rebase --abort git merge --abort git cherry-pick --abort git bisect reset # Restore file to version from specific commit git restore --source=abc123 path/to/file # Undo last commit but keep changes git reset --soft HEAD^ # Undo last commit and discard changes git reset --hard HEAD^ # Recover deleted branch (within 90 days) git reflog git branch recovered-branch abc123 ``` ## Resources - **references/git-rebase-guide.md**: Deep dive into interactive rebase - **references/git-conflict-resolution.md**: Advanced conflict resolution strategies - **references/git-history-rewriting.md**: Safely rewriting Git history - **assets/git-workflow-checklist.md**: Pre-PR cleanup checklist - **assets/git-aliases.md**: Useful Git aliases for advanced workflows - **scripts/git-clean-branches.sh**: Clean up merged and stale branches
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

sql-optimization-patterns

Master SQL query optimization, indexing strategies, and EXPLAIN

coding
⭐1
# SQL Optimization Patterns Transform slow database queries into lightning-fast operations through systematic optimization, proper indexing, and query plan analysis. ## When to Use This Skill - Debugging slow-running queries - Designing performant database schemas - Optimizing application response times - Reducing database load and costs - Improving scalability for growing datasets - Analyzing EXPLAIN query plans - Implementing efficient indexes - Resolving N+1 query problems ## Core Concepts ### 1. Query Execution Plans (EXPLAIN) Understanding EXPLAIN output is fundamental to optimization. **PostgreSQL EXPLAIN:** ```sql -- Basic explain EXPLAIN SELECT * FROM users WHERE email = 'user@example.com'; -- With actual execution stats EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'user@example.com'; -- Verbose output with more details EXPLAIN (ANALYZE, BUFFERS, VERBOSE) SELECT u.*, o.order_total FROM users u JOIN orders o ON u.id = o.user_id WHERE u.created_at > NOW() - INTERVAL '30 days'; ``` **Key Metrics to Watch:** - **Seq Scan**: Full table scan (usually slow for large tables) - **Index Scan**: Using index (good) - **Index Only Scan**: Using index without touching table (best) - **Nested Loop**: Join method (okay for small datasets) - **Hash Join**: Join method (good for larger datasets) - **Merge Join**: Join method (good for sorted data) - **Cost**: Estimated query cost (lower is better) - **Rows**: Estimated rows returned - **Actual Time**: Real execution time ### 2. Index Strategies Indexes are the most powerful optimization tool. **Index Types:** - **B-Tree**: Default, good for equality and range queries - **Hash**: Only for equality (=) comparisons - **GIN**: Full-text search, array queries, JSONB - **GiST**: Geometric data, full-text search - **BRIN**: Block Range INdex for very large tables with correlation ```sql -- Standard B-Tree index CREATE INDEX idx_users_email ON users(email); -- Composite index (order matters!) CREATE INDEX idx_orders_user_status ON orders(user_id, status); -- Partial index (index subset of rows) CREATE INDEX idx_active_users ON users(email) WHERE status = 'active'; -- Expression index CREATE INDEX idx_users_lower_email ON users(LOWER(email)); -- Covering index (include additional columns) CREATE INDEX idx_users_email_covering ON users(email) INCLUDE (name, created_at); -- Full-text search index CREATE INDEX idx_posts_search ON posts USING GIN(to_tsvector('english', title || ' ' || body)); -- JSONB index CREATE INDEX idx_metadata ON events USING GIN(metadata); ``` ### 3. Query Optimization Patterns **Avoid SELECT \*:** ```sql -- Bad: Fetches unnecessary columns SELECT * FROM users WHERE id = 123; -- Good: Fetch only what you need SELECT id, email, name FROM users WHERE id = 123; ``` **Use WHERE Clause Efficiently:** ```sql -- Bad: Function prevents index usage SELECT * FROM users WHERE LOWER(email) = 'user@example.com'; -- Good: Create functional index or use exact match CREATE INDEX idx_users_email_lower ON users(LOWER(email)); -- Then: SELECT * FROM users WHERE LOWER(email) = 'user@example.com'; -- Or store normalized data SELECT * FROM users WHERE email = 'user@example.com'; ``` **Optimize JOINs:** ```sql -- Bad: Cartesian product then filter SELECT u.name, o.total FROM users u, orders o WHERE u.id = o.user_id AND u.created_at > '2024-01-01'; -- Good: Filter before join SELECT u.name, o.total FROM users u JOIN orders o ON u.id = o.user_id WHERE u.created_at > '2024-01-01'; -- Better: Filter both tables SELECT u.name, o.total FROM (SELECT * FROM users WHERE created_at > '2024-01-01') u JOIN orders o ON u.id = o.user_id; ``` ## Optimization Patterns ### Pattern 1: Eliminate N+1 Queries **Problem: N+1 Query Anti-Pattern** ```python # Bad: Executes N+1 queries users = db.query("SELECT * FROM users LIMIT 10") for user in users: orders = db.query("SELECT * FROM orders WHERE user_id = ?", user.id) # Process orders ``` **Solution: Use JOINs or Batch Loading** ```sql -- Solution 1: JOIN SELECT u.id, u.name, o.id as order_id, o.total FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.id IN (1, 2, 3, 4, 5); -- Solution 2: Batch query SELECT * FROM orders WHERE user_id IN (1, 2, 3, 4, 5); ``` ```python # Good: Single query with JOIN or batch load # Using JOIN results = db.query(""" SELECT u.id, u.name, o.id as order_id, o.total FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.id IN (1, 2, 3, 4, 5) """) # Or batch load users = db.query("SELECT * FROM users LIMIT 10") user_ids = [u.id for u in users] orders = db.query( "SELECT * FROM orders WHERE user_id IN (?)", user_ids ) # Group orders by user_id orders_by_user = {} for order in orders: orders_by_user.setdefault(order.user_id, []).append(order) ``` ### Pattern 2: Optimize Pagination **Bad: OFFSET on Large Tables** ```sql -- Slow for large offsets SELECT * FROM users ORDER BY created_at DESC LIMIT 20 OFFSET 100000; -- Very slow! ``` **Good: Cursor-Based Pagination** ```sql -- Much faster: Use cursor (last seen ID) SELECT * FROM users WHERE created_at < '2024-01-15 10:30:00' -- Last cursor ORDER BY created_at DESC LIMIT 20; -- With composite sorting SELECT * FROM users WHERE (created_at, id) < ('2024-01-15 10:30:00', 12345) ORDER BY created_at DESC, id DESC LIMIT 20; -- Requires index CREATE INDEX idx_users_cursor ON users(created_at DESC, id DESC); ``` ### Pattern 3: Aggregate Efficiently **Optimize COUNT Queries:** ```sql -- Bad: Counts all rows SELECT COUNT(*) FROM orders; -- Slow on large tables -- Good: Use estimates for approximate counts SELECT reltuples::bigint AS estimate FROM pg_class WHERE relname = 'orders'; -- Good: Filter before counting SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '7 days'; -- Better: Use index-only scan CREATE INDEX idx_orders_created ON orders(created_at); SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '7 days'; ``` **Optimize GROUP BY:** ```sql -- Bad: Group by then filter SELECT user_id, COUNT(*) as order_count FROM orders GROUP BY user_id HAVING COUNT(*) > 10; -- Better: Filter first, then group (if possible) SELECT user_id, COUNT(*) as order_count FROM orders WHERE status = 'completed' GROUP BY user_id HAVING COUNT(*) > 10; -- Best: Use covering index CREATE INDEX idx_orders_user_status ON orders(user_id, status); ``` ### Pattern 4: Subquery Optimization **Transform Correlated Subqueries:** ```sql -- Bad: Correlated subquery (runs for each row) SELECT u.name, u.email, (SELECT COUNT(*) FROM orders o WHERE o.user_id = u.id) as order_count FROM users u; -- Good: JOIN with aggregation SELECT u.name, u.email, COUNT(o.id) as order_count FROM users u LEFT JOIN orders o ON o.user_id = u.id GROUP BY u.id, u.name, u.email; -- Better: Use window functions SELECT DISTINCT ON (u.id) u.name, u.email, COUNT(o.id) OVER (PARTITION BY u.id) as order_count FROM users u LEFT JOIN orders o ON o.user_id = u.id; ``` **Use CTEs for Clarity:** ```sql -- Using Common Table Expressions WITH recent_users AS ( SELECT id, name, email FROM users WHERE created_at > NOW() - INTERVAL '30 days' ), user_order_counts AS ( SELECT user_id, COUNT(*) as order_count FROM orders WHERE created_at > NOW() - INTERVAL '30 days' GROUP BY user_id ) SELECT ru.name, ru.email, COALESCE(uoc.order_count, 0) as orders FROM recent_users ru LEFT JOIN user_order_counts uoc ON ru.id = uoc.user_id; ``` ### Pattern 5: Batch Operations **Batch INSERT:** ```sql -- Bad: Multiple individual inserts INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com'); INSERT INTO users (name, email) VALUES ('Bob', 'bob@example.com'); INSERT INTO users (name, email) VALUES ('Carol', 'carol@example.com'); -- Good: Batch insert INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com'), ('Bob', 'bob@example.com'), ('Carol', 'carol@example.com'); -- Better: Use COPY for bulk inserts (PostgreSQL) COPY users (name, email) FROM '/tmp/users.csv' CSV HEADER; ``` **Batch UPDATE:** ```sql -- Bad: Update in loop UPDATE users SET status = 'active' WHERE id = 1; UPDATE users SET status = 'active' WHERE id = 2; -- ... repeat for many IDs -- Good: Single UPDATE with IN clause UPDATE users SET status = 'active' WHERE id IN (1, 2, 3, 4, 5, ...); -- Better: Use temporary table for large batches CREATE TEMP TABLE temp_user_updates (id INT, new_status VARCHAR); INSERT INTO temp_user_updates VALUES (1, 'active'), (2, 'active'), ...; UPDATE users u SET status = t.new_status FROM temp_user_updates t WHERE u.id = t.id; ``` ## Advanced Techniques ### Materialized Views Pre-compute expensive queries. ```sql -- Create materialized view CREATE MATERIALIZED VIEW user_order_summary AS SELECT u.id, u.name, COUNT(o.id) as total_orders, SUM(o.total) as total_spent, MAX(o.created_at) as last_order_date FROM users u LEFT JOIN orders o ON u.id = o.user_id GROUP BY u.id, u.name; -- Add index to materialized view CREATE INDEX idx_user_summary_spent ON user_order_summary(total_spent DESC); -- Refresh materialized view REFRESH MATERIALIZED VIEW user_order_summary; -- Concurrent refresh (PostgreSQL) REFRESH MATERIALIZED VIEW CONCURRENTLY user_order_summary; -- Query materialized view (very fast) SELECT * FROM user_order_summary WHERE total_spent > 1000 ORDER BY total_spent DESC; ``` ### Partitioning Split large tables for better performance. ```sql -- Range partitioning by date (PostgreSQL) CREATE TABLE orders ( id SERIAL, user_id INT, total DECIMAL, created_at TIMESTAMP ) PARTITION BY RANGE (created_at); -- Create partitions CREATE TABLE orders_2024_q1 PARTITION OF orders FOR VALUES FROM ('2024-01-01') TO ('2024-04-01'); CREATE TABLE orders_2024_q2 PARTITION OF orders FOR VALUES FROM ('2024-04-01') TO ('2024-07-01'); -- Queries automatically use appropriate partition SELECT * FROM orders WHERE created_at BETWEEN '2024-02-01' AND '2024-02-28'; -- Only scans orders_2024_q1 partition ``` ### Query Hints and Optimization ```sql -- Force index usage (MySQL) SELECT * FROM users USE INDEX (idx_users_email) WHERE email = 'user@example.com'; -- Parallel query (PostgreSQL) SET max_parallel_workers_per_gather = 4; SELECT * FROM large_table WHERE condition; -- Join hints (PostgreSQL) SET enable_nestloop = OFF; -- Force hash or merge join ``` ## Best Practices 1. **Index Selectively**: Too many indexes slow down writes 2. **Monitor Query Performance**: Use slow query logs 3. **Keep Statistics Updated**: Run ANALYZE regularly 4. **Use Appropriate Data Types**: Smaller types = better performance 5. **Normalize Thoughtfully**: Balance normalization vs performance 6. **Cache Frequently Accessed Data**: Use application-level caching 7. **Connection Pooling**: Reuse database connections 8. **Regular Maintenance**: VACUUM, ANALYZE, rebuild indexes ```sql -- Update statistics ANALYZE users; ANALYZE VERBOSE orders; -- Vacuum (PostgreSQL) VACUUM ANALYZE users; VACUUM FULL users; -- Reclaim space (locks table) -- Reindex REINDEX INDEX idx_users_email; REINDEX TABLE users; ``` ## Common Pitfalls - **Over-Indexing**: Each index slows down INSERT/UPDATE/DELETE - **Unused Indexes**: Waste space and slow writes - **Missing Indexes**: Slow queries, full table scans - **Implicit Type Conversion**: Prevents index usage - **OR Conditions**: Can't use indexes efficiently - **LIKE with Leading Wildcard**: `LIKE '%abc'` can't use index - **Function in WHERE**: Prevents index usage unless functional index exists ## Monitoring Queries ```sql -- Find slow queries (PostgreSQL) SELECT query, calls, total_time, mean_time FROM pg_stat_statements ORDER BY mean_time DESC LIMIT 10; -- Find missing indexes (PostgreSQL) SELECT schemaname, tablename, seq_scan, seq_tup_read, idx_scan, seq_tup_read / seq_scan AS avg_seq_tup_read FROM pg_stat_user_tables WHERE seq_scan > 0 ORDER BY seq_tup_read DESC LIMIT 10; -- Find unused indexes (PostgreSQL) SELECT schemaname, tablename, indexname, idx_scan, idx_tup_read, idx_tup_fetch FROM pg_stat_user_indexes WHERE idx_scan = 0 ORDER BY pg_relation_size(indexrelid) DESC; ``` ## Resources - **references/postgres-optimization-guide.md**: PostgreSQL-specific optimization - **references/mysql-optimization-guide.md**: MySQL/MariaDB optimization - **references/query-plan-analysis.md**: Deep dive into EXPLAIN plans - **assets/index-strategy-checklist.md**: When and how to create indexes - **assets/query-optimization-checklist.md**: Step-by-step optimization guide - **scripts/analyze-slow-queries.sql**: Identify slow queries in your database - **scripts/index-recommendations.sql**: Generate index recommendations
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

tailwind-design-system

Build scalable design systems with Tailwind CSS v4, design tokens,

coding
⭐1
# Tailwind Design System (v4) Build production-ready design systems with Tailwind CSS v4, including CSS-first configuration, design tokens, component variants, responsive patterns, and accessibility. > **Note**: This skill targets Tailwind CSS v4 (2024+). For v3 projects, refer to the [upgrade guide](https://tailwindcss.com/docs/upgrade-guide). ## When to Use This Skill - Creating a component library with Tailwind v4 - Implementing design tokens and theming with CSS-first configuration - Building responsive and accessible components - Standardizing UI patterns across a codebase - Migrating from Tailwind v3 to v4 - Setting up dark mode with native CSS features ## Key v4 Changes | v3 Pattern | v4 Pattern | | ------------------------------------- | --------------------------------------------------------------------- | | `tailwind.config.ts` | `@theme` in CSS | | `@tailwind base/components/utilities` | `@import "tailwindcss"` | | `darkMode: "class"` | `@custom-variant dark (&:where(.dark, .dark *))` | | `theme.extend.colors` | `@theme { --color-*: value }` | | `require("tailwindcss-animate")` | CSS `@keyframes` in `@theme` + `@starting-style` for entry animations | ## Quick Start ```css /* app.css - Tailwind v4 CSS-first configuration */ @import "tailwindcss"; /* Define your theme with @theme */ @theme { /* Semantic color tokens using OKLCH for better color perception */ --color-background: oklch(100% 0 0); --color-foreground: oklch(14.5% 0.025 264); --color-primary: oklch(14.5% 0.025 264); --color-primary-foreground: oklch(98% 0.01 264); --color-secondary: oklch(96% 0.01 264); --color-secondary-foreground: oklch(14.5% 0.025 264); --color-muted: oklch(96% 0.01 264); --color-muted-foreground: oklch(46% 0.02 264); --color-accent: oklch(96% 0.01 264); --color-accent-foreground: oklch(14.5% 0.025 264); --color-destructive: oklch(53% 0.22 27); --color-destructive-foreground: oklch(98% 0.01 264); --color-border: oklch(91% 0.01 264); --color-ring: oklch(14.5% 0.025 264); --color-card: oklch(100% 0 0); --color-card-foreground: oklch(14.5% 0.025 264); /* Ring offset for focus states */ --color-ring-offset: oklch(100% 0 0); /* Radius tokens */ --radius-sm: 0.25rem; --radius-md: 0.375rem; --radius-lg: 0.5rem; --radius-xl: 0.75rem; /* Animation tokens - keyframes inside @theme are output when referenced by --animate-* variables */ --animate-fade-in: fade-in 0.2s ease-out; --animate-fade-out: fade-out 0.2s ease-in; --animate-slide-in: slide-in 0.3s ease-out; --animate-slide-out: slide-out 0.3s ease-in; @keyframes fade-in { from { opacity: 0; } to { opacity: 1; } } @keyframes fade-out { from { opacity: 1; } to { opacity: 0; } } @keyframes slide-in { from { transform: translateY(-0.5rem); opacity: 0; } to { transform: translateY(0); opacity: 1; } } @keyframes slide-out { from { transform: translateY(0); opacity: 1; } to { transform: translateY(-0.5rem); opacity: 0; } } } /* Dark mode variant - use @custom-variant for class-based dark mode */ @custom-variant dark (&:where(.dark, .dark *)); /* Dark mode theme overrides */ .dark { --color-background: oklch(14.5% 0.025 264); --color-foreground: oklch(98% 0.01 264); --color-primary: oklch(98% 0.01 264); --color-primary-foreground: oklch(14.5% 0.025 264); --color-secondary: oklch(22% 0.02 264); --color-secondary-foreground: oklch(98% 0.01 264); --color-muted: oklch(22% 0.02 264); --color-muted-foreground: oklch(65% 0.02 264); --color-accent: oklch(22% 0.02 264); --color-accent-foreground: oklch(98% 0.01 264); --color-destructive: oklch(42% 0.15 27); --color-destructive-foreground: oklch(98% 0.01 264); --color-border: oklch(22% 0.02 264); --color-ring: oklch(83% 0.02 264); --color-card: oklch(14.5% 0.025 264); --color-card-foreground: oklch(98% 0.01 264); --color-ring-offset: oklch(14.5% 0.025 264); } /* Base styles */ @layer base { * { @apply border-border; } body { @apply bg-background text-foreground antialiased; } } ``` ## Core Concepts ### 1. Design Token Hierarchy ``` Brand Tokens (abstract) └── Semantic Tokens (purpose) └── Component Tokens (specific) Example: oklch(45% 0.2 260) → --color-primary → bg-primary ``` ### 2. Component Architecture ``` Base styles → Variants → Sizes → States → Overrides ``` ## Patterns ### Pattern 1: CVA (Class Variance Authority) Components ```typescript // components/ui/button.tsx import { Slot } from '@radix-ui/react-slot' import { cva, type VariantProps } from 'class-variance-authority' import { cn } from '@/lib/utils' const buttonVariants = cva( // Base styles - v4 uses native CSS variables 'inline-flex items-center justify-center whitespace-nowrap rounded-md text-sm font-medium transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50', { variants: { variant: { default: 'bg-primary text-primary-foreground hover:bg-primary/90', destructive: 'bg-destructive text-destructive-foreground hover:bg-destructive/90', outline: 'border border-border bg-background hover:bg-accent hover:text-accent-foreground', secondary: 'bg-secondary text-secondary-foreground hover:bg-secondary/80', ghost: 'hover:bg-accent hover:text-accent-foreground', link: 'text-primary underline-offset-4 hover:underline', }, size: { default: 'h-10 px-4 py-2', sm: 'h-9 rounded-md px-3', lg: 'h-11 rounded-md px-8', icon: 'size-10', }, }, defaultVariants: { variant: 'default', size: 'default', }, } ) export interface ButtonProps extends React.ButtonHTMLAttributes<HTMLButtonElement>, VariantProps<typeof buttonVariants> { asChild?: boolean } // React 19: No forwardRef needed export function Button({ className, variant, size, asChild = false, ref, ...props }: ButtonProps & { ref?: React.Ref<HTMLButtonElement> }) { const Comp = asChild ? Slot : 'button' return ( <Comp className={cn(buttonVariants({ variant, size, className }))} ref={ref} {...props} /> ) } // Usage <Button variant="destructive" size="lg">Delete</Button> <Button variant="outline">Cancel</Button> <Button asChild><Link href="/home">Home</Link></Button> ``` ### Pattern 2: Compound Components (React 19) ```typescript // components/ui/card.tsx import { cn } from '@/lib/utils' // React 19: ref is a regular prop, no forwardRef export function Card({ className, ref, ...props }: React.HTMLAttributes<HTMLDivElement> & { ref?: React.Ref<HTMLDivElement> }) { return ( <div ref={ref} className={cn( 'rounded-lg border border-border bg-card text-card-foreground shadow-sm', className )} {...props} /> ) } export function CardHeader({ className, ref, ...props }: React.HTMLAttributes<HTMLDivElement> & { ref?: React.Ref<HTMLDivElement> }) { return ( <div ref={ref} className={cn('flex flex-col space-y-1.5 p-6', className)} {...props} /> ) } export function CardTitle({ className, ref, ...props }: React.HTMLAttributes<HTMLHeadingElement> & { ref?: React.Ref<HTMLHeadingElement> }) { return ( <h3 ref={ref} className={cn('text-2xl font-semibold leading-none tracking-tight', className)} {...props} /> ) } export function CardDescription({ className, ref, ...props }: React.HTMLAttributes<HTMLParagraphElement> & { ref?: React.Ref<HTMLParagraphElement> }) { return ( <p ref={ref} className={cn('text-sm text-muted-foreground', className)} {...props} /> ) } export function CardContent({ className, ref, ...props }: React.HTMLAttributes<HTMLDivElement> & { ref?: React.Ref<HTMLDivElement> }) { return ( <div ref={ref} className={cn('p-6 pt-0', className)} {...props} /> ) } export function CardFooter({ className, ref, ...props }: React.HTMLAttributes<HTMLDivElement> & { ref?: React.Ref<HTMLDivElement> }) { return ( <div ref={ref} className={cn('flex items-center p-6 pt-0', className)} {...props} /> ) } // Usage <Card> <CardHeader> <CardTitle>Account</CardTitle> <CardDescription>Manage your account settings</CardDescription> </CardHeader> <CardContent> <form>...</form> </CardContent> <CardFooter> <Button>Save</Button> </CardFooter> </Card> ``` ### Pattern 3: Form Components ```typescript // components/ui/input.tsx import { cn } from '@/lib/utils' export interface InputProps extends React.InputHTMLAttributes<HTMLInputElement> { error?: string ref?: React.Ref<HTMLInputElement> } export function Input({ className, type, error, ref, ...props }: InputProps) { return ( <div className="relative"> <input type={type} className={cn( 'flex h-10 w-full rounded-md border border-border bg-background px-3 py-2 text-sm ring-offset-background file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50', error && 'border-destructive focus-visible:ring-destructive', className )} ref={ref} aria-invalid={!!error} aria-describedby={error ? `${props.id}-error` : undefined} {...props} /> {error && ( <p id={`${props.id}-error`} className="mt-1 text-sm text-destructive" role="alert" > {error} </p> )} </div> ) } // components/ui/label.tsx import { cva, type VariantProps } from 'class-variance-authority' const labelVariants = cva( 'text-sm font-medium leading-none peer-disabled:cursor-not-allowed peer-disabled:opacity-70' ) export function Label({ className, ref, ...props }: React.LabelHTMLAttributes<HTMLLabelElement> & { ref?: React.Ref<HTMLLabelElement> }) { return ( <label ref={ref} className={cn(labelVariants(), className)} {...props} /> ) } // Usage with React Hook Form + Zod import { useForm } from 'react-hook-form' import { zodResolver } from '@hookform/resolvers/zod' import { z } from 'zod' const schema = z.object({ email: z.string().email('Invalid email address'), password: z.string().min(8, 'Password must be at least 8 characters'), }) function LoginForm() { const { register, handleSubmit, formState: { errors } } = useForm({ resolver: zodResolver(schema), }) return ( <form onSubmit={handleSubmit(onSubmit)} className="space-y-4"> <div className="space-y-2"> <Label htmlFor="email">Email</Label> <Input id="email" type="email" {...register('email')} error={errors.email?.message} /> </div> <div className="space-y-2"> <Label htmlFor="password">Password</Label> <Input id="password" type="password" {...register('password')} error={errors.password?.message} /> </div> <Button type="submit" className="w-full">Sign In</Button> </form> ) } ``` ### Pattern 4: Responsive Grid System ```typescript // components/ui/grid.tsx import { cn } from '@/lib/utils' import { cva, type VariantProps } from 'class-variance-authority' const gridVariants = cva('grid', { variants: { cols: { 1: 'grid-cols-1', 2: 'grid-cols-1 sm:grid-cols-2', 3: 'grid-cols-1 sm:grid-cols-2 lg:grid-cols-3', 4: 'grid-cols-1 sm:grid-cols-2 lg:grid-cols-4', 5: 'grid-cols-2 sm:grid-cols-3 lg:grid-cols-5', 6: 'grid-cols-2 sm:grid-cols-3 lg:grid-cols-6', }, gap: { none: 'gap-0', sm: 'gap-2', md: 'gap-4', lg: 'gap-6', xl: 'gap-8', }, }, defaultVariants: { cols: 3, gap: 'md', }, }) interface GridProps extends React.HTMLAttributes<HTMLDivElement>, VariantProps<typeof gridVariants> {} export function Grid({ className, cols, gap, ...props }: GridProps) { return ( <div className={cn(gridVariants({ cols, gap, className }))} {...props} /> ) } // Container component const containerVariants = cva('mx-auto w-full px-4 sm:px-6 lg:px-8', { variants: { size: { sm: 'max-w-screen-sm', md: 'max-w-screen-md', lg: 'max-w-screen-lg', xl: 'max-w-screen-xl', '2xl': 'max-w-screen-2xl', full: 'max-w-full', }, }, defaultVariants: { size: 'xl', }, }) interface ContainerProps extends React.HTMLAttributes<HTMLDivElement>, VariantProps<typeof containerVariants> {} export function Container({ className, size, ...props }: ContainerProps) { return ( <div className={cn(containerVariants({ size, className }))} {...props} /> ) } // Usage <Container> <Grid cols={4} gap="lg"> {products.map((product) => ( <ProductCard key={product.id} product={product} /> ))} </Grid> </Container> ``` ### Pattern 5: Native CSS Animations (v4) ```css /* In your CSS file - native @starting-style for entry animations */ @theme { --animate-dialog-in: dialog-fade-in 0.2s ease-out; --animate-dialog-out: dialog-fade-out 0.15s ease-in; } @keyframes dialog-fade-in { from { opacity: 0; transform: scale(0.95) translateY(-0.5rem); } to { opacity: 1; transform: scale(1) translateY(0); } } @keyframes dialog-fade-out { from { opacity: 1; transform: scale(1) translateY(0); } to { opacity: 0; transform: scale(0.95) translateY(-0.5rem); } } /* Native popover animations using @starting-style */ [popover] { transition: opacity 0.2s, transform 0.2s, display 0.2s allow-discrete; opacity: 0; transform: scale(0.95); } [popover]:popover-open { opacity: 1; transform: scale(1); } @starting-style { [popover]:popover-open { opacity: 0; transform: scale(0.95); } } ``` ```typescript // components/ui/dialog.tsx - Using native popover API import * as DialogPrimitive from '@radix-ui/react-dialog' import { cn } from '@/lib/utils' const DialogPortal = DialogPrimitive.Portal export function DialogOverlay({ className, ref, ...props }: React.ComponentPropsWithoutRef<typeof DialogPrimitive.Overlay> & { ref?: React.Ref<HTMLDivElement> }) { return ( <DialogPrimitive.Overlay ref={ref} className={cn( 'fixed inset-0 z-50 bg-black/80', 'data-[state=open]:animate-fade-in data-[state=closed]:animate-fade-out', className )} {...props} /> ) } export function DialogContent({ className, children, ref, ...props }: React.ComponentPropsWithoutRef<typeof DialogPrimitive.Content> & { ref?: React.Ref<HTMLDivElement> }) { return ( <DialogPortal> <DialogOverlay /> <DialogPrimitive.Content ref={ref} className={cn( 'fixed left-1/2 top-1/2 z-50 grid w-full max-w-lg -translate-x-1/2 -translate-y-1/2 gap-4 border border-border bg-background p-6 shadow-lg sm:rounded-lg', 'data-[state=open]:animate-dialog-in data-[state=closed]:animate-dialog-out', className )} {...props} > {children} </DialogPrimitive.Content> </DialogPortal> ) } ``` ### Pattern 6: Dark Mode with CSS (v4) ```typescript // providers/ThemeProvider.tsx - Simplified for v4 'use client' import { createContext, useContext, useEffect, useState } from 'react' type Theme = 'dark' | 'light' | 'system' interface ThemeContextType { theme: Theme setTheme: (theme: Theme) => void resolvedTheme: 'dark' | 'light' } const ThemeContext = createContext<ThemeContextType | undefined>(undefined) export function ThemeProvider({ children, defaultTheme = 'system', storageKey = 'theme', }: { children: React.ReactNode defaultTheme?: Theme storageKey?: string }) { const [theme, setTheme] = useState<Theme>(defaultTheme) const [resolvedTheme, setResolvedTheme] = useState<'dark' | 'light'>('light') useEffect(() => { const stored = localStorage.getItem(storageKey) as Theme | null if (stored) setTheme(stored) }, [storageKey]) useEffect(() => { const root = document.documentElement root.classList.remove('light', 'dark') const resolved = theme === 'system' ? (window.matchMedia('(prefers-color-scheme: dark)').matches ? 'dark' : 'light') : theme root.classList.add(resolved) setResolvedTheme(resolved) // Update meta theme-color for mobile browsers const metaThemeColor = document.querySelector('meta[name="theme-color"]') if (metaThemeColor) { metaThemeColor.setAttribute('content', resolved === 'dark' ? '#09090b' : '#ffffff') } }, [theme]) return ( <ThemeContext.Provider value={{ theme, setTheme: (newTheme) => { localStorage.setItem(storageKey, newTheme) setTheme(newTheme) }, resolvedTheme, }}> {children} </ThemeContext.Provider> ) } export const useTheme = () => { const context = useContext(ThemeContext) if (!context) throw new Error('useTheme must be used within ThemeProvider') return context } // components/ThemeToggle.tsx import { Moon, Sun } from 'lucide-react' import { useTheme } from '@/providers/ThemeProvider' export function ThemeToggle() { const { resolvedTheme, setTheme } = useTheme() return ( <Button variant="ghost" size="icon" onClick={() => setTheme(resolvedTheme === 'dark' ? 'light' : 'dark')} > <Sun className="size-5 rotate-0 scale-100 transition-all dark:-rotate-90 dark:scale-0" /> <Moon className="absolute size-5 rotate-90 scale-0 transition-all dark:rotate-0 dark:scale-100" /> <span className="sr-only">Toggle theme</span> </Button> ) } ``` ## Utility Functions ```typescript // lib/utils.ts import { type ClassValue, clsx } from "clsx"; import { twMerge } from "tailwind-merge"; export function cn(...inputs: ClassValue[]) { return twMerge(clsx(inputs)); } // Focus ring utility export const focusRing = cn( "focus-visible:outline-none focus-visible:ring-2", "focus-visible:ring-ring focus-visible:ring-offset-2", ); // Disabled utility export const disabled = "disabled:pointer-events-none disabled:opacity-50"; ``` ## Advanced v4 Patterns ### Custom Utilities with `@utility` Define reusable custom utilities: ```css /* Custom utility for decorative lines */ @utility line-t { @apply relative before:absolute before:top-0 before:-left-[100vw] before:h-px before:w-[200vw] before:bg-gray-950/5 dark:before:bg-white/10; } /* Custom ut
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

parallel-feature-development

Coordinate parallel feature development with file ownership

coding
⭐1
# Parallel Feature Development Strategies for decomposing features into parallel work streams, establishing file ownership boundaries, avoiding conflicts, and integrating results from multiple implementer agents. ## When to Use This Skill - Decomposing a feature for parallel implementation - Establishing file ownership boundaries between agents - Designing interface contracts between parallel work streams - Choosing integration strategies (vertical slice vs horizontal layer) - Managing branch and merge workflows for parallel development ## File Ownership Strategies ### By Directory Assign each implementer ownership of specific directories: ``` implementer-1: src/components/auth/ implementer-2: src/api/auth/ implementer-3: tests/auth/ ``` **Best for**: Well-organized codebases with clear directory boundaries. ### By Module Assign ownership of logical modules (which may span directories): ``` implementer-1: Authentication module (login, register, logout) implementer-2: Authorization module (roles, permissions, guards) ``` **Best for**: Feature-oriented architectures, domain-driven design. ### By Layer Assign ownership of architectural layers: ``` implementer-1: UI layer (components, styles, layouts) implementer-2: Business logic layer (services, validators) implementer-3: Data layer (models, repositories, migrations) ``` **Best for**: Traditional MVC/layered architectures. ## Conflict Avoidance Rules ### The Cardinal Rule **One owner per file.** No file should be assigned to multiple implementers. ### When Files Must Be Shared If a file genuinely needs changes from multiple implementers: 1. **Designate a single owner** — One implementer owns the file 2. **Other implementers request changes** — Message the owner with specific change requests 3. **Owner applies changes sequentially** — Prevents merge conflicts 4. **Alternative: Extract interfaces** — Create a separate interface file that the non-owner can import without modifying ### Interface Contracts When implementers need to coordinate at boundaries: ```typescript // src/types/auth-contract.ts (owned by team-lead, read-only for implementers) export interface AuthResponse { token: string; user: UserProfile; expiresAt: number; } export interface AuthService { login(email: string, password: string): Promise<AuthResponse>; register(data: RegisterData): Promise<AuthResponse>; } ``` Both implementers import from the contract file but neither modifies it. ## Integration Patterns ### Vertical Slice Each implementer builds a complete feature slice (UI + API + tests): ``` implementer-1: Login feature (login form + login API + login tests) implementer-2: Register feature (register form + register API + register tests) ``` **Pros**: Each slice is independently testable, minimal integration needed. **Cons**: May duplicate shared utilities, harder with tightly coupled features. ### Horizontal Layer Each implementer builds one layer across all features: ``` implementer-1: All UI components (login form, register form, profile page) implementer-2: All API endpoints (login, register, profile) implementer-3: All tests (unit, integration, e2e) ``` **Pros**: Consistent patterns within each layer, natural specialization. **Cons**: More integration points, layer 3 depends on layers 1 and 2. ### Hybrid Mix vertical and horizontal based on coupling: ``` implementer-1: Login feature (vertical slice — UI + API + tests) implementer-2: Shared auth infrastructure (horizontal — middleware, JWT utils, types) ``` **Best for**: Most real-world features with some shared infrastructure. ## Branch Management ### Single Branch Strategy All implementers work on the same feature branch: - Simple setup, no merge overhead - Requires strict file ownership to avoid conflicts - Best for: small teams (2-3), well-defined boundaries ### Multi-Branch Strategy Each implementer works on a sub-branch: ``` feature/auth ā”œā”€ā”€ feature/auth-login (implementer-1) ā”œā”€ā”€ feature/auth-register (implementer-2) └── feature/auth-tests (implementer-3) ``` - More isolation, explicit merge points - Higher overhead, merge conflicts still possible in shared files - Best for: larger teams (4+), complex features
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

python-anti-patterns

Common Python anti-patterns to avoid. Use as a checklist when

coding
⭐1
# Python Anti-Patterns Checklist A reference checklist of common mistakes and anti-patterns in Python code. Review this before finalizing implementations to catch issues early. ## When to Use This Skill - Reviewing code before merge - Debugging mysterious issues - Teaching or learning Python best practices - Establishing team coding standards - Refactoring legacy code **Note:** This skill focuses on what to avoid. For guidance on positive patterns and architecture, see the `python-design-patterns` skill. ## Infrastructure Anti-Patterns ### Scattered Timeout/Retry Logic ```python # BAD: Timeout logic duplicated everywhere def fetch_user(user_id): try: return requests.get(url, timeout=30) except Timeout: logger.warning("Timeout fetching user") return None def fetch_orders(user_id): try: return requests.get(url, timeout=30) except Timeout: logger.warning("Timeout fetching orders") return None ``` **Fix:** Centralize in decorators or client wrappers. ```python # GOOD: Centralized retry logic @retry(stop=stop_after_attempt(3), wait=wait_exponential()) def http_get(url: str) -> Response: return requests.get(url, timeout=30) ``` ### Double Retry ```python # BAD: Retrying at multiple layers @retry(max_attempts=3) # Application retry def call_service(): return client.request() # Client also has retry configured! ``` **Fix:** Retry at one layer only. Know your infrastructure's retry behavior. ### Hard-Coded Configuration ```python # BAD: Secrets and config in code DB_HOST = "prod-db.example.com" API_KEY = "sk-12345" def connect(): return psycopg.connect(f"host={DB_HOST}...") ``` **Fix:** Use environment variables with typed settings. ```python # GOOD from pydantic_settings import BaseSettings class Settings(BaseSettings): db_host: str = Field(alias="DB_HOST") api_key: str = Field(alias="API_KEY") settings = Settings() ``` ## Architecture Anti-Patterns ### Exposed Internal Types ```python # BAD: Leaking ORM model to API @app.get("/users/{id}") def get_user(id: str) -> UserModel: # SQLAlchemy model return db.query(UserModel).get(id) ``` **Fix:** Use DTOs/response models. ```python # GOOD @app.get("/users/{id}") def get_user(id: str) -> UserResponse: user = db.query(UserModel).get(id) return UserResponse.from_orm(user) ``` ### Mixed I/O and Business Logic ```python # BAD: SQL embedded in business logic def calculate_discount(user_id: str) -> float: user = db.query("SELECT * FROM users WHERE id = ?", user_id) orders = db.query("SELECT * FROM orders WHERE user_id = ?", user_id) # Business logic mixed with data access if len(orders) > 10: return 0.15 return 0.0 ``` **Fix:** Repository pattern. Keep business logic pure. ```python # GOOD def calculate_discount(user: User, orders: list[Order]) -> float: # Pure business logic, easily testable if len(orders) > 10: return 0.15 return 0.0 ``` ## Error Handling Anti-Patterns ### Bare Exception Handling ```python # BAD: Swallowing all exceptions try: process() except Exception: pass # Silent failure - bugs hidden forever ``` **Fix:** Catch specific exceptions. Log or handle appropriately. ```python # GOOD try: process() except ConnectionError as e: logger.warning("Connection failed, will retry", error=str(e)) raise except ValueError as e: logger.error("Invalid input", error=str(e)) raise BadRequestError(str(e)) ``` ### Ignored Partial Failures ```python # BAD: Stops on first error def process_batch(items): results = [] for item in items: result = process(item) # Raises on error - batch aborted results.append(result) return results ``` **Fix:** Capture both successes and failures. ```python # GOOD def process_batch(items) -> BatchResult: succeeded = {} failed = {} for idx, item in enumerate(items): try: succeeded[idx] = process(item) except Exception as e: failed[idx] = e return BatchResult(succeeded, failed) ``` ### Missing Input Validation ```python # BAD: No validation def create_user(data: dict): return User(**data) # Crashes deep in code on bad input ``` **Fix:** Validate early at API boundaries. ```python # GOOD def create_user(data: dict) -> User: validated = CreateUserInput.model_validate(data) return User.from_input(validated) ``` ## Resource Anti-Patterns ### Unclosed Resources ```python # BAD: File never closed def read_file(path): f = open(path) return f.read() # What if this raises? ``` **Fix:** Use context managers. ```python # GOOD def read_file(path): with open(path) as f: return f.read() ``` ### Blocking in Async ```python # BAD: Blocks the entire event loop async def fetch_data(): time.sleep(1) # Blocks everything! response = requests.get(url) # Also blocks! ``` **Fix:** Use async-native libraries. ```python # GOOD async def fetch_data(): await asyncio.sleep(1) async with httpx.AsyncClient() as client: response = await client.get(url) ``` ## Type Safety Anti-Patterns ### Missing Type Hints ```python # BAD: No types def process(data): return data["value"] * 2 ``` **Fix:** Annotate all public functions. ```python # GOOD def process(data: dict[str, int]) -> int: return data["value"] * 2 ``` ### Untyped Collections ```python # BAD: Generic list without type parameter def get_users() -> list: ... ``` **Fix:** Use type parameters. ```python # GOOD def get_users() -> list[User]: ... ``` ## Testing Anti-Patterns ### Only Testing Happy Paths ```python # BAD: Only tests success case def test_create_user(): user = service.create_user(valid_data) assert user.id is not None ``` **Fix:** Test error conditions and edge cases. ```python # GOOD def test_create_user_success(): user = service.create_user(valid_data) assert user.id is not None def test_create_user_invalid_email(): with pytest.raises(ValueError, match="Invalid email"): service.create_user(invalid_email_data) def test_create_user_duplicate_email(): service.create_user(valid_data) with pytest.raises(ConflictError): service.create_user(valid_data) ``` ### Over-Mocking ```python # BAD: Mocking everything def test_user_service(): mock_repo = Mock() mock_cache = Mock() mock_logger = Mock() mock_metrics = Mock() # Test doesn't verify real behavior ``` **Fix:** Use integration tests for critical paths. Mock only external services. ## Quick Review Checklist Before finalizing code, verify: - [ ] No scattered timeout/retry logic (centralized) - [ ] No double retry (app + infrastructure) - [ ] No hard-coded configuration or secrets - [ ] No exposed internal types (ORM models, protobufs) - [ ] No mixed I/O and business logic - [ ] No bare `except Exception: pass` - [ ] No ignored partial failures in batches - [ ] No missing input validation - [ ] No unclosed resources (using context managers) - [ ] No blocking calls in async code - [ ] All public functions have type hints - [ ] Collections have type parameters - [ ] Error paths are tested - [ ] Edge cases are covered ## Common Fixes Summary | Anti-Pattern | Fix | |-------------|-----| | Scattered retry logic | Centralized decorators | | Hard-coded config | Environment variables + pydantic-settings | | Exposed ORM models | DTO/response schemas | | Mixed I/O + logic | Repository pattern | | Bare except | Catch specific exceptions | | Batch stops on error | Return BatchResult with successes/failures | | No validation | Validate at boundaries with Pydantic | | Unclosed resources | Context managers | | Blocking in async | Async-native libraries | | Missing types | Type annotations on all public APIs | | Only happy path tests | Test errors and edge cases |
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

shellcheck-configuration

Master ShellCheck static analysis configuration and usage for shell

coding
⭐1
# ShellCheck Configuration and Static Analysis Comprehensive guidance for configuring and using ShellCheck to improve shell script quality, catch common pitfalls, and enforce best practices through static code analysis. ## When to Use This Skill - Setting up linting for shell scripts in CI/CD pipelines - Analyzing existing shell scripts for issues - Understanding ShellCheck error codes and warnings - Configuring ShellCheck for specific project requirements - Integrating ShellCheck into development workflows - Suppressing false positives and configuring rule sets - Enforcing consistent code quality standards - Migrating scripts to meet quality gates ## ShellCheck Fundamentals ### What is ShellCheck? ShellCheck is a static analysis tool that analyzes shell scripts and detects problematic patterns. It supports: - Bash, sh, dash, ksh, and other POSIX shells - Over 100 different warnings and errors - Configuration for target shell and flags - Integration with editors and CI/CD systems ### Installation ```bash # macOS with Homebrew brew install shellcheck # Ubuntu/Debian apt-get install shellcheck # From source git clone https://github.com/koalaman/shellcheck.git cd shellcheck make build make install # Verify installation shellcheck --version ``` ## Configuration Files ### .shellcheckrc (Project Level) Create `.shellcheckrc` in your project root: ``` # Specify target shell shell=bash # Enable optional checks enable=avoid-nullary-conditions enable=require-variable-braces # Disable specific warnings disable=SC1091 disable=SC2086 ``` ### Environment Variables ```bash # Set default shell target export SHELLCHECK_SHELL=bash # Enable strict mode export SHELLCHECK_STRICT=true # Specify configuration file location export SHELLCHECK_CONFIG=~/.shellcheckrc ``` ## Common ShellCheck Error Codes ### SC1000-1099: Parser Errors ```bash # SC1004: Backslash continuation not followed by newline echo hello\ world # Error - needs line continuation # SC1008: Invalid data for operator `==' if [[ $var = "value" ]]; then # Space before == true fi ``` ### SC2000-2099: Shell Issues ```bash # SC2009: Consider using pgrep or pidof instead of grep|grep ps aux | grep -v grep | grep myprocess # Use pgrep instead # SC2012: Use `ls` only for viewing. Use `find` for reliable output for file in $(ls -la) # Better: use find or globbing # SC2015: Avoid using && and || instead of if-then-else [[ -f "$file" ]] && echo "found" || echo "not found" # Less clear # SC2016: Expressions don't expand in single quotes echo '$VAR' # Literal $VAR, not variable expansion # SC2026: This word is non-standard. Set POSIXLY_CORRECT # when using with scripts for other shells ``` ### SC2100-2199: Quoting Issues ```bash # SC2086: Double quote to prevent globbing and word splitting for i in $list; do # Should be: for i in $list or for i in "$list" echo "$i" done # SC2115: Literal tilde in path not expanded. Use $HOME instead ~/.bashrc # In strings, use "$HOME/.bashrc" # SC2181: Check exit code directly with `if`, not indirectly in a list some_command if [ $? -eq 0 ]; then # Better: if some_command; then # SC2206: Quote to prevent word splitting or set IFS array=( $items ) # Should use: array=( $items ) ``` ### SC3000-3999: POSIX Compliance Issues ```bash # SC3010: In POSIX sh, use 'case' instead of 'cond && foo' [[ $var == "value" ]] && do_something # Not POSIX # SC3043: In POSIX sh, use 'local' is undefined function my_func() { local var=value # Not POSIX in some shells } ``` ## Practical Configuration Examples ### Minimal Configuration (Strict POSIX) ```bash #!/bin/bash # Configure for maximum portability shellcheck \ --shell=sh \ --external-sources \ --check-sourced \ script.sh ``` ### Development Configuration (Bash with Relaxed Rules) ```bash #!/bin/bash # Configure for Bash development shellcheck \ --shell=bash \ --exclude=SC1091,SC2119 \ --enable=all \ script.sh ``` ### CI/CD Integration Configuration ```bash #!/bin/bash set -Eeuo pipefail # Analyze all shell scripts and fail on issues find . -type f -name "*.sh" | while read -r script; do echo "Checking: $script" shellcheck \ --shell=bash \ --format=gcc \ --exclude=SC1091 \ "$script" || exit 1 done ``` ### .shellcheckrc for Project ``` # Shell dialect to analyze against shell=bash # Enable optional checks enable=avoid-nullary-conditions,require-variable-braces,check-unassigned-uppercase # Disable specific warnings # SC1091: Not following sourced files (many false positives) disable=SC1091 # SC2119: Use function_name instead of function_name -- (arguments) disable=SC2119 # External files to source for context external-sources=true ``` ## Integration Patterns ### Pre-commit Hook Configuration ```bash #!/bin/bash # .git/hooks/pre-commit #!/bin/bash set -e # Find all shell scripts changed in this commit git diff --cached --name-only | grep '\.sh$' | while read -r script; do echo "Linting: $script" if ! shellcheck "$script"; then echo "ShellCheck failed on $script" exit 1 fi done ``` ### GitHub Actions Workflow ```yaml name: ShellCheck on: [push, pull_request] jobs: shellcheck: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Run ShellCheck run: | sudo apt-get install shellcheck find . -type f -name "*.sh" -exec shellcheck {} \; ``` ### GitLab CI Pipeline ```yaml shellcheck: stage: lint image: koalaman/shellcheck-alpine script: - find . -type f -name "*.sh" -exec shellcheck {} \; allow_failure: false ``` ## Handling ShellCheck Violations ### Suppressing Specific Warnings ```bash #!/bin/bash # Disable warning for entire line # shellcheck disable=SC2086 for file in $(ls -la); do echo "$file" done # Disable for entire script # shellcheck disable=SC1091,SC2119 # Disable multiple warnings (format varies) command_that_fails() { # shellcheck disable=SC2015 [ -f "$1" ] && echo "found" || echo "not found" } # Disable specific check for source directive # shellcheck source=./helper.sh source helper.sh ``` ### Common Violations and Fixes #### SC2086: Double quote to prevent word splitting ```bash # Problem for i in $list; do done # Solution for i in $list; do done # If $list is already quoted, or for i in "${list[@]}"; do done # If list is an array ``` #### SC2181: Check exit code directly ```bash # Problem some_command if [ $? -eq 0 ]; then echo "success" fi # Solution if some_command; then echo "success" fi ``` #### SC2015: Use if-then instead of && || ```bash # Problem [ -f "$file" ] && echo "exists" || echo "not found" # Solution - clearer intent if [ -f "$file" ]; then echo "exists" else echo "not found" fi ``` #### SC2016: Expressions don't expand in single quotes ```bash # Problem echo 'Variable value: $VAR' # Solution echo "Variable value: $VAR" ``` #### SC2009: Use pgrep instead of grep ```bash # Problem ps aux | grep -v grep | grep myprocess # Solution pgrep -f myprocess ``` ## Performance Optimization ### Checking Multiple Files ```bash #!/bin/bash # Sequential checking for script in *.sh; do shellcheck "$script" done # Parallel checking (faster) find . -name "*.sh" -print0 | \ xargs -0 -P 4 -n 1 shellcheck ``` ### Caching Results ```bash #!/bin/bash CACHE_DIR=".shellcheck_cache" mkdir -p "$CACHE_DIR" check_script() { local script="$1" local hash local cache_file hash=$(sha256sum "$script" | cut -d' ' -f1) cache_file="$CACHE_DIR/$hash" if [[ ! -f "$cache_file" ]]; then if shellcheck "$script" > "$cache_file" 2>&1; then touch "$cache_file.ok" else return 1 fi fi [[ -f "$cache_file.ok" ]] } find . -name "*.sh" | while read -r script; do check_script "$script" || exit 1 done ``` ## Output Formats ### Default Format ```bash shellcheck script.sh # Output: # script.sh:1:3: warning: foo is referenced but not assigned. [SC2154] ``` ### GCC Format (for CI/CD) ```bash shellcheck --format=gcc script.sh # Output: # script.sh:1:3: warning: foo is referenced but not assigned. ``` ### JSON Format (for parsing) ```bash shellcheck --format=json script.sh # Output: # [{"file": "script.sh", "line": 1, "column": 3, "level": "warning", "code": 2154, "message": "..."}] ``` ### Quiet Format ```bash shellcheck --format=quiet script.sh # Returns non-zero if issues found, no output otherwise ``` ## Best Practices 1. **Run ShellCheck in CI/CD** - Catch issues before merging 2. **Configure for your target shell** - Don't analyze bash as sh 3. **Document exclusions** - Explain why violations are suppressed 4. **Address violations** - Don't just disable warnings 5. **Enable strict mode** - Use `--enable=all` with careful exclusions 6. **Update regularly** - Keep ShellCheck current for new checks 7. **Use pre-commit hooks** - Catch issues locally before pushing 8. **Integrate with editors** - Get real-time feedback during development ## Resources - **ShellCheck GitHub**: https://github.com/koalaman/shellcheck - **ShellCheck Wiki**: https://www.shellcheck.net/wiki/ - **Error Code Reference**: https://www.shellcheck.net/
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

go-concurrency-patterns

Master Go concurrency with goroutines, channels, sync primitives,

coding
⭐1
# Go Concurrency Patterns Production patterns for Go concurrency including goroutines, channels, synchronization primitives, and context management. ## When to Use This Skill - Building concurrent Go applications - Implementing worker pools and pipelines - Managing goroutine lifecycles - Using channels for communication - Debugging race conditions - Implementing graceful shutdown ## Core Concepts ### 1. Go Concurrency Primitives | Primitive | Purpose | | ----------------- | -------------------------------- | | `goroutine` | Lightweight concurrent execution | | `channel` | Communication between goroutines | | `select` | Multiplex channel operations | | `sync.Mutex` | Mutual exclusion | | `sync.WaitGroup` | Wait for goroutines to complete | | `context.Context` | Cancellation and deadlines | ### 2. Go Concurrency Mantra ``` Don't communicate by sharing memory; share memory by communicating. ``` ## Quick Start ```go package main import ( "context" "fmt" "sync" "time" ) func main() { ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() results := make(chan string, 10) var wg sync.WaitGroup // Spawn workers for i := 0; i < 3; i++ { wg.Add(1) go worker(ctx, i, results, &wg) } // Close results when done go func() { wg.Wait() close(results) }() // Collect results for result := range results { fmt.Println(result) } } func worker(ctx context.Context, id int, results chan<- string, wg *sync.WaitGroup) { defer wg.Done() select { case <-ctx.Done(): return case results <- fmt.Sprintf("Worker %d done", id): } } ``` ## Patterns ### Pattern 1: Worker Pool ```go package main import ( "context" "fmt" "sync" ) type Job struct { ID int Data string } type Result struct { JobID int Output string Err error } func WorkerPool(ctx context.Context, numWorkers int, jobs <-chan Job) <-chan Result { results := make(chan Result, len(jobs)) var wg sync.WaitGroup for i := 0; i < numWorkers; i++ { wg.Add(1) go func(workerID int) { defer wg.Done() for job := range jobs { select { case <-ctx.Done(): return default: result := processJob(job) results <- result } } }(i) } go func() { wg.Wait() close(results) }() return results } func processJob(job Job) Result { // Simulate work return Result{ JobID: job.ID, Output: fmt.Sprintf("Processed: %s", job.Data), } } // Usage func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() jobs := make(chan Job, 100) // Send jobs go func() { for i := 0; i < 50; i++ { jobs <- Job{ID: i, Data: fmt.Sprintf("job-%d", i)} } close(jobs) }() // Process with 5 workers results := WorkerPool(ctx, 5, jobs) for result := range results { fmt.Printf("Result: %+v\n", result) } } ``` ### Pattern 2: Fan-Out/Fan-In Pipeline ```go package main import ( "context" "sync" ) // Stage 1: Generate numbers func generate(ctx context.Context, nums ...int) <-chan int { out := make(chan int) go func() { defer close(out) for _, n := range nums { select { case <-ctx.Done(): return case out <- n: } } }() return out } // Stage 2: Square numbers (can run multiple instances) func square(ctx context.Context, in <-chan int) <-chan int { out := make(chan int) go func() { defer close(out) for n := range in { select { case <-ctx.Done(): return case out <- n * n: } } }() return out } // Fan-in: Merge multiple channels into one func merge(ctx context.Context, cs ...<-chan int) <-chan int { var wg sync.WaitGroup out := make(chan int) // Start output goroutine for each input channel output := func(c <-chan int) { defer wg.Done() for n := range c { select { case <-ctx.Done(): return case out <- n: } } } wg.Add(len(cs)) for _, c := range cs { go output(c) } // Close out after all inputs are done go func() { wg.Wait() close(out) }() return out } func main() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() // Generate input in := generate(ctx, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10) // Fan out to multiple squarers c1 := square(ctx, in) c2 := square(ctx, in) c3 := square(ctx, in) // Fan in results for result := range merge(ctx, c1, c2, c3) { fmt.Println(result) } } ``` ### Pattern 3: Bounded Concurrency with Semaphore ```go package main import ( "context" "fmt" "golang.org/x/sync/semaphore" "sync" ) type RateLimitedWorker struct { sem *semaphore.Weighted } func NewRateLimitedWorker(maxConcurrent int64) *RateLimitedWorker { return &RateLimitedWorker{ sem: semaphore.NewWeighted(maxConcurrent), } } func (w *RateLimitedWorker) Do(ctx context.Context, tasks []func() error) []error { var ( wg sync.WaitGroup mu sync.Mutex errors []error ) for _, task := range tasks { // Acquire semaphore (blocks if at limit) if err := w.sem.Acquire(ctx, 1); err != nil { return []error{err} } wg.Add(1) go func(t func() error) { defer wg.Done() defer w.sem.Release(1) if err := t(); err != nil { mu.Lock() errors = append(errors, err) mu.Unlock() } }(task) } wg.Wait() return errors } // Alternative: Channel-based semaphore type Semaphore chan struct{} func NewSemaphore(n int) Semaphore { return make(chan struct{}, n) } func (s Semaphore) Acquire() { s <- struct{}{} } func (s Semaphore) Release() { <-s } ``` ### Pattern 4: Graceful Shutdown ```go package main import ( "context" "fmt" "os" "os/signal" "sync" "syscall" "time" ) type Server struct { shutdown chan struct{} wg sync.WaitGroup } func NewServer() *Server { return &Server{ shutdown: make(chan struct{}), } } func (s *Server) Start(ctx context.Context) { // Start workers for i := 0; i < 5; i++ { s.wg.Add(1) go s.worker(ctx, i) } } func (s *Server) worker(ctx context.Context, id int) { defer s.wg.Done() defer fmt.Printf("Worker %d stopped\n", id) ticker := time.NewTicker(time.Second) defer ticker.Stop() for { select { case <-ctx.Done(): // Cleanup fmt.Printf("Worker %d cleaning up...\n", id) time.Sleep(500 * time.Millisecond) // Simulated cleanup return case <-ticker.C: fmt.Printf("Worker %d working...\n", id) } } } func (s *Server) Shutdown(timeout time.Duration) { // Signal shutdown close(s.shutdown) // Wait with timeout done := make(chan struct{}) go func() { s.wg.Wait() close(done) }() select { case <-done: fmt.Println("Clean shutdown completed") case <-time.After(timeout): fmt.Println("Shutdown timed out, forcing exit") } } func main() { // Setup signal handling ctx, cancel := context.WithCancel(context.Background()) sigCh := make(chan os.Signal, 1) signal.Notify(sigCh, syscall.SIGINT, syscall.SIGTERM) server := NewServer() server.Start(ctx) // Wait for signal sig := <-sigCh fmt.Printf("\nReceived signal: %v\n", sig) // Cancel context to stop workers cancel() // Wait for graceful shutdown server.Shutdown(5 * time.Second) } ``` ### Pattern 5: Error Group with Cancellation ```go package main import ( "context" "fmt" "golang.org/x/sync/errgroup" "net/http" ) func fetchAllURLs(ctx context.Context, urls []string) ([]string, error) { g, ctx := errgroup.WithContext(ctx) results := make([]string, len(urls)) for i, url := range urls { i, url := i, url // Capture loop variables g.Go(func() error { req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { return fmt.Errorf("creating request for %s: %w", url, err) } resp, err := http.DefaultClient.Do(req) if err != nil { return fmt.Errorf("fetching %s: %w", url, err) } defer resp.Body.Close() results[i] = fmt.Sprintf("%s: %d", url, resp.StatusCode) return nil }) } // Wait for all goroutines to complete or one to fail if err := g.Wait(); err != nil { return nil, err // First error cancels all others } return results, nil } // With concurrency limit func fetchWithLimit(ctx context.Context, urls []string, limit int) ([]string, error) { g, ctx := errgroup.WithContext(ctx) g.SetLimit(limit) // Max concurrent goroutines results := make([]string, len(urls)) var mu sync.Mutex for i, url := range urls { i, url := i, url g.Go(func() error { result, err := fetchURL(ctx, url) if err != nil { return err } mu.Lock() results[i] = result mu.Unlock() return nil }) } if err := g.Wait(); err != nil { return nil, err } return results, nil } ``` ### Pattern 6: Concurrent Map with sync.Map ```go package main import ( "sync" ) // For frequent reads, infrequent writes type Cache struct { m sync.Map } func (c *Cache) Get(key string) (interface{}, bool) { return c.m.Load(key) } func (c *Cache) Set(key string, value interface{}) { c.m.Store(key, value) } func (c *Cache) GetOrSet(key string, value interface{}) (interface{}, bool) { return c.m.LoadOrStore(key, value) } func (c *Cache) Delete(key string) { c.m.Delete(key) } // For write-heavy workloads, use sharded map type ShardedMap struct { shards []*shard numShards int } type shard struct { sync.RWMutex data map[string]interface{} } func NewShardedMap(numShards int) *ShardedMap { m := &ShardedMap{ shards: make([]*shard, numShards), numShards: numShards, } for i := range m.shards { m.shards[i] = &shard{data: make(map[string]interface{})} } return m } func (m *ShardedMap) getShard(key string) *shard { // Simple hash h := 0 for _, c := range key { h = 31*h + int(c) } return m.shards[h%m.numShards] } func (m *ShardedMap) Get(key string) (interface{}, bool) { shard := m.getShard(key) shard.RLock() defer shard.RUnlock() v, ok := shard.data[key] return v, ok } func (m *ShardedMap) Set(key string, value interface{}) { shard := m.getShard(key) shard.Lock() defer shard.Unlock() shard.data[key] = value } ``` ### Pattern 7: Select with Timeout and Default ```go func selectPatterns() { ch := make(chan int) // Timeout pattern select { case v := <-ch: fmt.Println("Received:", v) case <-time.After(time.Second): fmt.Println("Timeout!") } // Non-blocking send/receive select { case ch <- 42: fmt.Println("Sent") default: fmt.Println("Channel full, skipping") } // Priority select (check high priority first) highPriority := make(chan int) lowPriority := make(chan int) for { select { case msg := <-highPriority: fmt.Println("High priority:", msg) default: select { case msg := <-highPriority: fmt.Println("High priority:", msg) case msg := <-lowPriority: fmt.Println("Low priority:", msg) } } } } ``` ## Race Detection ```bash # Run tests with race detector go test -race ./... # Build with race detector go build -race . # Run with race detector go run -race main.go ``` ## Best Practices ### Do's - **Use context** - For cancellation and deadlines - **Close channels** - From sender side only - **Use errgroup** - For concurrent operations with errors - **Buffer channels** - When you know the count - **Prefer channels** - Over mutexes when possible ### Don'ts - **Don't leak goroutines** - Always have exit path - **Don't close from receiver** - Causes panic - **Don't use shared memory** - Unless necessary - **Don't ignore context cancellation** - Check ctx.Done() - **Don't use time.Sleep for sync** - Use proper primitives ## Resources - [Go Concurrency Patterns](https://go.dev/blog/pipelines) - [Effective Go - Concurrency](https://go.dev/doc/effective_go#concurrency) - [Go by Example - Goroutines](https://gobyexample.com/goroutines)
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

rust-async-patterns

Master Rust async programming with Tokio, async traits, error

coding
⭐1
# Rust Async Patterns Production patterns for async Rust programming with Tokio runtime, including tasks, channels, streams, and error handling. ## When to Use This Skill - Building async Rust applications - Implementing concurrent network services - Using Tokio for async I/O - Handling async errors properly - Debugging async code issues - Optimizing async performance ## Core Concepts ### 1. Async Execution Model ``` Future (lazy) → poll() → Ready(value) | Pending ↑ ↓ Waker ← Runtime schedules ``` ### 2. Key Abstractions | Concept | Purpose | | ---------- | ---------------------------------------- | | `Future` | Lazy computation that may complete later | | `async fn` | Function returning impl Future | | `await` | Suspend until future completes | | `Task` | Spawned future running concurrently | | `Runtime` | Executor that polls futures | ## Quick Start ```toml # Cargo.toml [dependencies] tokio = { version = "1", features = ["full"] } futures = "0.3" async-trait = "0.1" anyhow = "1.0" tracing = "0.1" tracing-subscriber = "0.3" ``` ```rust use tokio::time::{sleep, Duration}; use anyhow::Result; #[tokio::main] async fn main() -> Result<()> { // Initialize tracing tracing_subscriber::fmt::init(); // Async operations let result = fetch_data("https://api.example.com").await?; println!("Got: {}", result); Ok(()) } async fn fetch_data(url: &str) -> Result<String> { // Simulated async operation sleep(Duration::from_millis(100)).await; Ok(format!("Data from {}", url)) } ``` ## Patterns ### Pattern 1: Concurrent Task Execution ```rust use tokio::task::JoinSet; use anyhow::Result; // Spawn multiple concurrent tasks async fn fetch_all_concurrent(urls: Vec<String>) -> Result<Vec<String>> { let mut set = JoinSet::new(); for url in urls { set.spawn(async move { fetch_data(&url).await }); } let mut results = Vec::new(); while let Some(res) = set.join_next().await { match res { Ok(Ok(data)) => results.push(data), Ok(Err(e)) => tracing::error!("Task failed: {}", e), Err(e) => tracing::error!("Join error: {}", e), } } Ok(results) } // With concurrency limit use futures::stream::{self, StreamExt}; async fn fetch_with_limit(urls: Vec<String>, limit: usize) -> Vec<Result<String>> { stream::iter(urls) .map(|url| async move { fetch_data(&url).await }) .buffer_unordered(limit) // Max concurrent tasks .collect() .await } // Select first to complete use tokio::select; async fn race_requests(url1: &str, url2: &str) -> Result<String> { select! { result = fetch_data(url1) => result, result = fetch_data(url2) => result, } } ``` ### Pattern 2: Channels for Communication ```rust use tokio::sync::{mpsc, broadcast, oneshot, watch}; // Multi-producer, single-consumer async fn mpsc_example() { let (tx, mut rx) = mpsc::channel::<String>(100); // Spawn producer let tx2 = tx.clone(); tokio::spawn(async move { tx2.send("Hello".to_string()).await.unwrap(); }); // Consume while let Some(msg) = rx.recv().await { println!("Got: {}", msg); } } // Broadcast: multi-producer, multi-consumer async fn broadcast_example() { let (tx, _) = broadcast::channel::<String>(100); let mut rx1 = tx.subscribe(); let mut rx2 = tx.subscribe(); tx.send("Event".to_string()).unwrap(); // Both receivers get the message let _ = rx1.recv().await; let _ = rx2.recv().await; } // Oneshot: single value, single use async fn oneshot_example() -> String { let (tx, rx) = oneshot::channel::<String>(); tokio::spawn(async move { tx.send("Result".to_string()).unwrap(); }); rx.await.unwrap() } // Watch: single producer, multi-consumer, latest value async fn watch_example() { let (tx, mut rx) = watch::channel("initial".to_string()); tokio::spawn(async move { loop { // Wait for changes rx.changed().await.unwrap(); println!("New value: {}", *rx.borrow()); } }); tx.send("updated".to_string()).unwrap(); } ``` ### Pattern 3: Async Error Handling ```rust use anyhow::{Context, Result, bail}; use thiserror::Error; #[derive(Error, Debug)] pub enum ServiceError { #[error("Network error: {0}")] Network(#[from] reqwest::Error), #[error("Database error: {0}")] Database(#[from] sqlx::Error), #[error("Not found: {0}")] NotFound(String), #[error("Timeout after {0:?}")] Timeout(std::time::Duration), } // Using anyhow for application errors async fn process_request(id: &str) -> Result<Response> { let data = fetch_data(id) .await .context("Failed to fetch data")?; let parsed = parse_response(&data) .context("Failed to parse response")?; Ok(parsed) } // Using custom errors for library code async fn get_user(id: &str) -> Result<User, ServiceError> { let result = db.query(id).await?; match result { Some(user) => Ok(user), None => Err(ServiceError::NotFound(id.to_string())), } } // Timeout wrapper use tokio::time::timeout; async fn with_timeout<T, F>(duration: Duration, future: F) -> Result<T, ServiceError> where F: std::future::Future<Output = Result<T, ServiceError>>, { timeout(duration, future) .await .map_err(|_| ServiceError::Timeout(duration))? } ``` ### Pattern 4: Graceful Shutdown ```rust use tokio::signal; use tokio::sync::broadcast; use tokio_util::sync::CancellationToken; async fn run_server() -> Result<()> { // Method 1: CancellationToken let token = CancellationToken::new(); let token_clone = token.clone(); // Spawn task that respects cancellation tokio::spawn(async move { loop { tokio::select! { _ = token_clone.cancelled() => { tracing::info!("Task shutting down"); break; } _ = do_work() => {} } } }); // Wait for shutdown signal signal::ctrl_c().await?; tracing::info!("Shutdown signal received"); // Cancel all tasks token.cancel(); // Give tasks time to cleanup tokio::time::sleep(Duration::from_secs(5)).await; Ok(()) } // Method 2: Broadcast channel for shutdown async fn run_with_broadcast() -> Result<()> { let (shutdown_tx, _) = broadcast::channel::<()>(1); let mut rx = shutdown_tx.subscribe(); tokio::spawn(async move { tokio::select! { _ = rx.recv() => { tracing::info!("Received shutdown"); } _ = async { loop { do_work().await } } => {} } }); signal::ctrl_c().await?; let _ = shutdown_tx.send(()); Ok(()) } ``` ### Pattern 5: Async Traits ```rust use async_trait::async_trait; #[async_trait] pub trait Repository { async fn get(&self, id: &str) -> Result<Entity>; async fn save(&self, entity: &Entity) -> Result<()>; async fn delete(&self, id: &str) -> Result<()>; } pub struct PostgresRepository { pool: sqlx::PgPool, } #[async_trait] impl Repository for PostgresRepository { async fn get(&self, id: &str) -> Result<Entity> { sqlx::query_as!(Entity, "SELECT * FROM entities WHERE id = $1", id) .fetch_one(&self.pool) .await .map_err(Into::into) } async fn save(&self, entity: &Entity) -> Result<()> { sqlx::query!( "INSERT INTO entities (id, data) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET data = $2", entity.id, entity.data ) .execute(&self.pool) .await?; Ok(()) } async fn delete(&self, id: &str) -> Result<()> { sqlx::query!("DELETE FROM entities WHERE id = $1", id) .execute(&self.pool) .await?; Ok(()) } } // Trait object usage async fn process(repo: &dyn Repository, id: &str) -> Result<()> { let entity = repo.get(id).await?; // Process... repo.save(&entity).await } ``` ### Pattern 6: Streams and Async Iteration ```rust use futures::stream::{self, Stream, StreamExt}; use async_stream::stream; // Create stream from async iterator fn numbers_stream() -> impl Stream<Item = i32> { stream! { for i in 0..10 { tokio::time::sleep(Duration::from_millis(100)).await; yield i; } } } // Process stream async fn process_stream() { let stream = numbers_stream(); // Map and filter let processed: Vec<_> = stream .filter(|n| futures::future::ready(*n % 2 == 0)) .map(|n| n * 2) .collect() .await; println!("{:?}", processed); } // Chunked processing async fn process_in_chunks() { let stream = numbers_stream(); let mut chunks = stream.chunks(3); while let Some(chunk) = chunks.next().await { println!("Processing chunk: {:?}", chunk); } } // Merge multiple streams async fn merge_streams() { let stream1 = numbers_stream(); let stream2 = numbers_stream(); let merged = stream::select(stream1, stream2); merged .for_each(|n| async move { println!("Got: {}", n); }) .await; } ``` ### Pattern 7: Resource Management ```rust use std::sync::Arc; use tokio::sync::{Mutex, RwLock, Semaphore}; // Shared state with RwLock (prefer for read-heavy) struct Cache { data: RwLock<HashMap<String, String>>, } impl Cache { async fn get(&self, key: &str) -> Option<String> { self.data.read().await.get(key).cloned() } async fn set(&self, key: String, value: String) { self.data.write().await.insert(key, value); } } // Connection pool with semaphore struct Pool { semaphore: Semaphore, connections: Mutex<Vec<Connection>>, } impl Pool { fn new(size: usize) -> Self { Self { semaphore: Semaphore::new(size), connections: Mutex::new((0..size).map(|_| Connection::new()).collect()), } } async fn acquire(&self) -> PooledConnection<'_> { let permit = self.semaphore.acquire().await.unwrap(); let conn = self.connections.lock().await.pop().unwrap(); PooledConnection { pool: self, conn: Some(conn), _permit: permit } } } struct PooledConnection<'a> { pool: &'a Pool, conn: Option<Connection>, _permit: tokio::sync::SemaphorePermit<'a>, } impl Drop for PooledConnection<'_> { fn drop(&mut self) { if let Some(conn) = self.conn.take() { let pool = self.pool; tokio::spawn(async move { pool.connections.lock().await.push(conn); }); } } } ``` ## Debugging Tips ```rust // Enable tokio-console for runtime debugging // Cargo.toml: tokio = { features = ["tracing"] } // Run: RUSTFLAGS="--cfg tokio_unstable" cargo run // Then: tokio-console // Instrument async functions use tracing::instrument; #[instrument(skip(pool))] async fn fetch_user(pool: &PgPool, id: &str) -> Result<User> { tracing::debug!("Fetching user"); // ... } // Track task spawning let span = tracing::info_span!("worker", id = %worker_id); tokio::spawn(async move { // Enters span when polled }.instrument(span)); ``` ## Best Practices ### Do's - **Use `tokio::select!`** - For racing futures - **Prefer channels** - Over shared state when possible - **Use `JoinSet`** - For managing multiple tasks - **Instrument with tracing** - For debugging async code - **Handle cancellation** - Check `CancellationToken` ### Don'ts - **Don't block** - Never use `std::thread::sleep` in async - **Don't hold locks across awaits** - Causes deadlocks - **Don't spawn unboundedly** - Use semaphores for limits - **Don't ignore errors** - Propagate with `?` or log - **Don't forget Send bounds** - For spawned futures ## Resources - [Tokio Tutorial](https://tokio.rs/tokio/tutorial) - [Async Book](https://rust-lang.github.io/async-book/) - [Tokio Console](https://github.com/tokio-rs/console)
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

code-review-excellence

Master effective code review practices to provide constructive

coding
⭐1
# Code Review Excellence Transform code reviews from gatekeeping to knowledge sharing through constructive feedback, systematic analysis, and collaborative improvement. ## When to Use This Skill - Reviewing pull requests and code changes - Establishing code review standards for teams - Mentoring junior developers through reviews - Conducting architecture reviews - Creating review checklists and guidelines - Improving team collaboration - Reducing code review cycle time - Maintaining code quality standards ## Core Principles ### 1. The Review Mindset **Goals of Code Review:** - Catch bugs and edge cases - Ensure code maintainability - Share knowledge across team - Enforce coding standards - Improve design and architecture - Build team culture **Not the Goals:** - Show off knowledge - Nitpick formatting (use linters) - Block progress unnecessarily - Rewrite to your preference ### 2. Effective Feedback **Good Feedback is:** - Specific and actionable - Educational, not judgmental - Focused on the code, not the person - Balanced (praise good work too) - Prioritized (critical vs nice-to-have) ```markdown āŒ Bad: "This is wrong." āœ… Good: "This could cause a race condition when multiple users access simultaneously. Consider using a mutex here." āŒ Bad: "Why didn't you use X pattern?" āœ… Good: "Have you considered the Repository pattern? It would make this easier to test. Here's an example: [link]" āŒ Bad: "Rename this variable." āœ… Good: "[nit] Consider `userCount` instead of `uc` for clarity. Not blocking if you prefer to keep it." ``` ### 3. Review Scope **What to Review:** - Logic correctness and edge cases - Security vulnerabilities - Performance implications - Test coverage and quality - Error handling - Documentation and comments - API design and naming - Architectural fit **What Not to Review Manually:** - Code formatting (use Prettier, Black, etc.) - Import organization - Linting violations - Simple typos ## Review Process ### Phase 1: Context Gathering (2-3 minutes) ```markdown Before diving into code, understand: 1. Read PR description and linked issue 2. Check PR size (>400 lines? Ask to split) 3. Review CI/CD status (tests passing?) 4. Understand the business requirement 5. Note any relevant architectural decisions ``` ### Phase 2: High-Level Review (5-10 minutes) ```markdown 1. **Architecture & Design** - Does the solution fit the problem? - Are there simpler approaches? - Is it consistent with existing patterns? - Will it scale? 2. **File Organization** - Are new files in the right places? - Is code grouped logically? - Are there duplicate files? 3. **Testing Strategy** - Are there tests? - Do tests cover edge cases? - Are tests readable? ``` ### Phase 3: Line-by-Line Review (10-20 minutes) ```markdown For each file: 1. **Logic & Correctness** - Edge cases handled? - Off-by-one errors? - Null/undefined checks? - Race conditions? 2. **Security** - Input validation? - SQL injection risks? - XSS vulnerabilities? - Sensitive data exposure? 3. **Performance** - N+1 queries? - Unnecessary loops? - Memory leaks? - Blocking operations? 4. **Maintainability** - Clear variable names? - Functions doing one thing? - Complex code commented? - Magic numbers extracted? ``` ### Phase 4: Summary & Decision (2-3 minutes) ```markdown 1. Summarize key concerns 2. Highlight what you liked 3. Make clear decision: - āœ… Approve - šŸ’¬ Comment (minor suggestions) - šŸ”„ Request Changes (must address) 4. Offer to pair if complex ``` ## Review Techniques ### Technique 1: The Checklist Method ```markdown ## Security Checklist - [ ] User input validated and sanitized - [ ] SQL queries use parameterization - [ ] Authentication/authorization checked - [ ] Secrets not hardcoded - [ ] Error messages don't leak info ## Performance Checklist - [ ] No N+1 queries - [ ] Database queries indexed - [ ] Large lists paginated - [ ] Expensive operations cached - [ ] No blocking I/O in hot paths ## Testing Checklist - [ ] Happy path tested - [ ] Edge cases covered - [ ] Error cases tested - [ ] Test names are descriptive - [ ] Tests are deterministic ``` ### Technique 2: The Question Approach Instead of stating problems, ask questions to encourage thinking: ```markdown āŒ "This will fail if the list is empty." āœ… "What happens if `items` is an empty array?" āŒ "You need error handling here." āœ… "How should this behave if the API call fails?" āŒ "This is inefficient." āœ… "I see this loops through all users. Have we considered the performance impact with 100k users?" ``` ### Technique 3: Suggest, Don't Command ````markdown ## Use Collaborative Language āŒ "You must change this to use async/await" āœ… "Suggestion: async/await might make this more readable: `typescript async function fetchUser(id: string) { const user = await db.query('SELECT * FROM users WHERE id = ?', id); return user; } ` What do you think?" āŒ "Extract this into a function" āœ… "This logic appears in 3 places. Would it make sense to extract it into a shared utility function?" ```` ### Technique 4: Differentiate Severity ```markdown Use labels to indicate priority: šŸ”“ [blocking] - Must fix before merge 🟔 [important] - Should fix, discuss if disagree 🟢 [nit] - Nice to have, not blocking šŸ’” [suggestion] - Alternative approach to consider šŸ“š [learning] - Educational comment, no action needed šŸŽ‰ [praise] - Good work, keep it up! Example: "šŸ”“ [blocking] This SQL query is vulnerable to injection. Please use parameterized queries." "🟢 [nit] Consider renaming `data` to `userData` for clarity." "šŸŽ‰ [praise] Excellent test coverage! This will catch edge cases." ``` ## Language-Specific Patterns ### Python Code Review ```python # Check for Python-specific issues # āŒ Mutable default arguments def add_item(item, items=[]): # Bug! Shared across calls items.append(item) return items # āœ… Use None as default def add_item(item, items=None): if items is None: items = [] items.append(item) return items # āŒ Catching too broad try: result = risky_operation() except: # Catches everything, even KeyboardInterrupt! pass # āœ… Catch specific exceptions try: result = risky_operation() except ValueError as e: logger.error(f"Invalid value: {e}") raise # āŒ Using mutable class attributes class User: permissions = [] # Shared across all instances! # āœ… Initialize in __init__ class User: def __init__(self): self.permissions = [] ``` ### TypeScript/JavaScript Code Review ```typescript // Check for TypeScript-specific issues // āŒ Using any defeats type safety function processData(data: any) { // Avoid any return data.value; } // āœ… Use proper types interface DataPayload { value: string; } function processData(data: DataPayload) { return data.value; } // āŒ Not handling async errors async function fetchUser(id: string) { const response = await fetch(`/api/users/${id}`); return response.json(); // What if network fails? } // āœ… Handle errors properly async function fetchUser(id: string): Promise<User> { try { const response = await fetch(`/api/users/${id}`); if (!response.ok) { throw new Error(`HTTP ${response.status}`); } return await response.json(); } catch (error) { console.error('Failed to fetch user:', error); throw error; } } // āŒ Mutation of props function UserProfile({ user }: Props) { user.lastViewed = new Date(); // Mutating prop! return <div>{user.name}</div>; } // āœ… Don't mutate props function UserProfile({ user, onView }: Props) { useEffect(() => { onView(user.id); // Notify parent to update }, [user.id]); return <div>{user.name}</div>; } ``` ## Advanced Review Patterns ### Pattern 1: Architectural Review ```markdown When reviewing significant changes: 1. **Design Document First** - For large features, request design doc before code - Review design with team before implementation - Agree on approach to avoid rework 2. **Review in Stages** - First PR: Core abstractions and interfaces - Second PR: Implementation - Third PR: Integration and tests - Easier to review, faster to iterate 3. **Consider Alternatives** - "Have we considered using [pattern/library]?" - "What's the tradeoff vs. the simpler approach?" - "How will this evolve as requirements change?" ``` ### Pattern 2: Test Quality Review ```typescript // āŒ Poor test: Implementation detail testing test('increments counter variable', () => { const component = render(<Counter />); const button = component.getByRole('button'); fireEvent.click(button); expect(component.state.counter).toBe(1); // Testing internal state }); // āœ… Good test: Behavior testing test('displays incremented count when clicked', () => { render(<Counter />); const button = screen.getByRole('button', { name: /increment/i }); fireEvent.click(button); expect(screen.getByText('Count: 1')).toBeInTheDocument(); }); // Review questions for tests: // - Do tests describe behavior, not implementation? // - Are test names clear and descriptive? // - Do tests cover edge cases? // - Are tests independent (no shared state)? // - Can tests run in any order? ``` ### Pattern 3: Security Review ```markdown ## Security Review Checklist ### Authentication & Authorization - [ ] Is authentication required where needed? - [ ] Are authorization checks before every action? - [ ] Is JWT validation proper (signature, expiry)? - [ ] Are API keys/secrets properly secured? ### Input Validation - [ ] All user inputs validated? - [ ] File uploads restricted (size, type)? - [ ] SQL queries parameterized? - [ ] XSS protection (escape output)? ### Data Protection - [ ] Passwords hashed (bcrypt/argon2)? - [ ] Sensitive data encrypted at rest? - [ ] HTTPS enforced for sensitive data? - [ ] PII handled according to regulations? ### Common Vulnerabilities - [ ] No eval() or similar dynamic execution? - [ ] No hardcoded secrets? - [ ] CSRF protection for state-changing operations? - [ ] Rate limiting on public endpoints? ``` ## Giving Difficult Feedback ### Pattern: The Sandwich Method (Modified) ```markdown Traditional: Praise + Criticism + Praise (feels fake) Better: Context + Specific Issue + Helpful Solution Example: "I noticed the payment processing logic is inline in the controller. This makes it harder to test and reuse. [Specific Issue] The calculateTotal() function mixes tax calculation, discount logic, and database queries, making it difficult to unit test and reason about. [Helpful Solution] Could we extract this into a PaymentService class? That would make it testable and reusable. I can pair with you on this if helpful." ``` ### Handling Disagreements ```markdown When author disagrees with your feedback: 1. **Seek to Understand** "Help me understand your approach. What led you to choose this pattern?" 2. **Acknowledge Valid Points** "That's a good point about X. I hadn't considered that." 3. **Provide Data** "I'm concerned about performance. Can we add a benchmark to validate the approach?" 4. **Escalate if Needed** "Let's get [architect/senior dev] to weigh in on this." 5. **Know When to Let Go** If it's working and not a critical issue, approve it. Perfection is the enemy of progress. ``` ## Best Practices 1. **Review Promptly**: Within 24 hours, ideally same day 2. **Limit PR Size**: 200-400 lines max for effective review 3. **Review in Time Blocks**: 60 minutes max, take breaks 4. **Use Review Tools**: GitHub, GitLab, or dedicated tools 5. **Automate What You Can**: Linters, formatters, security scans 6. **Build Rapport**: Emoji, praise, and empathy matter 7. **Be Available**: Offer to pair on complex issues 8. **Learn from Others**: Review others' review comments ## Common Pitfalls - **Perfectionism**: Blocking PRs for minor style preferences - **Scope Creep**: "While you're at it, can you also..." - **Inconsistency**: Different standards for different people - **Delayed Reviews**: Letting PRs sit for days - **Ghosting**: Requesting changes then disappearing - **Rubber Stamping**: Approving without actually reviewing - **Bike Shedding**: Debating trivial details extensively ## Templates ### PR Review Comment Template ```markdown ## Summary [Brief overview of what was reviewed] ## Strengths - [What was done well] - [Good patterns or approaches] ## Required Changes šŸ”“ [Blocking issue 1] šŸ”“ [Blocking issue 2] ## Suggestions šŸ’” [Improvement 1] šŸ’” [Improvement 2] ## Questions ā“ [Clarification needed on X] ā“ [Alternative approach consideration] ## Verdict āœ… Approve after addressing required changes ``` ## Resources - **references/code-review-best-practices.md**: Comprehensive review guidelines - **references/common-bugs-checklist.md**: Language-specific bugs to watch for - **references/security-review-guide.md**: Security-focused review checklist - **assets/pr-review-template.md**: Standard review comment template - **assets/review-checklist.md**: Quick reference checklist - **scripts/pr-analyzer.py**: Analyze PR complexity and suggest reviewers
šŸ‘0
šŸ‘ļø3
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

typescript-advanced-types

Master TypeScript's advanced type system including generics,

coding
⭐1
# TypeScript Advanced Types Comprehensive guidance for mastering TypeScript's advanced type system including generics, conditional types, mapped types, template literal types, and utility types for building robust, type-safe applications. ## When to Use This Skill - Building type-safe libraries or frameworks - Creating reusable generic components - Implementing complex type inference logic - Designing type-safe API clients - Building form validation systems - Creating strongly-typed configuration objects - Implementing type-safe state management - Migrating JavaScript codebases to TypeScript ## Core Concepts ### 1. Generics **Purpose:** Create reusable, type-flexible components while maintaining type safety. **Basic Generic Function:** ```typescript function identity<T>(value: T): T { return value; } const num = identity<number>(42); // Type: number const str = identity<string>("hello"); // Type: string const auto = identity(true); // Type inferred: boolean ``` **Generic Constraints:** ```typescript interface HasLength { length: number; } function logLength<T extends HasLength>(item: T): T { console.log(item.length); return item; } logLength("hello"); // OK: string has length logLength([1, 2, 3]); // OK: array has length logLength({ length: 10 }); // OK: object has length // logLength(42); // Error: number has no length ``` **Multiple Type Parameters:** ```typescript function merge<T, U>(obj1: T, obj2: U): T & U { return { ...obj1, ...obj2 }; } const merged = merge({ name: "John" }, { age: 30 }); // Type: { name: string } & { age: number } ``` ### 2. Conditional Types **Purpose:** Create types that depend on conditions, enabling sophisticated type logic. **Basic Conditional Type:** ```typescript type IsString<T> = T extends string ? true : false; type A = IsString<string>; // true type B = IsString<number>; // false ``` **Extracting Return Types:** ```typescript type ReturnType<T> = T extends (...args: any[]) => infer R ? R : never; function getUser() { return { id: 1, name: "John" }; } type User = ReturnType<typeof getUser>; // Type: { id: number; name: string; } ``` **Distributive Conditional Types:** ```typescript type ToArray<T> = T extends any ? T[] : never; type StrOrNumArray = ToArray<string | number>; // Type: string[] | number[] ``` **Nested Conditions:** ```typescript type TypeName<T> = T extends string ? "string" : T extends number ? "number" : T extends boolean ? "boolean" : T extends undefined ? "undefined" : T extends Function ? "function" : "object"; type T1 = TypeName<string>; // "string" type T2 = TypeName<() => void>; // "function" ``` ### 3. Mapped Types **Purpose:** Transform existing types by iterating over their properties. **Basic Mapped Type:** ```typescript type Readonly<T> = { readonly [P in keyof T]: T[P]; }; interface User { id: number; name: string; } type ReadonlyUser = Readonly<User>; // Type: { readonly id: number; readonly name: string; } ``` **Optional Properties:** ```typescript type Partial<T> = { [P in keyof T]?: T[P]; }; type PartialUser = Partial<User>; // Type: { id?: number; name?: string; } ``` **Key Remapping:** ```typescript type Getters<T> = { [K in keyof T as `get${Capitalize<string & K>}`]: () => T[K]; }; interface Person { name: string; age: number; } type PersonGetters = Getters<Person>; // Type: { getName: () => string; getAge: () => number; } ``` **Filtering Properties:** ```typescript type PickByType<T, U> = { [K in keyof T as T[K] extends U ? K : never]: T[K]; }; interface Mixed { id: number; name: string; age: number; active: boolean; } type OnlyNumbers = PickByType<Mixed, number>; // Type: { id: number; age: number; } ``` ### 4. Template Literal Types **Purpose:** Create string-based types with pattern matching and transformation. **Basic Template Literal:** ```typescript type EventName = "click" | "focus" | "blur"; type EventHandler = `on${Capitalize<EventName>}`; // Type: "onClick" | "onFocus" | "onBlur" ``` **String Manipulation:** ```typescript type UppercaseGreeting = Uppercase<"hello">; // "HELLO" type LowercaseGreeting = Lowercase<"HELLO">; // "hello" type CapitalizedName = Capitalize<"john">; // "John" type UncapitalizedName = Uncapitalize<"John">; // "john" ``` **Path Building:** ```typescript type Path<T> = T extends object ? { [K in keyof T]: K extends string ? `${K}` | `${K}.${Path<T[K]>}` : never; }[keyof T] : never; interface Config { server: { host: string; port: number; }; database: { url: string; }; } type ConfigPath = Path<Config>; // Type: "server" | "database" | "server.host" | "server.port" | "database.url" ``` ### 5. Utility Types **Built-in Utility Types:** ```typescript // Partial<T> - Make all properties optional type PartialUser = Partial<User>; // Required<T> - Make all properties required type RequiredUser = Required<PartialUser>; // Readonly<T> - Make all properties readonly type ReadonlyUser = Readonly<User>; // Pick<T, K> - Select specific properties type UserName = Pick<User, "name" | "email">; // Omit<T, K> - Remove specific properties type UserWithoutPassword = Omit<User, "password">; // Exclude<T, U> - Exclude types from union type T1 = Exclude<"a" | "b" | "c", "a">; // "b" | "c" // Extract<T, U> - Extract types from union type T2 = Extract<"a" | "b" | "c", "a" | "b">; // "a" | "b" // NonNullable<T> - Exclude null and undefined type T3 = NonNullable<string | null | undefined>; // string // Record<K, T> - Create object type with keys K and values T type PageInfo = Record<"home" | "about", { title: string }>; ``` ## Advanced Patterns ### Pattern 1: Type-Safe Event Emitter ```typescript type EventMap = { "user:created": { id: string; name: string }; "user:updated": { id: string }; "user:deleted": { id: string }; }; class TypedEventEmitter<T extends Record<string, any>> { private listeners: { [K in keyof T]?: Array<(data: T[K]) => void>; } = {}; on<K extends keyof T>(event: K, callback: (data: T[K]) => void): void { if (!this.listeners[event]) { this.listeners[event] = []; } this.listeners[event]!.push(callback); } emit<K extends keyof T>(event: K, data: T[K]): void { const callbacks = this.listeners[event]; if (callbacks) { callbacks.forEach((callback) => callback(data)); } } } const emitter = new TypedEventEmitter<EventMap>(); emitter.on("user:created", (data) => { console.log(data.id, data.name); // Type-safe! }); emitter.emit("user:created", { id: "1", name: "John" }); // emitter.emit("user:created", { id: "1" }); // Error: missing 'name' ``` ### Pattern 2: Type-Safe API Client ```typescript type HTTPMethod = "GET" | "POST" | "PUT" | "DELETE"; type EndpointConfig = { "/users": { GET: { response: User[] }; POST: { body: { name: string; email: string }; response: User }; }; "/users/:id": { GET: { params: { id: string }; response: User }; PUT: { params: { id: string }; body: Partial<User>; response: User }; DELETE: { params: { id: string }; response: void }; }; }; type ExtractParams<T> = T extends { params: infer P } ? P : never; type ExtractBody<T> = T extends { body: infer B } ? B : never; type ExtractResponse<T> = T extends { response: infer R } ? R : never; class APIClient<Config extends Record<string, Record<HTTPMethod, any>>> { async request<Path extends keyof Config, Method extends keyof Config[Path]>( path: Path, method: Method, ...[options]: ExtractParams<Config[Path][Method]> extends never ? ExtractBody<Config[Path][Method]> extends never ? [] : [{ body: ExtractBody<Config[Path][Method]> }] : [ { params: ExtractParams<Config[Path][Method]>; body?: ExtractBody<Config[Path][Method]>; }, ] ): Promise<ExtractResponse<Config[Path][Method]>> { // Implementation here return {} as any; } } const api = new APIClient<EndpointConfig>(); // Type-safe API calls const users = await api.request("/users", "GET"); // Type: User[] const newUser = await api.request("/users", "POST", { body: { name: "John", email: "john@example.com" }, }); // Type: User const user = await api.request("/users/:id", "GET", { params: { id: "123" }, }); // Type: User ``` ### Pattern 3: Builder Pattern with Type Safety ```typescript type BuilderState<T> = { [K in keyof T]: T[K] | undefined; }; type RequiredKeys<T> = { [K in keyof T]-?: {} extends Pick<T, K> ? never : K; }[keyof T]; type OptionalKeys<T> = { [K in keyof T]-?: {} extends Pick<T, K> ? K : never; }[keyof T]; type IsComplete<T, S> = RequiredKeys<T> extends keyof S ? S[RequiredKeys<T>] extends undefined ? false : true : false; class Builder<T, S extends BuilderState<T> = {}> { private state: S = {} as S; set<K extends keyof T>(key: K, value: T[K]): Builder<T, S & Record<K, T[K]>> { this.state[key] = value; return this as any; } build(this: IsComplete<T, S> extends true ? this : never): T { return this.state as T; } } interface User { id: string; name: string; email: string; age?: number; } const builder = new Builder<User>(); const user = builder .set("id", "1") .set("name", "John") .set("email", "john@example.com") .build(); // OK: all required fields set // const incomplete = builder // .set("id", "1") // .build(); // Error: missing required fields ``` ### Pattern 4: Deep Readonly/Partial ```typescript type DeepReadonly<T> = { readonly [P in keyof T]: T[P] extends object ? T[P] extends Function ? T[P] : DeepReadonly<T[P]> : T[P]; }; type DeepPartial<T> = { [P in keyof T]?: T[P] extends object ? T[P] extends Array<infer U> ? Array<DeepPartial<U>> : DeepPartial<T[P]> : T[P]; }; interface Config { server: { host: string; port: number; ssl: { enabled: boolean; cert: string; }; }; database: { url: string; pool: { min: number; max: number; }; }; } type ReadonlyConfig = DeepReadonly<Config>; // All nested properties are readonly type PartialConfig = DeepPartial<Config>; // All nested properties are optional ``` ### Pattern 5: Type-Safe Form Validation ```typescript type ValidationRule<T> = { validate: (value: T) => boolean; message: string; }; type FieldValidation<T> = { [K in keyof T]?: ValidationRule<T[K]>[]; }; type ValidationErrors<T> = { [K in keyof T]?: string[]; }; class FormValidator<T extends Record<string, any>> { constructor(private rules: FieldValidation<T>) {} validate(data: T): ValidationErrors<T> | null { const errors: ValidationErrors<T> = {}; let hasErrors = false; for (const key in this.rules) { const fieldRules = this.rules[key]; const value = data[key]; if (fieldRules) { const fieldErrors: string[] = []; for (const rule of fieldRules) { if (!rule.validate(value)) { fieldErrors.push(rule.message); } } if (fieldErrors.length > 0) { errors[key] = fieldErrors; hasErrors = true; } } } return hasErrors ? errors : null; } } interface LoginForm { email: string; password: string; } const validator = new FormValidator<LoginForm>({ email: [ { validate: (v) => v.includes("@"), message: "Email must contain @", }, { validate: (v) => v.length > 0, message: "Email is required", }, ], password: [ { validate: (v) => v.length >= 8, message: "Password must be at least 8 characters", }, ], }); const errors = validator.validate({ email: "invalid", password: "short", }); // Type: { email?: string[]; password?: string[]; } | null ``` ### Pattern 6: Discriminated Unions ```typescript type Success<T> = { status: "success"; data: T; }; type Error = { status: "error"; error: string; }; type Loading = { status: "loading"; }; type AsyncState<T> = Success<T> | Error | Loading; function handleState<T>(state: AsyncState<T>): void { switch (state.status) { case "success": console.log(state.data); // Type: T break; case "error": console.log(state.error); // Type: string break; case "loading": console.log("Loading..."); break; } } // Type-safe state machine type State = | { type: "idle" } | { type: "fetching"; requestId: string } | { type: "success"; data: any } | { type: "error"; error: Error }; type Event = | { type: "FETCH"; requestId: string } | { type: "SUCCESS"; data: any } | { type: "ERROR"; error: Error } | { type: "RESET" }; function reducer(state: State, event: Event): State { switch (state.type) { case "idle": return event.type === "FETCH" ? { type: "fetching", requestId: event.requestId } : state; case "fetching": if (event.type === "SUCCESS") { return { type: "success", data: event.data }; } if (event.type === "ERROR") { return { type: "error", error: event.error }; } return state; case "success": case "error": return event.type === "RESET" ? { type: "idle" } : state; } } ``` ## Type Inference Techniques ### 1. Infer Keyword ```typescript // Extract array element type type ElementType<T> = T extends (infer U)[] ? U : never; type NumArray = number[]; type Num = ElementType<NumArray>; // number // Extract promise type type PromiseType<T> = T extends Promise<infer U> ? U : never; type AsyncNum = PromiseType<Promise<number>>; // number // Extract function parameters type Parameters<T> = T extends (...args: infer P) => any ? P : never; function foo(a: string, b: number) {} type FooParams = Parameters<typeof foo>; // [string, number] ``` ### 2. Type Guards ```typescript function isString(value: unknown): value is string { return typeof value === "string"; } function isArrayOf<T>( value: unknown, guard: (item: unknown) => item is T, ): value is T[] { return Array.isArray(value) && value.every(guard); } const data: unknown = ["a", "b", "c"]; if (isArrayOf(data, isString)) { data.forEach((s) => s.toUpperCase()); // Type: string[] } ``` ### 3. Assertion Functions ```typescript function assertIsString(value: unknown): asserts value is string { if (typeof value !== "string") { throw new Error("Not a string"); } } function processValue(value: unknown) { assertIsString(value); // value is now typed as string console.log(value.toUpperCase()); } ``` ## Best Practices 1. **Use `unknown` over `any`**: Enforce type checking 2. **Prefer `interface` for object shapes**: Better error messages 3. **Use `type` for unions and complex types**: More flexible 4. **Leverage type inference**: Let TypeScript infer when possible 5. **Create helper types**: Build reusable type utilities 6. **Use const assertions**: Preserve literal types 7. **Avoid type assertions**: Use type guards instead 8. **Document complex types**: Add JSDoc comments 9. **Use strict mode**: Enable all strict compiler options 10. **Test your types**: Use type tests to verify type behavior ## Type Testing ```typescript // Type assertion tests type AssertEqual<T, U> = [T] extends [U] ? [U] extends [T] ? true : false : false; type Test1 = AssertEqual<string, string>; // true type Test2 = AssertEqual<string, number>; // false type Test3 = AssertEqual<string | number, string>; // false // Expect error helper type ExpectError<T extends never> = T; // Example usage type ShouldError = ExpectError<AssertEqual<string, number>>; ``` ## Common Pitfalls 1. **Over-using `any`**: Defeats the purpose of TypeScript 2. **Ignoring strict null checks**: Can lead to runtime errors 3. **Too complex types**: Can slow down compilation 4. **Not using discriminated unions**: Misses type narrowing opportunities 5. **Forgetting readonly modifiers**: Allows unintended mutations 6. **Circular type references**: Can cause compiler errors 7. **Not handling edge cases**: Like empty arrays or null values ## Performance Considerations - Avoid deeply nested conditional types - Use simple types when possible - Cache complex type computations - Limit recursion depth in recursive types - Use build tools to skip type checking in production ## Resources - **TypeScript Handbook**: https://www.typescriptlang.org/docs/handbook/ - **Type Challenges**: https://github.com/type-challenges/type-challenges - **TypeScript Deep Dive**: https://basarat.gitbook.io/typescript/ - **Effective TypeScript**: Book by Dan Vanderkam
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered
šŸ¤–system prompt•7 months ago

gitlab-ci-patterns

Build GitLab CI/CD pipelines with multi-stage workflows, caching,

coding
⭐1
# GitLab CI Patterns Comprehensive GitLab CI/CD pipeline patterns for automated testing, building, and deployment. ## Purpose Create efficient GitLab CI pipelines with proper stage organization, caching, and deployment strategies. ## When to Use - Automate GitLab-based CI/CD - Implement multi-stage pipelines - Configure GitLab Runners - Deploy to Kubernetes from GitLab - Implement GitOps workflows ## Basic Pipeline Structure ```yaml stages: - build - test - deploy variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" build: stage: build image: node:20 script: - npm ci - npm run build artifacts: paths: - dist/ expire_in: 1 hour cache: key: ${CI_COMMIT_REF_SLUG} paths: - node_modules/ test: stage: test image: node:20 script: - npm ci - npm run lint - npm test coverage: '/Lines\s*:\s*(\d+\.\d+)%/' artifacts: reports: coverage_report: coverage_format: cobertura path: coverage/cobertura-coverage.xml deploy: stage: deploy image: bitnami/kubectl:latest script: - kubectl apply -f k8s/ - kubectl rollout status deployment/my-app only: - main environment: name: production url: https://app.example.com ``` ## Docker Build and Push ```yaml build-docker: stage: build image: docker:24 services: - docker:24-dind before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY script: - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA . - docker build -t $CI_REGISTRY_IMAGE:latest . - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - docker push $CI_REGISTRY_IMAGE:latest only: - main - tags ``` ## Multi-Environment Deployment ```yaml .deploy_template: &deploy_template image: bitnami/kubectl:latest before_script: - kubectl config set-cluster k8s --server="$KUBE_URL" --insecure-skip-tls-verify=true - kubectl config set-credentials admin --token="$KUBE_TOKEN" - kubectl config set-context default --cluster=k8s --user=admin - kubectl config use-context default deploy:staging: <<: *deploy_template stage: deploy script: - kubectl apply -f k8s/ -n staging - kubectl rollout status deployment/my-app -n staging environment: name: staging url: https://staging.example.com only: - develop deploy:production: <<: *deploy_template stage: deploy script: - kubectl apply -f k8s/ -n production - kubectl rollout status deployment/my-app -n production environment: name: production url: https://app.example.com when: manual only: - main ``` ## Terraform Pipeline ```yaml stages: - validate - plan - apply variables: TF_ROOT: ${CI_PROJECT_DIR}/terraform TF_VERSION: "1.6.0" before_script: - cd ${TF_ROOT} - terraform --version validate: stage: validate image: hashicorp/terraform:${TF_VERSION} script: - terraform init -backend=false - terraform validate - terraform fmt -check plan: stage: plan image: hashicorp/terraform:${TF_VERSION} script: - terraform init - terraform plan -out=tfplan artifacts: paths: - ${TF_ROOT}/tfplan expire_in: 1 day apply: stage: apply image: hashicorp/terraform:${TF_VERSION} script: - terraform init - terraform apply -auto-approve tfplan dependencies: - plan when: manual only: - main ``` ## Security Scanning ```yaml include: - template: Security/SAST.gitlab-ci.yml - template: Security/Dependency-Scanning.gitlab-ci.yml - template: Security/Container-Scanning.gitlab-ci.yml trivy-scan: stage: test image: aquasec/trivy:latest script: - trivy image --exit-code 1 --severity HIGH,CRITICAL $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA allow_failure: true ``` ## Caching Strategies ```yaml # Cache node_modules build: cache: key: ${CI_COMMIT_REF_SLUG} paths: - node_modules/ policy: pull-push # Global cache cache: key: ${CI_COMMIT_REF_SLUG} paths: - .cache/ - vendor/ # Separate cache per job job1: cache: key: job1-cache paths: - build/ job2: cache: key: job2-cache paths: - dist/ ``` ## Dynamic Child Pipelines ```yaml generate-pipeline: stage: build script: - python generate_pipeline.py > child-pipeline.yml artifacts: paths: - child-pipeline.yml trigger-child: stage: deploy trigger: include: - artifact: child-pipeline.yml job: generate-pipeline strategy: depend ``` ## Reference Files - `assets/gitlab-ci.yml.template` - Complete pipeline template - `references/pipeline-stages.md` - Stage organization patterns ## Best Practices 1. **Use specific image tags** (node:20, not node:latest) 2. **Cache dependencies** appropriately 3. **Use artifacts** for build outputs 4. **Implement manual gates** for production 5. **Use environments** for deployment tracking 6. **Enable merge request pipelines** 7. **Use pipeline schedules** for recurring jobs 8. **Implement security scanning** 9. **Use CI/CD variables** for secrets 10. **Monitor pipeline performance** ## Related Skills - `github-actions-templates` - For GitHub Actions - `deployment-pipeline-design` - For architecture - `secrets-management` - For secrets handling
šŸ‘0
šŸ‘ļø0
šŸ¤– Auto-discovered