Skip to main content
EVOKORE// BROWSE
>

./browse/prompts

16 NODES
🤖system prompt•7 months ago

solidity-security

Master smart contract security best practices to prevent common

security
⭐1
# Solidity Security Master smart contract security best practices, vulnerability prevention, and secure Solidity development patterns. ## When to Use This Skill - Writing secure smart contracts - Auditing existing contracts for vulnerabilities - Implementing secure DeFi protocols - Preventing reentrancy, overflow, and access control issues - Optimizing gas usage while maintaining security - Preparing contracts for professional audits - Understanding common attack vectors ## Critical Vulnerabilities ### 1. Reentrancy Attacker calls back into your contract before state is updated. **Vulnerable Code:** ```solidity // VULNERABLE TO REENTRANCY contract VulnerableBank { mapping(address => uint256) public balances; function withdraw() public { uint256 amount = balances[msg.sender]; // DANGER: External call before state update (bool success, ) = msg.sender.call{value: amount}(""); require(success); balances[msg.sender] = 0; // Too late! } } ``` **Secure Pattern (Checks-Effects-Interactions):** ```solidity contract SecureBank { mapping(address => uint256) public balances; function withdraw() public { uint256 amount = balances[msg.sender]; require(amount > 0, "Insufficient balance"); // EFFECTS: Update state BEFORE external call balances[msg.sender] = 0; // INTERACTIONS: External call last (bool success, ) = msg.sender.call{value: amount}(""); require(success, "Transfer failed"); } } ``` **Alternative: ReentrancyGuard** ```solidity import "@openzeppelin/contracts/security/ReentrancyGuard.sol"; contract SecureBank is ReentrancyGuard { mapping(address => uint256) public balances; function withdraw() public nonReentrant { uint256 amount = balances[msg.sender]; require(amount > 0, "Insufficient balance"); balances[msg.sender] = 0; (bool success, ) = msg.sender.call{value: amount}(""); require(success, "Transfer failed"); } } ``` ### 2. Integer Overflow/Underflow **Vulnerable Code (Solidity < 0.8.0):** ```solidity // VULNERABLE contract VulnerableToken { mapping(address => uint256) public balances; function transfer(address to, uint256 amount) public { // No overflow check - can wrap around balances[msg.sender] -= amount; // Can underflow! balances[to] += amount; // Can overflow! } } ``` **Secure Pattern (Solidity >= 0.8.0):** ```solidity // Solidity 0.8+ has built-in overflow/underflow checks contract SecureToken { mapping(address => uint256) public balances; function transfer(address to, uint256 amount) public { // Automatically reverts on overflow/underflow balances[msg.sender] -= amount; balances[to] += amount; } } ``` **For Solidity < 0.8.0, use SafeMath:** ```solidity import "@openzeppelin/contracts/utils/math/SafeMath.sol"; contract SecureToken { using SafeMath for uint256; mapping(address => uint256) public balances; function transfer(address to, uint256 amount) public { balances[msg.sender] = balances[msg.sender].sub(amount); balances[to] = balances[to].add(amount); } } ``` ### 3. Access Control **Vulnerable Code:** ```solidity // VULNERABLE: Anyone can call critical functions contract VulnerableContract { address public owner; function withdraw(uint256 amount) public { // No access control! payable(msg.sender).transfer(amount); } } ``` **Secure Pattern:** ```solidity import "@openzeppelin/contracts/access/Ownable.sol"; contract SecureContract is Ownable { function withdraw(uint256 amount) public onlyOwner { payable(owner()).transfer(amount); } } // Or implement custom role-based access contract RoleBasedContract { mapping(address => bool) public admins; modifier onlyAdmin() { require(admins[msg.sender], "Not an admin"); _; } function criticalFunction() public onlyAdmin { // Protected function } } ``` ### 4. Front-Running **Vulnerable:** ```solidity // VULNERABLE TO FRONT-RUNNING contract VulnerableDEX { function swap(uint256 amount, uint256 minOutput) public { // Attacker sees this in mempool and front-runs uint256 output = calculateOutput(amount); require(output >= minOutput, "Slippage too high"); // Perform swap } } ``` **Mitigation:** ```solidity contract SecureDEX { mapping(bytes32 => bool) public usedCommitments; // Step 1: Commit to trade function commitTrade(bytes32 commitment) public { usedCommitments[commitment] = true; } // Step 2: Reveal trade (next block) function revealTrade( uint256 amount, uint256 minOutput, bytes32 secret ) public { bytes32 commitment = keccak256(abi.encodePacked( msg.sender, amount, minOutput, secret )); require(usedCommitments[commitment], "Invalid commitment"); // Perform swap } } ``` ## Security Best Practices ### Checks-Effects-Interactions Pattern ```solidity contract SecurePattern { mapping(address => uint256) public balances; function withdraw(uint256 amount) public { // 1. CHECKS: Validate conditions require(amount <= balances[msg.sender], "Insufficient balance"); require(amount > 0, "Amount must be positive"); // 2. EFFECTS: Update state balances[msg.sender] -= amount; // 3. INTERACTIONS: External calls last (bool success, ) = msg.sender.call{value: amount}(""); require(success, "Transfer failed"); } } ``` ### Pull Over Push Pattern ```solidity // Prefer this (pull) contract SecurePayment { mapping(address => uint256) public pendingWithdrawals; function recordPayment(address recipient, uint256 amount) internal { pendingWithdrawals[recipient] += amount; } function withdraw() public { uint256 amount = pendingWithdrawals[msg.sender]; require(amount > 0, "Nothing to withdraw"); pendingWithdrawals[msg.sender] = 0; payable(msg.sender).transfer(amount); } } // Over this (push) contract RiskyPayment { function distributePayments(address[] memory recipients, uint256[] memory amounts) public { for (uint i = 0; i < recipients.length; i++) { // If any transfer fails, entire batch fails payable(recipients[i]).transfer(amounts[i]); } } } ``` ### Input Validation ```solidity contract SecureContract { function transfer(address to, uint256 amount) public { // Validate inputs require(to != address(0), "Invalid recipient"); require(to != address(this), "Cannot send to contract"); require(amount > 0, "Amount must be positive"); require(amount <= balances[msg.sender], "Insufficient balance"); // Proceed with transfer balances[msg.sender] -= amount; balances[to] += amount; } } ``` ### Emergency Stop (Circuit Breaker) ```solidity import "@openzeppelin/contracts/security/Pausable.sol"; contract EmergencyStop is Pausable, Ownable { function criticalFunction() public whenNotPaused { // Function logic } function emergencyStop() public onlyOwner { _pause(); } function resume() public onlyOwner { _unpause(); } } ``` ## Gas Optimization ### Use `uint256` Instead of Smaller Types ```solidity // More gas efficient contract GasEfficient { uint256 public value; // Optimal function set(uint256 _value) public { value = _value; } } // Less efficient contract GasInefficient { uint8 public value; // Still uses 256-bit slot function set(uint8 _value) public { value = _value; // Extra gas for type conversion } } ``` ### Pack Storage Variables ```solidity // Gas efficient (3 variables in 1 slot) contract PackedStorage { uint128 public a; // Slot 0 uint64 public b; // Slot 0 uint64 public c; // Slot 0 uint256 public d; // Slot 1 } // Gas inefficient (each variable in separate slot) contract UnpackedStorage { uint256 public a; // Slot 0 uint256 public b; // Slot 1 uint256 public c; // Slot 2 uint256 public d; // Slot 3 } ``` ### Use `calldata` Instead of `memory` for Function Arguments ```solidity contract GasOptimized { // More gas efficient function processData(uint256[] calldata data) public pure returns (uint256) { return data[0]; } // Less efficient function processDataMemory(uint256[] memory data) public pure returns (uint256) { return data[0]; } } ``` ### Use Events for Data Storage (When Appropriate) ```solidity contract EventStorage { // Emitting events is cheaper than storage event DataStored(address indexed user, uint256 indexed id, bytes data); function storeData(uint256 id, bytes calldata data) public { emit DataStored(msg.sender, id, data); // Don't store in contract storage unless needed } } ``` ## Common Vulnerabilities Checklist ```solidity // Security Checklist Contract contract SecurityChecklist { /** * [ ] Reentrancy protection (ReentrancyGuard or CEI pattern) * [ ] Integer overflow/underflow (Solidity 0.8+ or SafeMath) * [ ] Access control (Ownable, roles, modifiers) * [ ] Input validation (require statements) * [ ] Front-running mitigation (commit-reveal if applicable) * [ ] Gas optimization (packed storage, calldata) * [ ] Emergency stop mechanism (Pausable) * [ ] Pull over push pattern for payments * [ ] No delegatecall to untrusted contracts * [ ] No tx.origin for authentication (use msg.sender) * [ ] Proper event emission * [ ] External calls at end of function * [ ] Check return values of external calls * [ ] No hardcoded addresses * [ ] Upgrade mechanism (if proxy pattern) */ } ``` ## Testing for Security ```javascript // Hardhat test example const { expect } = require("chai"); const { ethers } = require("hardhat"); describe("Security Tests", function () { it("Should prevent reentrancy attack", async function () { const [attacker] = await ethers.getSigners(); const VictimBank = await ethers.getContractFactory("SecureBank"); const bank = await VictimBank.deploy(); const Attacker = await ethers.getContractFactory("ReentrancyAttacker"); const attackerContract = await Attacker.deploy(bank.address); // Deposit funds await bank.deposit({ value: ethers.utils.parseEther("10") }); // Attempt reentrancy attack await expect( attackerContract.attack({ value: ethers.utils.parseEther("1") }), ).to.be.revertedWith("ReentrancyGuard: reentrant call"); }); it("Should prevent integer overflow", async function () { const Token = await ethers.getContractFactory("SecureToken"); const token = await Token.deploy(); // Attempt overflow await expect(token.transfer(attacker.address, ethers.constants.MaxUint256)) .to.be.reverted; }); it("Should enforce access control", async function () { const [owner, attacker] = await ethers.getSigners(); const Contract = await ethers.getContractFactory("SecureContract"); const contract = await Contract.deploy(); // Attempt unauthorized withdrawal await expect(contract.connect(attacker).withdraw(100)).to.be.revertedWith( "Ownable: caller is not the owner", ); }); }); ``` ## Audit Preparation ```solidity contract WellDocumentedContract { /** * @title Well Documented Contract * @dev Example of proper documentation for audits * @notice This contract handles user deposits and withdrawals */ /// @notice Mapping of user balances mapping(address => uint256) public balances; /** * @dev Deposits ETH into the contract * @notice Anyone can deposit funds */ function deposit() public payable { require(msg.value > 0, "Must send ETH"); balances[msg.sender] += msg.value; } /** * @dev Withdraws user's balance * @notice Follows CEI pattern to prevent reentrancy * @param amount Amount to withdraw in wei */ function withdraw(uint256 amount) public { // CHECKS require(amount <= balances[msg.sender], "Insufficient balance"); // EFFECTS balances[msg.sender] -= amount; // INTERACTIONS (bool success, ) = msg.sender.call{value: amount}(""); require(success, "Transfer failed"); } } ``` ## Resources - **references/reentrancy.md**: Comprehensive reentrancy prevention - **references/access-control.md**: Role-based access patterns - **references/overflow-underflow.md**: SafeMath and integer safety - **references/gas-optimization.md**: Gas saving techniques - **references/vulnerability-patterns.md**: Common vulnerability catalog - **assets/solidity-contracts-templates.sol**: Secure contract templates - **assets/security-checklist.md**: Pre-audit checklist - **scripts/analyze-contract.sh**: Static analysis tools ## Tools for Security Analysis - **Slither**: Static analysis tool - **Mythril**: Security analysis tool - **Echidna**: Fuzzing tool - **Manticore**: Symbolic execution - **Securify**: Automated security scanner ## Common Pitfalls 1. **Using `tx.origin` for Authentication**: Use `msg.sender` instead 2. **Unchecked External Calls**: Always check return values 3. **Delegatecall to Untrusted Contracts**: Can hijack your contract 4. **Floating Pragma**: Pin to specific Solidity version 5. **Missing Events**: Emit events for state changes 6. **Excessive Gas in Loops**: Can hit block gas limit 7. **No Upgrade Path**: Consider proxy patterns if upgrades needed
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

data-storytelling

Transform data into compelling narratives using visualization,

data
⭐1
# Data Storytelling Transform raw data into compelling narratives that drive decisions and inspire action. ## When to Use This Skill - Presenting analytics to executives - Creating quarterly business reviews - Building investor presentations - Writing data-driven reports - Communicating insights to non-technical audiences - Making recommendations based on data ## Core Concepts ### 1. Story Structure ``` Setup → Conflict → Resolution Setup: Context and baseline Conflict: The problem or opportunity Resolution: Insights and recommendations ``` ### 2. Narrative Arc ``` 1. Hook: Grab attention with surprising insight 2. Context: Establish the baseline 3. Rising Action: Build through data points 4. Climax: The key insight 5. Resolution: Recommendations 6. Call to Action: Next steps ``` ### 3. Three Pillars | Pillar | Purpose | Components | | ------------- | -------- | -------------------------------- | | **Data** | Evidence | Numbers, trends, comparisons | | **Narrative** | Meaning | Context, causation, implications | | **Visuals** | Clarity | Charts, diagrams, highlights | ## Story Frameworks ### Framework 1: The Problem-Solution Story ```markdown # Customer Churn Analysis ## The Hook "We're losing $2.4M annually to preventable churn." ## The Context - Current churn rate: 8.5% (industry average: 5%) - Average customer lifetime value: $4,800 - 500 customers churned last quarter ## The Problem Analysis of churned customers reveals a pattern: - 73% churned within first 90 days - Common factor: < 3 support interactions - Low feature adoption in first month ## The Insight [Show engagement curve visualization] Customers who don't engage in the first 14 days are 4x more likely to churn. ## The Solution 1. Implement 14-day onboarding sequence 2. Proactive outreach at day 7 3. Feature adoption tracking ## Expected Impact - Reduce early churn by 40% - Save $960K annually - Payback period: 3 months ## Call to Action Approve $50K budget for onboarding automation. ``` ### Framework 2: The Trend Story ```markdown # Q4 Performance Analysis ## Where We Started Q3 ended with $1.2M MRR, 15% below target. Team morale was low after missed goals. ## What Changed [Timeline visualization] - Oct: Launched self-serve pricing - Nov: Reduced friction in signup - Dec: Added customer success calls ## The Transformation [Before/after comparison chart] | Metric | Q3 | Q4 | Change | |----------------|--------|--------|--------| | Trial → Paid | 8% | 15% | +87% | | Time to Value | 14 days| 5 days | -64% | | Expansion Rate | 2% | 8% | +300% | ## Key Insight Self-serve + high-touch creates compound growth. Customers who self-serve AND get a success call have 3x higher expansion rate. ## Going Forward Double down on hybrid model. Target: $1.8M MRR by Q2. ``` ### Framework 3: The Comparison Story ```markdown # Market Opportunity Analysis ## The Question Should we expand into EMEA or APAC first? ## The Comparison [Side-by-side market analysis] ### EMEA - Market size: $4.2B - Growth rate: 8% - Competition: High - Regulatory: Complex (GDPR) - Language: Multiple ### APAC - Market size: $3.8B - Growth rate: 15% - Competition: Moderate - Regulatory: Varied - Language: Multiple ## The Analysis [Weighted scoring matrix visualization] | Factor | Weight | EMEA Score | APAC Score | | ----------- | ------ | ---------- | ---------- | | Market Size | 25% | 5 | 4 | | Growth | 30% | 3 | 5 | | Competition | 20% | 2 | 4 | | Ease | 25% | 2 | 3 | | **Total** | | **2.9** | **4.1** | ## The Recommendation APAC first. Higher growth, less competition. Start with Singapore hub (English, business-friendly). Enter EMEA in Year 2 with localization ready. ## Risk Mitigation - Timezone coverage: Hire 24/7 support - Cultural fit: Local partnerships - Payment: Multi-currency from day 1 ``` ## Visualization Techniques ### Technique 1: Progressive Reveal ```markdown Start simple, add layers: Slide 1: "Revenue is growing" [single line chart] Slide 2: "But growth is slowing" [add growth rate overlay] Slide 3: "Driven by one segment" [add segment breakdown] Slide 4: "Which is saturating" [add market share] Slide 5: "We need new segments" [add opportunity zones] ``` ### Technique 2: Contrast and Compare ```markdown Before/After: ┌─────────────────┬─────────────────┐ │ BEFORE │ AFTER │ │ │ │ │ Process: 5 days│ Process: 1 day │ │ Errors: 15% │ Errors: 2% │ │ Cost: $50/unit │ Cost: $20/unit │ └─────────────────┴─────────────────┘ This/That (emphasize difference): ┌─────────────────────────────────────┐ │ CUSTOMER A vs B │ │ ┌──────────┐ ┌──────────┐ │ │ │ ████████ │ │ ██ │ │ │ │ $45,000 │ │ $8,000 │ │ │ │ LTV │ │ LTV │ │ │ └──────────┘ └──────────┘ │ │ Onboarded No onboarding │ └─────────────────────────────────────┘ ``` ### Technique 3: Annotation and Highlight ```python import matplotlib.pyplot as plt import pandas as pd fig, ax = plt.subplots(figsize=(12, 6)) # Plot the main data ax.plot(dates, revenue, linewidth=2, color='#2E86AB') # Add annotation for key events ax.annotate( 'Product Launch\n+32% spike', xy=(launch_date, launch_revenue), xytext=(launch_date, launch_revenue * 1.2), fontsize=10, arrowprops=dict(arrowstyle='->', color='#E63946'), color='#E63946' ) # Highlight a region ax.axvspan(growth_start, growth_end, alpha=0.2, color='green', label='Growth Period') # Add threshold line ax.axhline(y=target, color='gray', linestyle='--', label=f'Target: ${target:,.0f}') ax.set_title('Revenue Growth Story', fontsize=14, fontweight='bold') ax.legend() ``` ## Presentation Templates ### Template 1: Executive Summary Slide ``` ┌─────────────────────────────────────────────────────────────┐ │ KEY INSIGHT │ │ ══════════════════════════════════════════════════════════│ │ │ │ "Customers who complete onboarding in week 1 │ │ have 3x higher lifetime value" │ │ │ ├──────────────────────┬──────────────────────────────────────┤ │ │ │ │ THE DATA │ THE IMPLICATION │ │ │ │ │ Week 1 completers: │ ✓ Prioritize onboarding UX │ │ • LTV: $4,500 │ ✓ Add day-1 success milestones │ │ • Retention: 85% │ ✓ Proactive week-1 outreach │ │ • NPS: 72 │ │ │ │ Investment: $75K │ │ Others: │ Expected ROI: 8x │ │ • LTV: $1,500 │ │ │ • Retention: 45% │ │ │ • NPS: 34 │ │ │ │ │ └──────────────────────┴──────────────────────────────────────┘ ``` ### Template 2: Data Story Flow ``` Slide 1: THE HEADLINE "We can grow 40% faster by fixing onboarding" Slide 2: THE CONTEXT Current state metrics Industry benchmarks Gap analysis Slide 3: THE DISCOVERY What the data revealed Surprising finding Pattern identification Slide 4: THE DEEP DIVE Root cause analysis Segment breakdowns Statistical significance Slide 5: THE RECOMMENDATION Proposed actions Resource requirements Timeline Slide 6: THE IMPACT Expected outcomes ROI calculation Risk assessment Slide 7: THE ASK Specific request Decision needed Next steps ``` ### Template 3: One-Page Dashboard Story ```markdown # Monthly Business Review: January 2024 ## THE HEADLINE Revenue up 15% but CAC increasing faster than LTV ## KEY METRICS AT A GLANCE ┌────────┬────────┬────────┬────────┐ │ MRR │ NRR │ CAC │ LTV │ │ $125K │ 108% │ $450 │ $2,200 │ │ ▲15% │ ▲3% │ ▲22% │ ▲8% │ └────────┴────────┴────────┴────────┘ ## WHAT'S WORKING ✓ Enterprise segment growing 25% MoM ✓ Referral program driving 30% of new logos ✓ Support satisfaction at all-time high (94%) ## WHAT NEEDS ATTENTION ✗ SMB acquisition cost up 40% ✗ Trial conversion down 5 points ✗ Time-to-value increased by 3 days ## ROOT CAUSE [Mini chart showing SMB vs Enterprise CAC trend] SMB paid ads becoming less efficient. CPC up 35% while conversion flat. ## RECOMMENDATION 1. Shift $20K/mo from paid to content 2. Launch SMB self-serve trial 3. A/B test shorter onboarding ## NEXT MONTH'S FOCUS - Launch content marketing pilot - Complete self-serve MVP - Reduce time-to-value to < 7 days ``` ## Writing Techniques ### Headlines That Work ```markdown BAD: "Q4 Sales Analysis" GOOD: "Q4 Sales Beat Target by 23% - Here's Why" BAD: "Customer Churn Report" GOOD: "We're Losing $2.4M to Preventable Churn" BAD: "Marketing Performance" GOOD: "Content Marketing Delivers 4x ROI vs. Paid" Formula: [Specific Number] + [Business Impact] + [Actionable Context] ``` ### Transition Phrases ```markdown Building the narrative: • "This leads us to ask..." • "When we dig deeper..." • "The pattern becomes clear when..." • "Contrast this with..." Introducing insights: • "The data reveals..." • "What surprised us was..." • "The inflection point came when..." • "The key finding is..." Moving to action: • "This insight suggests..." • "Based on this analysis..." • "The implication is clear..." • "Our recommendation is..." ``` ### Handling Uncertainty ```markdown Acknowledge limitations: • "With 95% confidence, we can say..." • "The sample size of 500 shows..." • "While correlation is strong, causation requires..." • "This trend holds for [segment], though [caveat]..." Present ranges: • "Impact estimate: $400K-$600K" • "Confidence interval: 15-20% improvement" • "Best case: X, Conservative: Y" ``` ## Best Practices ### Do's - **Start with the "so what"** - Lead with insight - **Use the rule of three** - Three points, three comparisons - **Show, don't tell** - Let data speak - **Make it personal** - Connect to audience goals - **End with action** - Clear next steps ### Don'ts - **Don't data dump** - Curate ruthlessly - **Don't bury the insight** - Front-load key findings - **Don't use jargon** - Match audience vocabulary - **Don't show methodology first** - Context, then method - **Don't forget the narrative** - Numbers need meaning ## Resources - [Storytelling with Data (Cole Nussbaumer)](https://www.storytellingwithdata.com/) - [The Pyramid Principle (Barbara Minto)](https://www.amazon.com/Pyramid-Principle-Logic-Writing-Thinking/dp/0273710516) - [Resonate (Nancy Duarte)](https://www.duarte.com/resonate/)
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

debugging-strategies

Master systematic debugging techniques, profiling tools, and root

coding
⭐1
# Debugging Strategies Transform debugging from frustrating guesswork into systematic problem-solving with proven strategies, powerful tools, and methodical approaches. ## When to Use This Skill - Tracking down elusive bugs - Investigating performance issues - Understanding unfamiliar codebases - Debugging production issues - Analyzing crash dumps and stack traces - Profiling application performance - Investigating memory leaks - Debugging distributed systems ## Core Principles ### 1. The Scientific Method **1. Observe**: What's the actual behavior? **2. Hypothesize**: What could be causing it? **3. Experiment**: Test your hypothesis **4. Analyze**: Did it prove/disprove your theory? **5. Repeat**: Until you find the root cause ### 2. Debugging Mindset **Don't Assume:** - "It can't be X" - Yes it can - "I didn't change Y" - Check anyway - "It works on my machine" - Find out why **Do:** - Reproduce consistently - Isolate the problem - Keep detailed notes - Question everything - Take breaks when stuck ### 3. Rubber Duck Debugging Explain your code and problem out loud (to a rubber duck, colleague, or yourself). Often reveals the issue. ## Systematic Debugging Process ### Phase 1: Reproduce ```markdown ## Reproduction Checklist 1. **Can you reproduce it?** - Always? Sometimes? Randomly? - Specific conditions needed? - Can others reproduce it? 2. **Create minimal reproduction** - Simplify to smallest example - Remove unrelated code - Isolate the problem 3. **Document steps** - Write down exact steps - Note environment details - Capture error messages ``` ### Phase 2: Gather Information ```markdown ## Information Collection 1. **Error Messages** - Full stack trace - Error codes - Console/log output 2. **Environment** - OS version - Language/runtime version - Dependencies versions - Environment variables 3. **Recent Changes** - Git history - Deployment timeline - Configuration changes 4. **Scope** - Affects all users or specific ones? - All browsers or specific ones? - Production only or also dev? ``` ### Phase 3: Form Hypothesis ```markdown ## Hypothesis Formation Based on gathered info, ask: 1. **What changed?** - Recent code changes - Dependency updates - Infrastructure changes 2. **What's different?** - Working vs broken environment - Working vs broken user - Before vs after 3. **Where could this fail?** - Input validation - Business logic - Data layer - External services ``` ### Phase 4: Test & Verify ```markdown ## Testing Strategies 1. **Binary Search** - Comment out half the code - Narrow down problematic section - Repeat until found 2. **Add Logging** - Strategic console.log/print - Track variable values - Trace execution flow 3. **Isolate Components** - Test each piece separately - Mock dependencies - Remove complexity 4. **Compare Working vs Broken** - Diff configurations - Diff environments - Diff data ``` ## Debugging Tools ### JavaScript/TypeScript Debugging ```typescript // Chrome DevTools Debugger function processOrder(order: Order) { debugger; // Execution pauses here const total = calculateTotal(order); console.log("Total:", total); // Conditional breakpoint if (order.items.length > 10) { debugger; // Only breaks if condition true } return total; } // Console debugging techniques console.log("Value:", value); // Basic console.table(arrayOfObjects); // Table format console.time("operation"); /* code */ console.timeEnd("operation"); // Timing console.trace(); // Stack trace console.assert(value > 0, "Value must be positive"); // Assertion // Performance profiling performance.mark("start-operation"); // ... operation code performance.mark("end-operation"); performance.measure("operation", "start-operation", "end-operation"); console.log(performance.getEntriesByType("measure")); ``` **VS Code Debugger Configuration:** ```json // .vscode/launch.json { "version": "0.2.0", "configurations": [ { "type": "node", "request": "launch", "name": "Debug Program", "program": "${workspaceFolder}/src/index.ts", "preLaunchTask": "tsc: build - tsconfig.json", "outFiles": ["${workspaceFolder}/dist/**/*.js"], "skipFiles": ["<node_internals>/**"] }, { "type": "node", "request": "launch", "name": "Debug Tests", "program": "${workspaceFolder}/node_modules/jest/bin/jest", "args": ["--runInBand", "--no-cache"], "console": "integratedTerminal" } ] } ``` ### Python Debugging ```python # Built-in debugger (pdb) import pdb def calculate_total(items): total = 0 pdb.set_trace() # Debugger starts here for item in items: total += item.price * item.quantity return total # Breakpoint (Python 3.7+) def process_order(order): breakpoint() # More convenient than pdb.set_trace() # ... code # Post-mortem debugging try: risky_operation() except Exception: import pdb pdb.post_mortem() # Debug at exception point # IPython debugging (ipdb) from ipdb import set_trace set_trace() # Better interface than pdb # Logging for debugging import logging logging.basicConfig(level=logging.DEBUG) logger = logging.getLogger(__name__) def fetch_user(user_id): logger.debug(f'Fetching user: {user_id}') user = db.query(User).get(user_id) logger.debug(f'Found user: {user}') return user # Profile performance import cProfile import pstats cProfile.run('slow_function()', 'profile_stats') stats = pstats.Stats('profile_stats') stats.sort_stats('cumulative') stats.print_stats(10) # Top 10 slowest ``` ### Go Debugging ```go // Delve debugger // Install: go install github.com/go-delve/delve/cmd/dlv@latest // Run: dlv debug main.go import ( "fmt" "runtime" "runtime/debug" ) // Print stack trace func debugStack() { debug.PrintStack() } // Panic recovery with debugging func processRequest() { defer func() { if r := recover(); r != nil { fmt.Println("Panic:", r) debug.PrintStack() } }() // ... code that might panic } // Memory profiling import _ "net/http/pprof" // Visit http://localhost:6060/debug/pprof/ // CPU profiling import ( "os" "runtime/pprof" ) f, _ := os.Create("cpu.prof") pprof.StartCPUProfile(f) defer pprof.StopCPUProfile() // ... code to profile ``` ## Advanced Debugging Techniques ### Technique 1: Binary Search Debugging ```bash # Git bisect for finding regression git bisect start git bisect bad # Current commit is bad git bisect good v1.0.0 # v1.0.0 was good # Git checks out middle commit # Test it, then: git bisect good # if it works git bisect bad # if it's broken # Continue until bug found git bisect reset # when done ``` ### Technique 2: Differential Debugging Compare working vs broken: ```markdown ## What's Different? | Aspect | Working | Broken | | ------------ | ----------- | -------------- | | Environment | Development | Production | | Node version | 18.16.0 | 18.15.0 | | Data | Empty DB | 1M records | | User | Admin | Regular user | | Browser | Chrome | Safari | | Time | During day | After midnight | Hypothesis: Time-based issue? Check timezone handling. ``` ### Technique 3: Trace Debugging ```typescript // Function call tracing function trace( target: any, propertyKey: string, descriptor: PropertyDescriptor, ) { const originalMethod = descriptor.value; descriptor.value = function (...args: any[]) { console.log(`Calling ${propertyKey} with args:`, args); const result = originalMethod.apply(this, args); console.log(`${propertyKey} returned:`, result); return result; }; return descriptor; } class OrderService { @trace calculateTotal(items: Item[]): number { return items.reduce((sum, item) => sum + item.price, 0); } } ``` ### Technique 4: Memory Leak Detection ```typescript // Chrome DevTools Memory Profiler // 1. Take heap snapshot // 2. Perform action // 3. Take another snapshot // 4. Compare snapshots // Node.js memory debugging if (process.memoryUsage().heapUsed > 500 * 1024 * 1024) { console.warn("High memory usage:", process.memoryUsage()); // Generate heap dump require("v8").writeHeapSnapshot(); } // Find memory leaks in tests let beforeMemory: number; beforeEach(() => { beforeMemory = process.memoryUsage().heapUsed; }); afterEach(() => { const afterMemory = process.memoryUsage().heapUsed; const diff = afterMemory - beforeMemory; if (diff > 10 * 1024 * 1024) { // 10MB threshold console.warn(`Possible memory leak: ${diff / 1024 / 1024}MB`); } }); ``` ## Debugging Patterns by Issue Type ### Pattern 1: Intermittent Bugs ```markdown ## Strategies for Flaky Bugs 1. **Add extensive logging** - Log timing information - Log all state transitions - Log external interactions 2. **Look for race conditions** - Concurrent access to shared state - Async operations completing out of order - Missing synchronization 3. **Check timing dependencies** - setTimeout/setInterval - Promise resolution order - Animation frame timing 4. **Stress test** - Run many times - Vary timing - Simulate load ``` ### Pattern 2: Performance Issues ```markdown ## Performance Debugging 1. **Profile first** - Don't optimize blindly - Measure before and after - Find bottlenecks 2. **Common culprits** - N+1 queries - Unnecessary re-renders - Large data processing - Synchronous I/O 3. **Tools** - Browser DevTools Performance tab - Lighthouse - Python: cProfile, line_profiler - Node: clinic.js, 0x ``` ### Pattern 3: Production Bugs ```markdown ## Production Debugging 1. **Gather evidence** - Error tracking (Sentry, Bugsnag) - Application logs - User reports - Metrics/monitoring 2. **Reproduce locally** - Use production data (anonymized) - Match environment - Follow exact steps 3. **Safe investigation** - Don't change production - Use feature flags - Add monitoring/logging - Test fixes in staging ``` ## Best Practices 1. **Reproduce First**: Can't fix what you can't reproduce 2. **Isolate the Problem**: Remove complexity until minimal case 3. **Read Error Messages**: They're usually helpful 4. **Check Recent Changes**: Most bugs are recent 5. **Use Version Control**: Git bisect, blame, history 6. **Take Breaks**: Fresh eyes see better 7. **Document Findings**: Help future you 8. **Fix Root Cause**: Not just symptoms ## Common Debugging Mistakes - **Making Multiple Changes**: Change one thing at a time - **Not Reading Error Messages**: Read the full stack trace - **Assuming It's Complex**: Often it's simple - **Debug Logging in Prod**: Remove before shipping - **Not Using Debugger**: console.log isn't always best - **Giving Up Too Soon**: Persistence pays off - **Not Testing the Fix**: Verify it actually works ## Quick Debugging Checklist ```markdown ## When Stuck, Check: - [ ] Spelling errors (typos in variable names) - [ ] Case sensitivity (fileName vs filename) - [ ] Null/undefined values - [ ] Array index off-by-one - [ ] Async timing (race conditions) - [ ] Scope issues (closure, hoisting) - [ ] Type mismatches - [ ] Missing dependencies - [ ] Environment variables - [ ] File paths (absolute vs relative) - [ ] Cache issues (clear cache) - [ ] Stale data (refresh database) ``` ## Resources - **references/debugging-tools-guide.md**: Comprehensive tool documentation - **references/performance-profiling.md**: Performance debugging guide - **references/production-debugging.md**: Debugging live systems - **assets/debugging-checklist.md**: Quick reference checklist - **assets/common-bugs.md**: Common bug patterns - **scripts/debug-helper.ts**: Debugging utility functions
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

git-advanced-workflows

Master advanced Git workflows including rebasing, cherry-picking,

coding
⭐1
# Git Advanced Workflows Master advanced Git techniques to maintain clean history, collaborate effectively, and recover from any situation with confidence. ## When to Use This Skill - Cleaning up commit history before merging - Applying specific commits across branches - Finding commits that introduced bugs - Working on multiple features simultaneously - Recovering from Git mistakes or lost commits - Managing complex branch workflows - Preparing clean PRs for review - Synchronizing diverged branches ## Core Concepts ### 1. Interactive Rebase Interactive rebase is the Swiss Army knife of Git history editing. **Common Operations:** - `pick`: Keep commit as-is - `reword`: Change commit message - `edit`: Amend commit content - `squash`: Combine with previous commit - `fixup`: Like squash but discard message - `drop`: Remove commit entirely **Basic Usage:** ```bash # Rebase last 5 commits git rebase -i HEAD~5 # Rebase all commits on current branch git rebase -i $(git merge-base HEAD main) # Rebase onto specific commit git rebase -i abc123 ``` ### 2. Cherry-Picking Apply specific commits from one branch to another without merging entire branches. ```bash # Cherry-pick single commit git cherry-pick abc123 # Cherry-pick range of commits (exclusive start) git cherry-pick abc123..def456 # Cherry-pick without committing (stage changes only) git cherry-pick -n abc123 # Cherry-pick and edit commit message git cherry-pick -e abc123 ``` ### 3. Git Bisect Binary search through commit history to find the commit that introduced a bug. ```bash # Start bisect git bisect start # Mark current commit as bad git bisect bad # Mark known good commit git bisect good v1.0.0 # Git will checkout middle commit - test it # Then mark as good or bad git bisect good # or: git bisect bad # Continue until bug found # When done git bisect reset ``` **Automated Bisect:** ```bash # Use script to test automatically git bisect start HEAD v1.0.0 git bisect run ./test.sh # test.sh should exit 0 for good, 1-127 (except 125) for bad ``` ### 4. Worktrees Work on multiple branches simultaneously without stashing or switching. ```bash # List existing worktrees git worktree list # Add new worktree for feature branch git worktree add ../project-feature feature/new-feature # Add worktree and create new branch git worktree add -b bugfix/urgent ../project-hotfix main # Remove worktree git worktree remove ../project-feature # Prune stale worktrees git worktree prune ``` ### 5. Reflog Your safety net - tracks all ref movements, even deleted commits. ```bash # View reflog git reflog # View reflog for specific branch git reflog show feature/branch # Restore deleted commit git reflog # Find commit hash git checkout abc123 git branch recovered-branch # Restore deleted branch git reflog git branch deleted-branch abc123 ``` ## Practical Workflows ### Workflow 1: Clean Up Feature Branch Before PR ```bash # Start with feature branch git checkout feature/user-auth # Interactive rebase to clean history git rebase -i main # Example rebase operations: # - Squash "fix typo" commits # - Reword commit messages for clarity # - Reorder commits logically # - Drop unnecessary commits # Force push cleaned branch (safe if no one else is using it) git push --force-with-lease origin feature/user-auth ``` ### Workflow 2: Apply Hotfix to Multiple Releases ```bash # Create fix on main git checkout main git commit -m "fix: critical security patch" # Apply to release branches git checkout release/2.0 git cherry-pick abc123 git checkout release/1.9 git cherry-pick abc123 # Handle conflicts if they arise git cherry-pick --continue # or git cherry-pick --abort ``` ### Workflow 3: Find Bug Introduction ```bash # Start bisect git bisect start git bisect bad HEAD git bisect good v2.1.0 # Git checks out middle commit - run tests npm test # If tests fail git bisect bad # If tests pass git bisect good # Git will automatically checkout next commit to test # Repeat until bug found # Automated version git bisect start HEAD v2.1.0 git bisect run npm test ``` ### Workflow 4: Multi-Branch Development ```bash # Main project directory cd ~/projects/myapp # Create worktree for urgent bugfix git worktree add ../myapp-hotfix hotfix/critical-bug # Work on hotfix in separate directory cd ../myapp-hotfix # Make changes, commit git commit -m "fix: resolve critical bug" git push origin hotfix/critical-bug # Return to main work without interruption cd ~/projects/myapp git fetch origin git cherry-pick hotfix/critical-bug # Clean up when done git worktree remove ../myapp-hotfix ``` ### Workflow 5: Recover from Mistakes ```bash # Accidentally reset to wrong commit git reset --hard HEAD~5 # Oh no! # Use reflog to find lost commits git reflog # Output shows: # abc123 HEAD@{0}: reset: moving to HEAD~5 # def456 HEAD@{1}: commit: my important changes # Recover lost commits git reset --hard def456 # Or create branch from lost commit git branch recovery def456 ``` ## Advanced Techniques ### Rebase vs Merge Strategy **When to Rebase:** - Cleaning up local commits before pushing - Keeping feature branch up-to-date with main - Creating linear history for easier review **When to Merge:** - Integrating completed features into main - Preserving exact history of collaboration - Public branches used by others ```bash # Update feature branch with main changes (rebase) git checkout feature/my-feature git fetch origin git rebase origin/main # Handle conflicts git status # Fix conflicts in files git add . git rebase --continue # Or merge instead git merge origin/main ``` ### Autosquash Workflow Automatically squash fixup commits during rebase. ```bash # Make initial commit git commit -m "feat: add user authentication" # Later, fix something in that commit # Stage changes git commit --fixup HEAD # or specify commit hash # Make more changes git commit --fixup abc123 # Rebase with autosquash git rebase -i --autosquash main # Git automatically marks fixup commits ``` ### Split Commit Break one commit into multiple logical commits. ```bash # Start interactive rebase git rebase -i HEAD~3 # Mark commit to split with 'edit' # Git will stop at that commit # Reset commit but keep changes git reset HEAD^ # Stage and commit in logical chunks git add file1.py git commit -m "feat: add validation" git add file2.py git commit -m "feat: add error handling" # Continue rebase git rebase --continue ``` ### Partial Cherry-Pick Cherry-pick only specific files from a commit. ```bash # Show files in commit git show --name-only abc123 # Checkout specific files from commit git checkout abc123 -- path/to/file1.py path/to/file2.py # Stage and commit git commit -m "cherry-pick: apply specific changes from abc123" ``` ## Best Practices 1. **Always Use --force-with-lease**: Safer than --force, prevents overwriting others' work 2. **Rebase Only Local Commits**: Don't rebase commits that have been pushed and shared 3. **Descriptive Commit Messages**: Future you will thank present you 4. **Atomic Commits**: Each commit should be a single logical change 5. **Test Before Force Push**: Ensure history rewrite didn't break anything 6. **Keep Reflog Aware**: Remember reflog is your safety net for 90 days 7. **Branch Before Risky Operations**: Create backup branch before complex rebases ```bash # Safe force push git push --force-with-lease origin feature/branch # Create backup before risky operation git branch backup-branch git rebase -i main # If something goes wrong git reset --hard backup-branch ``` ## Common Pitfalls - **Rebasing Public Branches**: Causes history conflicts for collaborators - **Force Pushing Without Lease**: Can overwrite teammate's work - **Losing Work in Rebase**: Resolve conflicts carefully, test after rebase - **Forgetting Worktree Cleanup**: Orphaned worktrees consume disk space - **Not Backing Up Before Experiment**: Always create safety branch - **Bisect on Dirty Working Directory**: Commit or stash before bisecting ## Recovery Commands ```bash # Abort operations in progress git rebase --abort git merge --abort git cherry-pick --abort git bisect reset # Restore file to version from specific commit git restore --source=abc123 path/to/file # Undo last commit but keep changes git reset --soft HEAD^ # Undo last commit and discard changes git reset --hard HEAD^ # Recover deleted branch (within 90 days) git reflog git branch recovered-branch abc123 ``` ## Resources - **references/git-rebase-guide.md**: Deep dive into interactive rebase - **references/git-conflict-resolution.md**: Advanced conflict resolution strategies - **references/git-history-rewriting.md**: Safely rewriting Git history - **assets/git-workflow-checklist.md**: Pre-PR cleanup checklist - **assets/git-aliases.md**: Useful Git aliases for advanced workflows - **scripts/git-clean-branches.sh**: Clean up merged and stale branches
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

sql-optimization-patterns

Master SQL query optimization, indexing strategies, and EXPLAIN

coding
⭐1
# SQL Optimization Patterns Transform slow database queries into lightning-fast operations through systematic optimization, proper indexing, and query plan analysis. ## When to Use This Skill - Debugging slow-running queries - Designing performant database schemas - Optimizing application response times - Reducing database load and costs - Improving scalability for growing datasets - Analyzing EXPLAIN query plans - Implementing efficient indexes - Resolving N+1 query problems ## Core Concepts ### 1. Query Execution Plans (EXPLAIN) Understanding EXPLAIN output is fundamental to optimization. **PostgreSQL EXPLAIN:** ```sql -- Basic explain EXPLAIN SELECT * FROM users WHERE email = 'user@example.com'; -- With actual execution stats EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'user@example.com'; -- Verbose output with more details EXPLAIN (ANALYZE, BUFFERS, VERBOSE) SELECT u.*, o.order_total FROM users u JOIN orders o ON u.id = o.user_id WHERE u.created_at > NOW() - INTERVAL '30 days'; ``` **Key Metrics to Watch:** - **Seq Scan**: Full table scan (usually slow for large tables) - **Index Scan**: Using index (good) - **Index Only Scan**: Using index without touching table (best) - **Nested Loop**: Join method (okay for small datasets) - **Hash Join**: Join method (good for larger datasets) - **Merge Join**: Join method (good for sorted data) - **Cost**: Estimated query cost (lower is better) - **Rows**: Estimated rows returned - **Actual Time**: Real execution time ### 2. Index Strategies Indexes are the most powerful optimization tool. **Index Types:** - **B-Tree**: Default, good for equality and range queries - **Hash**: Only for equality (=) comparisons - **GIN**: Full-text search, array queries, JSONB - **GiST**: Geometric data, full-text search - **BRIN**: Block Range INdex for very large tables with correlation ```sql -- Standard B-Tree index CREATE INDEX idx_users_email ON users(email); -- Composite index (order matters!) CREATE INDEX idx_orders_user_status ON orders(user_id, status); -- Partial index (index subset of rows) CREATE INDEX idx_active_users ON users(email) WHERE status = 'active'; -- Expression index CREATE INDEX idx_users_lower_email ON users(LOWER(email)); -- Covering index (include additional columns) CREATE INDEX idx_users_email_covering ON users(email) INCLUDE (name, created_at); -- Full-text search index CREATE INDEX idx_posts_search ON posts USING GIN(to_tsvector('english', title || ' ' || body)); -- JSONB index CREATE INDEX idx_metadata ON events USING GIN(metadata); ``` ### 3. Query Optimization Patterns **Avoid SELECT \*:** ```sql -- Bad: Fetches unnecessary columns SELECT * FROM users WHERE id = 123; -- Good: Fetch only what you need SELECT id, email, name FROM users WHERE id = 123; ``` **Use WHERE Clause Efficiently:** ```sql -- Bad: Function prevents index usage SELECT * FROM users WHERE LOWER(email) = 'user@example.com'; -- Good: Create functional index or use exact match CREATE INDEX idx_users_email_lower ON users(LOWER(email)); -- Then: SELECT * FROM users WHERE LOWER(email) = 'user@example.com'; -- Or store normalized data SELECT * FROM users WHERE email = 'user@example.com'; ``` **Optimize JOINs:** ```sql -- Bad: Cartesian product then filter SELECT u.name, o.total FROM users u, orders o WHERE u.id = o.user_id AND u.created_at > '2024-01-01'; -- Good: Filter before join SELECT u.name, o.total FROM users u JOIN orders o ON u.id = o.user_id WHERE u.created_at > '2024-01-01'; -- Better: Filter both tables SELECT u.name, o.total FROM (SELECT * FROM users WHERE created_at > '2024-01-01') u JOIN orders o ON u.id = o.user_id; ``` ## Optimization Patterns ### Pattern 1: Eliminate N+1 Queries **Problem: N+1 Query Anti-Pattern** ```python # Bad: Executes N+1 queries users = db.query("SELECT * FROM users LIMIT 10") for user in users: orders = db.query("SELECT * FROM orders WHERE user_id = ?", user.id) # Process orders ``` **Solution: Use JOINs or Batch Loading** ```sql -- Solution 1: JOIN SELECT u.id, u.name, o.id as order_id, o.total FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.id IN (1, 2, 3, 4, 5); -- Solution 2: Batch query SELECT * FROM orders WHERE user_id IN (1, 2, 3, 4, 5); ``` ```python # Good: Single query with JOIN or batch load # Using JOIN results = db.query(""" SELECT u.id, u.name, o.id as order_id, o.total FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.id IN (1, 2, 3, 4, 5) """) # Or batch load users = db.query("SELECT * FROM users LIMIT 10") user_ids = [u.id for u in users] orders = db.query( "SELECT * FROM orders WHERE user_id IN (?)", user_ids ) # Group orders by user_id orders_by_user = {} for order in orders: orders_by_user.setdefault(order.user_id, []).append(order) ``` ### Pattern 2: Optimize Pagination **Bad: OFFSET on Large Tables** ```sql -- Slow for large offsets SELECT * FROM users ORDER BY created_at DESC LIMIT 20 OFFSET 100000; -- Very slow! ``` **Good: Cursor-Based Pagination** ```sql -- Much faster: Use cursor (last seen ID) SELECT * FROM users WHERE created_at < '2024-01-15 10:30:00' -- Last cursor ORDER BY created_at DESC LIMIT 20; -- With composite sorting SELECT * FROM users WHERE (created_at, id) < ('2024-01-15 10:30:00', 12345) ORDER BY created_at DESC, id DESC LIMIT 20; -- Requires index CREATE INDEX idx_users_cursor ON users(created_at DESC, id DESC); ``` ### Pattern 3: Aggregate Efficiently **Optimize COUNT Queries:** ```sql -- Bad: Counts all rows SELECT COUNT(*) FROM orders; -- Slow on large tables -- Good: Use estimates for approximate counts SELECT reltuples::bigint AS estimate FROM pg_class WHERE relname = 'orders'; -- Good: Filter before counting SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '7 days'; -- Better: Use index-only scan CREATE INDEX idx_orders_created ON orders(created_at); SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '7 days'; ``` **Optimize GROUP BY:** ```sql -- Bad: Group by then filter SELECT user_id, COUNT(*) as order_count FROM orders GROUP BY user_id HAVING COUNT(*) > 10; -- Better: Filter first, then group (if possible) SELECT user_id, COUNT(*) as order_count FROM orders WHERE status = 'completed' GROUP BY user_id HAVING COUNT(*) > 10; -- Best: Use covering index CREATE INDEX idx_orders_user_status ON orders(user_id, status); ``` ### Pattern 4: Subquery Optimization **Transform Correlated Subqueries:** ```sql -- Bad: Correlated subquery (runs for each row) SELECT u.name, u.email, (SELECT COUNT(*) FROM orders o WHERE o.user_id = u.id) as order_count FROM users u; -- Good: JOIN with aggregation SELECT u.name, u.email, COUNT(o.id) as order_count FROM users u LEFT JOIN orders o ON o.user_id = u.id GROUP BY u.id, u.name, u.email; -- Better: Use window functions SELECT DISTINCT ON (u.id) u.name, u.email, COUNT(o.id) OVER (PARTITION BY u.id) as order_count FROM users u LEFT JOIN orders o ON o.user_id = u.id; ``` **Use CTEs for Clarity:** ```sql -- Using Common Table Expressions WITH recent_users AS ( SELECT id, name, email FROM users WHERE created_at > NOW() - INTERVAL '30 days' ), user_order_counts AS ( SELECT user_id, COUNT(*) as order_count FROM orders WHERE created_at > NOW() - INTERVAL '30 days' GROUP BY user_id ) SELECT ru.name, ru.email, COALESCE(uoc.order_count, 0) as orders FROM recent_users ru LEFT JOIN user_order_counts uoc ON ru.id = uoc.user_id; ``` ### Pattern 5: Batch Operations **Batch INSERT:** ```sql -- Bad: Multiple individual inserts INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com'); INSERT INTO users (name, email) VALUES ('Bob', 'bob@example.com'); INSERT INTO users (name, email) VALUES ('Carol', 'carol@example.com'); -- Good: Batch insert INSERT INTO users (name, email) VALUES ('Alice', 'alice@example.com'), ('Bob', 'bob@example.com'), ('Carol', 'carol@example.com'); -- Better: Use COPY for bulk inserts (PostgreSQL) COPY users (name, email) FROM '/tmp/users.csv' CSV HEADER; ``` **Batch UPDATE:** ```sql -- Bad: Update in loop UPDATE users SET status = 'active' WHERE id = 1; UPDATE users SET status = 'active' WHERE id = 2; -- ... repeat for many IDs -- Good: Single UPDATE with IN clause UPDATE users SET status = 'active' WHERE id IN (1, 2, 3, 4, 5, ...); -- Better: Use temporary table for large batches CREATE TEMP TABLE temp_user_updates (id INT, new_status VARCHAR); INSERT INTO temp_user_updates VALUES (1, 'active'), (2, 'active'), ...; UPDATE users u SET status = t.new_status FROM temp_user_updates t WHERE u.id = t.id; ``` ## Advanced Techniques ### Materialized Views Pre-compute expensive queries. ```sql -- Create materialized view CREATE MATERIALIZED VIEW user_order_summary AS SELECT u.id, u.name, COUNT(o.id) as total_orders, SUM(o.total) as total_spent, MAX(o.created_at) as last_order_date FROM users u LEFT JOIN orders o ON u.id = o.user_id GROUP BY u.id, u.name; -- Add index to materialized view CREATE INDEX idx_user_summary_spent ON user_order_summary(total_spent DESC); -- Refresh materialized view REFRESH MATERIALIZED VIEW user_order_summary; -- Concurrent refresh (PostgreSQL) REFRESH MATERIALIZED VIEW CONCURRENTLY user_order_summary; -- Query materialized view (very fast) SELECT * FROM user_order_summary WHERE total_spent > 1000 ORDER BY total_spent DESC; ``` ### Partitioning Split large tables for better performance. ```sql -- Range partitioning by date (PostgreSQL) CREATE TABLE orders ( id SERIAL, user_id INT, total DECIMAL, created_at TIMESTAMP ) PARTITION BY RANGE (created_at); -- Create partitions CREATE TABLE orders_2024_q1 PARTITION OF orders FOR VALUES FROM ('2024-01-01') TO ('2024-04-01'); CREATE TABLE orders_2024_q2 PARTITION OF orders FOR VALUES FROM ('2024-04-01') TO ('2024-07-01'); -- Queries automatically use appropriate partition SELECT * FROM orders WHERE created_at BETWEEN '2024-02-01' AND '2024-02-28'; -- Only scans orders_2024_q1 partition ``` ### Query Hints and Optimization ```sql -- Force index usage (MySQL) SELECT * FROM users USE INDEX (idx_users_email) WHERE email = 'user@example.com'; -- Parallel query (PostgreSQL) SET max_parallel_workers_per_gather = 4; SELECT * FROM large_table WHERE condition; -- Join hints (PostgreSQL) SET enable_nestloop = OFF; -- Force hash or merge join ``` ## Best Practices 1. **Index Selectively**: Too many indexes slow down writes 2. **Monitor Query Performance**: Use slow query logs 3. **Keep Statistics Updated**: Run ANALYZE regularly 4. **Use Appropriate Data Types**: Smaller types = better performance 5. **Normalize Thoughtfully**: Balance normalization vs performance 6. **Cache Frequently Accessed Data**: Use application-level caching 7. **Connection Pooling**: Reuse database connections 8. **Regular Maintenance**: VACUUM, ANALYZE, rebuild indexes ```sql -- Update statistics ANALYZE users; ANALYZE VERBOSE orders; -- Vacuum (PostgreSQL) VACUUM ANALYZE users; VACUUM FULL users; -- Reclaim space (locks table) -- Reindex REINDEX INDEX idx_users_email; REINDEX TABLE users; ``` ## Common Pitfalls - **Over-Indexing**: Each index slows down INSERT/UPDATE/DELETE - **Unused Indexes**: Waste space and slow writes - **Missing Indexes**: Slow queries, full table scans - **Implicit Type Conversion**: Prevents index usage - **OR Conditions**: Can't use indexes efficiently - **LIKE with Leading Wildcard**: `LIKE '%abc'` can't use index - **Function in WHERE**: Prevents index usage unless functional index exists ## Monitoring Queries ```sql -- Find slow queries (PostgreSQL) SELECT query, calls, total_time, mean_time FROM pg_stat_statements ORDER BY mean_time DESC LIMIT 10; -- Find missing indexes (PostgreSQL) SELECT schemaname, tablename, seq_scan, seq_tup_read, idx_scan, seq_tup_read / seq_scan AS avg_seq_tup_read FROM pg_stat_user_tables WHERE seq_scan > 0 ORDER BY seq_tup_read DESC LIMIT 10; -- Find unused indexes (PostgreSQL) SELECT schemaname, tablename, indexname, idx_scan, idx_tup_read, idx_tup_fetch FROM pg_stat_user_indexes WHERE idx_scan = 0 ORDER BY pg_relation_size(indexrelid) DESC; ``` ## Resources - **references/postgres-optimization-guide.md**: PostgreSQL-specific optimization - **references/mysql-optimization-guide.md**: MySQL/MariaDB optimization - **references/query-plan-analysis.md**: Deep dive into EXPLAIN plans - **assets/index-strategy-checklist.md**: When and how to create indexes - **assets/query-optimization-checklist.md**: Step-by-step optimization guide - **scripts/analyze-slow-queries.sql**: Identify slow queries in your database - **scripts/index-recommendations.sql**: Generate index recommendations
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

typescript-advanced-types

Master TypeScript's advanced type system including generics,

coding
⭐1
# TypeScript Advanced Types Comprehensive guidance for mastering TypeScript's advanced type system including generics, conditional types, mapped types, template literal types, and utility types for building robust, type-safe applications. ## When to Use This Skill - Building type-safe libraries or frameworks - Creating reusable generic components - Implementing complex type inference logic - Designing type-safe API clients - Building form validation systems - Creating strongly-typed configuration objects - Implementing type-safe state management - Migrating JavaScript codebases to TypeScript ## Core Concepts ### 1. Generics **Purpose:** Create reusable, type-flexible components while maintaining type safety. **Basic Generic Function:** ```typescript function identity<T>(value: T): T { return value; } const num = identity<number>(42); // Type: number const str = identity<string>("hello"); // Type: string const auto = identity(true); // Type inferred: boolean ``` **Generic Constraints:** ```typescript interface HasLength { length: number; } function logLength<T extends HasLength>(item: T): T { console.log(item.length); return item; } logLength("hello"); // OK: string has length logLength([1, 2, 3]); // OK: array has length logLength({ length: 10 }); // OK: object has length // logLength(42); // Error: number has no length ``` **Multiple Type Parameters:** ```typescript function merge<T, U>(obj1: T, obj2: U): T & U { return { ...obj1, ...obj2 }; } const merged = merge({ name: "John" }, { age: 30 }); // Type: { name: string } & { age: number } ``` ### 2. Conditional Types **Purpose:** Create types that depend on conditions, enabling sophisticated type logic. **Basic Conditional Type:** ```typescript type IsString<T> = T extends string ? true : false; type A = IsString<string>; // true type B = IsString<number>; // false ``` **Extracting Return Types:** ```typescript type ReturnType<T> = T extends (...args: any[]) => infer R ? R : never; function getUser() { return { id: 1, name: "John" }; } type User = ReturnType<typeof getUser>; // Type: { id: number; name: string; } ``` **Distributive Conditional Types:** ```typescript type ToArray<T> = T extends any ? T[] : never; type StrOrNumArray = ToArray<string | number>; // Type: string[] | number[] ``` **Nested Conditions:** ```typescript type TypeName<T> = T extends string ? "string" : T extends number ? "number" : T extends boolean ? "boolean" : T extends undefined ? "undefined" : T extends Function ? "function" : "object"; type T1 = TypeName<string>; // "string" type T2 = TypeName<() => void>; // "function" ``` ### 3. Mapped Types **Purpose:** Transform existing types by iterating over their properties. **Basic Mapped Type:** ```typescript type Readonly<T> = { readonly [P in keyof T]: T[P]; }; interface User { id: number; name: string; } type ReadonlyUser = Readonly<User>; // Type: { readonly id: number; readonly name: string; } ``` **Optional Properties:** ```typescript type Partial<T> = { [P in keyof T]?: T[P]; }; type PartialUser = Partial<User>; // Type: { id?: number; name?: string; } ``` **Key Remapping:** ```typescript type Getters<T> = { [K in keyof T as `get${Capitalize<string & K>}`]: () => T[K]; }; interface Person { name: string; age: number; } type PersonGetters = Getters<Person>; // Type: { getName: () => string; getAge: () => number; } ``` **Filtering Properties:** ```typescript type PickByType<T, U> = { [K in keyof T as T[K] extends U ? K : never]: T[K]; }; interface Mixed { id: number; name: string; age: number; active: boolean; } type OnlyNumbers = PickByType<Mixed, number>; // Type: { id: number; age: number; } ``` ### 4. Template Literal Types **Purpose:** Create string-based types with pattern matching and transformation. **Basic Template Literal:** ```typescript type EventName = "click" | "focus" | "blur"; type EventHandler = `on${Capitalize<EventName>}`; // Type: "onClick" | "onFocus" | "onBlur" ``` **String Manipulation:** ```typescript type UppercaseGreeting = Uppercase<"hello">; // "HELLO" type LowercaseGreeting = Lowercase<"HELLO">; // "hello" type CapitalizedName = Capitalize<"john">; // "John" type UncapitalizedName = Uncapitalize<"John">; // "john" ``` **Path Building:** ```typescript type Path<T> = T extends object ? { [K in keyof T]: K extends string ? `${K}` | `${K}.${Path<T[K]>}` : never; }[keyof T] : never; interface Config { server: { host: string; port: number; }; database: { url: string; }; } type ConfigPath = Path<Config>; // Type: "server" | "database" | "server.host" | "server.port" | "database.url" ``` ### 5. Utility Types **Built-in Utility Types:** ```typescript // Partial<T> - Make all properties optional type PartialUser = Partial<User>; // Required<T> - Make all properties required type RequiredUser = Required<PartialUser>; // Readonly<T> - Make all properties readonly type ReadonlyUser = Readonly<User>; // Pick<T, K> - Select specific properties type UserName = Pick<User, "name" | "email">; // Omit<T, K> - Remove specific properties type UserWithoutPassword = Omit<User, "password">; // Exclude<T, U> - Exclude types from union type T1 = Exclude<"a" | "b" | "c", "a">; // "b" | "c" // Extract<T, U> - Extract types from union type T2 = Extract<"a" | "b" | "c", "a" | "b">; // "a" | "b" // NonNullable<T> - Exclude null and undefined type T3 = NonNullable<string | null | undefined>; // string // Record<K, T> - Create object type with keys K and values T type PageInfo = Record<"home" | "about", { title: string }>; ``` ## Advanced Patterns ### Pattern 1: Type-Safe Event Emitter ```typescript type EventMap = { "user:created": { id: string; name: string }; "user:updated": { id: string }; "user:deleted": { id: string }; }; class TypedEventEmitter<T extends Record<string, any>> { private listeners: { [K in keyof T]?: Array<(data: T[K]) => void>; } = {}; on<K extends keyof T>(event: K, callback: (data: T[K]) => void): void { if (!this.listeners[event]) { this.listeners[event] = []; } this.listeners[event]!.push(callback); } emit<K extends keyof T>(event: K, data: T[K]): void { const callbacks = this.listeners[event]; if (callbacks) { callbacks.forEach((callback) => callback(data)); } } } const emitter = new TypedEventEmitter<EventMap>(); emitter.on("user:created", (data) => { console.log(data.id, data.name); // Type-safe! }); emitter.emit("user:created", { id: "1", name: "John" }); // emitter.emit("user:created", { id: "1" }); // Error: missing 'name' ``` ### Pattern 2: Type-Safe API Client ```typescript type HTTPMethod = "GET" | "POST" | "PUT" | "DELETE"; type EndpointConfig = { "/users": { GET: { response: User[] }; POST: { body: { name: string; email: string }; response: User }; }; "/users/:id": { GET: { params: { id: string }; response: User }; PUT: { params: { id: string }; body: Partial<User>; response: User }; DELETE: { params: { id: string }; response: void }; }; }; type ExtractParams<T> = T extends { params: infer P } ? P : never; type ExtractBody<T> = T extends { body: infer B } ? B : never; type ExtractResponse<T> = T extends { response: infer R } ? R : never; class APIClient<Config extends Record<string, Record<HTTPMethod, any>>> { async request<Path extends keyof Config, Method extends keyof Config[Path]>( path: Path, method: Method, ...[options]: ExtractParams<Config[Path][Method]> extends never ? ExtractBody<Config[Path][Method]> extends never ? [] : [{ body: ExtractBody<Config[Path][Method]> }] : [ { params: ExtractParams<Config[Path][Method]>; body?: ExtractBody<Config[Path][Method]>; }, ] ): Promise<ExtractResponse<Config[Path][Method]>> { // Implementation here return {} as any; } } const api = new APIClient<EndpointConfig>(); // Type-safe API calls const users = await api.request("/users", "GET"); // Type: User[] const newUser = await api.request("/users", "POST", { body: { name: "John", email: "john@example.com" }, }); // Type: User const user = await api.request("/users/:id", "GET", { params: { id: "123" }, }); // Type: User ``` ### Pattern 3: Builder Pattern with Type Safety ```typescript type BuilderState<T> = { [K in keyof T]: T[K] | undefined; }; type RequiredKeys<T> = { [K in keyof T]-?: {} extends Pick<T, K> ? never : K; }[keyof T]; type OptionalKeys<T> = { [K in keyof T]-?: {} extends Pick<T, K> ? K : never; }[keyof T]; type IsComplete<T, S> = RequiredKeys<T> extends keyof S ? S[RequiredKeys<T>] extends undefined ? false : true : false; class Builder<T, S extends BuilderState<T> = {}> { private state: S = {} as S; set<K extends keyof T>(key: K, value: T[K]): Builder<T, S & Record<K, T[K]>> { this.state[key] = value; return this as any; } build(this: IsComplete<T, S> extends true ? this : never): T { return this.state as T; } } interface User { id: string; name: string; email: string; age?: number; } const builder = new Builder<User>(); const user = builder .set("id", "1") .set("name", "John") .set("email", "john@example.com") .build(); // OK: all required fields set // const incomplete = builder // .set("id", "1") // .build(); // Error: missing required fields ``` ### Pattern 4: Deep Readonly/Partial ```typescript type DeepReadonly<T> = { readonly [P in keyof T]: T[P] extends object ? T[P] extends Function ? T[P] : DeepReadonly<T[P]> : T[P]; }; type DeepPartial<T> = { [P in keyof T]?: T[P] extends object ? T[P] extends Array<infer U> ? Array<DeepPartial<U>> : DeepPartial<T[P]> : T[P]; }; interface Config { server: { host: string; port: number; ssl: { enabled: boolean; cert: string; }; }; database: { url: string; pool: { min: number; max: number; }; }; } type ReadonlyConfig = DeepReadonly<Config>; // All nested properties are readonly type PartialConfig = DeepPartial<Config>; // All nested properties are optional ``` ### Pattern 5: Type-Safe Form Validation ```typescript type ValidationRule<T> = { validate: (value: T) => boolean; message: string; }; type FieldValidation<T> = { [K in keyof T]?: ValidationRule<T[K]>[]; }; type ValidationErrors<T> = { [K in keyof T]?: string[]; }; class FormValidator<T extends Record<string, any>> { constructor(private rules: FieldValidation<T>) {} validate(data: T): ValidationErrors<T> | null { const errors: ValidationErrors<T> = {}; let hasErrors = false; for (const key in this.rules) { const fieldRules = this.rules[key]; const value = data[key]; if (fieldRules) { const fieldErrors: string[] = []; for (const rule of fieldRules) { if (!rule.validate(value)) { fieldErrors.push(rule.message); } } if (fieldErrors.length > 0) { errors[key] = fieldErrors; hasErrors = true; } } } return hasErrors ? errors : null; } } interface LoginForm { email: string; password: string; } const validator = new FormValidator<LoginForm>({ email: [ { validate: (v) => v.includes("@"), message: "Email must contain @", }, { validate: (v) => v.length > 0, message: "Email is required", }, ], password: [ { validate: (v) => v.length >= 8, message: "Password must be at least 8 characters", }, ], }); const errors = validator.validate({ email: "invalid", password: "short", }); // Type: { email?: string[]; password?: string[]; } | null ``` ### Pattern 6: Discriminated Unions ```typescript type Success<T> = { status: "success"; data: T; }; type Error = { status: "error"; error: string; }; type Loading = { status: "loading"; }; type AsyncState<T> = Success<T> | Error | Loading; function handleState<T>(state: AsyncState<T>): void { switch (state.status) { case "success": console.log(state.data); // Type: T break; case "error": console.log(state.error); // Type: string break; case "loading": console.log("Loading..."); break; } } // Type-safe state machine type State = | { type: "idle" } | { type: "fetching"; requestId: string } | { type: "success"; data: any } | { type: "error"; error: Error }; type Event = | { type: "FETCH"; requestId: string } | { type: "SUCCESS"; data: any } | { type: "ERROR"; error: Error } | { type: "RESET" }; function reducer(state: State, event: Event): State { switch (state.type) { case "idle": return event.type === "FETCH" ? { type: "fetching", requestId: event.requestId } : state; case "fetching": if (event.type === "SUCCESS") { return { type: "success", data: event.data }; } if (event.type === "ERROR") { return { type: "error", error: event.error }; } return state; case "success": case "error": return event.type === "RESET" ? { type: "idle" } : state; } } ``` ## Type Inference Techniques ### 1. Infer Keyword ```typescript // Extract array element type type ElementType<T> = T extends (infer U)[] ? U : never; type NumArray = number[]; type Num = ElementType<NumArray>; // number // Extract promise type type PromiseType<T> = T extends Promise<infer U> ? U : never; type AsyncNum = PromiseType<Promise<number>>; // number // Extract function parameters type Parameters<T> = T extends (...args: infer P) => any ? P : never; function foo(a: string, b: number) {} type FooParams = Parameters<typeof foo>; // [string, number] ``` ### 2. Type Guards ```typescript function isString(value: unknown): value is string { return typeof value === "string"; } function isArrayOf<T>( value: unknown, guard: (item: unknown) => item is T, ): value is T[] { return Array.isArray(value) && value.every(guard); } const data: unknown = ["a", "b", "c"]; if (isArrayOf(data, isString)) { data.forEach((s) => s.toUpperCase()); // Type: string[] } ``` ### 3. Assertion Functions ```typescript function assertIsString(value: unknown): asserts value is string { if (typeof value !== "string") { throw new Error("Not a string"); } } function processValue(value: unknown) { assertIsString(value); // value is now typed as string console.log(value.toUpperCase()); } ``` ## Best Practices 1. **Use `unknown` over `any`**: Enforce type checking 2. **Prefer `interface` for object shapes**: Better error messages 3. **Use `type` for unions and complex types**: More flexible 4. **Leverage type inference**: Let TypeScript infer when possible 5. **Create helper types**: Build reusable type utilities 6. **Use const assertions**: Preserve literal types 7. **Avoid type assertions**: Use type guards instead 8. **Document complex types**: Add JSDoc comments 9. **Use strict mode**: Enable all strict compiler options 10. **Test your types**: Use type tests to verify type behavior ## Type Testing ```typescript // Type assertion tests type AssertEqual<T, U> = [T] extends [U] ? [U] extends [T] ? true : false : false; type Test1 = AssertEqual<string, string>; // true type Test2 = AssertEqual<string, number>; // false type Test3 = AssertEqual<string | number, string>; // false // Expect error helper type ExpectError<T extends never> = T; // Example usage type ShouldError = ExpectError<AssertEqual<string, number>>; ``` ## Common Pitfalls 1. **Over-using `any`**: Defeats the purpose of TypeScript 2. **Ignoring strict null checks**: Can lead to runtime errors 3. **Too complex types**: Can slow down compilation 4. **Not using discriminated unions**: Misses type narrowing opportunities 5. **Forgetting readonly modifiers**: Allows unintended mutations 6. **Circular type references**: Can cause compiler errors 7. **Not handling edge cases**: Like empty arrays or null values ## Performance Considerations - Avoid deeply nested conditional types - Use simple types when possible - Cache complex type computations - Limit recursion depth in recursive types - Use build tools to skip type checking in production ## Resources - **TypeScript Handbook**: https://www.typescriptlang.org/docs/handbook/ - **Type Challenges**: https://github.com/type-challenges/type-challenges - **TypeScript Deep Dive**: https://basarat.gitbook.io/typescript/ - **Effective TypeScript**: Book by Dan Vanderkam
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

prompt-engineering-patterns

Master advanced prompt engineering techniques to maximize LLM

coding
⭐1
# Prompt Engineering Patterns Master advanced prompt engineering techniques to maximize LLM performance, reliability, and controllability. ## When to Use This Skill - Designing complex prompts for production LLM applications - Optimizing prompt performance and consistency - Implementing structured reasoning patterns (chain-of-thought, tree-of-thought) - Building few-shot learning systems with dynamic example selection - Creating reusable prompt templates with variable interpolation - Debugging and refining prompts that produce inconsistent outputs - Implementing system prompts for specialized AI assistants - Using structured outputs (JSON mode) for reliable parsing ## Core Capabilities ### 1. Few-Shot Learning - Example selection strategies (semantic similarity, diversity sampling) - Balancing example count with context window constraints - Constructing effective demonstrations with input-output pairs - Dynamic example retrieval from knowledge bases - Handling edge cases through strategic example selection ### 2. Chain-of-Thought Prompting - Step-by-step reasoning elicitation - Zero-shot CoT with "Let's think step by step" - Few-shot CoT with reasoning traces - Self-consistency techniques (sampling multiple reasoning paths) - Verification and validation steps ### 3. Structured Outputs - JSON mode for reliable parsing - Pydantic schema enforcement - Type-safe response handling - Error handling for malformed outputs ### 4. Prompt Optimization - Iterative refinement workflows - A/B testing prompt variations - Measuring prompt performance metrics (accuracy, consistency, latency) - Reducing token usage while maintaining quality - Handling edge cases and failure modes ### 5. Template Systems - Variable interpolation and formatting - Conditional prompt sections - Multi-turn conversation templates - Role-based prompt composition - Modular prompt components ### 6. System Prompt Design - Setting model behavior and constraints - Defining output formats and structure - Establishing role and expertise - Safety guidelines and content policies - Context setting and background information ## Quick Start ```python from langchain_anthropic import ChatAnthropic from langchain_core.prompts import ChatPromptTemplate from pydantic import BaseModel, Field # Define structured output schema class SQLQuery(BaseModel): query: str = Field(description="The SQL query") explanation: str = Field(description="Brief explanation of what the query does") tables_used: list[str] = Field(description="List of tables referenced") # Initialize model with structured output llm = ChatAnthropic(model="claude-sonnet-4-6") structured_llm = llm.with_structured_output(SQLQuery) # Create prompt template prompt = ChatPromptTemplate.from_messages([ ("system", """You are an expert SQL developer. Generate efficient, secure SQL queries. Always use parameterized queries to prevent SQL injection. Explain your reasoning briefly."""), ("user", "Convert this to SQL: {query}") ]) # Create chain chain = prompt | structured_llm # Use result = await chain.ainvoke({ "query": "Find all users who registered in the last 30 days" }) print(result.query) print(result.explanation) ``` ## Key Patterns ### Pattern 1: Structured Output with Pydantic ```python from anthropic import Anthropic from pydantic import BaseModel, Field from typing import Literal import json class SentimentAnalysis(BaseModel): sentiment: Literal["positive", "negative", "neutral"] confidence: float = Field(ge=0, le=1) key_phrases: list[str] reasoning: str async def analyze_sentiment(text: str) -> SentimentAnalysis: """Analyze sentiment with structured output.""" client = Anthropic() message = client.messages.create( model="claude-sonnet-4-6", max_tokens=500, messages=[{ "role": "user", "content": f"""Analyze the sentiment of this text. Text: {text} Respond with JSON matching this schema: {{ "sentiment": "positive" | "negative" | "neutral", "confidence": 0.0-1.0, "key_phrases": ["phrase1", "phrase2"], "reasoning": "brief explanation" }}""" }] ) return SentimentAnalysis(**json.loads(message.content[0].text)) ``` ### Pattern 2: Chain-of-Thought with Self-Verification ```python from langchain_core.prompts import ChatPromptTemplate cot_prompt = ChatPromptTemplate.from_template(""" Solve this problem step by step. Problem: {problem} Instructions: 1. Break down the problem into clear steps 2. Work through each step showing your reasoning 3. State your final answer 4. Verify your answer by checking it against the original problem Format your response as: ## Steps [Your step-by-step reasoning] ## Answer [Your final answer] ## Verification [Check that your answer is correct] """) ``` ### Pattern 3: Few-Shot with Dynamic Example Selection ```python from langchain_voyageai import VoyageAIEmbeddings from langchain_core.example_selectors import SemanticSimilarityExampleSelector from langchain_chroma import Chroma # Create example selector with semantic similarity example_selector = SemanticSimilarityExampleSelector.from_examples( examples=[ {"input": "How do I reset my password?", "output": "Go to Settings > Security > Reset Password"}, {"input": "Where can I see my order history?", "output": "Navigate to Account > Orders"}, {"input": "How do I contact support?", "output": "Click Help > Contact Us or email support@example.com"}, ], embeddings=VoyageAIEmbeddings(model="voyage-3-large"), vectorstore_cls=Chroma, k=2 # Select 2 most similar examples ) async def get_few_shot_prompt(query: str) -> str: """Build prompt with dynamically selected examples.""" examples = await example_selector.aselect_examples({"input": query}) examples_text = "\n".join( f"User: {ex['input']}\nAssistant: {ex['output']}" for ex in examples ) return f"""You are a helpful customer support assistant. Here are some example interactions: {examples_text} Now respond to this query: User: {query} Assistant:""" ``` ### Pattern 4: Progressive Disclosure Start with simple prompts, add complexity only when needed: ```python PROMPT_LEVELS = { # Level 1: Direct instruction "simple": "Summarize this article: {text}", # Level 2: Add constraints "constrained": """Summarize this article in 3 bullet points, focusing on: - Key findings - Main conclusions - Practical implications Article: {text}""", # Level 3: Add reasoning "reasoning": """Read this article carefully. 1. First, identify the main topic and thesis 2. Then, extract the key supporting points 3. Finally, summarize in 3 bullet points Article: {text} Summary:""", # Level 4: Add examples "few_shot": """Read articles and provide concise summaries. Example: Article: "New research shows that regular exercise can reduce anxiety by up to 40%..." Summary: • Regular exercise reduces anxiety by up to 40% • 30 minutes of moderate activity 3x/week is sufficient • Benefits appear within 2 weeks of starting Now summarize this article: Article: {text} Summary:""" } ``` ### Pattern 5: Error Recovery and Fallback ```python from pydantic import BaseModel, ValidationError import json class ResponseWithConfidence(BaseModel): answer: str confidence: float sources: list[str] alternative_interpretations: list[str] = [] ERROR_RECOVERY_PROMPT = """ Answer the question based on the context provided. Context: {context} Question: {question} Instructions: 1. If you can answer confidently (>0.8), provide a direct answer 2. If you're somewhat confident (0.5-0.8), provide your best answer with caveats 3. If you're uncertain (<0.5), explain what information is missing 4. Always provide alternative interpretations if the question is ambiguous Respond in JSON: {{ "answer": "your answer or 'I cannot determine this from the context'", "confidence": 0.0-1.0, "sources": ["relevant context excerpts"], "alternative_interpretations": ["if question is ambiguous"] }} """ async def answer_with_fallback( context: str, question: str, llm ) -> ResponseWithConfidence: """Answer with error recovery and fallback.""" prompt = ERROR_RECOVERY_PROMPT.format(context=context, question=question) try: response = await llm.ainvoke(prompt) return ResponseWithConfidence(**json.loads(response.content)) except (json.JSONDecodeError, ValidationError) as e: # Fallback: try to extract answer without structure simple_prompt = f"Based on: {context}\n\nAnswer: {question}" simple_response = await llm.ainvoke(simple_prompt) return ResponseWithConfidence( answer=simple_response.content, confidence=0.5, sources=["fallback extraction"], alternative_interpretations=[] ) ``` ### Pattern 6: Role-Based System Prompts ```python SYSTEM_PROMPTS = { "analyst": """You are a senior data analyst with expertise in SQL, Python, and business intelligence. Your responsibilities: - Write efficient, well-documented queries - Explain your analysis methodology - Highlight key insights and recommendations - Flag any data quality concerns Communication style: - Be precise and technical when discussing methodology - Translate technical findings into business impact - Use clear visualizations when helpful""", "assistant": """You are a helpful AI assistant focused on accuracy and clarity. Core principles: - Always cite sources when making factual claims - Acknowledge uncertainty rather than guessing - Ask clarifying questions when the request is ambiguous - Provide step-by-step explanations for complex topics Constraints: - Do not provide medical, legal, or financial advice - Redirect harmful requests appropriately - Protect user privacy""", "code_reviewer": """You are a senior software engineer conducting code reviews. Review criteria: - Correctness: Does the code work as intended? - Security: Are there any vulnerabilities? - Performance: Are there efficiency concerns? - Maintainability: Is the code readable and well-structured? - Best practices: Does it follow language idioms? Output format: 1. Summary assessment (approve/request changes) 2. Critical issues (must fix) 3. Suggestions (nice to have) 4. Positive feedback (what's done well)""" } ``` ## Integration Patterns ### With RAG Systems ```python RAG_PROMPT = """You are a knowledgeable assistant that answers questions based on provided context. Context (retrieved from knowledge base): {context} Instructions: 1. Answer ONLY based on the provided context 2. If the context doesn't contain the answer, say "I don't have information about that in my knowledge base" 3. Cite specific passages using [1], [2] notation 4. If the question is ambiguous, ask for clarification Question: {question} Answer:""" ``` ### With Validation and Verification ```python VALIDATED_PROMPT = """Complete the following task: Task: {task} After generating your response, verify it meets ALL these criteria: ✓ Directly addresses the original request ✓ Contains no factual errors ✓ Is appropriately detailed (not too brief, not too verbose) ✓ Uses proper formatting ✓ Is safe and appropriate If verification fails on any criterion, revise before responding. Response:""" ``` ## Performance Optimization ### Token Efficiency ```python # Before: Verbose prompt (150+ tokens) verbose_prompt = """ I would like you to please take the following text and provide me with a comprehensive summary of the main points. The summary should capture the key ideas and important details while being concise and easy to understand. """ # After: Concise prompt (30 tokens) concise_prompt = """Summarize the key points concisely: {text} Summary:""" ``` ### Caching Common Prefixes ```python from anthropic import Anthropic client = Anthropic() # Use prompt caching for repeated system prompts response = client.messages.create( model="claude-sonnet-4-6", max_tokens=1000, system=[ { "type": "text", "text": LONG_SYSTEM_PROMPT, "cache_control": {"type": "ephemeral"} } ], messages=[{"role": "user", "content": user_query}] ) ``` ## Best Practices 1. **Be Specific**: Vague prompts produce inconsistent results 2. **Show, Don't Tell**: Examples are more effective than descriptions 3. **Use Structured Outputs**: Enforce schemas with Pydantic for reliability 4. **Test Extensively**: Evaluate on diverse, representative inputs 5. **Iterate Rapidly**: Small changes can have large impacts 6. **Monitor Performance**: Track metrics in production 7. **Version Control**: Treat prompts as code with proper versioning 8. **Document Intent**: Explain why prompts are structured as they are ## Common Pitfalls - **Over-engineering**: Starting with complex prompts before trying simple ones - **Example pollution**: Using examples that don't match the target task - **Context overflow**: Exceeding token limits with excessive examples - **Ambiguous instructions**: Leaving room for multiple interpretations - **Ignoring edge cases**: Not testing on unusual or boundary inputs - **No error handling**: Assuming outputs will always be well-formed - **Hardcoded values**: Not parameterizing prompts for reuse ## Success Metrics Track these KPIs for your prompts: - **Accuracy**: Correctness of outputs - **Consistency**: Reproducibility across similar inputs - **Latency**: Response time (P50, P95, P99) - **Token Usage**: Average tokens per request - **Success Rate**: Percentage of valid, parseable outputs - **User Satisfaction**: Ratings and feedback ## Resources - [Anthropic Prompt Engineering Guide](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering) - [Claude Prompt Caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) - [OpenAI Prompt Engineering](https://platform.openai.com/docs/guides/prompt-engineering) - [LangChain Prompts](https://python.langchain.com/docs/concepts/prompts/)
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

anti-reversing-techniques

Understand anti-reversing, obfuscation, and protection techniques

security
⭐1
> **AUTHORIZED USE ONLY**: This skill contains dual-use security techniques. Before proceeding with any bypass or analysis: > > 1. **Verify authorization**: Confirm you have explicit written permission from the software owner, or are operating within a legitimate security context (CTF, authorized pentest, malware analysis, security research) > 2. **Document scope**: Ensure your activities fall within the defined scope of your authorization > 3. **Legal compliance**: Understand that unauthorized bypassing of software protection may violate laws (CFAA, DMCA anti-circumvention, etc.) > > **Legitimate use cases**: Malware analysis, authorized penetration testing, CTF competitions, academic security research, analyzing software you own/have rights to # Anti-Reversing Techniques Understanding protection mechanisms encountered during authorized software analysis, security research, and malware analysis. This knowledge helps analysts bypass protections to complete legitimate analysis tasks. ## Anti-Debugging Techniques ### Windows Anti-Debugging #### API-Based Detection ```c // IsDebuggerPresent if (IsDebuggerPresent()) { exit(1); } // CheckRemoteDebuggerPresent BOOL debugged = FALSE; CheckRemoteDebuggerPresent(GetCurrentProcess(), &debugged); if (debugged) exit(1); // NtQueryInformationProcess typedef NTSTATUS (NTAPI *pNtQueryInformationProcess)( HANDLE, PROCESSINFOCLASS, PVOID, ULONG, PULONG); DWORD debugPort = 0; NtQueryInformationProcess( GetCurrentProcess(), ProcessDebugPort, // 7 &debugPort, sizeof(debugPort), NULL ); if (debugPort != 0) exit(1); // Debug flags DWORD debugFlags = 0; NtQueryInformationProcess( GetCurrentProcess(), ProcessDebugFlags, // 0x1F &debugFlags, sizeof(debugFlags), NULL ); if (debugFlags == 0) exit(1); // 0 means being debugged ``` **Bypass Approaches:** ```python # x64dbg: ScyllaHide plugin # Patches common anti-debug checks # Manual patching in debugger: # - Set IsDebuggerPresent return to 0 # - Patch PEB.BeingDebugged to 0 # - Hook NtQueryInformationProcess # IDAPython: Patch checks ida_bytes.patch_byte(check_addr, 0x90) # NOP ``` #### PEB-Based Detection ```c // Direct PEB access #ifdef _WIN64 PPEB peb = (PPEB)__readgsqword(0x60); #else PPEB peb = (PPEB)__readfsdword(0x30); #endif // BeingDebugged flag if (peb->BeingDebugged) exit(1); // NtGlobalFlag // Debugged: 0x70 (FLG_HEAP_ENABLE_TAIL_CHECK | // FLG_HEAP_ENABLE_FREE_CHECK | // FLG_HEAP_VALIDATE_PARAMETERS) if (peb->NtGlobalFlag & 0x70) exit(1); // Heap flags PDWORD heapFlags = (PDWORD)((PBYTE)peb->ProcessHeap + 0x70); if (*heapFlags & 0x50000062) exit(1); ``` **Bypass Approaches:** ```assembly ; In debugger, modify PEB directly ; x64dbg: dump at gs:[60] (x64) or fs:[30] (x86) ; Set BeingDebugged (offset 2) to 0 ; Clear NtGlobalFlag (offset 0xBC for x64) ``` #### Timing-Based Detection ```c // RDTSC timing uint64_t start = __rdtsc(); // ... some code ... uint64_t end = __rdtsc(); if ((end - start) > THRESHOLD) exit(1); // QueryPerformanceCounter LARGE_INTEGER start, end, freq; QueryPerformanceFrequency(&freq); QueryPerformanceCounter(&start); // ... code ... QueryPerformanceCounter(&end); double elapsed = (double)(end.QuadPart - start.QuadPart) / freq.QuadPart; if (elapsed > 0.1) exit(1); // Too slow = debugger // GetTickCount DWORD start = GetTickCount(); // ... code ... if (GetTickCount() - start > 1000) exit(1); ``` **Bypass Approaches:** ``` - Use hardware breakpoints instead of software - Patch timing checks - Use VM with controlled time - Hook timing APIs to return consistent values ``` #### Exception-Based Detection ```c // SEH-based detection __try { __asm { int 3 } // Software breakpoint } __except(EXCEPTION_EXECUTE_HANDLER) { // Normal execution: exception caught return; } // Debugger ate the exception exit(1); // VEH-based detection LONG CALLBACK VectoredHandler(PEXCEPTION_POINTERS ep) { if (ep->ExceptionRecord->ExceptionCode == EXCEPTION_BREAKPOINT) { ep->ContextRecord->Rip++; // Skip INT3 return EXCEPTION_CONTINUE_EXECUTION; } return EXCEPTION_CONTINUE_SEARCH; } ``` ### Linux Anti-Debugging ```c // ptrace self-trace if (ptrace(PTRACE_TRACEME, 0, NULL, NULL) == -1) { // Already being traced exit(1); } // /proc/self/status FILE *f = fopen("/proc/self/status", "r"); char line[256]; while (fgets(line, sizeof(line), f)) { if (strncmp(line, "TracerPid:", 10) == 0) { int tracer_pid = atoi(line + 10); if (tracer_pid != 0) exit(1); } } // Parent process check if (getppid() != 1 && strcmp(get_process_name(getppid()), "bash") != 0) { // Unusual parent (might be debugger) } ``` **Bypass Approaches:** ```bash # LD_PRELOAD to hook ptrace # Compile: gcc -shared -fPIC -o hook.so hook.c long ptrace(int request, ...) { return 0; // Always succeed } # Usage LD_PRELOAD=./hook.so ./target ``` ## Anti-VM Detection ### Hardware Fingerprinting ```c // CPUID-based detection int cpuid_info[4]; __cpuid(cpuid_info, 1); // Check hypervisor bit (bit 31 of ECX) if (cpuid_info[2] & (1 << 31)) { // Running in hypervisor } // CPUID brand string __cpuid(cpuid_info, 0x40000000); char vendor[13] = {0}; memcpy(vendor, &cpuid_info[1], 12); // "VMwareVMware", "Microsoft Hv", "KVMKVMKVM", "VBoxVBoxVBox" // MAC address prefix // VMware: 00:0C:29, 00:50:56 // VirtualBox: 08:00:27 // Hyper-V: 00:15:5D ``` ### Registry/File Detection ```c // Windows registry keys // HKLM\SOFTWARE\VMware, Inc.\VMware Tools // HKLM\SOFTWARE\Oracle\VirtualBox Guest Additions // HKLM\HARDWARE\ACPI\DSDT\VBOX__ // Files // C:\Windows\System32\drivers\vmmouse.sys // C:\Windows\System32\drivers\vmhgfs.sys // C:\Windows\System32\drivers\VBoxMouse.sys // Processes // vmtoolsd.exe, vmwaretray.exe // VBoxService.exe, VBoxTray.exe ``` ### Timing-Based VM Detection ```c // VM exits cause timing anomalies uint64_t start = __rdtsc(); __cpuid(cpuid_info, 0); // Causes VM exit uint64_t end = __rdtsc(); if ((end - start) > 500) { // Likely in VM (CPUID takes longer) } ``` **Bypass Approaches:** ``` - Use bare-metal analysis environment - Harden VM (remove guest tools, change MAC) - Patch detection code - Use specialized analysis VMs (FLARE-VM) ``` ## Code Obfuscation ### Control Flow Obfuscation #### Control Flow Flattening ```c // Original if (cond) { func_a(); } else { func_b(); } func_c(); // Flattened int state = 0; while (1) { switch (state) { case 0: state = cond ? 1 : 2; break; case 1: func_a(); state = 3; break; case 2: func_b(); state = 3; break; case 3: func_c(); return; } } ``` **Analysis Approach:** - Identify state variable - Map state transitions - Reconstruct original flow - Tools: D-810 (IDA), SATURN #### Opaque Predicates ```c // Always true, but complex to analyze int x = rand(); if ((x * x) >= 0) { // Always true real_code(); } else { junk_code(); // Dead code } // Always false if ((x * (x + 1)) % 2 == 1) { // Product of consecutive = even junk_code(); } ``` **Analysis Approach:** - Identify constant expressions - Symbolic execution to prove predicates - Pattern matching for known opaque predicates ### Data Obfuscation #### String Encryption ```c // XOR encryption char decrypt_string(char *enc, int len, char key) { char *dec = malloc(len + 1); for (int i = 0; i < len; i++) { dec[i] = enc[i] ^ key; } dec[len] = 0; return dec; } // Stack strings char url[20]; url[0] = 'h'; url[1] = 't'; url[2] = 't'; url[3] = 'p'; url[4] = ':'; url[5] = '/'; url[6] = '/'; // ... ``` **Analysis Approach:** ```python # FLOSS for automatic string deobfuscation floss malware.exe # IDAPython string decryption def decrypt_xor(ea, length, key): result = "" for i in range(length): byte = ida_bytes.get_byte(ea + i) result += chr(byte ^ key) return result ``` #### API Obfuscation ```c // Dynamic API resolution typedef HANDLE (WINAPI *pCreateFileW)(LPCWSTR, DWORD, DWORD, LPSECURITY_ATTRIBUTES, DWORD, DWORD, HANDLE); HMODULE kernel32 = LoadLibraryA("kernel32.dll"); pCreateFileW myCreateFile = (pCreateFileW)GetProcAddress( kernel32, "CreateFileW"); // API hashing DWORD hash_api(char *name) { DWORD hash = 0; while (*name) { hash = ((hash >> 13) | (hash << 19)) + *name++; } return hash; } // Resolve by hash comparison instead of string ``` **Analysis Approach:** - Identify hash algorithm - Build hash database of known APIs - Use HashDB plugin for IDA - Dynamic analysis to resolve at runtime ### Instruction-Level Obfuscation #### Dead Code Insertion ```asm ; Original mov eax, 1 ; With dead code push ebx ; Dead mov eax, 1 pop ebx ; Dead xor ecx, ecx ; Dead add ecx, ecx ; Dead ``` #### Instruction Substitution ```asm ; Original: xor eax, eax (set to 0) ; Substitutions: sub eax, eax mov eax, 0 and eax, 0 lea eax, [0] ; Original: mov eax, 1 ; Substitutions: xor eax, eax inc eax push 1 pop eax ``` ## Packing and Encryption ### Common Packers ``` UPX - Open source, easy to unpack Themida - Commercial, VM-based protection VMProtect - Commercial, code virtualization ASPack - Compression packer PECompact - Compression packer Enigma - Commercial protector ``` ### Unpacking Methodology ``` 1. Identify packer (DIE, Exeinfo PE, PEiD) 2. Static unpacking (if known packer): - UPX: upx -d packed.exe - Use existing unpackers 3. Dynamic unpacking: a. Find Original Entry Point (OEP) b. Set breakpoint on OEP c. Dump memory when OEP reached d. Fix import table (Scylla, ImpREC) 4. OEP finding techniques: - Hardware breakpoint on stack (ESP trick) - Break on common API calls (GetCommandLineA) - Trace and look for typical entry patterns ``` ### Manual Unpacking Example ``` 1. Load packed binary in x64dbg 2. Note entry point (packer stub) 3. Use ESP trick: - Run to entry - Set hardware breakpoint on [ESP] - Run until breakpoint hits (after PUSHAD/POPAD) 4. Look for JMP to OEP 5. At OEP, use Scylla to: - Dump process - Find imports (IAT autosearch) - Fix dump ``` ## Virtualization-Based Protection ### Code Virtualization ``` Original x86 code is converted to custom bytecode interpreted by embedded VM at runtime. Original: VM Protected: mov eax, 1 push vm_context add eax, 2 call vm_entry ; VM interprets bytecode ; equivalent to original ``` ### Analysis Approaches ``` 1. Identify VM components: - VM entry (dispatcher) - Handler table - Bytecode location - Virtual registers/stack 2. Trace execution: - Log handler calls - Map bytecode to operations - Understand instruction set 3. Lifting/devirtualization: - Map VM instructions back to native - Tools: VMAttack, SATURN, NoVmp 4. Symbolic execution: - Analyze VM semantically - angr, Triton ``` ## Bypass Strategies Summary ### General Principles 1. **Understand the protection**: Identify what technique is used 2. **Find the check**: Locate protection code in binary 3. **Patch or hook**: Modify check to always pass 4. **Use appropriate tools**: ScyllaHide, x64dbg plugins 5. **Document findings**: Keep notes on bypassed protections ### Tool Recommendations ``` Anti-debug bypass: ScyllaHide, TitanHide Unpacking: x64dbg + Scylla, OllyDumpEx Deobfuscation: D-810, SATURN, miasm VM analysis: VMAttack, NoVmp, manual tracing String decryption: FLOSS, custom scripts Symbolic execution: angr, Triton ``` ### Ethical Considerations This knowledge should only be used for: - Authorized security research - Malware analysis (defensive) - CTF competitions - Understanding protections for legitimate purposes - Educational purposes Never use to bypass protections for: - Software piracy - Unauthorized access - Malicious purposes
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

binary-analysis-patterns

Master binary analysis patterns including disassembly,

security
⭐1
# Binary Analysis Patterns Comprehensive patterns and techniques for analyzing compiled binaries, understanding assembly code, and reconstructing program logic. ## Disassembly Fundamentals ### x86-64 Instruction Patterns #### Function Prologue/Epilogue ```asm ; Standard prologue push rbp ; Save base pointer mov rbp, rsp ; Set up stack frame sub rsp, 0x20 ; Allocate local variables ; Leaf function (no calls) ; May skip frame pointer setup sub rsp, 0x18 ; Just allocate locals ; Standard epilogue mov rsp, rbp ; Restore stack pointer pop rbp ; Restore base pointer ret ; Leave instruction (equivalent) leave ; mov rsp, rbp; pop rbp ret ``` #### Calling Conventions **System V AMD64 (Linux, macOS)** ```asm ; Arguments: RDI, RSI, RDX, RCX, R8, R9, then stack ; Return: RAX (and RDX for 128-bit) ; Caller-saved: RAX, RCX, RDX, RSI, RDI, R8-R11 ; Callee-saved: RBX, RBP, R12-R15 ; Example: func(a, b, c, d, e, f, g) mov rdi, [a] ; 1st arg mov rsi, [b] ; 2nd arg mov rdx, [c] ; 3rd arg mov rcx, [d] ; 4th arg mov r8, [e] ; 5th arg mov r9, [f] ; 6th arg push [g] ; 7th arg on stack call func ``` **Microsoft x64 (Windows)** ```asm ; Arguments: RCX, RDX, R8, R9, then stack ; Shadow space: 32 bytes reserved on stack ; Return: RAX ; Example: func(a, b, c, d, e) sub rsp, 0x28 ; Shadow space + alignment mov rcx, [a] ; 1st arg mov rdx, [b] ; 2nd arg mov r8, [c] ; 3rd arg mov r9, [d] ; 4th arg mov [rsp+0x20], [e] ; 5th arg on stack call func add rsp, 0x28 ``` ### ARM Assembly Patterns #### ARM64 (AArch64) Calling Convention ```asm ; Arguments: X0-X7 ; Return: X0 (and X1 for 128-bit) ; Frame pointer: X29 ; Link register: X30 ; Function prologue stp x29, x30, [sp, #-16]! ; Save FP and LR mov x29, sp ; Set frame pointer ; Function epilogue ldp x29, x30, [sp], #16 ; Restore FP and LR ret ``` #### ARM32 Calling Convention ```asm ; Arguments: R0-R3, then stack ; Return: R0 (and R1 for 64-bit) ; Link register: LR (R14) ; Function prologue push {fp, lr} add fp, sp, #4 ; Function epilogue pop {fp, pc} ; Return by popping PC ``` ## Control Flow Patterns ### Conditional Branches ```asm ; if (a == b) cmp eax, ebx jne skip_block ; ... if body ... skip_block: ; if (a < b) - signed cmp eax, ebx jge skip_block ; Jump if greater or equal ; ... if body ... skip_block: ; if (a < b) - unsigned cmp eax, ebx jae skip_block ; Jump if above or equal ; ... if body ... skip_block: ``` ### Loop Patterns ```asm ; for (int i = 0; i < n; i++) xor ecx, ecx ; i = 0 loop_start: cmp ecx, [n] ; i < n jge loop_end ; ... loop body ... inc ecx ; i++ jmp loop_start loop_end: ; while (condition) jmp loop_check loop_body: ; ... body ... loop_check: cmp eax, ebx jl loop_body ; do-while loop_body: ; ... body ... cmp eax, ebx jl loop_body ``` ### Switch Statement Patterns ```asm ; Jump table pattern mov eax, [switch_var] cmp eax, max_case ja default_case jmp [jump_table + eax*8] ; Sequential comparison (small switch) cmp eax, 1 je case_1 cmp eax, 2 je case_2 cmp eax, 3 je case_3 jmp default_case ``` ## Data Structure Patterns ### Array Access ```asm ; array[i] - 4-byte elements mov eax, [rbx + rcx*4] ; rbx=base, rcx=index ; array[i] - 8-byte elements mov rax, [rbx + rcx*8] ; Multi-dimensional array[i][j] ; arr[i][j] = base + (i * cols + j) * element_size imul eax, [cols] add eax, [j] mov edx, [rbx + rax*4] ``` ### Structure Access ```c struct Example { int a; // offset 0 char b; // offset 4 // padding // offset 5-7 long c; // offset 8 short d; // offset 16 }; ``` ```asm ; Accessing struct fields mov rdi, [struct_ptr] mov eax, [rdi] ; s->a (offset 0) movzx eax, byte [rdi+4] ; s->b (offset 4) mov rax, [rdi+8] ; s->c (offset 8) movzx eax, word [rdi+16] ; s->d (offset 16) ``` ### Linked List Traversal ```asm ; while (node != NULL) list_loop: test rdi, rdi ; node == NULL? jz list_done ; ... process node ... mov rdi, [rdi+8] ; node = node->next (assuming next at offset 8) jmp list_loop list_done: ``` ## Common Code Patterns ### String Operations ```asm ; strlen pattern xor ecx, ecx strlen_loop: cmp byte [rdi + rcx], 0 je strlen_done inc ecx jmp strlen_loop strlen_done: ; ecx contains length ; strcpy pattern strcpy_loop: mov al, [rsi] mov [rdi], al test al, al jz strcpy_done inc rsi inc rdi jmp strcpy_loop strcpy_done: ; memcpy using rep movsb mov rdi, dest mov rsi, src mov rcx, count rep movsb ``` ### Arithmetic Patterns ```asm ; Multiplication by constant ; x * 3 lea eax, [rax + rax*2] ; x * 5 lea eax, [rax + rax*4] ; x * 10 lea eax, [rax + rax*4] ; x * 5 add eax, eax ; * 2 ; Division by power of 2 (signed) mov eax, [x] cdq ; Sign extend to EDX:EAX and edx, 7 ; For divide by 8 add eax, edx ; Adjust for negative sar eax, 3 ; Arithmetic shift right ; Modulo power of 2 and eax, 7 ; x % 8 ``` ### Bit Manipulation ```asm ; Test specific bit test eax, 0x80 ; Test bit 7 jnz bit_set ; Set bit or eax, 0x10 ; Set bit 4 ; Clear bit and eax, ~0x10 ; Clear bit 4 ; Toggle bit xor eax, 0x10 ; Toggle bit 4 ; Count leading zeros bsr eax, ecx ; Bit scan reverse xor eax, 31 ; Convert to leading zeros ; Population count (popcnt) popcnt eax, ecx ; Count set bits ``` ## Decompilation Patterns ### Variable Recovery ```asm ; Local variable at rbp-8 mov qword [rbp-8], rax ; Store to local mov rax, [rbp-8] ; Load from local ; Stack-allocated array lea rax, [rbp-0x40] ; Array starts at rbp-0x40 mov [rax], edx ; array[0] = value mov [rax+4], ecx ; array[1] = value ``` ### Function Signature Recovery ```asm ; Identify parameters by register usage func: ; rdi used as first param (System V) mov [rbp-8], rdi ; Save param to local ; rsi used as second param mov [rbp-16], rsi ; Identify return by RAX at end mov rax, [result] ret ``` ### Type Recovery ```asm ; 1-byte operations suggest char/bool movzx eax, byte [rdi] ; Zero-extend byte movsx eax, byte [rdi] ; Sign-extend byte ; 2-byte operations suggest short movzx eax, word [rdi] movsx eax, word [rdi] ; 4-byte operations suggest int/float mov eax, [rdi] movss xmm0, [rdi] ; Float ; 8-byte operations suggest long/double/pointer mov rax, [rdi] movsd xmm0, [rdi] ; Double ``` ## Ghidra Analysis Tips ### Improving Decompilation ```java // In Ghidra scripting // Fix function signature Function func = getFunctionAt(toAddr(0x401000)); func.setReturnType(IntegerDataType.dataType, SourceType.USER_DEFINED); // Create structure type StructureDataType struct = new StructureDataType("MyStruct", 0); struct.add(IntegerDataType.dataType, "field_a", null); struct.add(PointerDataType.dataType, "next", null); // Apply to memory createData(toAddr(0x601000), struct); ``` ### Pattern Matching Scripts ```python # Find all calls to dangerous functions for func in currentProgram.getFunctionManager().getFunctions(True): for ref in getReferencesTo(func.getEntryPoint()): if func.getName() in ["strcpy", "sprintf", "gets"]: print(f"Dangerous call at {ref.getFromAddress()}") ``` ## IDA Pro Patterns ### IDAPython Analysis ```python import idaapi import idautils import idc # Find all function calls def find_calls(func_name): for func_ea in idautils.Functions(): for head in idautils.Heads(func_ea, idc.find_func_end(func_ea)): if idc.print_insn_mnem(head) == "call": target = idc.get_operand_value(head, 0) if idc.get_func_name(target) == func_name: print(f"Call to {func_name} at {hex(head)}") # Rename functions based on strings def auto_rename(): for s in idautils.Strings(): for xref in idautils.XrefsTo(s.ea): func = idaapi.get_func(xref.frm) if func and "sub_" in idc.get_func_name(func.start_ea): # Use string as hint for naming pass ``` ## Best Practices ### Analysis Workflow 1. **Initial triage**: File type, architecture, imports/exports 2. **String analysis**: Identify interesting strings, error messages 3. **Function identification**: Entry points, exports, cross-references 4. **Control flow mapping**: Understand program structure 5. **Data structure recovery**: Identify structs, arrays, globals 6. **Algorithm identification**: Crypto, hashing, compression 7. **Documentation**: Comments, renamed symbols, type definitions ### Common Pitfalls - **Optimizer artifacts**: Code may not match source structure - **Inline functions**: Functions may be expanded inline - **Tail call optimization**: `jmp` instead of `call` + `ret` - **Dead code**: Unreachable code from optimization - **Position-independent code**: RIP-relative addressing
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

memory-forensics

Master memory forensics techniques including memory acquisition,

security
⭐1
# Memory Forensics Comprehensive techniques for acquiring, analyzing, and extracting artifacts from memory dumps for incident response and malware analysis. ## Memory Acquisition ### Live Acquisition Tools #### Windows ```powershell # WinPmem (Recommended) winpmem_mini_x64.exe memory.raw # DumpIt DumpIt.exe # Belkasoft RAM Capturer # GUI-based, outputs raw format # Magnet RAM Capture # GUI-based, outputs raw format ``` #### Linux ```bash # LiME (Linux Memory Extractor) sudo insmod lime.ko "path=/tmp/memory.lime format=lime" # /dev/mem (limited, requires permissions) sudo dd if=/dev/mem of=memory.raw bs=1M # /proc/kcore (ELF format) sudo cp /proc/kcore memory.elf ``` #### macOS ```bash # osxpmem sudo ./osxpmem -o memory.raw # MacQuisition (commercial) ``` ### Virtual Machine Memory ```bash # VMware: .vmem file is raw memory cp vm.vmem memory.raw # VirtualBox: Use debug console vboxmanage debugvm "VMName" dumpvmcore --filename memory.elf # QEMU virsh dump <domain> memory.raw --memory-only # Hyper-V # Checkpoint contains memory state ``` ## Volatility 3 Framework ### Installation and Setup ```bash # Install Volatility 3 pip install volatility3 # Install symbol tables (Windows) # Download from https://downloads.volatilityfoundation.org/volatility3/symbols/ # Basic usage vol -f memory.raw <plugin> # With symbol path vol -f memory.raw -s /path/to/symbols windows.pslist ``` ### Essential Plugins #### Process Analysis ```bash # List processes vol -f memory.raw windows.pslist # Process tree (parent-child relationships) vol -f memory.raw windows.pstree # Hidden process detection vol -f memory.raw windows.psscan # Process memory dumps vol -f memory.raw windows.memmap --pid <PID> --dump # Process environment variables vol -f memory.raw windows.envars --pid <PID> # Command line arguments vol -f memory.raw windows.cmdline ``` #### Network Analysis ```bash # Network connections vol -f memory.raw windows.netscan # Network connection state vol -f memory.raw windows.netstat ``` #### DLL and Module Analysis ```bash # Loaded DLLs per process vol -f memory.raw windows.dlllist --pid <PID> # Find hidden/injected DLLs vol -f memory.raw windows.ldrmodules # Kernel modules vol -f memory.raw windows.modules # Module dumps vol -f memory.raw windows.moddump --pid <PID> ``` #### Memory Injection Detection ```bash # Detect code injection vol -f memory.raw windows.malfind # VAD (Virtual Address Descriptor) analysis vol -f memory.raw windows.vadinfo --pid <PID> # Dump suspicious memory regions vol -f memory.raw windows.vadyarascan --yara-rules rules.yar ``` #### Registry Analysis ```bash # List registry hives vol -f memory.raw windows.registry.hivelist # Print registry key vol -f memory.raw windows.registry.printkey --key "Software\Microsoft\Windows\CurrentVersion\Run" # Dump registry hive vol -f memory.raw windows.registry.hivescan --dump ``` #### File System Artifacts ```bash # Scan for file objects vol -f memory.raw windows.filescan # Dump files from memory vol -f memory.raw windows.dumpfiles --pid <PID> # MFT analysis vol -f memory.raw windows.mftscan ``` ### Linux Analysis ```bash # Process listing vol -f memory.raw linux.pslist # Process tree vol -f memory.raw linux.pstree # Bash history vol -f memory.raw linux.bash # Network connections vol -f memory.raw linux.sockstat # Loaded kernel modules vol -f memory.raw linux.lsmod # Mount points vol -f memory.raw linux.mount # Environment variables vol -f memory.raw linux.envars ``` ### macOS Analysis ```bash # Process listing vol -f memory.raw mac.pslist # Process tree vol -f memory.raw mac.pstree # Network connections vol -f memory.raw mac.netstat # Kernel extensions vol -f memory.raw mac.lsmod ``` ## Analysis Workflows ### Malware Analysis Workflow ```bash # 1. Initial process survey vol -f memory.raw windows.pstree > processes.txt vol -f memory.raw windows.pslist > pslist.txt # 2. Network connections vol -f memory.raw windows.netscan > network.txt # 3. Detect injection vol -f memory.raw windows.malfind > malfind.txt # 4. Analyze suspicious processes vol -f memory.raw windows.dlllist --pid <PID> vol -f memory.raw windows.handles --pid <PID> # 5. Dump suspicious executables vol -f memory.raw windows.pslist --pid <PID> --dump # 6. Extract strings from dumps strings -a pid.<PID>.exe > strings.txt # 7. YARA scanning vol -f memory.raw windows.yarascan --yara-rules malware.yar ``` ### Incident Response Workflow ```bash # 1. Timeline of events vol -f memory.raw windows.timeliner > timeline.csv # 2. User activity vol -f memory.raw windows.cmdline vol -f memory.raw windows.consoles # 3. Persistence mechanisms vol -f memory.raw windows.registry.printkey \ --key "Software\Microsoft\Windows\CurrentVersion\Run" # 4. Services vol -f memory.raw windows.svcscan # 5. Scheduled tasks vol -f memory.raw windows.scheduled_tasks # 6. Recent files vol -f memory.raw windows.filescan | grep -i "recent" ``` ## Data Structures ### Windows Process Structures ```c // EPROCESS (Executive Process) typedef struct _EPROCESS { KPROCESS Pcb; // Kernel process block EX_PUSH_LOCK ProcessLock; LARGE_INTEGER CreateTime; LARGE_INTEGER ExitTime; // ... LIST_ENTRY ActiveProcessLinks; // Doubly-linked list ULONG_PTR UniqueProcessId; // PID // ... PEB* Peb; // Process Environment Block // ... } EPROCESS; // PEB (Process Environment Block) typedef struct _PEB { BOOLEAN InheritedAddressSpace; BOOLEAN ReadImageFileExecOptions; BOOLEAN BeingDebugged; // Anti-debug check // ... PVOID ImageBaseAddress; // Base address of executable PPEB_LDR_DATA Ldr; // Loader data (DLL list) PRTL_USER_PROCESS_PARAMETERS ProcessParameters; // ... } PEB; ``` ### VAD (Virtual Address Descriptor) ```c typedef struct _MMVAD { MMVAD_SHORT Core; union { ULONG LongFlags; MMVAD_FLAGS VadFlags; } u; // ... PVOID FirstPrototypePte; PVOID LastContiguousPte; // ... PFILE_OBJECT FileObject; } MMVAD; // Memory protection flags #define PAGE_EXECUTE 0x10 #define PAGE_EXECUTE_READ 0x20 #define PAGE_EXECUTE_READWRITE 0x40 #define PAGE_EXECUTE_WRITECOPY 0x80 ``` ## Detection Patterns ### Process Injection Indicators ```python # Malfind indicators # - PAGE_EXECUTE_READWRITE protection (suspicious) # - MZ header in non-image VAD region # - Shellcode patterns at allocation start # Common injection techniques # 1. Classic DLL Injection # - VirtualAllocEx + WriteProcessMemory + CreateRemoteThread # 2. Process Hollowing # - CreateProcess (SUSPENDED) + NtUnmapViewOfSection + WriteProcessMemory # 3. APC Injection # - QueueUserAPC targeting alertable threads # 4. Thread Execution Hijacking # - SuspendThread + SetThreadContext + ResumeThread ``` ### Rootkit Detection ```bash # Compare process lists vol -f memory.raw windows.pslist > pslist.txt vol -f memory.raw windows.psscan > psscan.txt diff pslist.txt psscan.txt # Hidden processes # Check for DKOM (Direct Kernel Object Manipulation) vol -f memory.raw windows.callbacks # Detect hooked functions vol -f memory.raw windows.ssdt # System Service Descriptor Table # Driver analysis vol -f memory.raw windows.driverscan vol -f memory.raw windows.driverirp ``` ### Credential Extraction ```bash # Dump hashes (requires hivelist first) vol -f memory.raw windows.hashdump # LSA secrets vol -f memory.raw windows.lsadump # Cached domain credentials vol -f memory.raw windows.cachedump # Mimikatz-style extraction # Requires specific plugins/tools ``` ## YARA Integration ### Writing Memory YARA Rules ```yara rule Suspicious_Injection { meta: description = "Detects common injection shellcode" strings: // Common shellcode patterns $mz = { 4D 5A } $shellcode1 = { 55 8B EC 83 EC } // Function prologue $api_hash = { 68 ?? ?? ?? ?? 68 ?? ?? ?? ?? E8 } // Push hash, call condition: $mz at 0 or any of ($shellcode*) } rule Cobalt_Strike_Beacon { meta: description = "Detects Cobalt Strike beacon in memory" strings: $config = { 00 01 00 01 00 02 } $sleep = "sleeptime" $beacon = "%s (admin)" wide condition: 2 of them } ``` ### Scanning Memory ```bash # Scan all process memory vol -f memory.raw windows.yarascan --yara-rules rules.yar # Scan specific process vol -f memory.raw windows.yarascan --yara-rules rules.yar --pid 1234 # Scan kernel memory vol -f memory.raw windows.yarascan --yara-rules rules.yar --kernel ``` ## String Analysis ### Extracting Strings ```bash # Basic string extraction strings -a memory.raw > all_strings.txt # Unicode strings strings -el memory.raw >> all_strings.txt # Targeted extraction from process dump vol -f memory.raw windows.memmap --pid 1234 --dump strings -a pid.1234.dmp > process_strings.txt # Pattern matching grep -E "(https?://|[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})" all_strings.txt ``` ### FLOSS for Obfuscated Strings ```bash # FLOSS extracts obfuscated strings floss malware.exe > floss_output.txt # From memory dump floss pid.1234.dmp ``` ## Best Practices ### Acquisition Best Practices 1. **Minimize footprint**: Use lightweight acquisition tools 2. **Document everything**: Record time, tool, and hash of capture 3. **Verify integrity**: Hash memory dump immediately after capture 4. **Chain of custody**: Maintain proper forensic handling ### Analysis Best Practices 1. **Start broad**: Get overview before deep diving 2. **Cross-reference**: Use multiple plugins for same data 3. **Timeline correlation**: Correlate memory findings with disk/network 4. **Document findings**: Keep detailed notes and screenshots 5. **Validate results**: Verify findings through multiple methods ### Common Pitfalls - **Stale data**: Memory is volatile, analyze promptly - **Incomplete dumps**: Verify dump size matches expected RAM - **Symbol issues**: Ensure correct symbol files for OS version - **Smear**: Memory may change during acquisition - **Encryption**: Some data may be encrypted in memory
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

protocol-reverse-engineering

Master network protocol reverse engineering including packet

security
⭐1
# Protocol Reverse Engineering Comprehensive techniques for capturing, analyzing, and documenting network protocols for security research, interoperability, and debugging. ## Traffic Capture ### Wireshark Capture ```bash # Capture on specific interface wireshark -i eth0 -k # Capture with filter wireshark -i eth0 -k -f "port 443" # Capture to file tshark -i eth0 -w capture.pcap # Ring buffer capture (rotate files) tshark -i eth0 -b filesize:100000 -b files:10 -w capture.pcap ``` ### tcpdump Capture ```bash # Basic capture tcpdump -i eth0 -w capture.pcap # With filter tcpdump -i eth0 port 8080 -w capture.pcap # Capture specific bytes tcpdump -i eth0 -s 0 -w capture.pcap # Full packet # Real-time display tcpdump -i eth0 -X port 80 ``` ### Man-in-the-Middle Capture ```bash # mitmproxy for HTTP/HTTPS mitmproxy --mode transparent -p 8080 # SSL/TLS interception mitmproxy --mode transparent --ssl-insecure # Dump to file mitmdump -w traffic.mitm # Burp Suite # Configure browser proxy to 127.0.0.1:8080 ``` ## Protocol Analysis ### Wireshark Analysis ``` # Display filters tcp.port == 8080 http.request.method == "POST" ip.addr == 192.168.1.1 tcp.flags.syn == 1 && tcp.flags.ack == 0 frame contains "password" # Following streams Right-click > Follow > TCP Stream Right-click > Follow > HTTP Stream # Export objects File > Export Objects > HTTP # Decryption Edit > Preferences > Protocols > TLS - (Pre)-Master-Secret log filename - RSA keys list ``` ### tshark Analysis ```bash # Extract specific fields tshark -r capture.pcap -T fields -e ip.src -e ip.dst -e tcp.port # Statistics tshark -r capture.pcap -q -z conv,tcp tshark -r capture.pcap -q -z endpoints,ip # Filter and extract tshark -r capture.pcap -Y "http" -T json > http_traffic.json # Protocol hierarchy tshark -r capture.pcap -q -z io,phs ``` ### Scapy for Custom Analysis ```python from scapy.all import * # Read pcap packets = rdpcap("capture.pcap") # Analyze packets for pkt in packets: if pkt.haslayer(TCP): print(f"Src: {pkt[IP].src}:{pkt[TCP].sport}") print(f"Dst: {pkt[IP].dst}:{pkt[TCP].dport}") if pkt.haslayer(Raw): print(f"Data: {pkt[Raw].load[:50]}") # Filter packets http_packets = [p for p in packets if p.haslayer(TCP) and (p[TCP].sport == 80 or p[TCP].dport == 80)] # Create custom packets pkt = IP(dst="target")/TCP(dport=80)/Raw(load="GET / HTTP/1.1\r\n") send(pkt) ``` ## Protocol Identification ### Common Protocol Signatures ``` HTTP - "HTTP/1." or "GET " or "POST " at start TLS/SSL - 0x16 0x03 (record layer) DNS - UDP port 53, specific header format SMB - 0xFF 0x53 0x4D 0x42 ("SMB" signature) SSH - "SSH-2.0" banner FTP - "220 " response, "USER " command SMTP - "220 " banner, "EHLO" command MySQL - 0x00 length prefix, protocol version PostgreSQL - 0x00 0x00 0x00 startup length Redis - "*" RESP array prefix MongoDB - BSON documents with specific header ``` ### Protocol Header Patterns ``` +--------+--------+--------+--------+ | Magic number / Signature | +--------+--------+--------+--------+ | Version | Flags | +--------+--------+--------+--------+ | Length | Message Type | +--------+--------+--------+--------+ | Sequence Number / Session ID | +--------+--------+--------+--------+ | Payload... | +--------+--------+--------+--------+ ``` ## Binary Protocol Analysis ### Structure Identification ```python # Common patterns in binary protocols # Length-prefixed message struct Message { uint32_t length; # Total message length uint16_t msg_type; # Message type identifier uint8_t flags; # Flags/options uint8_t reserved; # Padding/alignment uint8_t payload[]; # Variable-length payload }; # Type-Length-Value (TLV) struct TLV { uint8_t type; # Field type uint16_t length; # Field length uint8_t value[]; # Field data }; # Fixed header + variable payload struct Packet { uint8_t magic[4]; # "ABCD" signature uint32_t version; uint32_t payload_len; uint32_t checksum; # CRC32 or similar uint8_t payload[]; }; ``` ### Python Protocol Parser ```python import struct from dataclasses import dataclass @dataclass class MessageHeader: magic: bytes version: int msg_type: int length: int @classmethod def from_bytes(cls, data: bytes): magic, version, msg_type, length = struct.unpack( ">4sHHI", data[:12] ) return cls(magic, version, msg_type, length) def parse_messages(data: bytes): offset = 0 messages = [] while offset < len(data): header = MessageHeader.from_bytes(data[offset:]) payload = data[offset+12:offset+12+header.length] messages.append((header, payload)) offset += 12 + header.length return messages # Parse TLV structure def parse_tlv(data: bytes): fields = [] offset = 0 while offset < len(data): field_type = data[offset] length = struct.unpack(">H", data[offset+1:offset+3])[0] value = data[offset+3:offset+3+length] fields.append((field_type, value)) offset += 3 + length return fields ``` ### Hex Dump Analysis ```python def hexdump(data: bytes, width: int = 16): """Format binary data as hex dump.""" lines = [] for i in range(0, len(data), width): chunk = data[i:i+width] hex_part = ' '.join(f'{b:02x}' for b in chunk) ascii_part = ''.join( chr(b) if 32 <= b < 127 else '.' for b in chunk ) lines.append(f'{i:08x} {hex_part:<{width*3}} {ascii_part}') return '\n'.join(lines) # Example output: # 00000000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d HTTP/1.1 200 OK. # 00000010 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 .Content-Type: t ``` ## Encryption Analysis ### Identifying Encryption ```python # Entropy analysis - high entropy suggests encryption/compression import math from collections import Counter def entropy(data: bytes) -> float: if not data: return 0.0 counter = Counter(data) probs = [count / len(data) for count in counter.values()] return -sum(p * math.log2(p) for p in probs) # Entropy thresholds: # < 6.0: Likely plaintext or structured data # 6.0-7.5: Possibly compressed # > 7.5: Likely encrypted or random # Common encryption indicators # - High, uniform entropy # - No obvious structure or patterns # - Length often multiple of block size (16 for AES) # - Possible IV at start (16 bytes for AES-CBC) ``` ### TLS Analysis ```bash # Extract TLS metadata tshark -r capture.pcap -Y "ssl.handshake" \ -T fields -e ip.src -e ssl.handshake.ciphersuite # JA3 fingerprinting (client) tshark -r capture.pcap -Y "ssl.handshake.type == 1" \ -T fields -e ssl.handshake.ja3 # JA3S fingerprinting (server) tshark -r capture.pcap -Y "ssl.handshake.type == 2" \ -T fields -e ssl.handshake.ja3s # Certificate extraction tshark -r capture.pcap -Y "ssl.handshake.certificate" \ -T fields -e x509sat.printableString ``` ### Decryption Approaches ```bash # Pre-master secret log (browser) export SSLKEYLOGFILE=/tmp/keys.log # Configure Wireshark # Edit > Preferences > Protocols > TLS # (Pre)-Master-Secret log filename: /tmp/keys.log # Decrypt with private key (if available) # Only works for RSA key exchange # Edit > Preferences > Protocols > TLS > RSA keys list ``` ## Custom Protocol Documentation ### Protocol Specification Template ```markdown # Protocol Name Specification ## Overview Brief description of protocol purpose and design. ## Transport - Layer: TCP/UDP - Port: XXXX - Encryption: TLS 1.2+ ## Message Format ### Header (12 bytes) | Offset | Size | Field | Description | | ------ | ---- | ------- | ----------------------- | | 0 | 4 | Magic | 0x50524F54 ("PROT") | | 4 | 2 | Version | Protocol version (1) | | 6 | 2 | Type | Message type identifier | | 8 | 4 | Length | Payload length in bytes | ### Message Types | Type | Name | Description | | ---- | --------- | ---------------------- | | 0x01 | HELLO | Connection initiation | | 0x02 | HELLO_ACK | Connection accepted | | 0x03 | DATA | Application data | | 0x04 | CLOSE | Connection termination | ### Type 0x01: HELLO | Offset | Size | Field | Description | | ------ | ---- | ---------- | ------------------------ | | 0 | 4 | ClientID | Unique client identifier | | 4 | 2 | Flags | Connection flags | | 6 | var | Extensions | TLV-encoded extensions | ## State Machine ``` [INIT] --HELLO--> [WAIT_ACK] --HELLO_ACK--> [CONNECTED] | DATA/DATA | [CLOSED] <--CLOSE--+ ``` ## Examples ### Connection Establishment ``` Client -> Server: HELLO (ClientID=0x12345678) Server -> Client: HELLO_ACK (Status=OK) Client -> Server: DATA (payload) ``` ``` ### Wireshark Dissector (Lua) ```lua -- custom_protocol.lua local proto = Proto("custom", "Custom Protocol") -- Define fields local f_magic = ProtoField.string("custom.magic", "Magic") local f_version = ProtoField.uint16("custom.version", "Version") local f_type = ProtoField.uint16("custom.type", "Type") local f_length = ProtoField.uint32("custom.length", "Length") local f_payload = ProtoField.bytes("custom.payload", "Payload") proto.fields = { f_magic, f_version, f_type, f_length, f_payload } -- Message type names local msg_types = { [0x01] = "HELLO", [0x02] = "HELLO_ACK", [0x03] = "DATA", [0x04] = "CLOSE" } function proto.dissector(buffer, pinfo, tree) pinfo.cols.protocol = "CUSTOM" local subtree = tree:add(proto, buffer()) -- Parse header subtree:add(f_magic, buffer(0, 4)) subtree:add(f_version, buffer(4, 2)) local msg_type = buffer(6, 2):uint() subtree:add(f_type, buffer(6, 2)):append_text( " (" .. (msg_types[msg_type] or "Unknown") .. ")" ) local length = buffer(8, 4):uint() subtree:add(f_length, buffer(8, 4)) if length > 0 then subtree:add(f_payload, buffer(12, length)) end end -- Register for TCP port local tcp_table = DissectorTable.get("tcp.port") tcp_table:add(8888, proto) ``` ## Active Testing ### Fuzzing with Boofuzz ```python from boofuzz import * def main(): session = Session( target=Target( connection=TCPSocketConnection("target", 8888) ) ) # Define protocol structure s_initialize("HELLO") s_static(b"\x50\x52\x4f\x54") # Magic s_word(1, name="version") # Version s_word(0x01, name="type") # Type (HELLO) s_size("payload", length=4) # Length field s_block_start("payload") s_dword(0x12345678, name="client_id") s_word(0, name="flags") s_block_end() session.connect(s_get("HELLO")) session.fuzz() if __name__ == "__main__": main() ``` ### Replay and Modification ```python from scapy.all import * # Replay captured traffic packets = rdpcap("capture.pcap") for pkt in packets: if pkt.haslayer(TCP) and pkt[TCP].dport == 8888: send(pkt) # Modify and replay for pkt in packets: if pkt.haslayer(Raw): # Modify payload original = pkt[Raw].load modified = original.replace(b"client", b"CLIENT") pkt[Raw].load = modified # Recalculate checksums del pkt[IP].chksum del pkt[TCP].chksum send(pkt) ``` ## Best Practices ### Analysis Workflow 1. **Capture traffic**: Multiple sessions, different scenarios 2. **Identify boundaries**: Message start/end markers 3. **Map structure**: Fixed header, variable payload 4. **Identify fields**: Compare multiple samples 5. **Document format**: Create specification 6. **Validate understanding**: Implement parser/generator 7. **Test edge cases**: Fuzzing, boundary conditions ### Common Patterns to Look For - Magic numbers/signatures at message start - Version fields for compatibility - Length fields (often before variable data) - Type/opcode fields for message identification - Sequence numbers for ordering - Checksums/CRCs for integrity - Timestamps for timing - Session/connection identifiers
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

bash-defensive-patterns

Master defensive Bash programming techniques for production-grade

coding
⭐1
# Bash Defensive Patterns Comprehensive guidance for writing production-ready Bash scripts using defensive programming techniques, error handling, and safety best practices to prevent common pitfalls and ensure reliability. ## When to Use This Skill - Writing production automation scripts - Building CI/CD pipeline scripts - Creating system administration utilities - Developing error-resilient deployment automation - Writing scripts that must handle edge cases safely - Building maintainable shell script libraries - Implementing comprehensive logging and monitoring - Creating scripts that must work across different platforms ## Core Defensive Principles ### 1. Strict Mode Enable bash strict mode at the start of every script to catch errors early. ```bash #!/bin/bash set -Eeuo pipefail # Exit on error, unset variables, pipe failures ``` **Key flags:** - `set -E`: Inherit ERR trap in functions - `set -e`: Exit on any error (command returns non-zero) - `set -u`: Exit on undefined variable reference - `set -o pipefail`: Pipe fails if any command fails (not just last) ### 2. Error Trapping and Cleanup Implement proper cleanup on script exit or error. ```bash #!/bin/bash set -Eeuo pipefail trap 'echo "Error on line $LINENO"' ERR trap 'echo "Cleaning up..."; rm -rf "$TMPDIR"' EXIT TMPDIR=$(mktemp -d) # Script code here ``` ### 3. Variable Safety Always quote variables to prevent word splitting and globbing issues. ```bash # Wrong - unsafe cp $source $dest # Correct - safe cp "$source" "$dest" # Required variables - fail with message if unset : "${REQUIRED_VAR:?REQUIRED_VAR is not set}" ``` ### 4. Array Handling Use arrays safely for complex data handling. ```bash # Safe array iteration declare -a items=("item 1" "item 2" "item 3") for item in "${items[@]}"; do echo "Processing: $item" done # Reading output into array safely mapfile -t lines < <(some_command) readarray -t numbers < <(seq 1 10) ``` ### 5. Conditional Safety Use `[[ ]]` for Bash-specific features, `[ ]` for POSIX. ```bash # Bash - safer if [[ -f "$file" && -r "$file" ]]; then content=$(<"$file") fi # POSIX - portable if [ -f "$file" ] && [ -r "$file" ]; then content=$(cat "$file") fi # Test for existence before operations if [[ -z "${VAR:-}" ]]; then echo "VAR is not set or is empty" fi ``` ## Fundamental Patterns ### Pattern 1: Safe Script Directory Detection ```bash #!/bin/bash set -Eeuo pipefail # Correctly determine script directory SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)" SCRIPT_NAME="$(basename -- "${BASH_SOURCE[0]}")" echo "Script location: $SCRIPT_DIR/$SCRIPT_NAME" ``` ### Pattern 2: Comprehensive Function Templat ```bash #!/bin/bash set -Eeuo pipefail # Prefix for functions: handle_*, process_*, check_*, validate_* # Include documentation and error handling validate_file() { local -r file="$1" local -r message="${2:-File not found: $file}" if [[ ! -f "$file" ]]; then echo "ERROR: $message" >&2 return 1 fi return 0 } process_files() { local -r input_dir="$1" local -r output_dir="$2" # Validate inputs [[ -d "$input_dir" ]] || { echo "ERROR: input_dir not a directory" >&2; return 1; } # Create output directory if needed mkdir -p "$output_dir" || { echo "ERROR: Cannot create output_dir" >&2; return 1; } # Process files safely while IFS= read -r -d '' file; do echo "Processing: $file" # Do work done < <(find "$input_dir" -maxdepth 1 -type f -print0) return 0 } ``` ### Pattern 3: Safe Temporary File Handling ```bash #!/bin/bash set -Eeuo pipefail trap 'rm -rf -- "$TMPDIR"' EXIT # Create temporary directory TMPDIR=$(mktemp -d) || { echo "ERROR: Failed to create temp directory" >&2; exit 1; } # Create temporary files in directory TMPFILE1="$TMPDIR/temp1.txt" TMPFILE2="$TMPDIR/temp2.txt" # Use temporary files touch "$TMPFILE1" "$TMPFILE2" echo "Temp files created in: $TMPDIR" ``` ### Pattern 4: Robust Argument Parsing ```bash #!/bin/bash set -Eeuo pipefail # Default values VERBOSE=false DRY_RUN=false OUTPUT_FILE="" THREADS=4 usage() { cat <<EOF Usage: $0 [OPTIONS] Options: -v, --verbose Enable verbose output -d, --dry-run Run without making changes -o, --output FILE Output file path -j, --jobs NUM Number of parallel jobs -h, --help Show this help message EOF exit "${1:-0}" } # Parse arguments while [[ $# -gt 0 ]]; do case "$1" in -v|--verbose) VERBOSE=true shift ;; -d|--dry-run) DRY_RUN=true shift ;; -o|--output) OUTPUT_FILE="$2" shift 2 ;; -j|--jobs) THREADS="$2" shift 2 ;; -h|--help) usage 0 ;; --) shift break ;; *) echo "ERROR: Unknown option: $1" >&2 usage 1 ;; esac done # Validate required arguments [[ -n "$OUTPUT_FILE" ]] || { echo "ERROR: -o/--output is required" >&2; usage 1; } ``` ### Pattern 5: Structured Logging ```bash #!/bin/bash set -Eeuo pipefail # Logging functions log_info() { echo "[$(date +'%Y-%m-%d %H:%M:%S')] INFO: $*" >&2 } log_warn() { echo "[$(date +'%Y-%m-%d %H:%M:%S')] WARN: $*" >&2 } log_error() { echo "[$(date +'%Y-%m-%d %H:%M:%S')] ERROR: $*" >&2 } log_debug() { if [[ "${DEBUG:-0}" == "1" ]]; then echo "[$(date +'%Y-%m-%d %H:%M:%S')] DEBUG: $*" >&2 fi } # Usage log_info "Starting script" log_debug "Debug information" log_warn "Warning message" log_error "Error occurred" ``` ### Pattern 6: Process Orchestration with Signals ```bash #!/bin/bash set -Eeuo pipefail # Track background processes PIDS=() cleanup() { log_info "Shutting down..." # Terminate all background processes for pid in "${PIDS[@]}"; do if kill -0 "$pid" 2>/dev/null; then kill -TERM "$pid" 2>/dev/null || true fi done # Wait for graceful shutdown for pid in "${PIDS[@]}"; do wait "$pid" 2>/dev/null || true done } trap cleanup SIGTERM SIGINT # Start background tasks background_task & PIDS+=($!) another_task & PIDS+=($!) # Wait for all background processes wait ``` ### Pattern 7: Safe File Operations ```bash #!/bin/bash set -Eeuo pipefail # Use -i flag to move safely without overwriting safe_move() { local -r source="$1" local -r dest="$2" if [[ ! -e "$source" ]]; then echo "ERROR: Source does not exist: $source" >&2 return 1 fi if [[ -e "$dest" ]]; then echo "ERROR: Destination already exists: $dest" >&2 return 1 fi mv "$source" "$dest" } # Safe directory cleanup safe_rmdir() { local -r dir="$1" if [[ ! -d "$dir" ]]; then echo "ERROR: Not a directory: $dir" >&2 return 1 fi # Use -I flag to prompt before rm (BSD/GNU compatible) rm -rI -- "$dir" } # Atomic file writes atomic_write() { local -r target="$1" local -r tmpfile tmpfile=$(mktemp) || return 1 # Write to temp file first cat > "$tmpfile" # Atomic rename mv "$tmpfile" "$target" } ``` ### Pattern 8: Idempotent Script Design ```bash #!/bin/bash set -Eeuo pipefail # Check if resource already exists ensure_directory() { local -r dir="$1" if [[ -d "$dir" ]]; then log_info "Directory already exists: $dir" return 0 fi mkdir -p "$dir" || { log_error "Failed to create directory: $dir" return 1 } log_info "Created directory: $dir" } # Ensure configuration state ensure_config() { local -r config_file="$1" local -r default_value="$2" if [[ ! -f "$config_file" ]]; then echo "$default_value" > "$config_file" log_info "Created config: $config_file" fi } # Rerunning script multiple times should be safe ensure_directory "/var/cache/myapp" ensure_config "/etc/myapp/config" "DEBUG=false" ``` ### Pattern 9: Safe Command Substitution ```bash #!/bin/bash set -Eeuo pipefail # Use $() instead of backticks name=$(<"$file") # Modern, safe variable assignment from file output=$(command -v python3) # Get command location safely # Handle command substitution with error checking result=$(command -v node) || { log_error "node command not found" return 1 } # For multiple lines mapfile -t lines < <(grep "pattern" "$file") # NUL-safe iteration while IFS= read -r -d '' file; do echo "Processing: $file" done < <(find /path -type f -print0) ``` ### Pattern 10: Dry-Run Support ```bash #!/bin/bash set -Eeuo pipefail DRY_RUN="${DRY_RUN:-false}" run_cmd() { if [[ "$DRY_RUN" == "true" ]]; then echo "[DRY RUN] Would execute: $*" return 0 fi "$@" } # Usage run_cmd cp "$source" "$dest" run_cmd rm "$file" run_cmd chown "$owner" "$target" ``` ## Advanced Defensive Techniques ### Named Parameters Pattern ```bash #!/bin/bash set -Eeuo pipefail process_data() { local input_file="" local output_dir="" local format="json" # Parse named parameters while [[ $# -gt 0 ]]; do case "$1" in --input=*) input_file="${1#*=}" ;; --output=*) output_dir="${1#*=}" ;; --format=*) format="${1#*=}" ;; *) echo "ERROR: Unknown parameter: $1" >&2 return 1 ;; esac shift done # Validate required parameters [[ -n "$input_file" ]] || { echo "ERROR: --input is required" >&2; return 1; } [[ -n "$output_dir" ]] || { echo "ERROR: --output is required" >&2; return 1; } } ``` ### Dependency Checking ```bash #!/bin/bash set -Eeuo pipefail check_dependencies() { local -a missing_deps=() local -a required=("jq" "curl" "git") for cmd in "${required[@]}"; do if ! command -v "$cmd" &>/dev/null; then missing_deps+=("$cmd") fi done if [[ ${#missing_deps[@]} -gt 0 ]]; then echo "ERROR: Missing required commands: ${missing_deps[*]}" >&2 return 1 fi } check_dependencies ``` ## Best Practices Summary 1. **Always use strict mode** - `set -Eeuo pipefail` 2. **Quote all variables** - `"$variable"` prevents word splitting 3. **Use [[]] conditionals** - More robust than [ ] 4. **Implement error trapping** - Catch and handle errors gracefully 5. **Validate all inputs** - Check file existence, permissions, formats 6. **Use functions for reusability** - Prefix with meaningful names 7. **Implement structured logging** - Include timestamps and levels 8. **Support dry-run mode** - Allow users to preview changes 9. **Handle temporary files safely** - Use mktemp, cleanup with trap 10. **Design for idempotency** - Scripts should be safe to rerun 11. **Document requirements** - List dependencies and minimum versions 12. **Test error paths** - Ensure error handling works correctly 13. **Use `command -v`** - Safer than `which` for checking executables 14. **Prefer printf over echo** - More predictable across systems ## Resources - **Bash Strict Mode**: http://redsymbol.net/articles/unofficial-bash-strict-mode/ - **Google Shell Style Guide**: https://google.github.io/styleguide/shellguide.html - **Defensive BASH Programming**: https://www.lifepipe.net/
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

market-sizing-analysis

This skill should be used when the user asks to "calculate TAM",

business
⭐1
# Market Sizing Analysis Comprehensive market sizing methodologies for calculating Total Addressable Market (TAM), Serviceable Available Market (SAM), and Serviceable Obtainable Market (SOM) for startup opportunities. ## Overview Market sizing provides the foundation for startup strategy, fundraising, and business planning. Calculate market opportunity using three complementary methodologies: top-down (industry reports), bottom-up (customer segment calculations), and value theory (willingness to pay). ## Core Concepts ### The Three-Tier Market Framework **TAM (Total Addressable Market)** - Total revenue opportunity if achieving 100% market share - Defines the universe of potential customers - Used for long-term vision and market validation - Example: All email marketing software revenue globally **SAM (Serviceable Available Market)** - Portion of TAM targetable with current product/service - Accounts for geographic, segment, or capability constraints - Represents realistic addressable opportunity - Example: AI-powered email marketing for e-commerce in North America **SOM (Serviceable Obtainable Market)** - Realistic market share achievable in 3-5 years - Accounts for competition, resources, and market dynamics - Used for financial projections and fundraising - Example: 2-5% of SAM based on competitive landscape ### When to Use Each Methodology **Top-Down Analysis** - Use when established market research exists - Best for mature, well-defined markets - Validates market existence and growth - Starts with industry reports and narrows down **Bottom-Up Analysis** - Use when targeting specific customer segments - Best for new or niche markets - Most credible for investors - Builds from customer data and pricing **Value Theory** - Use when creating new market categories - Best for disruptive innovations - Estimates based on value creation - Calculates willingness to pay for problem solution ## Three-Methodology Framework ### Methodology 1: Top-Down Analysis Start with total market size and narrow to addressable segments. **Process:** 1. Identify total market category from research reports 2. Apply geographic filters (target regions) 3. Apply segment filters (target industries/customers) 4. Calculate competitive positioning adjustments **Formula:** ``` TAM = Total Market Category Size SAM = TAM × Geographic % × Segment % SOM = SAM × Realistic Capture Rate (2-5%) ``` **When to use:** Established markets with available research (e.g., SaaS, fintech, e-commerce) **Strengths:** Quick, uses credible data, validates market existence **Limitations:** May overestimate for new categories, less granular ### Methodology 2: Bottom-Up Analysis Build market size from customer segment calculations. **Process:** 1. Define target customer segments 2. Estimate number of potential customers per segment 3. Determine average revenue per customer 4. Calculate realistic penetration rates **Formula:** ``` TAM = Σ (Segment Size × Annual Revenue per Customer) SAM = TAM × (Segments You Can Serve / Total Segments) SOM = SAM × Realistic Penetration Rate (Year 3-5) ``` **When to use:** B2B, niche markets, specific customer segments **Strengths:** Most credible for investors, granular, defensible **Limitations:** Requires detailed customer research, time-intensive ### Methodology 3: Value Theory Calculate based on value created and willingness to pay. **Process:** 1. Identify problem being solved 2. Quantify current cost of problem (time, money, inefficiency) 3. Calculate value of solution (savings, gains, efficiency) 4. Estimate willingness to pay (typically 10-30% of value) 5. Multiply by addressable customer base **Formula:** ``` Value per Customer = Problem Cost × % Solved by Solution Price per Customer = Value × Willingness to Pay % (10-30%) TAM = Total Potential Customers × Price per Customer SAM = TAM × % Meeting Buy Criteria SOM = SAM × Realistic Adoption Rate ``` **When to use:** New categories, disruptive innovations, unclear existing markets **Strengths:** Shows value creation, works for new markets **Limitations:** Requires assumptions, harder to validate ## Step-by-Step Process ### Step 1: Define the Market Clearly specify what market is being measured. **Questions to answer:** - What problem is being solved? - Who are the target customers? - What's the product/service category? - What's the geographic scope? - What's the time horizon? **Example:** - Problem: E-commerce companies struggle with email marketing automation - Customers: E-commerce stores with >$1M annual revenue - Category: AI-powered email marketing software - Geography: North America initially, global expansion - Horizon: 3-5 year opportunity ### Step 2: Gather Data Sources Identify credible data for calculations. **Top-Down Sources:** - Industry research reports (Gartner, Forrester, IDC) - Government statistics (Census, BLS, trade associations) - Public company filings and earnings - Market research firms (Statista, CB Insights, PitchBook) **Bottom-Up Sources:** - Customer interviews and surveys - Sales data and CRM records - Industry databases (LinkedIn, ZoomInfo, Crunchbase) - Competitive intelligence - Academic research **Value Theory Sources:** - Customer problem quantification - Time/cost studies - ROI case studies - Pricing research and willingness-to-pay surveys ### Step 3: Calculate TAM Apply chosen methodology to determine total market. **For Top-Down:** 1. Find total category size from research 2. Document data source and year 3. Apply growth rate if needed 4. Validate with multiple sources **For Bottom-Up:** 1. Count total potential customers 2. Calculate average annual revenue per customer 3. Multiply to get TAM 4. Break down by segment **For Value Theory:** 1. Quantify total addressable customer base 2. Calculate value per customer 3. Estimate pricing based on value 4. Multiply for TAM ### Step 4: Calculate SAM Narrow TAM to serviceable addressable market. **Apply Filters:** - Geographic constraints (regions you can serve) - Product limitations (features you currently have) - Customer requirements (size, industry, use case) - Distribution channel access - Regulatory or compliance restrictions **Formula:** ``` SAM = TAM × (% matching all filters) ``` **Example:** - TAM: $10B global email marketing - Geographic filter: 40% (North America) - Product filter: 30% (e-commerce focus) - Feature filter: 60% (need AI capabilities) - SAM = $10B × 0.40 × 0.30 × 0.60 = $720M ### Step 5: Calculate SOM Determine realistic obtainable market share. **Consider:** - Current market share of competitors - Typical market share for new entrants (2-5%) - Resources available (funding, team, time) - Go-to-market effectiveness - Competitive advantages - Time to achieve (3-5 years typically) **Conservative Approach:** ``` SOM (Year 3) = SAM × 2% SOM (Year 5) = SAM × 5% ``` **Example:** - SAM: $720M - Year 3 SOM: $720M × 2% = $14.4M - Year 5 SOM: $720M × 5% = $36M ### Step 6: Validate and Triangulate Cross-check using multiple methods. **Validation Techniques:** 1. Compare top-down and bottom-up results (should be within 30%) 2. Check against public company revenues in space 3. Validate customer count assumptions 4. Sense-check pricing assumptions 5. Review with industry experts 6. Compare to similar market categories **Red Flags:** - TAM that's too small (< $1B for VC-backed startups) - TAM that's too large (unsupported by data) - SOM that's too aggressive (> 10% in 5 years for new entrant) - Inconsistency between methodologies (> 50% difference) ## Industry-Specific Considerations ### SaaS Markets **Key Metrics:** - Number of potential businesses in target segment - Average contract value (ACV) - Typical market penetration rates - Expansion revenue potential **TAM Calculation:** ``` TAM = Total Target Companies × Average ACV × (1 + Expansion Rate) ``` ### Marketplace Markets **Key Metrics:** - Gross Merchandise Value (GMV) of category - Take rate (% of GMV you capture) - Total transactions or users **TAM Calculation:** ``` TAM = Total Category GMV × Expected Take Rate ``` ### Consumer Markets **Key Metrics:** - Total addressable users/households - Average revenue per user (ARPU) - Engagement frequency **TAM Calculation:** ``` TAM = Total Users × ARPU × Purchase Frequency per Year ``` ### B2B Services **Key Metrics:** - Number of target companies by size/industry - Average project value or retainer - Typical buying frequency **TAM Calculation:** ``` TAM = Total Target Companies × Average Deal Size × Deals per Year ``` ## Presenting Market Sizing ### For Investors **Structure:** 1. Market definition and problem scope 2. TAM/SAM/SOM with methodology 3. Data sources and assumptions 4. Growth projections and drivers 5. Competitive landscape context **Key Points:** - Lead with bottom-up calculation (most credible) - Show triangulation with top-down - Explain conservative assumptions - Link to revenue projections - Highlight market growth rate ### For Strategy **Structure:** 1. Addressable customer segments 2. Prioritization by opportunity size 3. Entry strategy by segment 4. Expected penetration timeline 5. Resource requirements **Key Points:** - Focus on SAM and SOM - Show segment-level detail - Connect to go-to-market plan - Identify expansion opportunities - Discuss competitive positioning ## Common Mistakes to Avoid **Mistake 1: Confusing TAM with SAM** - Don't claim entire market as addressable - Apply realistic product/geographic constraints - Be honest about serviceable market **Mistake 2: Overly Aggressive SOM** - New entrants rarely capture > 5% in 5 years - Account for competition and resources - Show realistic ramp timeline **Mistake 3: Using Only Top-Down** - Investors prefer bottom-up validation - Top-down alone lacks credibility - Always triangulate with multiple methods **Mistake 4: Cherry-Picking Data** - Use consistent, recent data sources - Don't mix methodologies inappropriately - Document all assumptions clearly **Mistake 5: Ignoring Market Dynamics** - Account for market growth/decline - Consider competitive intensity - Factor in switching costs and barriers ## Additional Resources ### Reference Files For detailed methodologies and frameworks: - **`references/methodology-deep-dive.md`** - Comprehensive guide to each methodology with step-by-step worksheets - **`references/data-sources.md`** - Curated list of market research sources, databases, and tools - **`references/industry-templates.md`** - Specific templates for SaaS, marketplace, consumer, B2B, and fintech markets ### Example Files Working examples with complete calculations: - **`examples/saas-market-sizing.md`** - Complete TAM/SAM/SOM for a B2B SaaS product - **`examples/marketplace-sizing.md`** - Marketplace platform market opportunity calculation - **`examples/value-theory-example.md`** - Value-based market sizing for disruptive innovation Use these examples as templates for your own market sizing analysis. Each includes real numbers, data sources, and assumptions documented clearly. ## Quick Start To perform market sizing analysis: 1. **Define the market** - Problem, customers, category, geography 2. **Choose methodology** - Bottom-up (preferred) or top-down + triangulation 3. **Gather data** - Industry reports, customer data, competitive intelligence 4. **Calculate TAM** - Apply methodology formula 5. **Narrow to SAM** - Apply product, geographic, segment filters 6. **Estimate SOM** - 2-5% realistic capture rate 7. **Validate** - Cross-check with alternative methods 8. **Document** - Show methodology, sources, assumptions 9. **Present** - Structure for audience (investors, strategy, operations) For detailed step-by-step guidance on each methodology, reference the files in `references/` directory. For complete worked examples, see `examples/` directory.
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

startup-financial-modeling

This skill should be used when the user asks to "create financial

business
⭐1
# Startup Financial Modeling Build comprehensive 3-5 year financial models with revenue projections, cost structures, cash flow analysis, and scenario planning for early-stage startups. ## Overview Financial modeling provides the quantitative foundation for startup strategy, fundraising, and operational planning. Create realistic projections using cohort-based revenue modeling, detailed cost structures, and scenario analysis to support decision-making and investor presentations. ## Core Components ### Revenue Model **Cohort-Based Projections:** Build revenue from customer acquisition and retention by cohort. **Formula:** ``` MRR = Σ (Cohort Size × Retention Rate × ARPU) ARR = MRR × 12 ``` **Key Inputs:** - Monthly new customer acquisitions - Customer retention rates by month - Average revenue per user (ARPU) - Pricing and packaging assumptions - Expansion revenue (upsells, cross-sells) ### Cost Structure **Operating Expenses Categories:** 1. **Cost of Goods Sold (COGS)** - Hosting and infrastructure - Payment processing fees - Customer support (variable portion) - Third-party services per customer 2. **Sales & Marketing (S&M)** - Customer acquisition cost (CAC) - Marketing programs and advertising - Sales team compensation - Marketing tools and software 3. **Research & Development (R&D)** - Engineering team compensation - Product management - Design and UX - Development tools and infrastructure 4. **General & Administrative (G&A)** - Executive team - Finance, legal, HR - Office and facilities - Insurance and compliance ### Cash Flow Analysis **Components:** - Beginning cash balance - Cash inflows (revenue, fundraising) - Cash outflows (operating expenses, CapEx) - Ending cash balance - Monthly burn rate - Runway (months of cash remaining) **Formula:** ``` Runway = Current Cash Balance / Monthly Burn Rate Monthly Burn = Monthly Revenue - Monthly Expenses ``` ### Headcount Planning **Role-Based Hiring Plan:** Track headcount by department and role. **Key Metrics:** - Fully-loaded cost per employee - Revenue per employee - Headcount by department (% of total) **Typical Ratios (Early-Stage SaaS):** - Engineering: 40-50% - Sales & Marketing: 25-35% - G&A: 10-15% - Customer Success: 5-10% ## Financial Model Structure ### Three-Scenario Framework **Conservative Scenario (P10):** - Slower customer acquisition - Lower pricing or conversion - Higher churn rates - Extended sales cycles - Used for cash management **Base Scenario (P50):** - Most likely outcomes - Realistic assumptions - Primary planning scenario - Used for board reporting **Optimistic Scenario (P90):** - Faster growth - Better unit economics - Lower churn - Used for upside planning ### Time Horizon **Detailed Projections: 3 Years** - Monthly detail for Year 1 - Monthly detail for Year 2 - Quarterly detail for Year 3 **High-Level Projections: Years 4-5** - Annual projections - Key metrics only - Support long-term planning ## Step-by-Step Process ### Step 1: Define Business Model Clarify revenue model and pricing. **SaaS Model:** - Subscription pricing tiers - Annual vs. monthly contracts - Free trial or freemium approach - Expansion revenue strategy **Marketplace Model:** - GMV projections - Take rate (% of transactions) - Buyer and seller economics - Transaction frequency **Transactional Model:** - Transaction volume - Revenue per transaction - Frequency and seasonality ### Step 2: Build Revenue Projections Use cohort-based methodology for accuracy. **Monthly Customer Acquisition:** Define new customers acquired each month. **Retention Curve:** Model customer retention over time. **Typical SaaS Retention:** - Month 1: 100% - Month 3: 90% - Month 6: 85% - Month 12: 75% - Month 24: 70% **Revenue Calculation:** For each cohort, calculate retained customers × ARPU for each month. ### Step 3: Model Cost Structure Break down costs by category and behavior. **Fixed vs. Variable:** - Fixed: Salaries, software, rent - Variable: Hosting, payment processing, support **Scaling Assumptions:** - COGS as % of revenue - S&M as % of revenue (CAC payback) - R&D growth rate - G&A as % of total expenses ### Step 4: Create Hiring Plan Model headcount growth by role and department. **Inputs:** - Starting headcount - Hiring velocity by role - Fully-loaded compensation by role - Benefits and taxes (typically 1.3-1.4x salary) **Example:** ``` Engineer: $150K salary × 1.35 = $202K fully-loaded Sales Rep: $100K OTE × 1.30 = $130K fully-loaded ``` ### Step 5: Project Cash Flow Calculate monthly cash position and runway. **Monthly Cash Flow:** ``` Beginning Cash + Revenue Collected (consider payment terms) - Operating Expenses Paid - CapEx = Ending Cash ``` **Runway Calculation:** ``` If Ending Cash < 0: Funding Need = Negative Cash Balance Runway = 0 Else: Runway = Ending Cash / Average Monthly Burn ``` ### Step 6: Calculate Key Metrics Track metrics that matter for stage. **Revenue Metrics:** - MRR / ARR - Growth rate (MoM, YoY) - Revenue by segment or cohort **Unit Economics:** - CAC (Customer Acquisition Cost) - LTV (Lifetime Value) - CAC Payback Period - LTV / CAC Ratio **Efficiency Metrics:** - Burn multiple (Net Burn / Net New ARR) - Magic number (Net New ARR / S&M Spend) - Rule of 40 (Growth % + Profit Margin %) **Cash Metrics:** - Monthly burn rate - Runway (months) - Cash efficiency ### Step 7: Scenario Analysis Create three scenarios with different assumptions. **Variable Assumptions:** - Customer acquisition rate (±30%) - Churn rate (±20%) - Average contract value (±15%) - CAC (±25%) **Fixed Assumptions:** - Pricing structure - Core operating expenses - Hiring plan (adjust timing, not roles) ## Business Model Templates ### SaaS Financial Model **Revenue Drivers:** - New MRR (customers × ARPU) - Expansion MRR (upsells) - Contraction MRR (downgrades) - Churned MRR (lost customers) **Key Ratios:** - Gross margin: 75-85% - S&M as % revenue: 40-60% (early stage) - CAC payback: < 12 months - Net retention: 100-120% **Example Projection:** ``` Year 1: $500K ARR, 50 customers, $100K MRR by Dec Year 2: $2.5M ARR, 200 customers, $208K MRR by Dec Year 3: $8M ARR, 600 customers, $667K MRR by Dec ``` ### Marketplace Financial Model **Revenue Drivers:** - GMV (Gross Merchandise Value) - Take rate (% of GMV) - Net revenue = GMV × Take rate **Key Ratios:** - Take rate: 10-30% depending on category - CAC for buyers vs. sellers - Contribution margin: 60-70% **Example Projection:** ``` Year 1: $5M GMV, 15% take rate = $750K revenue Year 2: $20M GMV, 15% take rate = $3M revenue Year 3: $60M GMV, 15% take rate = $9M revenue ``` ### E-Commerce Financial Model **Revenue Drivers:** - Traffic (visitors) - Conversion rate - Average order value (AOV) - Purchase frequency **Key Ratios:** - Gross margin: 40-60% - Contribution margin: 20-35% - CAC payback: 3-6 months ### Services / Agency Financial Model **Revenue Drivers:** - Billable hours or projects - Hourly rate or project fee - Utilization rate - Team capacity **Key Ratios:** - Gross margin: 50-70% - Utilization: 70-85% - Revenue per employee ## Fundraising Integration ### Funding Scenario Modeling **Pre-Money Valuation:** Based on metrics and comparables. **Dilution:** ``` Post-Money = Pre-Money + Investment Dilution % = Investment / Post-Money ``` **Use of Funds:** Allocate funding to extend runway and achieve milestones. **Example:** ``` Raise: $5M at $20M pre-money Post-Money: $25M Dilution: 20% Use of Funds: - Product Development: $2M (40%) - Sales & Marketing: $2M (40%) - G&A and Operations: $0.5M (10%) - Working Capital: $0.5M (10%) ``` ### Milestone-Based Planning **Identify Key Milestones:** - Product launch - First $1M ARR - Break-even on CAC - Series A fundraise **Funding Amount:** Ensure runway to achieve next milestone + 6 months buffer. ## Common Pitfalls **Pitfall 1: Overly Optimistic Revenue** - New startups rarely hit aggressive projections - Use conservative customer acquisition assumptions - Model realistic churn rates **Pitfall 2: Underestimating Costs** - Add 20% buffer to expense estimates - Include fully-loaded compensation - Account for software and tools **Pitfall 3: Ignoring Cash Flow Timing** - Revenue ≠ cash (payment terms) - Expenses paid before revenue collected - Model cash conversion carefully **Pitfall 4: Static Headcount** - Hiring takes time (3-6 months to fill roles) - Ramp time for productivity (3-6 months) - Account for attrition (10-15% annually) **Pitfall 5: Not Scenario Planning** - Single scenario is never accurate - Always model conservative case - Plan for what you'll do if base case fails ## Model Validation **Sanity Checks:** - [ ] Revenue growth rate is achievable (3x in Year 2, 2x in Year 3) - [ ] Unit economics are realistic (LTV/CAC > 3, payback < 18 months) - [ ] Burn multiple is reasonable (< 2.0 in Year 2-3) - [ ] Headcount scales with revenue (revenue per employee growing) - [ ] Gross margin is appropriate for business model - [ ] S&M spending aligns with CAC and growth targets **Benchmark Against Peers:** Compare key metrics to similar companies at similar stage. **Investor Feedback:** Share model with advisors or investors for feedback on assumptions. ## Additional Resources ### Reference Files For detailed model structures and advanced techniques: - **`references/model-templates.md`** - Complete financial model templates by business model - **`references/unit-economics.md`** - Deep dive on CAC, LTV, payback, and efficiency metrics - **`references/fundraising-scenarios.md`** - Modeling funding rounds and dilution ### Example Files Working financial models with formulas: - **`examples/saas-financial-model.md`** - Complete 3-year SaaS model with cohort analysis - **`examples/marketplace-model.md`** - Marketplace GMV and take rate projections - **`examples/scenario-analysis.md`** - Three-scenario framework with sensitivities ## Quick Start To create a startup financial model: 1. **Define business model** - Revenue drivers and pricing 2. **Project revenue** - Cohort-based with retention 3. **Model costs** - COGS, S&M, R&D, G&A by month 4. **Plan headcount** - Hiring by role and department 5. **Calculate cash flow** - Revenue - expenses = burn/runway 6. **Compute metrics** - CAC, LTV, burn multiple, runway 7. **Create scenarios** - Conservative, base, optimistic 8. **Validate assumptions** - Sanity check and benchmark 9. **Integrate fundraising** - Model funding rounds and milestones For complete templates and formulas, reference the `references/` and `examples/` files.
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

responsive-design

Implement modern responsive layouts using container queries, fluid

coding
⭐1
# Responsive Design Master modern responsive design techniques to create interfaces that adapt seamlessly across all screen sizes and device contexts. ## When to Use This Skill - Implementing mobile-first responsive layouts - Using container queries for component-based responsiveness - Creating fluid typography and spacing scales - Building complex layouts with CSS Grid and Flexbox - Designing breakpoint strategies for design systems - Implementing responsive images and media - Creating adaptive navigation patterns - Building responsive tables and data displays ## Core Capabilities ### 1. Container Queries - Component-level responsiveness independent of viewport - Container query units (cqi, cqw, cqh) - Style queries for conditional styling - Fallbacks for browser support ### 2. Fluid Typography & Spacing - CSS clamp() for fluid scaling - Viewport-relative units (vw, vh, dvh) - Fluid type scales with min/max bounds - Responsive spacing systems ### 3. Layout Patterns - CSS Grid for 2D layouts - Flexbox for 1D distribution - Intrinsic layouts (content-based sizing) - Subgrid for nested grid alignment ### 4. Breakpoint Strategy - Mobile-first media queries - Content-based breakpoints - Design token integration - Feature queries (@supports) ## Quick Reference ### Modern Breakpoint Scale ```css /* Mobile-first breakpoints */ /* Base: Mobile (< 640px) */ @media (min-width: 640px) { /* sm: Landscape phones, small tablets */ } @media (min-width: 768px) { /* md: Tablets */ } @media (min-width: 1024px) { /* lg: Laptops, small desktops */ } @media (min-width: 1280px) { /* xl: Desktops */ } @media (min-width: 1536px) { /* 2xl: Large desktops */ } /* Tailwind CSS equivalent */ /* sm: @media (min-width: 640px) */ /* md: @media (min-width: 768px) */ /* lg: @media (min-width: 1024px) */ /* xl: @media (min-width: 1280px) */ /* 2xl: @media (min-width: 1536px) */ ``` ## Key Patterns ### Pattern 1: Container Queries ```css /* Define a containment context */ .card-container { container-type: inline-size; container-name: card; } /* Query the container, not the viewport */ @container card (min-width: 400px) { .card { display: grid; grid-template-columns: 200px 1fr; gap: 1rem; } .card-image { aspect-ratio: 1; } } @container card (min-width: 600px) { .card { grid-template-columns: 250px 1fr; } .card-title { font-size: 1.5rem; } } /* Container query units */ .card-title { /* 5% of container width, clamped between 1rem and 2rem */ font-size: clamp(1rem, 5cqi, 2rem); } ``` ```tsx // React component with container queries function ResponsiveCard({ title, image, description }) { return ( <div className="@container"> <article className="flex flex-col @md:flex-row @md:gap-4"> <img src={image} alt="" className="w-full @md:w-48 @lg:w-64 aspect-video @md:aspect-square object-cover" /> <div className="p-4 @md:p-0"> <h2 className="text-lg @md:text-xl @lg:text-2xl font-semibold"> {title} </h2> <p className="mt-2 text-muted-foreground @md:line-clamp-3"> {description} </p> </div> </article> </div> ); } ``` ### Pattern 2: Fluid Typography ```css /* Fluid type scale using clamp() */ :root { /* Min size, preferred (fluid), max size */ --text-xs: clamp(0.75rem, 0.7rem + 0.25vw, 0.875rem); --text-sm: clamp(0.875rem, 0.8rem + 0.375vw, 1rem); --text-base: clamp(1rem, 0.9rem + 0.5vw, 1.125rem); --text-lg: clamp(1.125rem, 1rem + 0.625vw, 1.25rem); --text-xl: clamp(1.25rem, 1rem + 1.25vw, 1.5rem); --text-2xl: clamp(1.5rem, 1.25rem + 1.25vw, 2rem); --text-3xl: clamp(1.875rem, 1.5rem + 1.875vw, 2.5rem); --text-4xl: clamp(2.25rem, 1.75rem + 2.5vw, 3.5rem); } /* Usage */ h1 { font-size: var(--text-4xl); } h2 { font-size: var(--text-3xl); } h3 { font-size: var(--text-2xl); } p { font-size: var(--text-base); } /* Fluid spacing scale */ :root { --space-xs: clamp(0.25rem, 0.2rem + 0.25vw, 0.5rem); --space-sm: clamp(0.5rem, 0.4rem + 0.5vw, 0.75rem); --space-md: clamp(1rem, 0.8rem + 1vw, 1.5rem); --space-lg: clamp(1.5rem, 1.2rem + 1.5vw, 2.5rem); --space-xl: clamp(2rem, 1.5rem + 2.5vw, 4rem); } ``` ```tsx // Utility function for fluid values function fluidValue( minSize: number, maxSize: number, minWidth = 320, maxWidth = 1280, ) { const slope = (maxSize - minSize) / (maxWidth - minWidth); const yAxisIntersection = -minWidth * slope + minSize; return `clamp(${minSize}rem, ${yAxisIntersection.toFixed(4)}rem + ${(slope * 100).toFixed(4)}vw, ${maxSize}rem)`; } // Generate fluid type scale const fluidTypeScale = { sm: fluidValue(0.875, 1), base: fluidValue(1, 1.125), lg: fluidValue(1.25, 1.5), xl: fluidValue(1.5, 2), "2xl": fluidValue(2, 3), }; ``` ### Pattern 3: CSS Grid Responsive Layout ```css /* Auto-fit grid - items wrap automatically */ .grid-auto { display: grid; grid-template-columns: repeat(auto-fit, minmax(min(300px, 100%), 1fr)); gap: 1.5rem; } /* Auto-fill grid - maintains empty columns */ .grid-auto-fill { display: grid; grid-template-columns: repeat(auto-fill, minmax(250px, 1fr)); gap: 1rem; } /* Responsive grid with named areas */ .page-layout { display: grid; grid-template-areas: "header" "main" "sidebar" "footer"; gap: 1rem; } @media (min-width: 768px) { .page-layout { grid-template-columns: 1fr 300px; grid-template-areas: "header header" "main sidebar" "footer footer"; } } @media (min-width: 1024px) { .page-layout { grid-template-columns: 250px 1fr 300px; grid-template-areas: "header header header" "nav main sidebar" "footer footer footer"; } } .header { grid-area: header; } .main { grid-area: main; } .sidebar { grid-area: sidebar; } .footer { grid-area: footer; } ``` ```tsx // Responsive grid component function ResponsiveGrid({ children, minItemWidth = "250px", gap = "1.5rem" }) { return ( <div className="grid" style={{ gridTemplateColumns: `repeat(auto-fit, minmax(min(${minItemWidth}, 100%), 1fr))`, gap, }} > {children} </div> ); } // Usage with Tailwind function ProductGrid({ products }) { return ( <div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-3 xl:grid-cols-4 gap-4 md:gap-6"> {products.map((product) => ( <ProductCard key={product.id} product={product} /> ))} </div> ); } ``` ### Pattern 4: Responsive Navigation ```tsx function ResponsiveNav({ items }) { const [isOpen, setIsOpen] = useState(false); return ( <nav className="relative"> {/* Mobile menu button */} <button className="lg:hidden p-2" onClick={() => setIsOpen(!isOpen)} aria-expanded={isOpen} aria-controls="nav-menu" > <span className="sr-only">Toggle navigation</span> {isOpen ? <X /> : <Menu />} </button> {/* Navigation links */} <ul id="nav-menu" className={cn( // Base: hidden on mobile "absolute top-full left-0 right-0 bg-background border-b", "flex flex-col", // Mobile: slide down isOpen ? "flex" : "hidden", // Desktop: always visible, horizontal "lg:static lg:flex lg:flex-row lg:border-0 lg:bg-transparent", )} > {items.map((item) => ( <li key={item.href}> <a href={item.href} className={cn( "block px-4 py-3", "lg:px-3 lg:py-2", "hover:bg-muted lg:hover:bg-transparent lg:hover:text-primary", )} > {item.label} </a> </li> ))} </ul> </nav> ); } ``` ### Pattern 5: Responsive Images ```tsx // Responsive image with art direction function ResponsiveHero() { return ( <picture> {/* Art direction: different crops for different screens */} <source media="(min-width: 1024px)" srcSet="/hero-wide.webp" type="image/webp" /> <source media="(min-width: 768px)" srcSet="/hero-medium.webp" type="image/webp" /> <source srcSet="/hero-mobile.webp" type="image/webp" /> {/* Fallback */} <img src="/hero-mobile.jpg" alt="Hero image description" className="w-full h-auto" loading="eager" fetchpriority="high" /> </picture> ); } // Responsive image with srcset for resolution switching function ProductImage({ product }) { return ( <img src={product.image} srcSet={` ${product.image}?w=400 400w, ${product.image}?w=800 800w, ${product.image}?w=1200 1200w `} sizes="(max-width: 640px) 100vw, (max-width: 1024px) 50vw, 33vw" alt={product.name} className="w-full h-auto object-cover" loading="lazy" /> ); } ``` ### Pattern 6: Responsive Tables ```tsx // Responsive table with horizontal scroll function ResponsiveTable({ data, columns }) { return ( <div className="w-full overflow-x-auto"> <table className="w-full min-w-[600px]"> <thead> <tr> {columns.map((col) => ( <th key={col.key} className="text-left p-3"> {col.label} </th> ))} </tr> </thead> <tbody> {data.map((row, i) => ( <tr key={i} className="border-t"> {columns.map((col) => ( <td key={col.key} className="p-3"> {row[col.key]} </td> ))} </tr> ))} </tbody> </table> </div> ); } // Card-based table for mobile function ResponsiveDataTable({ data, columns }) { return ( <> {/* Desktop table */} <table className="hidden md:table w-full"> {/* ... standard table */} </table> {/* Mobile cards */} <div className="md:hidden space-y-4"> {data.map((row, i) => ( <div key={i} className="border rounded-lg p-4 space-y-2"> {columns.map((col) => ( <div key={col.key} className="flex justify-between"> <span className="font-medium text-muted-foreground"> {col.label} </span> <span>{row[col.key]}</span> </div> ))} </div> ))} </div> </> ); } ``` ## Viewport Units ```css /* Standard viewport units */ .full-height { height: 100vh; /* May cause issues on mobile */ } /* Dynamic viewport units (recommended for mobile) */ .full-height-dynamic { height: 100dvh; /* Accounts for mobile browser UI */ } /* Small viewport (minimum) */ .min-full-height { min-height: 100svh; } /* Large viewport (maximum) */ .max-full-height { max-height: 100lvh; } /* Viewport-relative font sizing */ .hero-title { /* 5vw with min/max bounds */ font-size: clamp(2rem, 5vw, 4rem); } ``` ## Best Practices 1. **Mobile-First**: Start with mobile styles, enhance for larger screens 2. **Content Breakpoints**: Set breakpoints based on content, not devices 3. **Fluid Over Fixed**: Use fluid values for typography and spacing 4. **Container Queries**: Use for component-level responsiveness 5. **Test Real Devices**: Simulators don't catch all issues 6. **Performance**: Optimize images, lazy load off-screen content 7. **Touch Targets**: Maintain 44x44px minimum on mobile 8. **Logical Properties**: Use inline/block for internationalization ## Common Issues - **Horizontal Overflow**: Content breaking out of viewport - **Fixed Widths**: Using px instead of relative units - **Viewport Height**: 100vh issues on mobile browsers - **Font Size**: Text too small on mobile - **Touch Targets**: Buttons too small to tap accurately - **Aspect Ratio**: Images squishing or stretching - **Z-Index Stacking**: Overlays breaking on different screens ## Resources - [CSS Container Queries](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_container_queries) - [Utopia Fluid Type Calculator](https://utopia.fyi/type/calculator/) - [Every Layout](https://every-layout.dev/) - [Responsive Images Guide](https://web.dev/responsive-images/) - [CSS Grid Garden](https://cssgridgarden.com/)
👍0
👁️0
🤖 Auto-discovered
🤖system prompt•7 months ago

code-review-excellence

Master effective code review practices to provide constructive

coding
⭐1
# Code Review Excellence Transform code reviews from gatekeeping to knowledge sharing through constructive feedback, systematic analysis, and collaborative improvement. ## When to Use This Skill - Reviewing pull requests and code changes - Establishing code review standards for teams - Mentoring junior developers through reviews - Conducting architecture reviews - Creating review checklists and guidelines - Improving team collaboration - Reducing code review cycle time - Maintaining code quality standards ## Core Principles ### 1. The Review Mindset **Goals of Code Review:** - Catch bugs and edge cases - Ensure code maintainability - Share knowledge across team - Enforce coding standards - Improve design and architecture - Build team culture **Not the Goals:** - Show off knowledge - Nitpick formatting (use linters) - Block progress unnecessarily - Rewrite to your preference ### 2. Effective Feedback **Good Feedback is:** - Specific and actionable - Educational, not judgmental - Focused on the code, not the person - Balanced (praise good work too) - Prioritized (critical vs nice-to-have) ```markdown ❌ Bad: "This is wrong." ✅ Good: "This could cause a race condition when multiple users access simultaneously. Consider using a mutex here." ❌ Bad: "Why didn't you use X pattern?" ✅ Good: "Have you considered the Repository pattern? It would make this easier to test. Here's an example: [link]" ❌ Bad: "Rename this variable." ✅ Good: "[nit] Consider `userCount` instead of `uc` for clarity. Not blocking if you prefer to keep it." ``` ### 3. Review Scope **What to Review:** - Logic correctness and edge cases - Security vulnerabilities - Performance implications - Test coverage and quality - Error handling - Documentation and comments - API design and naming - Architectural fit **What Not to Review Manually:** - Code formatting (use Prettier, Black, etc.) - Import organization - Linting violations - Simple typos ## Review Process ### Phase 1: Context Gathering (2-3 minutes) ```markdown Before diving into code, understand: 1. Read PR description and linked issue 2. Check PR size (>400 lines? Ask to split) 3. Review CI/CD status (tests passing?) 4. Understand the business requirement 5. Note any relevant architectural decisions ``` ### Phase 2: High-Level Review (5-10 minutes) ```markdown 1. **Architecture & Design** - Does the solution fit the problem? - Are there simpler approaches? - Is it consistent with existing patterns? - Will it scale? 2. **File Organization** - Are new files in the right places? - Is code grouped logically? - Are there duplicate files? 3. **Testing Strategy** - Are there tests? - Do tests cover edge cases? - Are tests readable? ``` ### Phase 3: Line-by-Line Review (10-20 minutes) ```markdown For each file: 1. **Logic & Correctness** - Edge cases handled? - Off-by-one errors? - Null/undefined checks? - Race conditions? 2. **Security** - Input validation? - SQL injection risks? - XSS vulnerabilities? - Sensitive data exposure? 3. **Performance** - N+1 queries? - Unnecessary loops? - Memory leaks? - Blocking operations? 4. **Maintainability** - Clear variable names? - Functions doing one thing? - Complex code commented? - Magic numbers extracted? ``` ### Phase 4: Summary & Decision (2-3 minutes) ```markdown 1. Summarize key concerns 2. Highlight what you liked 3. Make clear decision: - ✅ Approve - 💬 Comment (minor suggestions) - 🔄 Request Changes (must address) 4. Offer to pair if complex ``` ## Review Techniques ### Technique 1: The Checklist Method ```markdown ## Security Checklist - [ ] User input validated and sanitized - [ ] SQL queries use parameterization - [ ] Authentication/authorization checked - [ ] Secrets not hardcoded - [ ] Error messages don't leak info ## Performance Checklist - [ ] No N+1 queries - [ ] Database queries indexed - [ ] Large lists paginated - [ ] Expensive operations cached - [ ] No blocking I/O in hot paths ## Testing Checklist - [ ] Happy path tested - [ ] Edge cases covered - [ ] Error cases tested - [ ] Test names are descriptive - [ ] Tests are deterministic ``` ### Technique 2: The Question Approach Instead of stating problems, ask questions to encourage thinking: ```markdown ❌ "This will fail if the list is empty." ✅ "What happens if `items` is an empty array?" ❌ "You need error handling here." ✅ "How should this behave if the API call fails?" ❌ "This is inefficient." ✅ "I see this loops through all users. Have we considered the performance impact with 100k users?" ``` ### Technique 3: Suggest, Don't Command ````markdown ## Use Collaborative Language ❌ "You must change this to use async/await" ✅ "Suggestion: async/await might make this more readable: `typescript async function fetchUser(id: string) { const user = await db.query('SELECT * FROM users WHERE id = ?', id); return user; } ` What do you think?" ❌ "Extract this into a function" ✅ "This logic appears in 3 places. Would it make sense to extract it into a shared utility function?" ```` ### Technique 4: Differentiate Severity ```markdown Use labels to indicate priority: 🔴 [blocking] - Must fix before merge 🟡 [important] - Should fix, discuss if disagree 🟢 [nit] - Nice to have, not blocking 💡 [suggestion] - Alternative approach to consider 📚 [learning] - Educational comment, no action needed 🎉 [praise] - Good work, keep it up! Example: "🔴 [blocking] This SQL query is vulnerable to injection. Please use parameterized queries." "🟢 [nit] Consider renaming `data` to `userData` for clarity." "🎉 [praise] Excellent test coverage! This will catch edge cases." ``` ## Language-Specific Patterns ### Python Code Review ```python # Check for Python-specific issues # ❌ Mutable default arguments def add_item(item, items=[]): # Bug! Shared across calls items.append(item) return items # ✅ Use None as default def add_item(item, items=None): if items is None: items = [] items.append(item) return items # ❌ Catching too broad try: result = risky_operation() except: # Catches everything, even KeyboardInterrupt! pass # ✅ Catch specific exceptions try: result = risky_operation() except ValueError as e: logger.error(f"Invalid value: {e}") raise # ❌ Using mutable class attributes class User: permissions = [] # Shared across all instances! # ✅ Initialize in __init__ class User: def __init__(self): self.permissions = [] ``` ### TypeScript/JavaScript Code Review ```typescript // Check for TypeScript-specific issues // ❌ Using any defeats type safety function processData(data: any) { // Avoid any return data.value; } // ✅ Use proper types interface DataPayload { value: string; } function processData(data: DataPayload) { return data.value; } // ❌ Not handling async errors async function fetchUser(id: string) { const response = await fetch(`/api/users/${id}`); return response.json(); // What if network fails? } // ✅ Handle errors properly async function fetchUser(id: string): Promise<User> { try { const response = await fetch(`/api/users/${id}`); if (!response.ok) { throw new Error(`HTTP ${response.status}`); } return await response.json(); } catch (error) { console.error('Failed to fetch user:', error); throw error; } } // ❌ Mutation of props function UserProfile({ user }: Props) { user.lastViewed = new Date(); // Mutating prop! return <div>{user.name}</div>; } // ✅ Don't mutate props function UserProfile({ user, onView }: Props) { useEffect(() => { onView(user.id); // Notify parent to update }, [user.id]); return <div>{user.name}</div>; } ``` ## Advanced Review Patterns ### Pattern 1: Architectural Review ```markdown When reviewing significant changes: 1. **Design Document First** - For large features, request design doc before code - Review design with team before implementation - Agree on approach to avoid rework 2. **Review in Stages** - First PR: Core abstractions and interfaces - Second PR: Implementation - Third PR: Integration and tests - Easier to review, faster to iterate 3. **Consider Alternatives** - "Have we considered using [pattern/library]?" - "What's the tradeoff vs. the simpler approach?" - "How will this evolve as requirements change?" ``` ### Pattern 2: Test Quality Review ```typescript // ❌ Poor test: Implementation detail testing test('increments counter variable', () => { const component = render(<Counter />); const button = component.getByRole('button'); fireEvent.click(button); expect(component.state.counter).toBe(1); // Testing internal state }); // ✅ Good test: Behavior testing test('displays incremented count when clicked', () => { render(<Counter />); const button = screen.getByRole('button', { name: /increment/i }); fireEvent.click(button); expect(screen.getByText('Count: 1')).toBeInTheDocument(); }); // Review questions for tests: // - Do tests describe behavior, not implementation? // - Are test names clear and descriptive? // - Do tests cover edge cases? // - Are tests independent (no shared state)? // - Can tests run in any order? ``` ### Pattern 3: Security Review ```markdown ## Security Review Checklist ### Authentication & Authorization - [ ] Is authentication required where needed? - [ ] Are authorization checks before every action? - [ ] Is JWT validation proper (signature, expiry)? - [ ] Are API keys/secrets properly secured? ### Input Validation - [ ] All user inputs validated? - [ ] File uploads restricted (size, type)? - [ ] SQL queries parameterized? - [ ] XSS protection (escape output)? ### Data Protection - [ ] Passwords hashed (bcrypt/argon2)? - [ ] Sensitive data encrypted at rest? - [ ] HTTPS enforced for sensitive data? - [ ] PII handled according to regulations? ### Common Vulnerabilities - [ ] No eval() or similar dynamic execution? - [ ] No hardcoded secrets? - [ ] CSRF protection for state-changing operations? - [ ] Rate limiting on public endpoints? ``` ## Giving Difficult Feedback ### Pattern: The Sandwich Method (Modified) ```markdown Traditional: Praise + Criticism + Praise (feels fake) Better: Context + Specific Issue + Helpful Solution Example: "I noticed the payment processing logic is inline in the controller. This makes it harder to test and reuse. [Specific Issue] The calculateTotal() function mixes tax calculation, discount logic, and database queries, making it difficult to unit test and reason about. [Helpful Solution] Could we extract this into a PaymentService class? That would make it testable and reusable. I can pair with you on this if helpful." ``` ### Handling Disagreements ```markdown When author disagrees with your feedback: 1. **Seek to Understand** "Help me understand your approach. What led you to choose this pattern?" 2. **Acknowledge Valid Points** "That's a good point about X. I hadn't considered that." 3. **Provide Data** "I'm concerned about performance. Can we add a benchmark to validate the approach?" 4. **Escalate if Needed** "Let's get [architect/senior dev] to weigh in on this." 5. **Know When to Let Go** If it's working and not a critical issue, approve it. Perfection is the enemy of progress. ``` ## Best Practices 1. **Review Promptly**: Within 24 hours, ideally same day 2. **Limit PR Size**: 200-400 lines max for effective review 3. **Review in Time Blocks**: 60 minutes max, take breaks 4. **Use Review Tools**: GitHub, GitLab, or dedicated tools 5. **Automate What You Can**: Linters, formatters, security scans 6. **Build Rapport**: Emoji, praise, and empathy matter 7. **Be Available**: Offer to pair on complex issues 8. **Learn from Others**: Review others' review comments ## Common Pitfalls - **Perfectionism**: Blocking PRs for minor style preferences - **Scope Creep**: "While you're at it, can you also..." - **Inconsistency**: Different standards for different people - **Delayed Reviews**: Letting PRs sit for days - **Ghosting**: Requesting changes then disappearing - **Rubber Stamping**: Approving without actually reviewing - **Bike Shedding**: Debating trivial details extensively ## Templates ### PR Review Comment Template ```markdown ## Summary [Brief overview of what was reviewed] ## Strengths - [What was done well] - [Good patterns or approaches] ## Required Changes 🔴 [Blocking issue 1] 🔴 [Blocking issue 2] ## Suggestions 💡 [Improvement 1] 💡 [Improvement 2] ## Questions ❓ [Clarification needed on X] ❓ [Alternative approach consideration] ## Verdict ✅ Approve after addressing required changes ``` ## Resources - **references/code-review-best-practices.md**: Comprehensive review guidelines - **references/common-bugs-checklist.md**: Language-specific bugs to watch for - **references/security-review-guide.md**: Security-focused review checklist - **assets/pr-review-template.md**: Standard review comment template - **assets/review-checklist.md**: Quick reference checklist - **scripts/pr-analyzer.py**: Analyze PR complexity and suggest reviewers
👍0
👁️3
🤖 Auto-discovered