Factory AI Testing Automation Hooks

Factory AI Testing Automation Hooks

This guide covers how to use Factory's lifecycle hooks for testing automation, with AIWG integration patterns. For a general hooks overview, see the Factory Quickstart — Hooks section.

Overview

Factory provides 9 lifecycle hook events. A subset of 7 are well-suited to testing automation: running test suites, enforcing coverage, blocking destructive operations, and capturing coverage artifacts. The remaining two (`Notification`, `SessionEnd`) are general-purpose lifecycle hooks.

Hooks vs AIWG Skills for Testing

MechanismWhen to Use
HooksDeterministic triggers — run tests on every file save, block operations, pre-flight checks. No AI judgment needed.
AIWG SkillsIntelligent workflows — analyze test failures, fix flaky tests, suggest coverage improvements. AI reasoning required.

Use both: hooks handle the mechanical triggers, AIWG skills handle the remediation.

Prerequisites

  • Factory CLI installed and authenticated
  • `jq` (for JSON processing in shell hooks)
  • Test framework installed (`npm test`, `pytest`, `cargo test`, etc.)
  • AIWG deployed to your project: `aiwg use sdlc --provider factory`

Quick Setup

Deploy AIWG with testing hooks:

aiwg use sdlc --provider factory

Then add hook configuration to `~/.factory/settings.json` or `.factory/settings.json`. The sections below cover each pattern.

Hook-by-Hook Guide

1. PostToolUse — Run Tests After Code Changes

What it does: Fires after every tool invocation. Use it to automatically run tests when source files change.

AIWG context: Complements the `/generate-tests` and `/flaky-fix` skills — hooks run tests mechanically; AIWG skills fix them intelligently.

Configuration:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|MultiEdit|Create",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'if echo \"$FACTORY_TOOL_INPUT\" | jq -e \".file_path\" | grep -qE \"\\.(ts|js|py|rs)$\"; then npm test --silent 2>&1 | tail -20; fi'"
          }
        ]
      }
    ]
  }
}

With MD5 caching (skip redundant runs):

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|MultiEdit|Create",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'HASH=$(find src -name \"*.ts\" | sort | xargs md5sum 2>/dev/null | md5sum | cut -c1-8); CACHE_FILE=\"/tmp/factory_test_cache_$HASH\"; if [ ! -f \"$CACHE_FILE\" ]; then npm test --silent && touch \"$CACHE_FILE\"; fi'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Change matcher to specific tools: `"Edit"` (single-file only)
  • Adjust file extension filter to match your project: `\.(go|java|rb)$`
  • Replace `npm test` with your project's test command

2. SessionStart — Pre-flight Checks

What it does: Fires when a new Factory session starts. Use it to ensure a clean testing environment before the droid begins work.

AIWG context: This is where `aiwg refresh --dry-run` fits to check for AIWG framework updates before a long session.

Configuration:

{
  "hooks": {
    "SessionStart": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'aiwg refresh --dry-run; npm ci --silent 2>/dev/null || npm install --silent; echo \"Pre-flight complete\"'"
          }
        ]
      }
    ]
  }
}

With coverage baseline (establish baseline before changes):

{
  "hooks": {
    "SessionStart": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'npm run test:coverage --silent 2>/dev/null | grep -E \"^(All files|Statements)\" > /tmp/factory_coverage_baseline.txt; echo \"Coverage baseline captured\"'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Add environment-specific checks: `node --version`, `docker ps`
  • Seed test databases for integration test environments
  • Load environment variables from `.env.test`

3. Stop — Full Test Suite on Task Completion

What it does: Fires when the droid finishes its primary task. Use it to run a complete test suite and report results before the session ends.

AIWG context: Works with the `/auto-test-execution` and `/test-coverage` AIWG skills — the hook confirms the baseline passes; AIWG skills investigate failures.

Configuration:

{
  "hooks": {
    "Stop": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "npm test 2>&1 | tail -30"
          }
        ]
      }
    ]
  }
}

With configurable minimum coverage (uses `DROID_MIN_COVERAGE` env var):

{
  "hooks": {
    "Stop": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'MIN_COV=${DROID_MIN_COVERAGE:-80}; RESULT=$(npm run test:coverage --silent 2>/dev/null | grep \"All files\" | awk \"{print \\$10}\"); echo \"Coverage: ${RESULT}% (min: ${MIN_COV}%)\"; [ $(echo \"${RESULT} >= ${MIN_COV}\" | bc) -eq 1 ] && echo \"PASS\" || echo \"WARN: Coverage below threshold\"'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Set `DROID_MIN_COVERAGE=90` in your shell profile for stricter enforcement
  • Add `|| echo "WARN: Tests failed"` to make failures non-blocking (hook failure blocks the session)
  • Timeout: add `timeout 120` prefix for test suites that may hang

4. PreToolUse — Block Dangerous Operations

What it does: Fires before every tool call. Use it to prevent destructive operations that would corrupt test state.

AIWG context: Acts as a safety layer beneath the `/security-gate` AIWG skill — the hook enforces hard blocks mechanically.

Configuration (block deletion of test files):

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'CMD=$(echo \"$FACTORY_TOOL_INPUT\" | jq -r \".command // empty\"); if echo \"$CMD\" | grep -qE \"^(rm|del|rmdir).*test\"; then echo \"BLOCKED: Cannot delete test files\"; exit 1; fi'"
          }
        ]
      }
    ]
  }
}

Block force-push during active test runs:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'CMD=$(echo \"$FACTORY_TOOL_INPUT\" | jq -r \".command // empty\"); if echo \"$CMD\" | grep -qE \"git push.*--force\"; then echo \"BLOCKED: Force push not allowed\"; exit 1; fi'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Expand the blocked pattern list for your project
  • Use exit code 0 for warnings (non-blocking), exit 1 for hard blocks
  • Log blocked attempts: `echo "$(date): BLOCKED $CMD" >> /tmp/factory_blocks.log`

5. SubagentStop — Validate Subagent Results

What it does: Fires when a subagent finishes. Use it to run targeted tests on files the subagent modified and accumulate a coverage trace.

AIWG context: Enables AIWG trace logging — the `/regression-report` and `/cross-task-learner` skills can read these traces.

Configuration:

{
  "hooks": {
    "SubagentStop": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'echo \"{\\\"timestamp\\\": \\\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\\\", \\\"event\\\": \\\"subagent_stop\\\"}\" >> .aiwg/working/factory-trace.jsonl; npm run test:changed --silent 2>/dev/null || echo \"No changed-file tests available\"'"
          }
        ]
      }
    ]
  }
}

With AIWG smart hooks (conditional based on `AIWG_SMART_HOOKS` flag):

{
  "hooks": {
    "SubagentStop": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'if [ \"${AIWG_SMART_HOOKS:-true}\" = \"true\" ]; then npm run test:changed --silent 2>&1 | tail -15; fi'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Set `AIWG_SMART_HOOKS=false` to disable during rapid iteration sessions
  • Replace `test:changed` with `jest --onlyChanged` or `pytest --lf` for your framework
  • Extend the JSONL log schema to capture task metadata from `$FACTORY_SESSION_ID`

6. UserPromptSubmit — Parse Test Targets from Prompts

What it does: Fires when the user submits a prompt. Use it to extract test file paths mentioned in the prompt and set up targeted test runs.

AIWG context: Works with the `/setup-tdd` and `/test-sync` AIWG skills — the hook handles the mechanical extraction; skills handle test generation.

Configuration:

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'PROMPT=$(echo \"$FACTORY_PROMPT\" | tr -d \"\\
\"); FILES=$(echo \"$PROMPT\" | grep -oE \"[a-zA-Z0-9_/.-]+\\.(ts|js|py|go|rs)\"); if [ -n \"$FILES\" ]; then echo \"Test targets detected: $FILES\" > /tmp/factory_test_targets.txt; fi'"
          }
        ]
      }
    ]
  }
}

Coverage ignore pattern extraction:

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'PROMPT=$(echo \"$FACTORY_PROMPT\"); if echo \"$PROMPT\" | grep -qi \"skip.*test\\|no test\\|ignore.*coverage\"; then echo \"AIWG_SKIP_COVERAGE=true\" >> /tmp/factory_session_env.sh; echo \"Coverage enforcement suspended for this prompt\"; fi'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Extend file extension list: `\\.(ts|js|py|go|rs|java|rb|cpp)$`
  • Parse ticket/issue references to look up test requirements
  • Set environment variables read by later hooks

7. PreCompact — Preserve Test Artifacts

What it does: Fires before Factory compacts the context window. Use it to persist test results and coverage data that would otherwise be lost from context.

AIWG context: Coverage artifacts preserved here can be read by `/regression-baseline` and `/regression-report` AIWG skills in subsequent sessions.

Configuration:

{
  "hooks": {
    "PreCompact": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'STAMP=$(date +%Y%m%d_%H%M%S); mkdir -p .aiwg/working/test-snapshots; [ -f coverage/coverage-summary.json ] && cp coverage/coverage-summary.json \".aiwg/working/test-snapshots/coverage-${STAMP}.json\"; [ -f /tmp/factory_coverage_baseline.txt ] && cp /tmp/factory_coverage_baseline.txt \".aiwg/working/test-snapshots/baseline-${STAMP}.txt\"; echo \"Test artifacts preserved\"'"
          }
        ]
      }
    ]
  }
}

With configurable coverage per directory (using `DROID_MIN_COVERAGE`):

{
  "hooks": {
    "PreCompact": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'STAMP=$(date +%Y%m%d_%H%M%S); mkdir -p .aiwg/working/test-snapshots; npm run test:coverage --silent 2>/dev/null && cp coverage/coverage-summary.json \".aiwg/working/test-snapshots/pre-compact-${STAMP}.json\" 2>/dev/null; echo \"Coverage snapshot saved\"'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Adjust snapshot directory to match your artifact retention policy
  • Add `git stash` before compaction to preserve unstaged test changes
  • Emit a coverage summary to stdout — it appears in the droid's compact context

Configuration Reference

settings.json Structure

{
  "hooks": {
    "<EventName>": [
      {
        "matcher": "<ToolNameOrGlob>",
        "hooks": [
          {
            "type": "command",
            "command": "<shell command>",
            "timeout": 120
          }
        ]
      }
    ]
  }
}

Scopes:

  • `~/.factory/settings.json` — applies to all projects
  • `.factory/settings.json` — project-local, committed to git (team-shared)

Project-local overrides user-level when both are present.

Matcher Patterns

PatternMatches
`"*"`All tool invocations
`"Edit"`Single-file edit only
`"Edit\MultiEdit\Create"`All file-write operations
`"Bash"`Shell command execution
`"Edit:*.ts"`Edit of TypeScript files only (path glob)

Timeout

Default: 60 seconds. For test suites:

{
  "type": "command",
  "command": "npm test",
  "timeout": 120
}

Set `"timeout": 0` for non-blocking async execution (hook result is ignored).

Environment Variables

VariableDefaultPurpose
`DROID_MIN_COVERAGE``80`Minimum coverage threshold (%) for Stop hook enforcement
`AIWG_SMART_HOOKS``true`Enable/disable AIWG-aware hook logic

Set in your shell profile:

export DROID_MIN_COVERAGE=90
export AIWG_SMART_HOOKS=true

AIWG vs Vanilla Factory: What Changes

FeatureVanilla FactoryAIWG-Enhanced
Test runnerShell command onlyShell command + `/auto-test-execution` skill for failure analysis
Coverage enforcementManual threshold check`DROID_MIN_COVERAGE` env var + AIWG artifact snapshots
Failure remediationRe-prompt manually`/flaky-fix` and `/test-sync` skills read trace logs
Cross-session coverageLost on session endPersisted to `.aiwg/working/test-snapshots/` via PreCompact
Subagent trackingNot trackedJSONL trace in `.aiwg/working/factory-trace.jsonl`

Best Practices

Async Execution for Non-Blocking Workflows

Long test suites can block the droid mid-task. For non-critical hooks, run async:

# Async (fire-and-forget, hook doesn't block)
"command": "npm test > /tmp/factory_test_out.txt 2>&1 &"

Use synchronous (blocking) hooks only for hard gates (Stop, PreToolUse blocks).

MD5 Caching to Skip Redundant Runs

Avoid re-running tests when source hasn't changed:

HASH=$(find src -name "*.ts" | sort | xargs md5sum | md5sum | cut -c1-8)
CACHE_FILE="/tmp/factory_test_cache_$HASH"
[ ! -f "$CACHE_FILE" ] && npm test && touch "$CACHE_FILE"

Configurable Coverage per Directory

Use `jest --collectCoverageFrom` or `pytest --cov=<module>` for targeted coverage:

# Frontend-only coverage
npm test -- --coverage --collectCoverageFrom="src/ui/**"

# Backend-only coverage  
npm test -- --coverage --collectCoverageFrom="src/api/**"

Coverage Ignore Patterns

Configure in your test framework, not in hooks:

// jest.config.json
{
  "coveragePathIgnorePatterns": [
    "/node_modules/",
    "/test/fixtures/",
    "/.aiwg/"
  ]
}

Troubleshooting

Hook not firing Check `hooksDisabled` in settings:

/settings
# Verify hooksDisabled is false or absent

Coverage false positives from AIWG working files Add `.aiwg/` to your coverage ignore list in jest/pytest config (see above).

Environment discrepancies between hook and test run Hooks run in a minimal shell environment. Use your project's test scripts:

# Preferred — uses project's npm/package.json test env
"command": "npm test"

# Avoid — may miss PATH, env vars
"command": "node --experimental-vm-modules jest"

Hook timeout on large test suites Increase the timeout:

{ "type": "command", "command": "npm test", "timeout": 180 }

Or split into unit tests (fast, synchronous) and integration tests (async background).

SubagentStop fires too frequently Use the `AIWG_SMART_HOOKS` flag to disable during rapid iteration:

export AIWG_SMART_HOOKS=false

Cross-References