Factory AI Testing Automation Hooks

Factory AI Testing Automation Hooks

Prompt-first procedure: Describe the outcome you want in your agent conversation. The agent should select and load the appropriate AIWG assets, explain material changes, request any needed approval, and report verification evidence. Exact commands and flags appear only in the CLI reference.

This guide covers how to use Factory's lifecycle hooks for testing automation, with AIWG integration patterns. For a general hooks overview, see the Factory Quickstart — Hooks section.

Overview

Factory provides 9 lifecycle hook events. A subset of 7 are well-suited to testing automation: running test suites, enforcing coverage, blocking destructive operations, and capturing coverage artifacts. The remaining two (`Notification`, `SessionEnd`) are general-purpose lifecycle hooks.

Hooks vs AIWG Skills for Testing

MechanismWhen to Use
HooksDeterministic triggers — run tests on every file save, block operations, pre-flight checks. No AI judgment needed.
AIWG SkillsIntelligent workflows — analyze test failures, fix flaky tests, suggest coverage improvements. AI reasoning required.

Use both: hooks handle the mechanical triggers, AIWG skills handle the remediation.

Prerequisites

  • Factory CLI installed and authenticated
  • `jq` (for JSON processing in shell hooks)
  • Test framework installed (`npm test`, `pytest`, `cargo test`, etc.)
  • AIWG deployed to your project: the agent-owned use operation

Quick Setup

Deploy AIWG with testing hooks:

Use AIWG to complete this documented outcome: Deploy AIWG with testing hooks
Have it inspect the current state, explain the plan, ask before material
changes, and report the result with verification evidence.

Then add hook configuration to `~/.factory/settings.json` or `.factory/settings.json`. The sections below cover each pattern.

Hook-by-Hook Guide

1. PostToolUse — Run Tests After Code Changes

What it does: Fires after every tool invocation. Use it to automatically run tests when source files change.

AIWG context: Complements the `/generate-tests` and `/flaky-fix` skills — hooks run tests mechanically; AIWG skills fix them intelligently.

Configuration:

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|MultiEdit|Create",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'if echo \"$FACTORY_TOOL_INPUT\" | jq -e \".file_path\" | grep -qE \"\\.(ts|js|py|rs)$\"; then npm test the silent option 2>&1 | tail -20; fi'"
          }
        ]
      }
    ]
  }
}

With MD5 caching (skip redundant runs):

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Edit|MultiEdit|Create",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'HASH=$(find src -name \"*.ts\" | sort | xargs md5sum 2>/dev/null | md5sum | cut -c1-8); CACHE_FILE=\"/tmp/factory_test_cache_$HASH\"; if [ ! -f \"$CACHE_FILE\" ]; then npm test the silent option && touch \"$CACHE_FILE\"; fi'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Change matcher to specific tools: `"Edit"` (single-file only)
  • Adjust file extension filter to match your project: `\.(go|java|rb)$`
  • Replace `npm test` with your project's test command

2. SessionStart — Pre-flight Checks

What it does: Fires when a new Factory session starts. Use it to ensure a clean testing environment before the droid begins work.

AIWG context: This is where the agent-owned refresh operation fits to check for AIWG framework updates before a long session.

Configuration:

{
  "hooks": {
    "SessionStart": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'the agent-owned refresh operation the dry-run option; npm ci the silent option 2>/dev/null || npm install the silent option; echo \"Pre-flight complete\"'"
          }
        ]
      }
    ]
  }
}

With coverage baseline (establish baseline before changes):

{
  "hooks": {
    "SessionStart": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'npm run test:coverage the silent option 2>/dev/null | grep -E \"^(All files|Statements)\" > /tmp/factory_coverage_baseline.txt; echo \"Coverage baseline captured\"'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Add environment-specific checks: `node the version option`, `docker ps`
  • Seed test databases for integration test environments
  • Load environment variables from `.env.test`

3. Stop — Full Test Suite on Task Completion

What it does: Fires when the droid finishes its primary task. Use it to run a complete test suite and report results before the session ends.

AIWG context: Works with the `/auto-test-execution` and `/test-coverage` AIWG skills — the hook confirms the baseline passes; AIWG skills investigate failures.

Configuration:

{
  "hooks": {
    "Stop": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "npm test 2>&1 | tail -30"
          }
        ]
      }
    ]
  }
}

With configurable minimum coverage (uses `DROID_MIN_COVERAGE` env var):

{
  "hooks": {
    "Stop": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'MIN_COV=${DROID_MIN_COVERAGE:-80}; RESULT=$(npm run test:coverage the silent option 2>/dev/null | grep \"All files\" | awk \"{print \\$10}\"); echo \"Coverage: ${RESULT}% (min: ${MIN_COV}%)\"; [ $(echo \"${RESULT} >= ${MIN_COV}\" | bc) -eq 1 ] && echo \"PASS\" || echo \"WARN: Coverage below threshold\"'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Set `DROID_MIN_COVERAGE=90` in your shell profile for stricter enforcement
  • Add `|| echo "WARN: Tests failed"` to make failures non-blocking (hook failure blocks the session)
  • Timeout: add `timeout 120` prefix for test suites that may hang

4. PreToolUse — Block Dangerous Operations

What it does: Fires before every tool call. Use it to prevent destructive operations that would corrupt test state.

AIWG context: Acts as a safety layer beneath the `/security-gate` AIWG skill — the hook enforces hard blocks mechanically.

Configuration (block deletion of test files):

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'CMD=$(echo \"$FACTORY_TOOL_INPUT\" | jq -r \".command // empty\"); if echo \"$CMD\" | grep -qE \"^(rm|del|rmdir).*test\"; then echo \"BLOCKED: Cannot delete test files\"; exit 1; fi'"
          }
        ]
      }
    ]
  }
}

Block force-push during active test runs:

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'CMD=$(echo \"$FACTORY_TOOL_INPUT\" | jq -r \".command // empty\"); if echo \"$CMD\" | grep -qE \"git push.*the force option\"; then echo \"BLOCKED: Force push not allowed\"; exit 1; fi'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Expand the blocked pattern list for your project
  • Use exit code 0 for warnings (non-blocking), exit 1 for hard blocks
  • Log blocked attempts: `echo "$(date): BLOCKED $CMD" >> /tmp/factory_blocks.log`

5. SubagentStop — Validate Subagent Results

What it does: Fires when a subagent finishes. Use it to run targeted tests on files the subagent modified and accumulate a coverage trace.

AIWG context: Enables AIWG trace logging — the `/regression-report` and `/cross-task-learner` skills can read these traces.

Configuration:

{
  "hooks": {
    "SubagentStop": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'echo \"{\\\"timestamp\\\": \\\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\\\", \\\"event\\\": \\\"subagent_stop\\\"}\" >> .aiwg/working/factory-trace.jsonl; npm run test:changed the silent option 2>/dev/null || echo \"No changed-file tests available\"'"
          }
        ]
      }
    ]
  }
}

With AIWG smart hooks (conditional based on `AIWG_SMART_HOOKS` flag):

{
  "hooks": {
    "SubagentStop": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'if [ \"${AIWG_SMART_HOOKS:-true}\" = \"true\" ]; then npm run test:changed the silent option 2>&1 | tail -15; fi'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Set `AIWG_SMART_HOOKS=false` to disable during rapid iteration sessions
  • Replace `test:changed` with `jest --onlyChanged` or `pytest the lf option` for your framework
  • Extend the JSONL log schema to capture task metadata from `$FACTORY_SESSION_ID`

6. UserPromptSubmit — Parse Test Targets from Prompts

What it does: Fires when the user submits a prompt. Use it to extract test file paths mentioned in the prompt and set up targeted test runs.

AIWG context: Works with the `/setup-tdd` and `/test-sync` AIWG skills — the hook handles the mechanical extraction; skills handle test generation.

Configuration:

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'PROMPT=$(echo \"$FACTORY_PROMPT\" | tr -d \"\\
\"); FILES=$(echo \"$PROMPT\" | grep -oE \"[a-zA-Z0-9_/.-]+\\.(ts|js|py|go|rs)\"); if [ -n \"$FILES\" ]; then echo \"Test targets detected: $FILES\" > /tmp/factory_test_targets.txt; fi'"
          }
        ]
      }
    ]
  }
}

Coverage ignore pattern extraction:

{
  "hooks": {
    "UserPromptSubmit": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'PROMPT=$(echo \"$FACTORY_PROMPT\"); if echo \"$PROMPT\" | grep -qi \"skip.*test\\|no test\\|ignore.*coverage\"; then echo \"AIWG_SKIP_COVERAGE=true\" >> /tmp/factory_session_env.sh; echo \"Coverage enforcement suspended for this prompt\"; fi'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Extend file extension list: `\\.(ts|js|py|go|rs|java|rb|cpp)$`
  • Parse ticket/issue references to look up test requirements
  • Set environment variables read by later hooks

7. PreCompact — Preserve Test Artifacts

What it does: Fires before Factory compacts the context window. Use it to persist test results and coverage data that would otherwise be lost from context.

AIWG context: Coverage artifacts preserved here can be read by `/regression-baseline` and `/regression-report` AIWG skills in subsequent sessions.

Configuration:

{
  "hooks": {
    "PreCompact": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'STAMP=$(date +%Y%m%d_%H%M%S); mkdir -p .aiwg/working/test-snapshots; [ -f coverage/coverage-summary.json ] && cp coverage/coverage-summary.json \".aiwg/working/test-snapshots/coverage-${STAMP}.json\"; [ -f /tmp/factory_coverage_baseline.txt ] && cp /tmp/factory_coverage_baseline.txt \".aiwg/working/test-snapshots/baseline-${STAMP}.txt\"; echo \"Test artifacts preserved\"'"
          }
        ]
      }
    ]
  }
}

With configurable coverage per directory (using `DROID_MIN_COVERAGE`):

{
  "hooks": {
    "PreCompact": [
      {
        "matcher": "*",
        "hooks": [
          {
            "type": "command",
            "command": "bash -c 'STAMP=$(date +%Y%m%d_%H%M%S); mkdir -p .aiwg/working/test-snapshots; npm run test:coverage the silent option 2>/dev/null && cp coverage/coverage-summary.json \".aiwg/working/test-snapshots/pre-compact-${STAMP}.json\" 2>/dev/null; echo \"Coverage snapshot saved\"'"
          }
        ]
      }
    ]
  }
}

Customization options:

  • Adjust snapshot directory to match your artifact retention policy
  • Add `git stash` before compaction to preserve unstaged test changes
  • Emit a coverage summary to stdout — it appears in the droid's compact context

Configuration Reference

settings.json Structure

{
  "hooks": {
    "<EventName>": [
      {
        "matcher": "<ToolNameOrGlob>",
        "hooks": [
          {
            "type": "command",
            "command": "<shell command>",
            "timeout": 120
          }
        ]
      }
    ]
  }
}

Scopes:

  • `~/.factory/settings.json` — applies to all projects
  • `.factory/settings.json` — project-local, committed to git (team-shared)

Project-local overrides user-level when both are present.

Matcher Patterns

PatternMatches
`"*"`All tool invocations
`"Edit"`Single-file edit only
`"Edit\MultiEdit\Create"`All file-write operations
`"Bash"`Shell command execution
`"Edit:*.ts"`Edit of TypeScript files only (path glob)

Timeout

Default: 60 seconds. For test suites:

{
  "type": "command",
  "command": "npm test",
  "timeout": 120
}

Set `"timeout": 0` for non-blocking async execution (hook result is ignored).

Environment Variables

VariableDefaultPurpose
`DROID_MIN_COVERAGE``80`Minimum coverage threshold (%) for Stop hook enforcement
`AIWG_SMART_HOOKS``true`Enable/disable AIWG-aware hook logic

Set in your shell profile:

export DROID_MIN_COVERAGE=90
export AIWG_SMART_HOOKS=true

AIWG vs Vanilla Factory: What Changes

FeatureVanilla FactoryAIWG-Enhanced
Test runnerShell command onlyShell command + `/auto-test-execution` skill for failure analysis
Coverage enforcementManual threshold check`DROID_MIN_COVERAGE` env var + AIWG artifact snapshots
Failure remediationRe-prompt manually`/flaky-fix` and `/test-sync` skills read trace logs
Cross-session coverageLost on session endPersisted to `.aiwg/working/test-snapshots/` via PreCompact
Subagent trackingNot trackedJSONL trace in `.aiwg/working/factory-trace.jsonl`

Best Practices

Async Execution for Non-Blocking Workflows

Long test suites can block the droid mid-task. For non-critical hooks, run async:

# Async (fire-and-forget, hook doesn't block)
"command": "npm test > /tmp/factory_test_out.txt 2>&1 &"

Use synchronous (blocking) hooks only for hard gates (Stop, PreToolUse blocks).

MD5 Caching to Skip Redundant Runs

Avoid re-running tests when source hasn't changed:

HASH=$(find src -name "*.ts" | sort | xargs md5sum | md5sum | cut -c1-8)
CACHE_FILE="/tmp/factory_test_cache_$HASH"
[ ! -f "$CACHE_FILE" ] && npm test && touch "$CACHE_FILE"

Configurable Coverage per Directory

Use `jest --collectCoverageFrom` or `pytest the cov option=<module>` for targeted coverage:

# Frontend-only coverage
npm test -- the coverage option --collectCoverageFrom="src/ui/**"

# Backend-only coverage  
npm test -- the coverage option --collectCoverageFrom="src/api/**"

Coverage Ignore Patterns

Configure in your test framework, not in hooks:

// jest.config.json
{
  "coveragePathIgnorePatterns": [
    "/node_modules/",
    "/test/fixtures/",
    "/.aiwg/"
  ]
}

Troubleshooting

Hook not firing Check `hooksDisabled` in settings:

/settings
# Verify hooksDisabled is false or absent

Coverage false positives from AIWG working files Add `.aiwg/` to your coverage ignore list in jest/pytest config (see above).

Environment discrepancies between hook and test run Hooks run in a minimal shell environment. Use your project's test scripts:

# Preferred — uses project's npm/package.json test env
"command": "npm test"

# Avoid — may miss PATH, env vars
"command": "node the experimental-vm-modules option jest"

Hook timeout on large test suites Increase the timeout:

{ "type": "command", "command": "npm test", "timeout": 180 }

Or split into unit tests (fast, synchronous) and integration tests (async background).

SubagentStop fires too frequently Use the `AIWG_SMART_HOOKS` flag to disable during rapid iteration:

export AIWG_SMART_HOOKS=false

Cross-References