Testing Requirements
Test standards for contributions
AIWG Testing Requirements
This document defines testing requirements for contributions to the AIWG toolkit.
Regression Testing Requirements
When Full Regression Testing is Required
Significant changes to AIWG require full regression testing and validation across all supported providers before merging. "Significant changes" include:
| Change Type | Full Regression Required | Reason |
|---|---|---|
| New agents | Yes | Agents deploy to all provider formats |
| New commands | Yes | Commands may have provider-specific behavior |
| New skills | Yes | Skills are deployed with frameworks |
| CLI changes | Yes | CLI handles all provider deployments |
| Manifest changes | Yes | Manifests control deployment behavior |
| Framework modifications | Yes | Frameworks are core to all providers |
| Addon modifications | Yes | Addons deploy across providers |
| Template changes | No (unless structural) | Templates are provider-agnostic |
| Documentation only | No | No deployment impact |
Provider Validation Matrix
All significant changes must be validated against these providers:
| Provider | Deployment Target | Validation Command |
|---|---|---|
| Claude Code | `.claude/agents/`, `.claude/commands/` | `aiwg use sdlc` |
| GitHub Copilot | `.github/agents/` | `aiwg use sdlc --provider copilot` |
| Warp | `WARP.md` | `aiwg use sdlc --provider warp` |
| Factory AI | `.factory/droids/` | `aiwg use sdlc --provider factory` |
| OpenCode | `.opencode/agent/` | `aiwg use sdlc --provider opencode` |
| Cursor | `.cursor/rules/` | `aiwg use sdlc --provider cursor` |
| OpenAI/Codex | `.codex/agents/` | `aiwg use sdlc --provider codex` |
Validation Checklist
Before submitting a PR with significant changes:
# 1. Run full test suite
npm test
# 2. Lint all modified markdown files
npm exec markdownlint-cli2 "path/to/modified/**/*.md"
# 3. Deploy to each provider and verify
aiwg use sdlc --provider claude --force
aiwg use sdlc --provider factory --force
aiwg use sdlc --provider openai --force
# 4. Verify new content is deployed correctly
ls .claude/commands/your-new-command.md
ls .factory/commands/your-new-command.md
ls .codex/commands/your-new-command.md
# 5. Check manifest is updated
cat agentic/code/addons/your-addon/manifest.json | jq '.commands'Known Provider Limitations
General-Purpose Commands (aiwg-utils)
Current behavior: Commands from `aiwg-utils` addon currently only deploy to Claude Code (`.claude/commands/`), not to other providers.
Impact: Commands like `/aiwg-refresh`, `/aiwg-regenerate`, `/workspace-*` are only available in Claude Code sessions.
Tracking: This is a known CLI limitation to be addressed in future releases.
Provider-Specific Features
| Feature | Claude | Factory | OpenAI | Cursor | Warp |
|---|---|---|---|---|---|
| Agents | ✓ | ✓ | ✓ | ✓ | ✓ |
| SDLC Commands | ✓ | ✓ | ✓ | ✓ | ✓ |
| Utils Commands | ✓ | ✗ | ✗ | ✗ | ✗ |
| Skills | ✓ | ✓ | ✓ | ✓ | ✓ |
| MCP Integration | ✓ | ✓ | ✗ | ✗ | ✗ |
Test Categories
Unit Tests
- Location: `test/unit/`
- Run: `npx vitest run test/unit/`
- Coverage: Core CLI functionality, parsers, validators
Integration Tests
- Location: `test/integration/`
- Run: `npx vitest run test/integration/`
- Coverage: Provider deployments, CLI end-to-end
Provider-Specific Tests
- Claude Code: `test/integration/claude-code-deployment.test.ts`
- Factory AI: `test/integration/factory-deployment.test.ts`
- OpenAI/Codex: `test/integration/openai-deployment.test.ts`
Minimum Test Coverage
| Category | Minimum Coverage | Current |
|---|---|---|
| Line coverage | 80% | Enforced via CI |
| Branch coverage | 75% | Enforced via CI |
| Function coverage | 90% | Enforced via CI |
PR Requirements
For Significant Changes
1. All tests passing: 0 failures allowed 2. Provider validation: Verified deployment to at least Claude, Factory, OpenAI 3. Lint clean: No markdown lint errors in modified files 4. Manifest updated: New commands/agents/skills added to manifest 5. README updated: New features documented in relevant README
For Documentation-Only Changes
1. Lint clean: No markdown lint errors 2. Links valid: All internal links resolve
Automated CI Checks
The following are enforced automatically on all PRs:
- `npm test` - Full test suite
- `markdownlint-cli2` - Markdown linting
- Type checking via TypeScript
- Coverage thresholds
Manual Verification
Some checks require manual verification before merge:
- [ ] Deployed to Claude Code and verified new content appears
- [ ] Deployed to at least one other provider (Factory or OpenAI)
- [ ] Tested command/agent functionality in actual session
- [ ] Verified no regressions in existing functionality