test: add review guidelines skill for testing bot guideline adherence#61
test: add review guidelines skill for testing bot guideline adherence#61factory-nizar wants to merge 3 commits intodevfrom
Conversation
Adds a .factory/skills/review-guidelines/SKILL.md with a mix of naming conventions, error handling rules, TypeScript-specific rules, structural constraints, comment formatting rules, import ordering, and security rules to validate that the review bot picks up and enforces custom guidelines. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
|
Droid finished @factory-nizar's task —— View job ✅ Review complete — no issues found. This PR adds a |
…l branch - Point droid-review.yml at feat/review-guidelines-to-skill so the bot uses the updated code that invokes the review-guidelines skill - Add example-violations.ts with intentional violations: - Single-letter variables (i, x) - Booleans without is/has/should/can prefix - Silent catch block - Use of `any` type - Comments ending with periods - Else after return / deep nesting (>3 levels) - No explicit return type on exported function - Bare process.env non-null assertion - Unescaped shell interpolation in execSync - Default import instead of named - `type` instead of `interface` for object shape - TODO without author tag - Logging sensitive token data Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
There was a problem hiding this comment.
This PR adds a review guidelines skill file (.factory/skills/review-guidelines/SKILL.md) containing repository-specific conventions for the review bot to enforce. No functional code changes are included — only a Markdown documentation file — so there are no bugs or issues to flag.
|
@droid review |
|
Droid finished @factory-nizar's task —— View job 🔍 Review complete — 6 issues found (3× P0, 3× P1).
Pin the workflow action to an immutable ref and relocate the unsafe fixture out of |
There was a problem hiding this comment.
This PR is primarily about adding review-guideline scaffolding for bot testing, but it also changes CI to use a mutable action ref and adds an intentionally unsafe TypeScript fixture under src/. Pin the workflow action to an immutable ref and relocate/contain the unsafe fixture so it cannot be shipped or accidentally imported at runtime.
The droid config is read from the checked-out repo (PR branch), not from the action ref. Without this, the subagent cannot invoke the review-guidelines skill. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
|
@droid review |
|
Droid finished @factory-nizar's task —— View job 🔍 Review complete — 1 issue found (1× P1).
The |
|
Droid finished @factory-nizar's task —— View job 🔍 Review validated — 2 issues approved. Both are concrete TypeScript typecheck breaks in |
| try { | ||
| const raw = require(path); | ||
| return raw; | ||
| } catch (e) { |
There was a problem hiding this comment.
[P0] Typecheck break: unused catch binding under noUnusedLocals
This file is included by tsconfig ("include": ["src/**/*"]) and the repo enables "noUnusedLocals": true; the catch (e) binding is never read, so tsc --noEmit will fail with an unused-local error. Fix by removing the binding (catch { ... }) or by using/renaming it (e.g., catch (_error) { ... }).
| } | ||
|
|
||
| // VIOLATION: default import, not organized in 3 groups | ||
| import path from "path"; |
There was a problem hiding this comment.
[P0] Typecheck break: default import from Node builtin without esModuleInterop
tsconfig.json does not enable esModuleInterop/allowSyntheticDefaultImports, so import path from "path" will fail typechecking (Node builtins are export = modules in @types/node). Use import * as path from "node:path" or switch to named imports (e.g., import { join, resolve } from "node:path") to keep the file typecheckable.
| const raw = require(path); | ||
| return raw; | ||
| } catch (e) { | ||
| // swallowed silently. |
There was a problem hiding this comment.
[P0] Typecheck break: unused catch binding under noUnusedLocals
This file is included by tsconfig ("include": ["src/**/*"]) and the repo enables "noUnusedLocals": true; the catch (e) binding is never read, so tsc --noEmit will fail with an unused-local error. Fix by removing the binding (catch { ... }) or by using/renaming it (e.g., catch (_error) { ... }).
|
|
||
| // VIOLATION: default import, not organized in 3 groups | ||
| import path from "path"; | ||
| import { resolve } from "path"; |
There was a problem hiding this comment.
[P0] Typecheck break: default import from Node builtin without esModuleInterop
tsconfig.json does not enable esModuleInterop/allowSyntheticDefaultImports, so import path from "path" will fail typechecking (Node builtins are export = modules in @types/node). Use import * as path from "node:path" or switch to named imports (e.g., import { join, resolve } from "node:path") to keep the file typecheckable.
Purpose
Adds a
.factory/skills/review-guidelines/SKILL.mdto this repo so we can test whether the review bot picks up and enforces custom guidelines.Guidelines included
The file contains a deliberate mix of conventions the bot should flag:
is/has/should/can), no single-letter vars, SCREAMING_SNAKE_CASE constantsinterfaceovertypefor object shapes, noany, explicit return types on exports, preferreadonlyprocess.env.X!Several of these rules are intentionally strict or unusual (e.g., no periods in comments, no single-letter loop vars) to make it easy to tell if the bot is actually reading and applying the guidelines vs just doing its default review.