feat: make AI review verdicts advisory

Removes the separate review check and relies on review labels/comments instead
Marks workflow/tooling failures with review:automation-failed
Updates review guidance to reflect the advisory verdict model
This commit is contained in:
Bohdan Triapitsyn
2026-07-22 10:33:25 +03:00
parent a62731bd6e
commit 733bd37c37
3 changed files with 25 additions and 121 deletions
+20 -116
View File
@@ -17,8 +17,6 @@ concurrency:
jobs: jobs:
review: review:
# Manual comment workflows run on the default branch, so their native job
# check cannot represent the reviewed PR HEAD. Publish that check explicitly.
name: automation name: automation
if: | if: |
github.event_name == 'pull_request_target' || github.event_name == 'pull_request_target' ||
@@ -26,7 +24,6 @@ jobs:
(github.event_name == 'pull_request_review_comment' && github.event.comment.user.login != 'openchamber-bot[bot]' && (github.event.comment.body == '/oc-review' || startsWith(github.event.comment.body, '/oc-review ') || github.event.comment.body == '@openchamber-bot review' || startsWith(github.event.comment.body, '@openchamber-bot review '))) (github.event_name == 'pull_request_review_comment' && github.event.comment.user.login != 'openchamber-bot[bot]' && (github.event.comment.body == '/oc-review' || startsWith(github.event.comment.body, '/oc-review ') || github.event.comment.body == '@openchamber-bot review' || startsWith(github.event.comment.body, '@openchamber-bot review ')))
runs-on: ubuntu-latest runs-on: ubuntu-latest
permissions: permissions:
checks: write
contents: read contents: read
pull-requests: write pull-requests: write
issues: write issues: write
@@ -62,44 +59,10 @@ jobs:
echo "head_repo_owner=$(printf '%s' "$pr_json" | jq -r '.headRepositoryOwner.login')" echo "head_repo_owner=$(printf '%s' "$pr_json" | jq -r '.headRepositoryOwner.login')"
} >> "$GITHUB_OUTPUT" } >> "$GITHUB_OUTPUT"
- name: Start review check
id: review-check
env:
GH_TOKEN: ${{ github.token }}
REVIEW_HEAD_SHA: ${{ steps.pr.outputs.head_sha }}
run: |
check_run="$(jq -n \
--arg head_sha "$REVIEW_HEAD_SHA" \
--arg details_url "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" \
'{
name: "review",
head_sha: $head_sha,
status: "in_progress",
details_url: $details_url,
output: {
title: "OpenChamber review in progress",
summary: "Reviewing the current pull request HEAD."
}
}')"
check_id="$(printf '%s' "$check_run" | gh api \
--method POST \
"repos/${GITHUB_REPOSITORY}/check-runs" \
--input - \
--jq '.id')"
echo "id=$check_id" >> "$GITHUB_OUTPUT"
- name: Generate review app token
id: app-token
uses: actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349 # v2.2.2
with:
app-id: ${{ secrets.OC_REVIEW_APP_ID }}
private-key: ${{ secrets.OC_REVIEW_APP_PRIVATE_KEY }}
- name: Clear review status for draft - name: Clear review status for draft
if: steps.pr.outputs.draft == 'true' if: steps.pr.outputs.draft == 'true'
env: env:
GH_TOKEN: ${{ steps.app-token.outputs.token }} GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ steps.pr.outputs.number }} PR_NUMBER: ${{ steps.pr.outputs.number }}
run: | run: |
remove_args=() remove_args=()
@@ -113,6 +76,14 @@ jobs:
gh pr edit "$PR_NUMBER" "${remove_args[@]}" gh pr edit "$PR_NUMBER" "${remove_args[@]}"
fi fi
- name: Generate review app token
id: app-token
if: steps.pr.outputs.draft == 'false'
uses: actions/create-github-app-token@fee1f7d63c2ff003460e3d139729b119787bc349 # v2.2.2
with:
app-id: ${{ secrets.OC_REVIEW_APP_ID }}
private-key: ${{ secrets.OC_REVIEW_APP_PRIVATE_KEY }}
- name: Check review safety - name: Check review safety
if: steps.pr.outputs.draft == 'false' if: steps.pr.outputs.draft == 'false'
id: safety id: safety
@@ -226,9 +197,7 @@ jobs:
$CHANGED_SENSITIVE_FILES $CHANGED_SENSITIVE_FILES
\`\`\` \`\`\`
Automated review cannot clear changes to its own policy or trust boundary. A maintainer must review and explicitly override this failing check." Automated review cannot clear changes to its own policy or trust boundary. A maintainer must review it directly."
exit 1
- name: Debounce new commits - name: Debounce new commits
if: steps.pr.outputs.draft == 'false' && steps.safety.outputs.safe == 'true' && github.event_name == 'pull_request_target' && github.event.action == 'synchronize' if: steps.pr.outputs.draft == 'false' && steps.safety.outputs.safe == 'true' && github.event_name == 'pull_request_target' && github.event.action == 'synchronize'
@@ -396,88 +365,23 @@ jobs:
echo "- Status: \`$review_label\`" echo "- Status: \`$review_label\`"
} >> "$GITHUB_STEP_SUMMARY" } >> "$GITHUB_STEP_SUMMARY"
if [ "$verdict" != "pass" ]; then - name: Mark automation failure
echo "Review verdict is $verdict; only pass satisfies this check." >&2 if: always() && steps.pr.outputs.draft == 'false' && steps.verdict.outcome != 'success' && steps.safety.outputs.safe != 'false'
exit 1
fi
- name: Publish review check
if: always() && steps.review-check.outputs.id != ''
env: env:
GH_TOKEN: ${{ github.token }} GH_TOKEN: ${{ github.token }}
CHECK_RUN_ID: ${{ steps.review-check.outputs.id }}
PR_NUMBER: ${{ steps.pr.outputs.number }} PR_NUMBER: ${{ steps.pr.outputs.number }}
REVIEW_HEAD_SHA: ${{ steps.pr.outputs.head_sha }} REVIEW_HEAD_SHA: ${{ steps.pr.outputs.head_sha }}
DRAFT: ${{ steps.pr.outputs.draft }}
VERDICT_OUTCOME: ${{ steps.verdict.outcome }}
run: | run: |
current_head="$(gh pr view "$PR_NUMBER" --json headRefOid --jq '.headRefOid')" current_head="$(gh pr view "$PR_NUMBER" --json headRefOid --jq '.headRefOid')"
review_label="$(gh pr view "$PR_NUMBER" --json labels --jq '[.labels[].name | select(startswith("review:"))] | first // ""')"
set_review_status() {
local target_label="$1"
local remove_args=()
while IFS= read -r label; do
case "$label" in
review:*) remove_args+=(--remove-label "$label") ;;
esac
done < <(gh pr view "$PR_NUMBER" --json labels --jq '.labels[].name')
if [ -n "$target_label" ]; then
gh pr edit "$PR_NUMBER" "${remove_args[@]}" --add-label "$target_label"
elif [ "${#remove_args[@]}" -gt 0 ]; then
gh pr edit "$PR_NUMBER" "${remove_args[@]}"
fi
review_label="$target_label"
}
if [ "$current_head" = "$REVIEW_HEAD_SHA" ]; then
if [ "$DRAFT" = "true" ]; then
set_review_status ""
elif [ "$VERDICT_OUTCOME" != "success" ]; then
case "$review_label" in
review:needs-evidence|review:blocked|review:human-required|review:automation-failed) ;;
*) set_review_status "review:automation-failed" ;;
esac
fi
fi
if [ "$current_head" != "$REVIEW_HEAD_SHA" ]; then if [ "$current_head" != "$REVIEW_HEAD_SHA" ]; then
conclusion="cancelled" exit 0
title="Review superseded by a newer HEAD"
summary="The pull request HEAD moved before this review completed."
elif [ "$DRAFT" = "true" ]; then
conclusion="neutral"
title="Review skipped for draft pull request"
summary="Mark the pull request ready for review to start the readiness check."
elif [ "$VERDICT_OUTCOME" = "success" ] && [ "$review_label" = "review:ready" ]; then
conclusion="success"
title="OpenChamber review passed"
summary="The reviewed HEAD is ready for maintainer review."
else
conclusion="failure"
title="OpenChamber review did not pass"
summary="Current readiness state: ${review_label:-review:automation-failed}."
fi fi
check_run="$(jq -n \ remove_args=()
--arg conclusion "$conclusion" \ while IFS= read -r label; do
--arg title "$title" \ case "$label" in
--arg summary "$summary" \ review:*) remove_args+=(--remove-label "$label") ;;
--arg details_url "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" \ esac
'{ done < <(gh pr view "$PR_NUMBER" --json labels --jq '.labels[].name')
status: "completed",
conclusion: $conclusion,
details_url: $details_url,
output: {
title: $title,
summary: $summary
}
}')"
printf '%s' "$check_run" | gh api \ gh pr edit "$PR_NUMBER" "${remove_args[@]}" --add-label "review:automation-failed"
--method PATCH \
"repos/${GITHUB_REPOSITORY}/check-runs/${CHECK_RUN_ID}" \
--input - >/dev/null
+1 -1
View File
@@ -179,7 +179,7 @@ Choose exactly one review verdict:
- `blocked`: at least one concrete correctness, security, mandatory-guidance, or contribution-contract blocker must be fixed. - `blocked`: at least one concrete correctness, security, mandatory-guidance, or contribution-contract blocker must be fixed.
- `human-review-required`: the PR changes review policy/automation or another trust boundary that automation must not clear by itself, or safe automated review is otherwise impossible. - `human-review-required`: the PR changes review policy/automation or another trust boundary that automation must not clear by itself, or safe automated review is otherwise impossible.
Verdict precedence is `human-review-required`, `blocked`, `needs-evidence`, then `pass`. CI status is intentionally outside this verdict: a review may return `pass` while a separate required check fails, and both gates must pass independently before merge. Verdict precedence is `human-review-required`, `blocked`, `needs-evidence`, then `pass`. CI status is intentionally outside this verdict: a review may return `pass` while a separate required check fails. The AI verdict is advisory, is communicated through the `review:*` label and review comment, and must not fail the pull request check.
## Comment style ## Comment style
+4 -4
View File
@@ -219,10 +219,10 @@ the previous readiness label before it starts, and only `review:ready` means
the pull request is ready to enter the maintainer review queue. Draft pull the pull request is ready to enter the maintainer review queue. Draft pull
requests have no readiness label. requests have no readiness label.
The workflow publishes a separate `review` check on the exact reviewed HEAD; AI review verdicts are advisory and never fail the pull request check. Readiness
only `review:ready` passes it. The `automation` job reports workflow execution is communicated only through the `review:*` label and immutable review comment.
independently so a manual `/oc-review` result can update readiness without The `automation` job fails only when the workflow itself cannot complete or
being attached to the default-branch commit that triggered the command. verify a trustworthy result, in which case it applies `review:automation-failed`.
Each completed review creates a new comment tied to its reviewed HEAD so the Each completed review creates a new comment tied to its reviewed HEAD so the
conversation remains chronological. Previous review comments are not rewritten. conversation remains chronological. Previous review comments are not rewritten.