How to Run a Quarterly Reviewer Calibration Session

A reviewer calibration session should compare how reviewers apply the same rules to the same content. Have participants score examples independently, discuss differences using evidence, agree on decision boundaries, and update the rubric or training record. Quarterly is a practical house cadence, not an externally required frequency.

Choose examples that test real boundaries

Select five to ten recent or constructed cases that represent current work. Remove personal and confidential information. Include:

• A clean asset that should not attract invented feedback
• A clear release blocker
• A mandatory but easily corrected issue
• An optional creative preference
• A claim near the edge of approved support
• A disclosure affected by captions or placement
• Conflicting brief language
• A version or rights-scope mistake

Do not choose only memorable failures. The set should test false positives, uncertainty, escalation, and feedback quality as well as issue detection.

Use examples from the creator content approval workflow, preserving the brief version and evidence available when the original decision occurred.

Collect independent decisions first

Before the meeting, ask each participant to record:

1. Approve, request changes, or escalate
2. Every finding and timestamp
3. Applicable rule or source
4. Severity
5. Creator-facing action
6. Confidence or unresolved question

Independent work prevents the first senior voice from becoming the apparent consensus. It also shows whether people reached the same label for different reasons.

Use the new reviewer onboarding model for trainees, but include experienced decision owners in calibration too. Seniority does not make an ambiguous rule clear.

Discuss evidence before preferences

For each case, reveal the distribution without naming individuals. Start with the largest material disagreement.

Ask:

• What does the asset actually communicate?
• Which current rule applies?
• Is that rule clear and within scope?
• What release action follows?
• Who has authority if the rule is uncertain?
• What is the smallest complete creator action?

Separate a legitimate alternate interpretation from a missed instruction. If the source supports two reasonable outcomes, improve the source rather than forcing agreement through authority alone.

Record decisions at the right level

Calibration outputs may include:

• A clarified definition in the severity rubric
• A new positive or negative example
• An escalation trigger
• Revised approved wording
• A brief-template improvement
• A finding that requires specialist review
• Confirmation that teams may use more than one acceptable creative solution

Date every update, name the owner, and state when it becomes effective. Do not retroactively score old work against a new rule.

If the discussion involves advertising claims, remember that the FTC advertising FAQ describes a reasonable-basis requirement for objective U.S. advertising claims. The specific evidence and interpretation still depend on the claim and product.

Measure agreement carefully

Useful measures include:

• Agreement on final action
• Agreement on known findings
• False-positive rate against the approved set
• Severity agreement
• Escalation agreement
• Feedback actionability

Report the sample size and case mix. A percentage from six deliberately difficult examples is not a production error rate. Use the results to identify categories needing clearer rules or additional sampling.

Do not reward agreement for its own sake. A group can consistently apply a wrong or outdated rule. Source validity and qualified ownership come first.

Follow through after the session

Publish a concise change note with updated examples and effective dates. Ask reviewers to acknowledge material changes. Update automation, templates, and training where the old rule was encoded.

For the next several weeks, sample affected categories and check whether disagreement or creator questions fall. If not, read the cases and revisit the rule.

Invite reviewers to submit future calibration candidates during normal work. A finding that caused repeated escalation or contradictory feedback is often more useful than a theoretical quiz.

Use a 60-minute agenda

A practical agenda is:

• 5 minutes: scope and source updates
• 10 minutes: independent results overview
• 30 minutes: two or three material disagreements
• 10 minutes: rule and example decisions
• 5 minutes: owners, effective dates, and follow-up sample

Move unresolved specialist questions out with a named owner and deadline. Do not invent an answer to finish the meeting.

Keep a calibration register with the session date, participants, example IDs, source versions, areas of agreement, material disagreements, decisions, owners, and effective dates. Restrict access where examples contain sensitive campaign information.

At the next session, begin by checking whether the prior actions were implemented. A recurring disagreement after training may indicate a workflow or source problem rather than a need for another presentation. Retire examples whose product, policy, or platform context is obsolete, while preserving their historical use.

The takeaway

Use realistic cases, collect independent answers, resolve differences from current sources, document rule changes, and sample the affected work afterward. Choose a cadence based on change and risk; quarterly is one workable operating rhythm, not a universal mandate.