1. Review: The Heart of Any Competition
However well a competition is run, if the review goes wrong — unfair results, scoring errors, judge bias — the credibility of the entire event collapses.
Reviewing is not simply “getting a few people to take a look.” A professional review system has to answer five core questions:
- How are entries assigned? How many judges review each entry? Random assignment or assignment by subject area?
- How is scoring done? A single overall score or dimension-based scoring? Should the highest and lowest scores be dropped?
- How is fairness guaranteed? Can judges see contestant information? How do you prevent favoritism?
- How are disputes handled? What if judges disagree sharply? How are contestant appeals processed?
- How are results presented? How are rankings, shortlists and judge comments generated automatically?
The AKI platform has managed the review process for 1,000+ competitions, covering clients from the Ministry of Education to Oriental Yuhong. This article systematizes that experience to help you build a professional, fair and efficient review system.
2. Four Key Questions in Review Process Design
2.1 Review Rounds: Is One Round Enough?
For competitions with more than 100 entries, we recommend at least two review rounds.
| Round | Purpose | Review Method | Pass Rate |
|---|---|---|---|
| Preliminary Round (Open Screening) | Fast screening to eliminate clearly unqualified entries | Distributed online review, each judge reviews 10-20 entries | 30-50% |
| Semi-Finals | Detailed assessment to select finalists | Mainly online review, with an optional judge discussion session | 20-30% |
| Finals | Final ranking and award decisions | Offline defense or live-streamed online pitch | Based on the number of awards |
2.2 Judge Assignment: How Many Entries per Judge?
The core of judge assignment is “cross coverage” — each entry is scored independently by at least three judges, and each judge's workload stays within a manageable range (no more than 30 entries per day).
The AKI platform's intelligent assignment algorithm can allocate automatically according to the following rules: each entry is randomly assigned to 3-5 judges, each judge receives an evenly distributed workload, and judges from the same school or organization recuse themselves from contestants from their own institution.
2.3 Scoring Methods: Overall Score vs. Dimension-Based Scoring
Use overall-score scoring for preliminary rounds (fast screening) and dimension-based scoring for finals (detailed assessment). Dimension-based scoring significantly reduces the bias of “one strong aspect masking weaknesses elsewhere.” A typical dimension design:
- Innovativeness(30%): originality of the topic or solution
- Completeness(25%): the actual finished quality of the entry
- Technical Quality / Professionalism(20%): technical difficulty or professional depth
- Presentation(15%): presentation of the work and defense performance
- Social Value(10%): practical application prospects or social significance
3. Core Features of an Online Review System
A good online review system does far more than “scoring.” The following is the checklist of capabilities it must cover:
| Module | Core Capability | Why It Matters |
|---|---|---|
| Entry Viewing | Online preview of images, video and documents, no download required | Judges work in one place instead of switching between tools |
| Dimension-Based Scoring | Custom scoring dimensions and weightings with automatic weighted calculation | Detailed assessment that reduces “impression-based” bias |
| Anonymous Mode | Hides identifying information such as contestant name and school | Prevents favoritism and ensures fairness |
| Outlier Removal | Automatically drops the highest and lowest scores, then averages the rest | Reduces the impact of extreme scores from individual judges |
| Judge Comments | Required or optional comments, with quick input from templates | Gives contestants useful feedback |
| Progress Monitoring | Real-time view of each judge's completion progress | Lets you follow up with slow judges and keep review on schedule |
| Automatic Ranking | Automatically generates rankings and shortlists from weighted total scores | Eliminates manual Excel calculation, with zero errors |
| Dispute Handling | Flags entries with excessive score divergence and triggers re-review | Protects review quality and reduces missed or incorrect decisions |
4. Anti-Cheating Mechanisms: Three Lines of Defense for Fairness
A review system without anti-cheating mechanisms is like a house without locks.
First Line of Defense: Anonymous Review
The system automatically hides identifying information such as contestants' real names, schools and organizations. Judges see only the entry number and the entry itself, and cannot identify the contestant. This is the most basic and most important safeguard of fairness.
Second Line of Defense: Statistical Anti-Cheating
- Outlier Removal: automatically drops the highest and lowest score for each entry and averages the rest. Even if a judge deliberately scores too high or too low, the impact is greatly reduced
- Deviation Alerts: the system checks automatically — if a judge's scores deviate from the average by more than 1.5 standard deviations on a sustained basis, the system flags it for administrator review
- Distribution Check: verifies that each judge's score distribution is reasonable — if a judge scores every entry above 90 or below 60, they are not discriminating properly
Third Line of Defense: Operation Audit
The system keeps a complete log of every review action: who viewed which entry at what time, what score was given, and whether a score was modified. All actions are traceable and auditable. In a dispute, the log is the strongest evidence available.
5. How to Define Review Criteria
How clearly the review criteria are defined directly determines how credible the results are. Below is a standard process for defining review criteria:
- Define Scoring Dimensions: select 3-5 dimensions based on the competition type (see section 2.3)
- Set Weightings: the share of each dimension (totaling 100%)
- Quantify Scoring Levels: give each dimension an explicit scoring description.
Example: “Innovativeness 90-100: proposes a solution unprecedented in the industry; 70-89: significant innovation built on existing work; 50-69: some improvement; below 50: largely reproduces an existing approach” - Judge Training: before review begins, brief all judges on the scoring criteria and key considerations to ensure consistent interpretation
- Trial Scoring Calibration: have all judges first score 3-5 “calibration samples” and check whether their scores converge. Judges whose scores deviate significantly should be contacted individually
6. Build vs. Buy: Choosing a Review System
The first instinct of many organizers is to “have a developer write one” or to “make do with a survey tool.” Both approaches have real drawbacks:
| Option | Cost | Advantages | Disadvantages |
|---|---|---|---|
| In-House Development | RMB 20,000-50,000 + 2-4 weeks | Fully customizable | Long development cycle; specialized features such as anti-cheating and score aggregation are easily overlooked |
| Generic Survey Tools | Free - RMB 1,500 | Quick to get started | Lacks specialized features such as anonymous review, outlier removal and assignment algorithms |
| Professional Competition Platform (Recommended) | RMB 1,500-5,000 per competition | Ready to use out of the box, complete feature set | Takes 1-2 hours to learn the configuration |
For most competition organizers, buying the review module of a professional competition platform offers the best value. Take AKI as an example: its review module includes every specialized feature — anonymous review, dimension-based scoring, outlier removal, deviation alerts and automatic ranking — with full packages starting at RMB 1,500 per competition. It has been proven in the field across thousands of competitions, including the China International College Students' Innovation Competition of the Ministry of Education and the China Innovation & Entrepreneurship Competition of the Ministry of Industry and Information Technology.
7. Frequently Asked Questions (FAQ)
What is the difference between online and offline review?
Online review offers flexible scheduling for judges, no geographic limits, automatic score aggregation that avoids manual Excel errors, and anonymous review that protects fairness. The drawback is the lack of face-to-face discussion among judges. A hybrid model is recommended: online review for preliminary rounds (efficient) and offline defense review for finals (in-depth).
How can cheating in online review be prevented?
Three layers of protection: 1) anonymous mode, which hides identifying information such as contestant name and school; 2) outlier removal, which automatically drops the highest and lowest score for each entry; 3) deviation alerts, where the system flags judges whose scores deviate excessively. In addition, a complete operation log makes every review action traceable.
Which scoring methods should a review system support?
At least three modes: overall-score scoring (a single total score, suited to initial screening), weighted dimension scoring (separate scores across multiple dimensions aggregated by weighting, suited to professional review) and ranking (ordering entries directly, suited to finals). Switching flexibly between these three modes is the key to practicality.
What if there are not enough judges?
Three remedies: 1) invite strong past winners to serve as “junior judges” (they can absorb part of the initial screening workload); 2) open some review stages to public voting, keeping its weight within 10-20%; 3) share judge resources with partner institutions or industry associations. Ensure at least three judges score every entry.
How much does a review system cost?
On professional competition platforms such as AKI, the review module is usually bundled into the overall package, starting at RMB 1,500 per competition. Generic tools such as survey apps are not a recommended substitute — they lack specialized features such as anonymous review, outlier removal and entry assignment. If review goes wrong, the loss of credibility far exceeds the cost of a review system.