Inter-rater Observational Workflow using Reports

Inter-rater Observation can be performed by an Admin or Group System Admin viewing the Tag or Keyword reports and using excel to filter the results.

Inter-rater Observation can also be done by using the Data Viewer as an Admin. Click here to see how.

1. Create two Scorer roles.
   

   Set “Scorer” as the role when creating a new user.

2. Associate a Client or Login Client to that Scorer Role.”Scorer assign Keywords and/or Tags to the video. (Scorer role is blinded to other scorer annotations)
   

   Associate Client to both Scorers

3. Scorer adds tags and keywords to the video in the Client’s folder.
   

   Scorer adds annotations to client video.

4. Admin logs into the system and exports the Tag or Keyword report.
   

   Admin can export Tag or Keyword report.

5. Use Excel to filter the results for the filename you want to review.
   

   Filter columns in excel for the filename you want to compare.

6. Look for discrepancies in the report between scorers.
   

   Use the filtered results to compare annotations between scorers.