Our team planned a norming session activity for our Primary Participants to (1) become acquainted with the DEI Audit initial rubrics, (2) understand and provide feedback on the rating process, and (3) reach a level of consistency with rating criteria. In other words, to test the interrater reliability of the audit survey instrument before implementing it.  

These sessions proved to be productive and insightful, plus it opened up difficult conversations on how to best approach this process. After participant feedback was collected and collated, we created a post-rubric norming session document that details the outcomes and observations from our activities. The feedback also informed revisions on the survey (e.g., modifying, eliminating, or adding question items).  

We split the Primary Participants into two groups and performed the following activities: 

1. Introduce the rubrics (we started with 6 separate rubrics, based on the Areas of Exploration)

2. Review corpus sample #1 and apply rubric #1

3. Discuss results and issues

4. Review corpus sample #2 and apply rubric #2

5. Discuss results and issues

6. Solicit general feedback