Local Contract Nurse RN - Utilization Review
REMOTE, MN
PRIDE Health
5x8 hrs, Days
Referral Bonus

REMOTE ROLE!!!!

General Job Information
Job Function Description
The Sr. Clinical Quality will review annotated prior authorization cases used in AI model development. In this role, the Manager of Clinical Quality will ensure that annotated cases meet high standards for accuracy, consistency and inter-rater reliability (IRR). This role will work cross-functionally with clinical annotators, quality leads, and data science teams to identify annotation errors, provide structured feedback, and support continuous quality improvement across prior authorization use cases. This role plays a key part in driving annotation quality, supporting IRR, and leading model calibration sessions that directly influence model performance.
The ideal candidate is skilled in clinical review and quality auditing and has experience facilitating group calibration sessions that align clinical annotators on complex cases, interpretation of payer criteria, and annotation best practices.
General Job Profile
- Reviews the work of others.
- Develops innovative approaches.
- Sought out as expert.
- Serves as a leader/ mentor.
Job Scope and Guidelines
- Conduct independent quality audits of annotated cases to evaluate accuracy, clinical relevance, and adherence to guidelines
-Monitor inter-rater reliability (IRR) using metrics like agreement rates and F1 scores
-Identify and document annotation discrepancies, patterns of error, and ambiguous cases
-Deliver structured, case-based feedback to annotators and clinical reviewers
-Lead recurring calibration sessions to align annotators on clinical interpretation and annotation schema
-Select and prepare representative calibration cases, including edge cases, disagreements, and model-suggested revisions
-Collaborate with clinical leads and data scientists to identify areas of annotation drift that may affect model performance
-Document outcomes from calibration sessions, update training materials, and adjust annotation guidelines accordingly
-Develop and maintain QA scoring rubrics, quality dashboards, and audit documentation
-Partner with AI/ML team to provide insights on annotation quality trends impacting model precision, recall, and F1 score
-Recommend process or tool changes based on audit findings and calibration feedback
-Support onboarding and ongoing education for clinical annotators, with an emphasis on evolving clinical guidelines and clinical nuances.
Additional Scope and Guidelines
Not Applicable
Qualification
- Bachelor’s or Master’s degree in Nursing
-Active, unrestricted RN License
-3+ years of experience in clinical quality review, utilization management, or prior authorization
-Familiarity with CPT/HCPCS, ICD-10 coding, and national and payer-specific medical necessity guidelines
-Strong experience reviewing clinical documentation (e.g., progress notes, imaging, referrals, orders)
-Prior experience with clinical annotation, AI/ML projects, or calibration efforts preferred
-Excellent facilitation, communication, and critical thinking skills.
-Ability to synthesize findings across multiple reviewers and escalation annotation ambiguities when needed.
Values Based Competencies
1. Integrity Value: Act Ethically
· Comply with Applicable Laws, Regulations and Policies
· Demonstrate Integrity
2. Compassion Value: Focus on Customers
· Identify and Exceed Customer Expectations
· Improve the Customer Experience
3. Relationships Value: Act as a Team Player
· Collaborate with Others
· Demonstrate Diversity Awareness
· Learn and Develop
4. Relationships Value: Communicate Effectively
· Influence Others
· Listen Actively
· Speak and Write Clearly
5. Innovation Value: Support Change and Innovation
· Contribute Innovative Ideas
· Work Effectively in a Changing Environment
6. Performance Value: Make Fact-Based Decisions
· Apply Business Knowledge
· Use Sound Judgement
7. Performance Value: Deliver Quality Results
· Drive for Results
· Manage Time Effectively
· Produce High-Quality Work