Director of Test Development

Job Summary:

The Director of Test Development provides leadership and direction for the Test Development team. This position is crucial for ensuring the accurate and timely development of over 50 high-stakes examinations and longitudinal assessments annually. The role involves significant independence, judgment, and initiative to ensure project completion with a focus on timeliness and accuracy.

Essential Duties and Responsibilities:

• Oversee all aspects of the test development processes and assessment products while integrating best practices, current research, and innovations in testing within the department.
• Provide test development expertise to staff and subject matter experts (SMEs), while building and maintaining the necessary strong relationships to achieve desired results.
• Oversee recruitment for the Certification Development Experts, General Pediatrics Examination Committee, and General Pediatrics MOCA-Peds Committee, as well as Medical Editors for the Subboards.
• Design test development processes, workflows, and committee/subboard meetings to ensure high-quality exams and assessments.
• Manage the team by setting clear objectives, establishing priorities, tracking projects, and delegating responsibilities appropriately.
• Promote a culture of collaboration, particularly between the test development team and the entire organization, particularly the psychometrics team, to strengthen integration and cooperation to drive assessment forward.
• Coordinate and partner with vendors to ensure the development and delivery of high-quality assessments.
• Explore, plan, lead, and implement innovative approaches to testing and item development (eg AI for item generation), including but not limited to process improvements and strategic initiatives.

Other Functions:

• Support the training and orientation of SMEs.
• Represent the ABP at national professional organizations and serve as a testing expert on various internal committees.
• Communicate effectively with external stakeholders, including the American Academy of Pediatrics (AAP), American Board of Medical Specialties (ABMS), other member boards, diplomates, and the wider testing community.
• Engage the department in research activities and share results with national testing audiences.
• Oversee the creation and maintenance of standard policy and procedure documents relating to test development activities ensuring compliance with industry standards.
• Initiate, lead, and participate in special projects outside of standard role and responsibilities.
• Performs other duties as assigned.

Supervisory Responsibilities:

This role currently supervises a team of 8 members.

Required Qualifications:

• Minimum of a master’s degree in Educational Measurement, Educational Statistics, Industrial/Organizational Psychology, Education, Psychology, or a related field. Additional experience may be considered in lieu of the educational requirement
• Minimum of 7 years of related experience in assessment and testing, particularly certification
• At least 5 years of personnel management experience
• Demonstrated experience in managing vendors

Preferred Qualifications:

·        Excellent leadership, motivational, and organizational skills 

• Strong communication skills, both written and in-person

·        Solid time and task management skills 

·        Strong analytical skills: ability to solve complex problems 

• Working knowledge of LLMs and AI.

·        Strong innovative skills, both in conceptualization and implementation 

·        Ability to integrate technological improvements and solutions 

·        Proficiency with item-banking best practices and software (eg, ExamDeveloper) 

·        Expert knowledge of computer and Internet-based testing 

·        Proficient with project management tools and applications 

·        Expertise in Microsoft Office applications, particularly Excel and SharePoint 

·        Superior communication skills, both oral and written 

·        Demonstrated public speaking and presentation skills 

·        Excellent facilitation skills