Psychometrics and Quantitative Research Director

The Director, Psychometrics and Quantitative Research (PQR) is a critical member of the leadership team for the CAL-WIDA Partnership at CAL. The CAL-WIDA Partnership team works closely with WIDA, our primary client, to execute WIDA’s vision with respect to their testing programs. The CAL-WIDA Partnership is responsible for developing test content and conducting psychometric analyses for WIDA’s suite of assessments.

Under the guidance of the Senior Director of the CAL-WIDA Partnership, the Director, PQR oversees and leads the work of the PQR team. This includes overseeing: operational and field test psychometric analyses; the preparation and quality control of annual technical reports; and the development of ancillary files including business rules and computer code that feeds into the psychometric analyses. The Director, PQR is also responsible for conceptualizing and overseeing special research activities related to WIDA’s suite of assessments, including verification studies, scale maintenance studies, and the like. The Director, PQR also provides leadership and guidance in quantitative and research methodologies to other researchers at CAL. The Director, PQR also assists the Senior Director in leading overall CAL-WIDA partnership activities and serves on CAL’s Extended Leadership Team.

Essential Duties

Psychometrics

• Direct the work of the (PQR) team, working with the PQR Project Manager and PQR Team Manager to ensure all work flows smoothly with appropriate staff assignments and resourcing for the CAL-WIDA Partnership program
• Direct the work of the PQR team in ensuring quality control of all deliverables
• Provide oversight on the technical quality of language assessment and analysis activities conducted by the PQR team
• Lead multiple technical operational psychometric and research projects for the CAL-WIDA Partnership program
• Provide guidance and oversight to the PQR psychometric staff on technical aspects of their work
• Lead, design, conduct, and document psychometric and statistical analyses for WIDA ACCESS and other WIDA Consortium-related projects
• Investigate, evaluate, recommend, and implement new approaches to conducting psychometric and statistical analyses
• Review and approve written research reports and technical documentation, such as the Annual Technical Report on WIDA ACCESS
• Direct ongoing psychometric activities related to CAL’s BEST assessments, including input on study design, oversight of work to maintain accreditation of the BEST assessments, and providing input on technical reports.

Administration

• Serve as the supervisor of record for CAL psychometricians, the PQR Project Manager, and the PQR Team Manager
• Provide leadership in planning and implementing CAL-WIDA partnership activities
• Assist with new project development and monitor current project budgets

CAL-wide Activities

• Provide guidance to CAL researchers on proposal design, research methodology, and quantitative analytical approaches
• Participate in writing quantitative-oriented portions of proposals as requested
• Provide leadership and oversight to maintain quality of psychometric aspects of BEST assessments.
• Develop and maintain a current list of professionals with expertise in psychometrics and quantitative methodology as potential contractors for CAL

External to CAL

• Represent CAL PQR in WIDA Technical Advisory Committee meeting
• Serve as CAL’s representative in WIDA Subcommittee and WIDA Board meeting, as needed

Other Duties as Assigned

Qualifications

Education and Experience

The Director, PQR must have

• Ph.D. in Educational Measurement, Psychometrics, or related field
• More than ten years of extensive experience in directing, designing, and conducting applied Psychometric analyses for high stakes operational testing programs.
• Strong demonstrated experience overseeing psychometricians and analysts in successfully applying the Rasch model to a variety of psychometric issues.
• Experience in leading analyses on performance and polytomously-scored tasks.
• Demonstrated proficiency in using SPSS, Winsteps, Facets, R and other statistical and psychometric software, and mentoring other staff in the use of these packages.

The ideal candidate for this role has

• Experience working within a high-stakes, standardized psychometric program in a K-12 U.S. Educational Context, and preferably in language testing.
• Experience overseeing and directing a team of psychometricians and research analysts.
• Experience communicating effectively in English with clients and internal and external stakeholders in high-stakes contexts.
• Experience overseeing a team in producing timely error-free deliverables.

Knowledge, Skills, and Abilities

The Director, PQR must be able to

• Apply theories of psychometric analysis to practical problems in language testing
• Thoroughly understand, and communicate effectively with internal and external stakeholders regarding, highly-complex test designs with multiple interconnected parts
• Oversee the development of quality control procedures for creating and maintaining databases, conducting statistical and psychometric analyses, conducting other quantitative research, and producing technical reports.
• Mentor staff in programming and developing documentation for statistical software such as SPSS, Winsteps, Stata, SAS, and R.
• Review and write narrative to accompany psychometric analyses for technical reports and other documentation.
• Coordinate staff across multiple tasks to meet deadlines and quality control expectations
• Work with leaders of other functional teams to meet unit goals
• Display leadership and flexibility to motivate staff to meet team goals
• Work independently as well as cooperatively in a fast-paced team environment

The ideal candidate for this role has

• Demonstrated ability to oversee and lead a diverse, multi-disciplinary professional team of researchers and language testers.
• Demonstrated ability to communicate both orally and in writing with a wide variety of audiences, including researchers, test developers, and client leadership teams.
• Strong demonstrated experience with computerized adaptive assessments.
• Experience in conducting standard setting study and/or alignment studies.

Travel Requirements

• This position requires occasional travel (1-3 times per year).

Additional Information

COMPENSATION:

Starting at $114,300 annually based on years of experience and skills. This position is benefits eligible.

PHYSICAL DEMANDS 

The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. While performing the duties of this job, the employee is regularly required to effectively communicate. The employee may be required to sit or stand in a stationary position for more than 50% of the time. This position constantly operates a computer and other office productivity equipment. The employee may be required to move boxes of 30lbs. CAL adheres to ADA compliance and guidelines as a common practice.

ADDITIONAL INFORMATION 

This job description summarizes the main duties of the job. It neither prescribes nor restricts the exact tasks that may be assigned to carry out these duties. This document should not be construed in any way to represent a contract of employment. Management reserves the right to review and revise this document at any time. 

AFFIRMATIVE ACTION/EQUAL OPPORTUNITY EMPLOYMENT STATEMENT 

The Center for Applied Linguistics (CAL) provides equal employment opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, sexual orientation, gender identity, disability, or genetics. In addition to federal law requirements, CAL complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training. 

CAL expressly prohibits any form of workplace harassment based on race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status. Improper interference with the ability of CAL employees to perform their job duties may result in discipline up to and including discharge.