 |
James Austin, Director of Assessment Services, Center on Education and Training for Employment, College of Education and Human Ecology |
Thursday, July 20, 2017
|
||
Presentation Title: Item Generation for Testing Occupational knowledge-Skill: An Investigative Program
Items, and many of them, are the lifeblood of testing programs; nowhere is this more evident than in high stakes and computer adaptive testing systems. Yet “theory” usually pertains to test scoring. Typically, subject matter experts in a content domain write items which go through validation and empirical tryout. Important gatekeeping decisions are based on score interpretations, thus the revision of the joint standards in 2014 provides new guidance for testing and assessment. In this presentation, I cover selected research and practice in item generation and its new variant, automatic item generation or AIG. When many think of item writing, it is either in a formative / classroom or standardized format. Items generated by traditional models (persons creating items) or item models applied by persons, or algorithms (apps/programs working in the background to build items for a pool). My work occurs in an operational statewide testing program for secondary students in Grades 11-12 in career-technical education (CTE; globally known as TVET) courses organized by broad career field (Agriculture, Health Sciences) and subordinate pathway – related occupations such as Allied Health & Nursing. Test forms (40-item fixed forms, not yet adaptive) are delivered through a portal maintained by my work group (www.webxam.org). I review the literature in this area, outline my strategy using item models, and present possible directions based on my understanding of the literature.
About Dr. Austin
James T. Austin earned a Ph.D. in Industrial/Organizational Psychology from Virginia Tech in 1987, then completed a Postdoctoral Traineeship in Quantitative Psychology at the University of Illinois in Urbana from 1987-1989. He served as a visiting faculty member at the University of Illinois and New York University, and was on the regular faculty at The Ohio State University from 1991-1997. In 1997, he was hired as a Research Specialist 2 at the Center on Education and Training for Employment in the College of Education and Human Ecology at The Ohio State University. Promoted to his current position of Senior Research Specialist in 2005, and Director of Assessment Services since 2010, he is responsible for managing (with Robert Mahlman and Brooke Parker) a 12-person unit engaged in a range of testing assessment projects. These projects involve developing and maintaining assessments for education and business clients, including the Ohio Department of Education, associations and bodies involved in certification of personnel, and government agencies. A new line of business entails consulting with certificate and certification programs seeking voluntary accreditation (under ISO-17024 or National Commission on Certifying Agencies). Dr. Austin’s research interests include statistical and methodological issues as well as testing (item generation) and job performance measurement. Dr. Austin has numerous citations to his publications, which include published articles in Psychological Bulletin, Journal of Applied Psychology, Personnel Psychology, Journal of Career-Technical Education, and a chapter in the Annual Review of Psychology. He helped develop and is lead presenter of a three-day workshop on principles and practices of basic test construction for a variety of participants since 2004 (40 iterations), including customized versions of the workshop for international clients (Canada, Malaysia, and Singapore).