Date of Award
10-1-1990
Document Type
Dissertation - NSU Access Only
Degree Name
Doctor of Education
Department
Center for the Advancement of Education
Abstract
The purpose of this Major Applied Research Project (MARP) was to develop a valid and reliable test instrument and to establish a reasonable cut score for the National Fire Academy's (NFA) Chemistry of Hazardous Materials course final exam. In addition, a test technical manual was written based on the American Psychological Association's Standards for Educational and Psychological Testing. 1985. There were nine research questions addressed in this MARP: (1) Were test development procedures based on correct educational methodology? (2) Does the test have content validity? (3) Does the test have domain validity? (4) Has an optimal cut score for the test been achieved? (5) Does the test have decision validity? (6) Does the test have threshold loss function reliability? (7) Does the test have internal consistency reliability? (8) Are the test items effective? and (9) ls the technical manual for the test clear, accurate, and complete? Nine classes of the Chemistry of Hazardous Materials course were selected by the NFA Assistant Superintendent for Leadership and Hazardous Materials to participate in this study. The classes were selected because they were scheduled to be conducted during the timeframe of this MARP. The total number of students was 221. Five classes were pretested and posttested and four classes were posttested only. The selection of which classes to pretest was by random drawing. The development process included writing educational objectives and test questions to create a criterion-referenced test. Specific procedures were used to ensure the test's validity and reliability. The types of validity and reliability were content validity, decision validity, threshold loss function reliability and internal consistency reliability. The optimal cur score was selected empirically based on the test results of instructed and uninstructed students using the criterion groups validation model. Item analysis procedures included sensitivity to instructional effects calculations, revise of test items for bias, and student feedback on the questions. The test technical manual was written based on applicable APA standards. Finally the development procedures were conducted by the NFA development team which consisted of three people. The work of the development team was evaluated by eleven individuals. The findings indicate that the test instrument was developed on correct educational methodology and is valid and reliable for the purpose of the instrument. Specifically the test has item, sampling, and domain validity. The test has 96 percent decision validity and a false positive error rate of .008 and a false negative error rate of .024 based on the optimal cut score of 70. The Pₒ reliability index is .974, the KR-21 is .839, and the standard error of measure around the cut score is +/- 3 points. All the test questions have a positive sensitivity to instructional effects score and the items are free of bias, and did not confuse the students. Finally, the test technical manual has completeness, accuracy, and clarity. A total of seven recommendations were made. The first three are that the test be used as the final exam in the Chemistry of Hazardous Materials course with a passing score of 70 and that the test technical manual be distributed to all Chemistry of Hazardous Materials faculty. The remaining four recommendations are more future oriented. Future test data should be stored in the computer database and additional psychometric analysis conducted. Finally, it was recommended that the procedures used for this study be applied to other NFA test instruments and that future course development projects include test validity and reliability procedures using this study as a model.