Title of presentation: Measuring the educational IMPACT of your program: How much is it really worth?
Purpose of presentation: (why is this important and who is the target audience)
Why is this important? It was Friedrich Nietzsche who said, “Evaluation is creation: hear it, you creators! . . . It is only through evaluation that value exists: and without evaluation the nut of existence would be hollow. Hear it, you creators!” Now and then an educational program has cause to reflect on its worth to institution and stakeholders and determine how much it needs to re-create. It too must be certain that its “nut of existence” is not “hollow.” The “now and then” of program evaluation is usually caused by external factors, e.g., introduction of new technologies, administrative changes, accreditation, etc., but the hope is always that the program will be responsive to evaluative findings and internally re-create itself as a result of the program evaluation. The Department of Evening Classes at Brigham Young University recently undertook such a program-searching endeavor, including a survey of its 18,000 students, caused by administrative changes in the department and in preparation for a year-end unit review conducted every seven years by the university’s Office of Assessment and Planning. The evaluative and research model used in the evaluation will be explained and the results of the study discussed.
Target audience? This session should be of interest to most attendees at this conference since the evaluation model used to measure the program impact and worth of the Department of Evening Classes at BYU could be replicated by other programs. However, it is program administrators who will benefit most from the experience of these two administrators who undertook a mixed methods approach to measuring program impact and worth by asking its stakeholders some tough evaluative questions.
Objectives of the presentation (what are you planning to do) My co-researcher and I will discuss the basic steps of program evaluation and impact analysis and how they were employed by the Department of Evening Classes. We will also review the development and administrations of a web-based survey to the 18,000 students enrolled in evening classes during winter semester 2008 and discuss results. We will also summarize findings from interviews about the value of the Evening Class program with nearly 50 department chairs during 2007.
Practical applications (how can your results/strategies be used by others)
The focus of this study is best summarized by Mohr who said, “The crux of impact analysis is a comparison of what did happen after implementing the program with what would have happened had the program not been implemented” (pp. 2–3). The approach used and the survey instrument developed will be of most value to session attendees. The survey, which will be made available to attendees, asked students why they enrolled in a course taught in the evening, what would have been the outcome if the class had not been available at that time, and how important it is to have courses available in the evening.
Relationship to the conference theme
This presentation best fits under the administration track of “evaluation and quality.”
Information (data or theoretical base) to support what is advocated.
The seminal work in this area of program evaluation was prepared by The Joint Committee on Standards for Educational Evaluation and others: “The Program Evaluation Standards: 2nd Edition: How to Assess Evaluations of Educational Programs.” This study was also informed by the important work of Lawrence Mohr in his book (1988), “Impact analysis for program evaluation.”
2nd-Day Hands On: How to do your own program evaluation?
We will field questions by attendees about how they can best conduct their own program evaluation. We will also go into more detail about how the survey instrument was developed, how it was administered, and how results were analyzed and interpreted. Finally, and most importantly, we will help attendees prepare a plan for their own program evaluation.
Biography
Scott is the Director of Evening Classes at Brigham Young University where he received his PhD in Instructional Science, his MEd in Community Education, and his BS in Business Management. He is the former director of BYU’s Center for Instructional Design and chair of the University of Continuing Education (UCEA) Distance Learning Community of Practice (DLCoP). He has received multiple national awards for his research in the areas of distance education and assessment and measurement. He is the co-editor of the three volume book series, “Online Assessment and Measurement” and serves on multiple editorial boards for academic journals.