Reflections on the lived experience of 160 evaluations: The professional costs of resisting pressure to de-contextualize field studies
Location
1048
Format Type
Event
Format Type
Paper
Start Date
January 2018
End Date
January 2018
Abstract
This case study presentation is based on the presenters 15 years at a research university leading an evaluation institute while conducting over 160 evaluations. As our Institute work completed, we had little time to separately and jointly reflect on our work. Unfortunately, this lived experience of leading multiple ongoing field research projects provided little time for meaningful reflection in action (Schön, 1983) due to project and administrative deadlines. Upon thoughtful practice reflection (Schwandt, 2015), we recognized that the most challenging aspect of our work was not technical – our research practices represented well-established practice norms. Instead, our greatest challenge was capturing and reporting on the project story, including multiple participant stories and perspectives. Funders wanted RCTs and “summary data tables” while we sought to temper and augmented these narrow expectations with contextualized stakeholder perspectives and stories addressing how they were affected by a project.
Our larger project narrative encompassing the experiences of stakeholders was difficult, costly, and time consuming. We resisted pressures to be mere funding agency data collectors, and in that we generated some conflict, lost contract opportunities and uninformed criticism at some level of professional disillusionment. This presentation will address how we conducted evaluation (Skolits, Morrow and Burr, 2011), and how we attempted to practice evaluation in a manner encompassing the establishment of meaning reflecting more contextual, qualitative oriented approaches (Guba and Lincoln, 1981; Patton, 2014). We conclude that our reflections motivate us to challenge the status quo of evaluation practice and funding.
References
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, Calif: Sage Publications. Thousand Oaks: Sage Publications
Patton, M. Q. (2014). Qualitative research and evaluation methods: Intergrating theory and practice.
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.
Schwandt, T. (2015a) Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice Stanford University Press: Stanford, CA, USA
Schwandt, T. (2015b) Dictionary of Qualitative Inquiry, 4th ed.: Thousand Oaks, CA: Sage Publications.
Skolits G. J., Morrow J. A., Burr E. M. (2009). Re-conceptualizing evaluator roles. American Journal of Evaluation, 30, 275–295.
Reflections on the lived experience of 160 evaluations: The professional costs of resisting pressure to de-contextualize field studies
1048
This case study presentation is based on the presenters 15 years at a research university leading an evaluation institute while conducting over 160 evaluations. As our Institute work completed, we had little time to separately and jointly reflect on our work. Unfortunately, this lived experience of leading multiple ongoing field research projects provided little time for meaningful reflection in action (Schön, 1983) due to project and administrative deadlines. Upon thoughtful practice reflection (Schwandt, 2015), we recognized that the most challenging aspect of our work was not technical – our research practices represented well-established practice norms. Instead, our greatest challenge was capturing and reporting on the project story, including multiple participant stories and perspectives. Funders wanted RCTs and “summary data tables” while we sought to temper and augmented these narrow expectations with contextualized stakeholder perspectives and stories addressing how they were affected by a project.
Our larger project narrative encompassing the experiences of stakeholders was difficult, costly, and time consuming. We resisted pressures to be mere funding agency data collectors, and in that we generated some conflict, lost contract opportunities and uninformed criticism at some level of professional disillusionment. This presentation will address how we conducted evaluation (Skolits, Morrow and Burr, 2011), and how we attempted to practice evaluation in a manner encompassing the establishment of meaning reflecting more contextual, qualitative oriented approaches (Guba and Lincoln, 1981; Patton, 2014). We conclude that our reflections motivate us to challenge the status quo of evaluation practice and funding.
References
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, Calif: Sage Publications. Thousand Oaks: Sage Publications
Patton, M. Q. (2014). Qualitative research and evaluation methods: Intergrating theory and practice.
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.
Schwandt, T. (2015a) Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice Stanford University Press: Stanford, CA, USA
Schwandt, T. (2015b) Dictionary of Qualitative Inquiry, 4th ed.: Thousand Oaks, CA: Sage Publications.
Skolits G. J., Morrow J. A., Burr E. M. (2009). Re-conceptualizing evaluator roles. American Journal of Evaluation, 30, 275–295.
Comments
Breakout Session B