Title

Reflections on the lived experience of 160 evaluations: The professional costs of resisting pressure to de-contextualize field studies

Location

1048

Format Type

Event

Format Type

Paper

Start Date

January 2018

End Date

January 2018

Abstract

This case study presentation is based on the presenters 15 years at a research university leading an evaluation institute while conducting over 160 evaluations. As our Institute work completed, we had little time to separately and jointly reflect on our work. Unfortunately, this lived experience of leading multiple ongoing field research projects provided little time for meaningful reflection in action (Schön, 1983) due to project and administrative deadlines. Upon thoughtful practice reflection (Schwandt, 2015), we recognized that the most challenging aspect of our work was not technical – our research practices represented well-established practice norms. Instead, our greatest challenge was capturing and reporting on the project story, including multiple participant stories and perspectives. Funders wanted RCTs and “summary data tables” while we sought to temper and augmented these narrow expectations with contextualized stakeholder perspectives and stories addressing how they were affected by a project.

Our larger project narrative encompassing the experiences of stakeholders was difficult, costly, and time consuming. We resisted pressures to be mere funding agency data collectors, and in that we generated some conflict, lost contract opportunities and uninformed criticism at some level of professional disillusionment. This presentation will address how we conducted evaluation (Skolits, Morrow and Burr, 2011), and how we attempted to practice evaluation in a manner encompassing the establishment of meaning reflecting more contextual, qualitative oriented approaches (Guba and Lincoln, 1981; Patton, 2014). We conclude that our reflections motivate us to challenge the status quo of evaluation practice and funding.

References

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, Calif: Sage Publications. Thousand Oaks: Sage Publications

Patton, M. Q. (2014). Qualitative research and evaluation methods: Intergrating theory and practice.

Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.

Schwandt, T. (2015a) Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice Stanford University Press: Stanford, CA, USA

Schwandt, T. (2015b) Dictionary of Qualitative Inquiry, 4th ed.: Thousand Oaks, CA: Sage Publications.

Skolits G. J., Morrow J. A., Burr E. M. (2009). Re-conceptualizing evaluator roles. American Journal of Evaluation, 30, 275–295.

Comments

Breakout Session B

This document is currently not available here.

COinS
 
Jan 11th, 3:15 PM Jan 11th, 3:35 PM

Reflections on the lived experience of 160 evaluations: The professional costs of resisting pressure to de-contextualize field studies

1048

This case study presentation is based on the presenters 15 years at a research university leading an evaluation institute while conducting over 160 evaluations. As our Institute work completed, we had little time to separately and jointly reflect on our work. Unfortunately, this lived experience of leading multiple ongoing field research projects provided little time for meaningful reflection in action (Schön, 1983) due to project and administrative deadlines. Upon thoughtful practice reflection (Schwandt, 2015), we recognized that the most challenging aspect of our work was not technical – our research practices represented well-established practice norms. Instead, our greatest challenge was capturing and reporting on the project story, including multiple participant stories and perspectives. Funders wanted RCTs and “summary data tables” while we sought to temper and augmented these narrow expectations with contextualized stakeholder perspectives and stories addressing how they were affected by a project.

Our larger project narrative encompassing the experiences of stakeholders was difficult, costly, and time consuming. We resisted pressures to be mere funding agency data collectors, and in that we generated some conflict, lost contract opportunities and uninformed criticism at some level of professional disillusionment. This presentation will address how we conducted evaluation (Skolits, Morrow and Burr, 2011), and how we attempted to practice evaluation in a manner encompassing the establishment of meaning reflecting more contextual, qualitative oriented approaches (Guba and Lincoln, 1981; Patton, 2014). We conclude that our reflections motivate us to challenge the status quo of evaluation practice and funding.

References

Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, Calif: Sage Publications. Thousand Oaks: Sage Publications

Patton, M. Q. (2014). Qualitative research and evaluation methods: Intergrating theory and practice.

Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.

Schwandt, T. (2015a) Evaluation Foundations Revisited: Cultivating a Life of the Mind for Practice Stanford University Press: Stanford, CA, USA

Schwandt, T. (2015b) Dictionary of Qualitative Inquiry, 4th ed.: Thousand Oaks, CA: Sage Publications.

Skolits G. J., Morrow J. A., Burr E. M. (2009). Re-conceptualizing evaluator roles. American Journal of Evaluation, 30, 275–295.