AEA365 | A Tip-a-Day by and for Evaluators

Feb/10

24

Jeff Sheldon on the Readiness for Organizational Learning and Evaluation instrument

My name is Jeff Sheldon and I am a doctoral student at the School of Behavioral and Organizational Sciences at Claremont Graduate University concentrating in evaluation and applied research methods.  A challenge I’ve encountered in my own practice is organizations that simply aren’t ready to be engaged in the evaluation process.

Rad Resource: Readiness for Organizational Learning and Evaluation (ROLE).  The ROLE (Preskill & Torres, 2000) was designed to help us determine the level of readiness for implementing organizational learning, evaluation practices, and supporting processes.  I’ve implemented the ROLE with schools and other types organizations, using the results to: 1) identify the existence of learning organization characteristics; 2) diagnose interest in conducting evaluation that facilitates organizational learning; 3) identify areas of strength to leverage evaluative inquiry processes; and 4) identify areas in need of organizational change and development.  ROLE items reflects the research on organizational learning, evaluative inquiry processes, and evaluation practices, and suggest that an organization must have certain internal elements in place if it is to support and encourage organizational learning.  Evaluation research indicates an organization’s culture and context influence the extent to which evaluative inquiry occurs in support of learning and decision making.  The ROLE consists of 78 items grouped into six major constructs: 1) Culture, 2) Leadership, 3) Systems and Structures, 4) Communication, 5) Teams, and 6) Evaluation. Four of these – Culture, Leadership, Systems and Structures, and Communication – are further divided into subconstructs.  Individual responses to ROLE items created a composite perspective of the internal context of an organization that determines the extent to which organizational learning, evaluation practices, and systems are present.

The ROLE instrument is available as an appendix in Russ-Eft, D. & Preskill, H. (2001) Evaluation in organizations: A systematic approach to enhancing learning, performance, and change.  New York, NY: Basic Books.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

·

5 comments

  • Jara Dean-Coffey · March 7, 2010 at 5:28 pm

    Thanks Jeff for the additional info on context and application of ROLE. Hi Hallie, thanks for the offer to all to request a copy from you and good to know it is being added to the FSG toolbox. I think I have it but will follow-up if that is not the case.

    Reply

  • Hallie Preskill · March 1, 2010 at 9:26 am

    Thank you, Jeff, for this very supportive endorsement of the ROLE survey (and thank you too, Jara!). We developed this tool in response to folks who had read our book, Evaluative Inquiry for Learning in Organizations (Preskill & Torres, 1999), and were saying, “I like what you’re suggesting, but where do I start to build an evaluative learning culture and practice?” I believe the survey is best suited as a diagnositic instrument to gauge the readiness of an organization to embrace evaluative thinking and practice. While the numbers (means scores, standard deviations, frequences, etc.) are often quite interesting, the real value comes in the conversation organization members have about how and why the results came out the way they did. From there, a department or orgainzation can find points of leverage (where there are strengths), and areas that might need to be shored up or attended to during an evaluation effort (if it is to support learning).

    While the survey is in the appendix of the first(2001) and second edition (2009) of Evaluation in Organizations (Russ-Eft and Preskill, Perseus Books), I am happy to send it electronically to anyone who wishes to use it. It will soon be on the FSG Social Impact Advisors website (www.fsg-impact.org), but for now, if anyone wants a copy (there is no charge), they can contact me at hallie.preskill@fsg-impact.org.

    Reply

  • Jeff Sheldon · February 25, 2010 at 7:07 pm

    I most recently used a modified vesion (i.e., specific to an academic setting) of the ROLE with nine different schools here in Southern California and found: 1) that none of them were conducting evaluation or had conducted evaluations of any kind save for student assessments; 2) none of them were remotely interested in conducting evaluation, writ large; and 3) only two of them had a context that would actually support evaluative inquiry and organizational learning (kind of ironic given they are institutions of learning). I wrote a report for each school based on their spefic results and I offered to present in person, but got no takers. My read on the situation was that they wanted the information for their own purposes (don’t know what those were), but were not keen on acting on it in any meaningful way, at least not with me. Had I met with them I might have been able to tease out their evaluation needs based on where they were vis-a-vis the results of their ROLE. I would then have been able to start where they were, if that makes sense.

    That said, I think the keys to administration of the ROLE are: 1) modifying it to the setting (i.e., writing the items to reflect the parlance used by that type of organization – you can do so, but you have to keep an eye on changing the instrument’s reliability, which in it’s original form is exceptional); 2) giving it in person to anyone in the organization who is likely to be involved in evaluation (e.g., , I’ve administered it to school administration, faculty, and staff) – it would be best if you could get a census of the entire organization, but that may not be possible especially if it’s large; and, 3) administering it during initial meetings with the organization/group. I think if you tell them in your early conversatios that the ROLE is an integral part of your evaluation work and explain how it is also a benefit to them then (i.e., provides useful information about their organization they might not otherwise have known; you’ll present the findings at a subsequent meeting provided you want to go ahead with evaluation work after analyzing the data) it’s a relative easy sell; you could also have it written into your contract. It takes most people about 15 – 20 minutes to complete the instrument and responses are very easy to analyze. I don’t think there is a best scenario per se as I see it as a tool that can be used with any organization early on in the evaluation as long as they know it’s part of what you do. I hope this has been helpful and if you want to chat more off-line you can email me at: jeffrey.sheldon@cgu.edu

    Cheers,
    Jeff

    Reply

  • Jara Dean-Coffey · February 25, 2010 at 11:24 am

    Hi Jeff,

    I have known wbout ROLE for some time as I am a big Torres fan. Do you have any tips on determing the best scenario in which to use? Also, any thoughts on administration, level of effort, etc.?

    We have been looking for an opportunity to do so in our work but have not found it yet. Stil looking.

    Reply

  • Admin comment by Susan Kistler · February 24, 2010 at 5:39 am

    Jeff, thanks so much for encouraging use of this tool from Russ-Eft and Preskill. This link takes you to an except from the ROLE instrument found in google books for those who want to gain a better understanding of the types of questions and format (note that it has only the first few pages of the multi-page instrument) http://ow.ly/1aGC0

    What has been the outcome when you have used it with organizations?

    Reply

Leave a Reply

<<

>>

Archives

To top