My name is Jennifer Sullivan Sulewski and I am a Research Associate at the Institute for Community Inclusion (ICI) at the University of Massachusetts Boston. At ICI I have worked on a variety of research and evaluation projects related to services and supports for people with disabilities. I currently am most involved with Work Without Limits, a public-private partnership funded by the Massachusetts Medicaid Infrastructure and Comprehensive Employment Opportunities (MI-CEO) grant to strengthen the Massachusetts workforce and advance work opportunities for youth and adults with disabilities in Massachusetts.

Universal Design refers to “the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design” (http://www.design.ncsu.edu/cud/about_ud/udprincipleshtmlformat.html). The principles of universal design can be applied to evaluation to ensure that all relevant populations are included at every stage of the work, from project design to sharing of findings.

Hot Tip: Universal design is helpful to think about even if your evaluation is not specifically focused on disability programs or issues. You will likely encounter people with disabilities or members of other vulnerable populations whatever your focus, so it is good to be prepared. Moreover, good universal design works better for everyone, even those without disabilities or other barriers. Think of bar patrons watching TV with closed captioning on, or people with strollers or rolling suitcases using elevators and curb cuts.

Hot Tip: One key aspect of applying universal design to evaluation work is to think about all the different ways people communicate or access information. For example, if you are conducting a survey on-line, you might offer the option to do it on paper or over the phone if respondents prefer not to respond by computer. Or if you are doing interviews by phone you may find that some respondents prefer to speak in person or respond by e-mail.

Rad Resource: For more tips, join the AEA Coffee Break Webinar on this topic on Thursday, August 5 at 2PM – click here to learn more and sign up. The Disability and Other Vulnerable Populations TIG is also hosting multiple sessions related to Universal Design, including a skill building workshop and a roundtable, at the upcoming American Evaluation Association Annual Conference in San Antonio the first week in November, so look for us there!

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources.

communication, Disabilities, Inclusive, Universal Design

I am American Evaluation Association Executive Director Susan Kistler and I contribute each Saturday’s aea365 post. I have a dirty little secret – I’m obsessed with fonts! Stephanie Evergreen recently gave a great AEA Coffee Break Webinar on Graphic Design for Evaluators (she’ll also be expanding on this topic at AEA’s Annual Conference this November). Stephanie expounded on the importance of color choice and font selection to make great reports and presentations.

Rad Resources: WhatFontIs and WhattheFont both allow you to upload a picture of a font or specify a URL and they will return a best guess at the pictured font. WhatFontIs includes the option to display only free or similar free fonts and then download them on the spot – and it has hundreds in its archive (also browsable). Alternatively, WhattheFont also has a forum where font geeks will help you identify a font if you run into a dead end. Here is a great walk-through of from the MakeUseOf blog. Both of these sites are in beta, and aren’t perfect, but I’ve been impressed with the options that they’ve provided – and the sheer volume of free fonts available from WhatFontIs for free.

Stephanie encouraged her webinar attendees to use kuler to identify particular colors appropriate to a report based on those used by a client. Using tools such as WhatFontIs takes this concept one step further.

Rad Resources: This short article from Chuck Green gives examples of sets of fonts that work together to convey a mood or message. I have no eye for such things and find this guidance invaluable.

Hot Tip: Know your vocabulary when talking about fonts with a designer or using fonts in your word processing program. Here are three definitions that will help you along:

• Serifs: Serifs are the little ‘feet’ that appear on many fonts. Fonts come in two types – Serif fonts such as Times New Roman and Sans serif fonts such as Arial.
• Kerning: Kerning refers to adjusting the space between letters so that the white space is similar from letter to letter, for instance pushing a ‘A ‘and ‘W’ up close to one another producing ‘AW’ so that they actually overlap in vertical space. You can adjust kerning in Microsoft Word under the character spacing. Kerning is used in particular when creating headlines or banners.
• Proportional Typefaces: Almost all typefaces today are proportional, allotting varying amounts of horizontal space to a letter based on its shape so that an ‘l’ receives less space than an ‘m’. Old typewriters used monospace fonts.

Rad Resource: A moment of fun for the font obsessed – take a look at this comic that reflects on font choice http://ow.ly/2hf2e.

Note: These insights are my own and do not necessarily represent the opinion of the American Evaluation Association.

communication, font, presentations

My name is Kathleen Norris and I am an Assistant Professor and Program Coordinator within the doctoral program in Learning, Leadership, and Community at Plymouth State University.

An arts organization I work with was stuck when it came to program evaluation. They wanted it, knew they should have it, but didn’t know how to begin. We discovered that a large part of the challenge was that they did not have a way of talking about this fairly complex organization that could be understood by everyone in the organization.

Hot Tip: As we met to work on this it became apparent that the organization was the “sun” in an entire solar system with planets, moons, various gravitational pulls and distant stars. Once this metaphor was established, everyone could use it when talking about the organization and it helped to engage several members who had not previously contributed in our discussions. When new “bodies” came into the conversation, we could determine whether they were planets, moons, zooming comets or space junk, etc. Further work with the board and staff allowed opportunities for the members to draw (literally) what “mission” means to them, and then discuss the organization’s mission using the drawings they had created. Some sketched traditional California Spanish Missions, some identified with “Mission Impossible” and a variety of other meanings of “mission” and then we were able to talk about how their understanding of mission in general was like the mission of the organization and from there move to a deeper connection to the real mission of the organization. Now that we are engaging in a deeper analysis of the work of the organization, being able to categorize the work within the metaphor of the solar system, for example, has made the evaluation work seem less abstract and actually more fun.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Kathleen? She’ll be presenting as part of the Evaluation 2010 Conference Program, November 10-13 in San Antonio.

Evaluation planning, metaphors, mission

My name is Marijata Daniel-Echols. I am Director of Research at the HighScope Educational Research Foundation. HighScope is best known for our work in preschool curriculum development and for the Perry Preschool Study. Our Research Department conducts evaluations of early childhood programs (e.g. state funded preschool initiatives) and general research on early childhood theory and practice. Because our work is focused on turning data into actual policy and changes in practice, we spend a great deal of time working with program partners who do not necessarily have a strong background in research or evaluation methods.

Lesson Learned: Researcher-program partnerships can be both a point of strength and a challenge. Program evaluation is often seen by administrators as threatening or focused solely on accountability as opposed to informing a cycle of ongoing program improvement. Evaluators often hold unrealistic expectations of the capacity of their partners to meet rigorous evaluation design and data collection standards given real world constraints.

When researcher-program (or evaluator-client) partnerships are successful, they can result in clear, relevant, and useful data. In my 10 years of experience conducting research in partnership with programs, there are a couple of basic tenets to follow:

• Having clear expectations of what everyone has to gain, lose, and must contribute to the evaluation process is essential. Acknowledge the trepidations and anticipations on both sides and keep them in mind during the evaluation planning, implementation, and dissemination phases of your work.

• Involve your partner in the evaluation process from the beginning. Early understanding and buy-in increases the likelihood that they will work hard to protect the integrity of your design. Solicit from them what they hope to learn from the evaluation process and to what uses they plan on putting the resulting report(s).

• Have your partner work with you to delineate their theory of change and create a logic model that guides your methods and instrumentation. That process will make clear to everyone what questions the evaluation will and will not be able to answer and what information has to be collected.

• Share preliminary findings with your partner and ask for their interpretation of the findings. In addition to creating a sense of ownership and understanding of the data, you will gain useful contextual insights that will help you draw conclusions and make suggestions for improvement.

• Work with your partner to create an evaluation report dissemination plan. Keep in mind that there are probably several different types of stakeholders with different data reporting needs.

• Finally, recognize that providing relevant, useful evaluation data leads naturally into consultation – ongoing support from you will increase the likelihood that the data is in fact used to make real changes in policy and practice.

This contribution is from the aea365 Tip-a-Day alerts, by and for evaluators, from the American Evaluation Association. If you’d like to learn more from Marijata, consider attending her session at the AEA Annual Conference this November in San Antonio. Search the conference program to find Marijata’s session or any of over 600 to be presented.

Childhood, partnership

My name is Andrea Velasquez, and I am a doctoral student at Brigham Young University. For the last four years, I have been an instructor of an undergraduate class that teaches pre-service teachers how to use technology effectively in elementary and secondary education settings. One of the principal frameworks that we use to teach pre-service teachers how to distinguish between all the facets of designing effective instruction with technology is TPACK, or Technological Pedagogical Content Knowledge (Mishra & Koehler, 2006). This framework states that in any effective technology mediated instruction, technology, pedagogy and content are three components that not only co-exist, but also interact and have an impact on each other. The research examining TPACK can be useful to the field of evaluation by applying it to evaluations of technology-mediated instruction. Distinguishing between these three components- technology, instructional content, and instructional strategies- can help evaluators identify appropriate questions and alleviate the complexity of evaluating e-learning.

Hot Tip: When designing an evaluation of technology-mediated instruction, after determining context and stakeholders, consider technology, pedagogy and content as evaluands. Then, identify criteria and questions for judging each evaluand. Before continuing the evaluation, also identify criteria and questions that take into account how each component impacts the others. These questions should address the compatibility between these components. For example, if an online high school uses video technologies to communicate with students, an evaluation of such a program should take into account the video technologies, the strategies the teacher uses to teach the class (i.e. group work, field experiences, presentations), and the content of the instruction he is teaching. Besides addressing each of these three components, the evaluation should address the relationships that exist between each of these components at each stage of the evaluation process. This approach to evaluation ensures a more holistic evaluation of the technology use in relation to the context and the needs of the students and stakeholders.

Rad Resource: This site is a resource that is maintained by the developers of the TPACK framework. It has updated research articles and many other resources for understanding the practical applications of the framework http://www.tpck.org/tpck/index.php?title=Main_Page

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Want to learn more from Andrea? She’ll be presenting as part of the Evaluation 2010 Conference Program, November 10-13 in San Antonio.

instructional design, pedagogy, TPACK

Hi, my name is Anne Vo and I am a doctoral student in the Graduate School of Education at the University of California, Los Angeles (UCLA). I have been using conversation analysis (CA) to study how program evaluation is taught in informal settings. CA is a research method and tool that was developed during the 1960’s and 1970’s in the field of sociology for the purposes of examining “talk-in-interaction” – the way in which interactions are organized through conversation. It is also used broadly by scholars in disciplines such as anthropology and linguistics. But, evaluators who find that they need to answer questions about the nature and quality of interaction and social relationships in their evaluations may also find CA useful. So, what you will find below are some things to consider while doing CA, a link to an online module for those who may be interested in learning more about what “doing CA” entails, and some references for further exploration.

As with every research and evaluation method, CA has its advantages and disadvantages. In this case, they are quite similar to what we normally encounter when using observational or ethnographic methods. A few methodological considerations, strengths, weaknesses, and pitfalls to consider appear below.

Rad Resource: The following link will take you to an online training module that will walk you through the CA transcription process. This module was created by one of the method’s developers, Dr. Emanuel Schegloff. http://www.sscnet.ucla.edu/soc/faculty/schegloff/TranscriptionProject/index.html

Rad Resources: These provide a great introduction to CA:

• Sacks, H., Schegloff, E.A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4): 696-735.

And, the following books are useful references for those who have long-term interest in CA:

• Ochs, E., Schegloff, E.A. & Thompson, S.A. (Eds.). (1996). Interaction and grammar. New York, NY: Cambridge University Press.
• Schegloff, E.A. (2007). Sequence organization in interaction: A primer in conversation analysis: Volume 1. New York, NY: Cambridge University Press.

Hot Tip: Methodological Considerations.

• Large social constructs that are often studied using CA include human interaction, social relationships, and context.
• In CA, these social constructs are indexed by units of analyses not limited to: turn taking, turn constructional units, and sequence organization.
• “Doing CA” involves meticulous transcription of audio or video recordings using CA notation. Analysis involves concurrent use of the audio/video data with the transcript, but the recordings remain the primary data sources.

Strengths:

• Because CA is inherently a fine-grained tool, one of its strengths is its precision, which can be used to identify nuances that might have otherwise been overlooked.
• It also allows analysis to occur specifically at the conversational level, which is not generally afforded by other methods.

Limitations of the method:

• CA transcription is a time consuming and costly activity.
• And, it requires specialized training in CA methods if it is going to be done well.

Pitfalls:

• The analytic process is sensitive to misinterpretation and over-interpretation so member checking becomes doubly crucial here.
• Likewise, the analyst should be wary of the tendency to essentialize the phenomena that they are observing; rather, she or he should always triangulate these initial patterns before arriving at conclusions.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

conversation analysis

My name is Michael Quinn Patton and I am an independent evaluation consultant. That means I make my living meeting my clients’ information needs. Over the last few years, I have found increasing demand for innovative evaluation approaches to evaluate innovations. In other words, social innovators and funders of innovative initiatives want and need an evaluation approach that they perceive to be a good match with the nature and scope of innovations they are attempting.  Out of working with these social innovators emerged an approach I’ve called developmental evaluation that applies complexity concepts to enhance innovation and support evaluation use.

Hot Tip: Innovations are different from standard projects and programs.  Innovators are often different from people implementing typical programs.  Innovators are in a hurry, value rapid, real time feedback, have a high tolerance for ambiguity, embrace uncertainty, learn quickly, and adapt rapidly to changed conditions. They’re not always sure where they’re heading, so they resist being boxed in by concrete, pre-set targets. They’re propelled into action more by vision than by clear, specific and measurable outcomes. They want an evaluation approach attuned to their fast pace and innovative spirit. They are at home in complex dynamic systems. Such systems characterize the world in which they live and work. Thus, they want an evaluation approach attuned to complexity.

Hot tip: Complex situations challenge traditional evaluation practices. Complexity can be defined as situations in which how to achieve desired results is not known (high uncertainty), key stakeholders disagree about what to do and how to do it, and many factors are interacting in a dynamic environment that undermine efforts at control, making predictions and static models problematic.  Complexity concepts include nonlinearity (small actions can produce large reactions), emergence (patterns emerge from self-organization among interacting agents), and dynamic adaptations (interacting elements and agents respond and adapt to each other).

Hot tip: Developmental evaluation aims to meet the needs of social innovators by applying complexity concepts to enhance innovation and use. Developmental evaluation focuses on what is being developed through innovative engagement.

Rad Resources:
•    Developmental evaluation: Applying complexity concepts to enhance innovation and use by Michael Quinn Patton (Guilford Press, 2010).*
•    A developmental evaluation primer. Jamie Gamble. (2008). Montréal: The J.W. McConnell Family Foundation.
•     DE 201: A Practitioner’s Guide to Developmental Evaluation by Elizabeth Dozois,
Marc Langlois and Natasha Blanchet-Cohen. Montréal: The J.W. McConnell Family Foundation.
•    AEA Annual Conference professional development workshop on Developmental Evaluation, with Michael Quinn Patton, November 8-9, San Antonio.

*AEA Members receive 20% off on all books ordered directly from Guilford. If you are a member, sign into the AEA website at http://eval.org/ and select “Publications Discount Codes” from the “Members Only” menu to access the discount codes and process.

developmental evaluation, funders, innovation

My name is Lori Wingate. I am a Principal Research Associate at The Evaluation Center at Western Michigan University. Two closely related topics I return to frequently in my research, practice, and teaching are metaevaluation and the Program Evaluation Standards (Joint Committee, 1994).  Here I share some lessons learned from my recent dissertation research on the use of the Program Evaluation Standards a rating tool for metaevaluation.

The Program Evaluation Standards are a set of 30 standards organized in four domains: utility, feasibility, propriety, and accuracy. Correspondingly, they are grounded in the principles that evaluations should be useful, practical, ethical, and valid.

Because of their applicability to a broad array evaluation contexts and widespread acceptance, they are often used as criteria in metaevaluation. Although the Standards provide a useful metaevaluation framework, there are some significant challenges to their application when a metaevaluation is focused on evaluation reports, without opportunity to gather additional information about the evaluation’s conduct.

This claim is based on my personal experience in using the Standards to evaluate reports, and is strongly supported by the findings from my study of interrater agreement in metaevaluation. Although agreement was generally low across all the standards, the uncalibrated raters had the least agreement on standards in the feasibility and propriety domains, which are largely concerned with issues related to the manner in which an evaluation is carried out. With only reports in hand to judge the evaluation, raters had to infer quite a bit in order to make judgments about evaluation process.

If you’re thinking of conducting a metaevaluation in which you will use the Program Evaluation Standards as criteria and you have only evaluation reports for data, here are some tips and resources that may help make it a more valid and useful endeavor:

Hot Tip: Select only those standards on which judgments can be made based on information that is typically included in evaluation reports.

Rad Resources: Check out the Program Evaluation Standards at www.jcsee.org.  Watch for a new edition to be published this year. A review of Dan Stufflebeam’s Program Evaluation Metaevaluation Checklist will help you get started in determining which standards will be feasible for use in your metaevaluation.

Hot Tip: If you want to look at several reports produced for a by a single organization or in a single content area, spend some time developing tailored criteria for that context.

Rad Resource: ALNAP’S Quality Proforma is an instrument designed for assessing humanitarian action evaluation reports. The criteria are tailored to the domain in which the evaluations were conducted and are focused on report quality.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

metaevaluation, standards

My name is Susan Kistler, AEA’s Executive Director, and I contribute each Saturday’s aea365 post. I love finding ways to make data understandable and useful. My very first aea365 post was on Data Visualization and I gave a presentation at the 2010 AEA/CDC Summer Evaluation Institute on the Democratization of Data Inquiry (the handout, with links to example tools, may be downloaded here).

Today, I want to take a different tack and think about the intersection of art, data, and representation, being careful not to imply that beautiful graphs are not art (Tufte immediately comes to mind). Yet I hope to move beyond the printed page or the webpage through providing two examples from recent museum exhibits.

Lessons Learned: In August of 2009, I attended Roman Ondak’s “Measuring the Universe” at the Museum of Modern Art in New York. The performance exhibition consisted of having docents mark the name, height, and date of people entering a large white walled room, with each exhibit goer standing against the wall. The result was a compelling representation of participatory research in which each person contributed a data point to the resulting artwork which built over the course of a couple of months to depict the distribution of heights of New York city museum goers. My two pictures above do not do the exhibition justice – see the MoMA exhibit online for more information. I found the piece to be powerful as was the eagerness with which museum-goers, from children to seniors, wanted to participate and be recognized via measurement and recording.

Lessons Learned: In July of 2010, I visited The Tech Museum in San Jose. They had a small exhibit full of internally lit globes. Upon each globe, someone had painstakingly painted or brushed colors, added text and numbers, and illustrated a range of demographic data – the glowing yellow one above shows percent of the world’s energy use by region and the browner one is life expectancy. Again, my flat pictures do not do justice to the beauty and intrigue of the globes around which visitors from many countries walked or ducked or stood on tip toe to locate their selves within the broader context. Like the MoMA exhibit, the globes created a compelling expression of data in a way with which users wanted to interact and learn.

My question is to aea365 readers to ask if you have examples to share of data representation that bridged the gap between tables and graphs and artistic expression. What have you done? What has worked? Share in the comments section of this post (click back to the post online if you are receiving this via email or RSS), or send me an email at susan@eval.org and perhaps we can develop an exhibit space!

No tags

Greetings, My name is Mehmet Dali Ozturk. I am the Assistant Vice President of Research, Evaluation and Development at Arizona State University Office of Education Partnerships (VPEP), an office that works with P-20, public and private sector partners to enhance the academic performance of students in high need communities.

Along with my colleagues Brian Garbarini and Kerry Lawton, I have been working to develop sound and reliable evaluations to assess educational partnerships and their ability to promote systemic change. One of my ongoing projects has been the evaluation of ASYouth, a program developed to provide a holistic support system to the University, schools, and parents so that disadvantaged children have the opportunity to participate in university-based summer enrichment activities.

Based on this experience, we offer the following advice to evaluators working on university-based outreach programs:

Hot Tip: Create a Multi-Disciplinary Evaluation Team

Although most University-led summer enrichment programs are directed towards similar goals, the activities often focus on a multitude of subjects ranging from drama, music and art to intensive math and science courses. Given this, evaluation teams that recruit individuals with expertise in a variety of academic subjects are well-equipped to develop evaluation designs and assessment tools appropriate to these programs.

Hot Tip: Ensure Linguistic and Cultural Relevance

Evaluations should be developed and conducted by evaluation teams that possess cultural competency to the target population. This allows for the development of culturally sensitive assessment materials that can be translated into the heritage language of the program participants at a fraction of the cost of hiring outside consultants. In addition, when survey methods are used, culturally-appropriate measures will result in higher initial response rates. The need for fewer follow-ups can greatly reduce the cost of successful evaluations.

Hot Tip: Embed Evaluation into Program Design

Due to limited resources, evaluation expertise, and/or capacity, many summer enrichment programs do not include rigorous evaluation components. In these cases, evaluation is merely an afterthought, making it very difficult to ensure valid data collection or implement a design with appropriate controls.

This aea365 contribution is part of College Access Programs week sponsored by AEA’s College Access Programs Topical Interest Group. Be sure to subscribe to AEA’s Headlines and Resources weekly update in order to tap into great CAP resources, and to consider attending CAP-sponsored sessions this November at Evaluation 2010.

cultural competency, program design, summer outreach programs