AEA365 | A Tip-a-Day by and for Evaluators

CAT | Uncategorized

I am Amy Hilgendorf, Associate Director for Engaged Research at the Center for Community and Nonprofit Studies of the University of Wisconsin-Madison. For the past three years, I have had the pleasure of convening a community of practice of evaluation practitioners here on our campus. What started with two graduate students and their interest in getting to know evaluators on campus and creating space for learning together, has grown to become much more.

Hot Tip:

A lot of good can come from just a little bit of effort! While there’s a lot you can read out there about communities of practice and how to support them, ours runs with just a few simple practices:

  • Come together regularly (typically once a month)
  • Highlight the work of a member and/or a topic of shared interest for evaluators
  • Provide food and reserve time for networking
  • Help members communicate with one another (like through an email list or with social media)

With just these few practices, we have grown to a network of more than 80 evaluation practitioners, students, and appreciators, and have started to incorporate members not affiliated with the university. I am especially proud of the exchanges we have had in our monthly gatherings, including thoughtful conversations around evaluation ethics enhancing social justice through evaluation practice.

Lesson Learned:

When you build a network of smart and passionate people, valuable developments will rise organically. Through our community of practice, I have learned about approaches and methods I knew little about before, such as Ripple Effect Mapping and critical cartography. We have also gained the inside scoop on AEA’s Graduate Education Diversity Initiative (GEDI) from our university’s resident intern and some of us are working together to coordinate applications to host GEDI’s next year. And this summer, we plan to use some of our time together to develop ideas for how we can build and support a pipeline at our university for more evaluators of diverse and underserved backgrounds.

Cool Trick:

Building relationships, including professional ones, often comes down to getting to know each other and having fun together. So make sure to reserve time and space for networking and fun in a community of practice. One of our best attended sessions has been our summer happy hour at the student union by the lake, with a cold Wisconsin brew in hand.

From our community of practice to yours, thank you.

The American Evaluation Association is celebrating The Wisconsin Idea in Action Week coordinated by the LEAD Center. The LEAD (Learning through Evaluation, Adaptation, and Dissemination) Center is housed within the Wisconsin Center for Education Research (WCER) at the School of EducationUniversity of Wisconsin-Madison and advances the quality of teaching and learning by evaluating the effectiveness and impact of educational innovations, policies, and practices within higher education. The contributions all this week to aea365 come from student and adult evaluators living in and practicing evaluation from the state of WI. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi!  I’m Kathy Newcomer, president of the American Evaluation Association.  I would like to learn and share what the AEA 2017 Conference theme “From Learning to Action” means to evaluators! To help share the theme across as wide an audience as possible, we’re inviting folks to submit brief videos that explore just this.   Links to winning videos will be posted on the AEA web page, and the top video will be featured during the opening plenary.   Up to five winners will also get great prizes!

Hot Tip: Your video doesn’t have to have high tech special effects or big name Hollywood stars.   By June 16th, submit a brief 45 to 60 second video addressing one or more of these questions:

  • What does the Evaluation 2017 theme From Learning to Action mean to you personally or professionally?
  • What could the theme mean to other evaluators?
  • What can the theme mean for the evaluation profession?

See more on how to enter your video here.

Hot Tip:  Consider the conference sub themes:

Learning to Enhance Evaluation Practices:  Evaluation theory and practice has been dynamically developing with innovative and expanding approaches. What are new developments in practicing and teaching evaluation that may advance our contribution to the generation of knowledge about effective human action?

Learning What Works and Why: Evaluation studies have been providing evidence about the effectiveness, efficiency, and utility of public programs and policies. We have been learning about mechanisms that contribute to the successes or failures of interventions. What have we learned about what works and why in different sectors and contexts, that could be useful for policy practitioners in improving public policies?

Learning from Others:  New communities such as behavioral insight teams, social labs, big data analysts, and design thinkers offer new insights to inform effective programs and policies. What can we learn from other communities, including evaluation communities outside of the US, to advance evaluation practice and knowledge about promising tools and approaches?

Learning About Evaluation Users and Uses:  For years’ evaluators have been struggling to increase meaningful use of evaluation by stakeholders. What have we learned about users of our work, their ways of acquiring and using knowledge, and useful ways to support them in applying evaluation findings to improve practice?

Need more ideas? See more on the conference theme here.

Rad Resource: Not sure how to get started making your brief video?

AEA365 Curator note: We generally feature posts by AEA staff and AEA365 Curators on Saturdays, and are now pleased to offer occasional Saturday blog posts from our esteemed AEA Board members!

Hi, I am Dominica McBride with Become: Center for Community Engagement and Social Change and serve you on the AEA Board of Directors.

John F. Kennedy said, “It is from numberless diverse acts of courage and belief that human history is shaped. Each time a [person] stands up for an ideal, or acts to improve the lot of others, or strikes out against injustice, he sends forth a tiny ripple of hope, and crossing each other from a million different centers of energy and daring those ripples build a current which can sweep down the mightiest walls of oppression and resistance.”

In the midst of national leaders acting against our values as an organization, explicitly marginalizing many who find a professional home in AEA and harming communities that many of us serve, I believe we are called as professionals and human beings to make ripples.

In the face of a grim reality, I have hope, especially given what I know about us as evaluators. We are connected to various organizations that are connected to many people, from residents to leaders. We’re able to critically and empirically explore the intersection of our content area and the sociopolitical context and how we may use our position and expertise to move forward on a broader issue. We have a unique set of skills – to gather information, think critically, analyze, synthesize and communicate. We are able to partner with organizations and leaders in many ways to use our skillset towards action around an issue.

With this potential, there are various possibilities for a new or refined role for evaluators to make a necessary difference in this environment. For example, we could:

  • Advocate or mobilize our partners, clients and communities to move in a common direction
  • Build resilience in the systems and institutions that are being depleted of resources
  • Help communities construct new systems and programs that work for and, in many cases, could be run by them

Hot Tips:

Begin one-on-one meetings with your clients, partners, colleagues or fellow community members. Remember to reach out and listen to those not often included in evaluation, such as returning citizens from incarceration, single mothers struggling to get by, and disenfranchised youth. Listen for recurring themes about what matters to them and what may motivate them to act collectively.

After those meetings, convene groups around that common issue to develop a plan of action and ground that action in evidence.

 

Rad Resource:

To learn more about advocacy, mobilizing and organizing and for examples on successful collective action, read Jane McAlevey’s book No Shortcuts: Organizing for Power in the New Gilded Age.

*If you’re interested in exploring or working together around these possibilities, please reach out to me at dmcbride@becomecenter.org or 312-394-9274.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hi, we are Tim Sheldon and Jane Fields, Research Associates at the Center for Applied Research and Educational Improvement (CAREI) at the University of Minnesota. We serve as external evaluators for EngrTEAMS, a five-year, $8 million project funded by the National Science Foundation. The project is a partnership involving the University of Minnesota’s Science, Technology, Engineering, and Mathematics Education Center (the STEM Center) and Center for Compact and Efficient Fluid Power (CCEFP); Purdue University’s Institute for P-12 Engineering Research and Learning (INSPIRE!); and several school districts. EngrTEAMS is designed to increase students’ learning of science content, as well as mathematical concepts related to data analysis and measurement, by using an engineering design-based approach to teacher professional development and curriculum development.

Context:

As the external evaluators for this project, we based our evaluation framework on Guskey’s five levels of professional development (PD) evaluation (Guskey, 2002). He suggests evaluating (1) participant perceptions of the PD; (2) the knowledge and skills gained by participants; (3) the support from, and impact on, the organization; (4) participants use of their new knowledge and skills; and (5) the impact on student outcomes. In Guskey’s model, the aspects to be evaluated begin after delivery of the PD; that is, the framework does not specifically suggest assessing differences in participants or organizations prior to the delivery of the PD.

In the case of EngrTEAMS and other PD we have evaluated, we have noticed that even though participants receive the same training (i.e., the same “treatment”), their capacity to apply the new knowledge and skills (Guskey level 4) is not the same. What might explain this? We suggest that one way to better understand and explain these differences in implementation (and eventually student outcomes) is to also better understand participants and their organizations prior to the PD. Not all participants start the PD in the same place; for example, participants come to the PD with different levels of prior knowledge, different attitudes about the PD, different classroom management abilities, and different levels of organizational support.

Lesson learned:

When possible, assess implementation readiness of participants and their organizations prior to the delivery of the PD. This may include obtaining information about organizational readiness to support novel approaches, as well as participants’ prior content knowledge and classroom experience, their perception of school or district buy-in, and participants’ attitudes about the training and future adoption of what they will be learning.

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi. My name is Jonny Morell. I’m Director of Evaluation at Syntek Technologies, a Principal at 4.669… Evaluation and Planning, and editor of Evaluation and Program Planning.

I recently did an evaluation that made me realize a few things. 1) Project timelines are a type of logic model. After all, they identify program activities and relationships among those activities. 2) Murphy was not quite right. It’s not that anything that can go wrong will, but that some things will go wrong. 3) Things go wrong for different reasons. 4) The reasons things go wrong can provide a lot of useful information about program behavior and outcome. For instance: Did most of the delays come from labor/management conflict, failure to anticipate demands on people’s time, or the demands of a better business climate? Which of these delays were longest, or hardest to resolve? Which affected outcome, budget, or stakeholder expectations? For the important delays, what were the critical incidents that caused the delay? What constellation of small factors combined to cause the delay? I built an entire evaluation around using timelines to answer these kinds of questions.

Lessons Learned: Organizing evaluation around the reasons and consequences of schedule slips provides a lot of useful knowledge that is hard to get by other means.

Rad Resource: I have a blog post that goes into a lot more detail on what I did and how I did it.

Timelines, Critical Incidents and Systems: A Nice Way to Understand Programs

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

 

·

Greetings AEA colleagues. We are Carla Hillerns and Pei-Pei Lei – survey enthusiasts in the Office of Survey Research at the University of Massachusetts Medical School. In 2014, we shared a post about effective email subject lines for internet survey invitations. Today we’d like to focus on the body of the email. Here are strategies for writing email invitations that motivate recipients to participate in your survey.

Hot Tips:

  • Personalize the salutation. Whenever possible, begin the invitation with the recipient’s name, such as “Dear Carla Hillerns” or “Dear Ms. Lei.” Personalization helps people know that they’re the intended recipient of the invitation.
  • Do not bury the lead. Use the first line or two of the email to invite the recipient to take the survey. Some people might open your email on mobile devices, which have significantly smaller screen sizes than most computers.
  • Include the essentials. A survey invitation should accomplish the following:
    • Explain why the individual is chosen for the survey
    • Request participation in the survey
    • Explain why participation is important
    • Provide clear instructions for accessing the survey
    • Address key concerns, such as confidentiality, and provide a way for recipients to ask questions about the survey, such as a telephone number and email address
    • Express appreciation
    • Include sender information that conveys the survey’s legitimacy and significance
  • Less is more. The most frequent problem we’ve seen is an overly wordy invitation. Follow the modified KISS principle – Keep It Short and Simple. Common issues that complicate invitations are:
    • Overlong sentences
    • Redundant points
    • Extra background details
    • Cryptic wording, such as acronyms and technical jargon
    • Intricate instructions for accessing and/or completing the survey

Cool Trick:

  • Pre-notify, if appropriate. Examples of pre-notifications include an advance letter from a key sponsor or an announcement at a meeting. Pre-notification can be a great way to relay compelling information about the survey so that the email invitation can focus on its purpose.

Lesson Learned:

Rad Resources:

  • Emily Lauer and Courtney Dutra’s AEA365 post on using Plain Language offers useful tips that can be applied to all aspects of survey design and implementation, including the initial invitation email, any reminders emails, and the survey itself.
  • Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 4th Edition by Don A. Dillman, Jolene D. Smyth, and Leah Melani Christian provides lots of helpful guidance for crafting invitations and implementing internet surveys.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi, I’m Harlan Luxenberg.  I’m the President and CEO of Professional Data Analysts, a small company that specializes in evaluating public health programs. I have reviewed hundreds of applications for evaluator positions over the years, and I wanted to share what I find to be the essential ingredients for an application that we want to follow up with.

Hot Tips:

  • Do your homework. Research the company you’re applying to and understand what makes them unique. Make your cover letter stand out by displaying your intimate knowledge of the company in your cover letter. It also never hurts to ask for an informational interview before applying to make a connection and learn more about the company.
  • Don’t forget you’re applying for a job and not a graduate program. A company wants to hear about what you bring to the table, but also wants to know that you understand what they’re looking for. Don’t just list your skills and experience and assume that the person reviewing applications will be able to draw the connections about how those skills serve their needs. Make sure you explicitly address what the company is looking for.
  • Be excited!!! If you’re excited about a specific aspect of the company, share that. Whether it’s about an employee that you heard speak at AEA, a topic area you want to learn more about, or something else unique to the company – it’s important to say what draws you to the employer. If you’re excited about working with them, they are more likely to be excited about working with you.
  • Don’t overdo it. Your application should be professional, concise, and in PDF format, but don’t retell your resume in your cover letter. Don’t add extraneous formats, styling, fonts, infographics, etc. Employers are looking for someone to interview, they don’t need to know your whole story yet.
  • Proofread! This should be a no-brainer, but probably 90% of applications we receive have grammatical issues. Even if it’s very minor, it reflects poorly on you and tells the prospective employer that you don’t pay attention to details. Always have others review your cover letter and resume.

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hello Evaluation Learners! I’m Sheila B. Robinson, aea365’s Lead Curator and sometimes Saturday contributor. Today, I’m writing about AEA’s Summer Evaluation Institute, a perennial fabulous learning opportunity. Anyone who knows me knows that I love learning and meeting with evaluation colleagues, and this is the perfect opportunity for both.

Registration is now open for the 2017 AEA Summer Evaluation Institute – June 4 – June 7 in Atlanta, GA. Here’s a quick preview of just a few  of the 28 high quality courses offered. Note: Descriptions are truncated, so please visit the site for complete descriptions:

Rad Resources: 

This year’s institute will feature an Appreciative Evaluation Keynote by Anastasia (Tessie) Tzavaras Catsambas, founder and CEO/CFO of EnCompass LLC, an organization that provides services in evaluation, learning, leadership and organizational development. Catsambas will demonstrate the scientific basis for using Appreciative Inquiry in evaluation, its contribution to getting better data by minimizing bias, and its role in increasing evaluation use.

Institute course offerings include:

Nonparametric Statistics — What to Do When Your Data Breaks the Rules – Jennifer Catrambone

This session walks participants through nonparametric statistics, techniques designed to be used on small, uneven, or skewed samples. Participants will leave with a stand-alone handout that clearly identifies situations in which nonparametric statistics should be used, explains when and why they are appropriate, illustrates how to run the techniques in SPSS (including annotated screen shots), how to interpret the output, and how to write up the results.

Focus Groups for Qualitative Topics – Michelle Revels

As a qualitative research method, focus groups are an important tool to help researchers understand the motivators and determinants of a given behavior. This course, based on the seminal work of Richard Krueger and David Morgan, provides a practical introduction to focus group research.

Evaluating Organizational Collaboration and Networks – Rebecca Woodland

“Collaboration” is a ubiquitous, yet misunderstood, under-empiricized and un-operationalized construct. Program leaders and organizational stakeholders looking to do collaboration and build networks struggle to identify, practice and evaluate it with efficacy. In this workshop, we will explore how the principles of collaboration theory can be used to plan, evaluate, and improve collaboration in the context of organizations/programs, partnerships, and networks.

A Participatory Method for Engaging Stakeholders with Evaluation Findings – Adrienne E. Adams

In this workshop, learn how to facilitate the “Expectations to Change (E2C)” process, a six-step, interactive, workshop-based method for guiding evaluation stakeholders from establishing performance standards (i.e., “expectations”) to formulating action steps toward desired programmatic change. The E2C process is designed to engage stakeholders with their evaluation findings as a means of promoting evaluation use and building evaluation capacity. The distinguishing feature of this process is that it is uniquely suited for contexts in which the aim is to assess performance on a set of indicators by comparing actual performance to planned performance standards for the purpose of program improvement.

Hot Tip: Act fast to register for the 2017 AEA Summer Evaluation Institute! Courses do fill up!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Good morning!  I’m Liz Zadnik, aea365’s Outreach Coordinator and Saturday contributor.  Part of my role on the curating team is working with evaluators and researchers interested in generating content for the blog.  Writing for the web is a little different than drafting an evaluation report, policy brief, or peer-review journal article – it requires a slightly more conversational and informal tone.  I’ve pulled together a few tips and resources for folks interested in refining their online-writing style.

Hot Tip: Frontload your information. Basically, put the most interesting or poignant nuggets first.  This is a little different than most of the resources you may usually write – results or findings are typically contextualized first and then outlined later.  Not online.  Blog and website visitors are looking for something – give them what they want.  They’ll peruse a page, scanning for keywords.  If they don’t see what they’re looking for, they’ll leave.  

Lesson Learned: White space is your friend.  Many people equate dense paragraphs with quality – that won’t do for online content!  Embrace patches of white space – throughout the page and also within the content.  “How do I do that?!”  Well, you can use bulleted or numbered lists, images, or line breaks between paragraphs.  Don’t worry if you feel it looks sparse – your readers will thank you!    

Hot Tip: Get active!  With your voice, that is.  Writing for the web is intended to keep the visitor engaged for short period of time.  Folks have something in mind when they visit a site and want to be spoken to directly.  Active voice helps create that atmosphere – it also makes blocks of text for readable and scannable.  

FROM “The participants’ questions were gathered by the meeting facilitator.” (passive)

TO “The meeting facilitator gathered participants’ questions.” (active)

Just to be clear, passive voice isn’t bad.  It has its place in scientific and academic writing.  But blogs and websites are different and should look and sound different.  This style can be difficult to practice at first, but I’ve found it has strengthened my writing both professionally and personally.   

Rad Resources:

  • Usability.gov offers a checklist and more tips on effectively writing for the web.
  • Writing Spaces pulled together a style guide a few years ago – it has some nice background on different platforms and “genres” of web writing
  • Speaking of style guides, Sum of Us offers a very thoughtful one, A Progressive’s Style Guide, for folks interested in harnessing language as a tool for social change. 

I would also encourage you to pay attention to blogs and websites you really like.  How do they use white space?  How/Do they offer a scannable page for visitors?  What information do they offer?  

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello, my name is Jayne Corso and I am the Community Manager for AEA. Posting on multiple social media sites requires good imagery, and on a low budget this can be tough. Images make your content eye-catching and can even add context to a post. On all channels, posting with images out preforms those without images. Canva is an easy and free way to create your own graphics, charts, infographics, and images. Today, I will show you how to create an image using free Canva formats, layouts, and photos.

Rad Resource: Choose your format

Each social media channel has a preferred image size. This size will allow your photos to be clearly viewed in a newsfeed. Canva takes the guess work out, and helps you create images specifically for each channel. They have an array of sizes you can choose from. You can even create a custom design by entering your own dimensions. For this example, we will be choosing the Facebook post format.

Rad Resource: Find a Layout

Canva offer many free layout that you can edit with your own content. Simply click on the layout you like and it will be added to your canvas.

Rad Resource: Edit your image

Once you have selected your desired layout, you can now add photos and text to your image. If you have a photo you would like to use, simply upload it to Canva under “uploads”. If you don’t have a photo, you’re in luck. Canva offers high quality stock photos for free. Browse the collection and find the one that works for your graphic. Once you find the photo, drag it onto the canvas.

Next, click on the text of your image and update the content. You can also change the color of text and backgrounds as you desire.

Once you are happy with your creation, download your image by selecting the “download” button in the right corner. Now you can post it to Facebook and promote your webinar!

I look forward to seeing lots of designs in my newsfeed!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · ·

Older posts >>

Archives

To top