AEA365 | A Tip-a-Day by and for Evaluators

CAT | Uncategorized

Happy Saturday, folks!  I’m Liz Zadnik, aea365’s Outreach Coordinator.  I live in the Mid-Atlantic region of the country and was snowed in a few weeks ago.  The storm wasn’t as bad as it could have been (for us…thankfully), but I had a chance to spend some time catching up on my reading resolution.  

Rad Resource: First off, I need to again express my appreciation for AEA’s member access to journals and publications from the field. I love New Directions for Evaluation and was excited to see “Planning and Facilitating Working Sessions with Evaluation Stakeholders.”  Part of my “day job” is engaging stakeholders in conversations about nuanced topics and complex issues.  The inclusion of a case example helped me operationalize concepts and give me some great ideas for my own practice.

View of desk with three plants lined up from left to right with a whiteboard in the background

Lessons Learned: A big factor in successful group project is navigating potential issues or influences within the group of stakeholders.  This includes both investigating the attitudes and dynamics of group members, as well as your own biases as the facilitator.  The article encourages evaluators to learn about possible political, historical, and/or social contexts that may prevent or hinder group cohesiveness and trust.  Is it (in)appropriate to bring everyone together initially?  Or do distinct groups need to be engaged before a collective can be established?  

There’s also a great table with skills and questions for facilitators, each topic has examples and items to explore.  What caught my eye – most likely because it’s something that has tripped me up personally in the past – was a set of questions about previous group facilitation experience.  It’s challenging not to bring past experiences with you to the present, but a lack of patience or quickness to make assumptions about dynamics and process can really impede creativity, innovation, and thoughtful problem-solving.  

I also loved how the author outlines thoughtful considerations and steps for facilitating and operationalized those considerations with a case example.  Particularly during the description of the debrief – I am a huge fan of self-reflection and really appreciated its inclusion within the facilitation process.  

I would definitely recommend the article to anyone who wants to up their facilitation game and is looking for guidance on how best to engage project stakeholders!   

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Tricia Wind and I work in the evaluation section of the International Development Research Centre in Canada – an organization that funds research in Africa, Asia and Latin America.  My section regularly does quality assessments of the evaluations that are commissioned by our program staff colleagues as well as by the organizations we fund.

Lessons Learned:

We have seen how quality depends not just on the consultants who undertake evaluations, but also on the program managers who commission them. Commissioners decide the scope of an evaluation and its timing. They define questions and facilitate use. They approve evaluation budgets. Commissioning decisions are key to evaluation quality.

Seeing that most evaluation resources are targeted to evaluators, IDRC teamed up with BetterEvaluation to produce a new interactive, online guide to support program managers.  It guides program managers in their roles and decision-making before, during and after an evaluation to ensure the evaluation is well designed, use-oriented and appropriately positioned within an organization.

Rad Resource:

The Program Manager’s Guide walks program managers through nine typical steps of commissioning and managing an evaluation. It provides high-level overviews of the steps, more detailed sub-steps and, and also links to further resources available on the rest of the rich BetterEvaluation website. It is available in English and French.

The GeneraTor: The guide is accompanied by a tool, called the GeneraToR, which prompts users to document the decisions they are making about an evaluation (its scope, uses, questions, timing, budget, evaluator qualifications, deliverables, governance, etc.) in an online form.  The form becomes a customized terms of reference that can be downloaded to share with stakeholders. The terms of reference are foundational for other documents for the evaluation, such as requests for proposals (rfps), consulting contracts, workplans and stakeholder engagement plans.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Mar/17

22

Engaging with EvalYouth by Khalil Bitar

I am Khalil Bitar (EvalYouth Vice-Chair). Along with Bianca Montrosse-Moorhead and Marie Gervais (EvalYouth Co-Chairs), I am very glad to have connected with all of you throughout our recent sponsored week. During the week, we presented EvalYouth, its achievements so far, and our plans for the near future. Despite the prominent work EvalYouth has achieved thus far, there is still more work to do.  EvalYouth hopes to build on our successes to achieve a lot more in 2017 and 2018.  Today, I’d like to tell you more about how to engage with EvalYouth.

Hot Tips:

During our sponsored week, you learned about the work of Task Force 1, Task Force 2, and Task Force 3.  We plan to start a fourth task force in 2017 focusing on youth inclusion in evaluation.  To do all this, we need the engagement of more members who are passionate about the future of Evaluation.

There are multiple ways to engage with EvalYouth:

Rad Resources:

Take a moment to read EvalYouth’s Concept Note, which details the network’s goals and objectives, governance structures, and a lot more.

Bianca, Marie, and I hope that we were successful in shedding light on EvalYouth and its work during EvalYouth week on aea365.  We very much look forward to hearing from you!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello AEA365!  I’m Paul Collier. Over the last two years I worked as the Data and Evaluation Manager at the San Francisco Child Abuse Prevention Center (SFCAPC), a mid-size nonprofit focused on ending child abuse in San Francisco. In my time there and as a freelancer, I can’t count the number of times I’ve fielded questions from staff about data their organization has collected. They often go something like this…

collier-post-imagea

How frustrating! But as someone serving as an internal evaluator or data analyst at an organization, I have to remind my staff questions like these are my friend. When my staff asked me questions about their data, I knew they’re engaged and interested in using it. But I often found the first questions they asked weren’t the questions that would really help them make decisions or improve their programs. This post is about helping your staff think critically and ask smarter questions about their data.

Hot Tip: Focus on highly strategic questions

Questions that can be answered with existing data come in all shapes and sizes. I like to consider first whether the results may help the organization improve or refine our programs. For example, questions testing the cause-and-effect relationships in our logic model or assumptions in our theory of change can and should inform programming. A second aspect of a strategic question is whether our team has expectations for the result. I often realized that our staff didn’t have expectations around average improvement or effect size, so I would find a few studies using comparable assessments and interventions to identify some benchmarks. Perhaps the most useful aspect of a strategic question is whether our staff can take action based on the results. I found that if my staff can’t envision how the results might actually be used, its wiser to help them think through this before spending my time (and theirs) analyzing the data.

Cool Trick: Plan for Analysis.

To be more strategic about the analysis questions I focused on, I built time between the request for analysis and doing the work. An initial conversation with the program manager or staff to learn more about the context of a question usually helped me refine it to be more specific and actionable. I found that batching analysis for a certain time in the year was also a useful planning approach that protected my time. I preferred to have this ‘analysis period’ in the winter, because my organization set its budget in the spring. This way, any changes to programming that resulted from the process could be planned for in the following year’s budget.

Rad Resources:

As you can tell, I think helping staff ask smarter questions is one of the most valuable things I do as an internal evaluator. For more reading on this topic, check out:

  • Michael Hyatt’s Blog on Asking More Powerful Questions: Michael Hyatt is a business author who provides some clear and easy to understand advice to aspiring leaders on asking questions.
  • Peter Block’s book, Flawless Consulting: Block’s Flawless Consulting provides many helpful suggestions for structuring analysis processes so they influence action. There are also several great chapters about overcoming resistance in clients, which I’ve found highly relevant for dealing with inevitable resistance in results within my team.
  • Rodger Peng, Ph.D.’s E-Book, The Art of Data Science: Peng illustrates what a practical data analysis approach looks like, framing each step as a process of setting expectations and understanding why results did or did not meet those expectations.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Say cheese! It’s me, Sheila B Robinson, AEA365’s Lead Curator and sometimes Saturday contributor. We have featured a number of blog posts over the years on finding and using images to complement your evaluation work, whether you are in the business of blogging, presenting, teaching, creating reports, or other areas. It’s more important than ever not only to become familiar with where to find images, but also with how you can and cannot use them legally.

Lesson Learned:

  1. Creative Commons is not the same as “copyright free.” According to Creativecommons.org,  The Creative Commons copyright licenses and tools forge a balance inside the traditional “all rights reserved” setting that copyright law creates. Our tools give everyone from individual creators to large companies and institutions a simple, standardized way to grant copyright permissions to their creative work. The combination of our tools and our users is a vast and growing digital commons, a pool of content that can be copied, distributed, edited, remixed, and built upon, all within the boundaries of copyright law.
  2. There are many Creative Commons licenses and it’s important to understand their differences. There are 6 main types ranging from more restrictive to less restrictive. Each license comes with language that helps the user understand whether attribution is required, and whether the product can be changed in any way or used for commercial purposes. Read this page to learn about each license.
  3. There is one type of license with NO restrictions! CC0 1.0 means that the designer has dedicated the work to the public domain by waiving all of his or her rights to the work worldwide under copyright law, including all related and neighboring rights, to the extent allowed by law. You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission.

Hot Tips: Where do you find good images? While there are countless websites that offer free and paid images, icons, and other graphics:

  1.  28 Places to Download Free Images for Websites and Blogs includes an updated list of and links to photo sites along with definitions of public domain and CC0.
  2. Nolan Haims Creative offers blog subscribers access to a bunch of great free resources, including a wonderful “taxonomic” reference list of photo sites

Cool Trick: Once you have a collection of images, what do you do with them? Check out Echo Rivera‘s email course on creating your own visual database. Echo found that searching for images while she was creating presentations wasn’t good for her workflow, so she advises creating visual database that organizes visuals in ways that make them easily accessible when you need them (minding the different types of licenses). Read this blog post for her explanation and rationale for this technique.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, my name is Jayne Corso and I am the Community Manager for the AEA. LinkedIn stands out as the social platform for professional development and industry sharing. It is a great resource for presenting yourself as an experienced evaluator as well as finding resources and networking opportunities that will benefit your practice and strategies. I have compiled a few tips that will help you create a stronger personal profile, and identified LinkedIn resources.

Hot Tip: Enhance your Profile

Go beyond just including a photo, work experience, and education – really enhance your profile by including your publications, skills, awards, independent course work, volunteer experience, or organizations you belong to. All of these features allow you to have a robust, well-rounded profile and will highlight your expertise as an evaluator.

Hot Tip: Use key words

Create a list of keywords that accurately communicate your expertise. For example, evaluation, visual data, statistics, research, and monitoring are searchable key words that resonate with evaluation. To improve your profile, incorporate these keywords repeatedly in your profile descriptions. This will allow your profile to be ranked high when the words are searched within LinkedIn. Placing keywords in your profile headline is also a great way to show your expertise and helps other users make an informed decision about connecting with you.

Hot Tip: Customize your LinkedIn URL.

When you join LinkedIn, the site creates a generic URL for your profile that includes a series of numbers. Similar to a website URL, these numbers do not resonate high in a search. Placing your name or keywords into your URL will improve the visibility of your profile. Here is a list of Instructions for how to customize your URL.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Greetings aea365-ers! I’m Sheila B. Robinson, Lead Curator and sometimes Saturday contributor. At this point you’re probably thinking, “Putty-what??? Multi-what??? Are those even real words?” According to Emilie Wapnick, they certainly are. Before I tell you about Emilie, let me ask you these questions:

1.) Have you had one or more jobs in fields other than evaluation?

2.) Do you have a degree or have you taken courses in another field of study, perhaps even one (or more!) unrelated to evaluation?

3.) Have you changed your interests (personal or professional) over time?

4.) Do you continue to pursue learning in fields outside of evaluation just based on personal interest?

If you answer “yes” to any or all of these, you might be a multipotentialite. Emilie, leader of the Puttylike Tribe, describes herself and her professional careers as characterized by “shape-shifting, exploration and evolution.” She has pursued multiple interests over time and created the Tribe to bring together fellow multipotentialites. According to Emilie, a multipotentialite is simply “a person who has many different interests and creative pursuits in life.”

Image credit: UFV Science Rocks via Flickr

Image credit: UFV Science Rocks via Flickr

I started my career as a public school classroom teacher and have held several other positions in education as well including evaluator. But, my first major in college was music and I spent 10 years working for a music studio teaching private piano lessons! In my spare time, I read books and articles in areas including business, psychology, nutrition, and medicine. Oh, and I’m in a Meetup group learning website design! Since becoming involved in AEA I’ve met many evaluators with backgrounds and interests at least as varied as my own.

If you think you might be a multipotentialite, you’re in good company! One of our most revered thought leaders, Michael Scriven, is certainly one. Were you under the impression that Scriven is just an evaluator? Not so! According to a 2014 Psychology Today article, Life Advice from the Smartest and Wisest Person I Know: Practical Wisdom from a True Polymath, Scriven has “published in the leading journal in eleven disciplines: from philosophy to mathematics, evaluation to parapsychology. He’s also relentlessly practical: with impressive expertise in design of word processors, knives, and sports cars.” Did you catch a key part of that last sentence? ELEVEN disciplines!!! Wow…

In a way, it’s not surprising that evaluators are likely multipotentialites. After all, Scriven maintains that evaluation itself is considered a transdiscipline (and even the “alpha discipline”): “…like logic and statistics, evaluation is a major transdiscipline because all disciplines rely on the evaluation process to judge the value of the entities within their own purview…” (Alkin, 2013, p. 33 from  Evaluation Roots: A Wider Perspective of Theorists’ Views and Influences). Evaluators must be puttylike in order to succeed in our field. Just check out this proposed set of evaluator competencies*!

Rad Resources: Emilie Wapnick’s popular TED talk has over 3.6 million views, and she just published How to Be Everything: A Guide for Those Who (Still) Don’t Kohn What They Want to Be When They Grow Up.

*You’ll need an AEA membership or free website account to access these.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Happy Saturday readers!  I’m Liz Zadnik, aea365 Outreach Coordinator and Saturday contributor with some tips on using images in blog posts.  Breaking up paragraphs with high-quality stock images, graphics, and/or visualizations help readers stay on your site, digest information, and motivate them to share your brilliance with the world.    

Cool Trip: Get to know your computer’s already-installed image editor.  Did you know PowerPoint (or Keynote for Mac users) has image-editing capabilities?  You can remove backgrounds, crop, recolor, and create an image that’s perfect for your needs and branding.  You can edit or create an image in the application and then “Save as a Picture” to your desktop for later use.  Just make sure the image is one that can be edited and shared.  We’ll cover that in a second.

Screenshot of Microsoft PowerPoint Picture Tools Ribbon with Annotations

You don’t need Photoshop to create beautiful images that will be engaging and look professional.  Try not to be intimidated by “photo editing” and exercise those creative muscles!   

Hot Tips: Selecting an image may seem easy enough, but there are a number of things to keep in mind:

  • Check out the quality of the image.  You may have seen the letters “dpi” accompany an image – this means dots (or pixels) per inch and gives you a sense of how crisp an image is.  For example, something that is five inches wide and 1000 pixels high would have 200dpi.  The higher the resolution, the more likely you are to be able to zoom in or crop without losing that “crispness.”  
  • Find candid and casual shots of people.  Move away from traditional stock photography to an image where the reader can imagine they’re a part of things.  
  • Make sure the image size fits with your webpage columns.  If you have a blog, check to see you how wide your main content column is – this will help you in avoiding images slipping over into your sidebar or messing with text.  It’s also important to make sure the image is aligned in such a way that text isn’t broken up along the side.  
  • Avoid images with watermarks.  First off, these are copyrighted or protected images for subscribers to stock photography sites and providers.  I understand that Creative Commons licensing can get confusing, but we need to respect our creator colleagues.  Secondly, it also doesn’t look very professional.   

Person with back to camera sitting at desk with a bright sun flare

Lesson Learned:  Something I’ve been trying to work on is making sure images are accessible using alternative text (“alttext”).  This includes crafting thoughtful and brief descriptions of images for folks using screen readers and other navigation devices, as well as when images cannot load on a specific page.  Visitors don’t lose the essence of your post just because they can’t access the visual content.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Greetings Colleagues!  We are Becca Sanders (Hood River, Oregon) and Heather Robinson (Rochester, New York) of Iteration Evaluation LLC. Our forte is the “…ahhh… there is no button on your system for that” analysis requests.  We work mostly with behavioral health organizations whose current informational needs often far exceed what existing systems can pull off.

Enter VLOOKUP.  VLOOKUP is a tool in Excel that can be used to amalgamate data data that lives on seperate spreadsheets.  It’s like a 4 wheel drive vehicle that can get stakeholders what they need across the potholes that result from siloed management information systems. For example, say we have a direct service provider with a caseload of folks he or she is trying to help.  We also have spreadsheet X, that lives in one place, and gives the provider a list of something important—say emergency room visits.  And we have spreadsheet Y that lives elsewhere and contains some other important piece of information about individuals the provider is trying to help—say existence of a chronic condition.  The provider wants this information all in one place across a particular group of individuals—say a caseload– in order to do their job well.  Enter VLOOKUP as the solution; a solution while you wait for relational table builds, system integration, or the like.  One common identifier across different spreadsheets (for example, a person’s ID number) PLUS a bazillion exported spreadsheets, none of which quite do the trick, PLUS VLOOKUP is all you need.  VLOOKUP essentially says “hey, here is an ID number for an individual in this table and that one (and that one and that one ad nauseum).  Let’s use it to pull in all values from one spreadsheet to the other (and so on) until all the information you need is in one place!” In that way, VLOOKUP allows you to deal with situations where the snapshot you are trying to take involves variables that live in different systems – especially systems that don’t speak to one another very well.  Get good at it! It will take you far!  It is like a Subaru in the winter which can get you through just about any wacky data set up you are faced with.

Rad Resource: Need more of a leg up to get started? Check out the free resource www.chandoo.org!  Chandoo has a free blog, with a mission to help anyone become awesome in EXCEL.

Go VLOOKUP! Don’t wait to kick your car into VLOOKUP 4 Wheel Drive!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings!  I am Tania Rempert, Evaluation Coach at Planning, Implementation, Evaluation Org.  This post is written together with my colleagues Molly Baltman from the McCormick Foundation, Mary Reynolds from Casa Central, and Anne Wells from Children’s Research Triangle.  We would like to share our experience speaking at the Office of Social Innovation White House convening on Outcomes Focused Technical Assistance (OFTA).

The purpose of this convening was to advance an outcomes mindset in government, across the public, private, and philanthropic sectors.  David Wilkinson shared the vision of OFTA to focus on building the capacity of social service providers using data to inform smarter service delivery and to implement evidence-based practices in local communities.  Wilkinson began the convening by pointing out,

“Government pays for 90% of the funding for social services in this country, but typically pays for outputs and compliance rather than outcomes and impact.  As a result, many social service providers do not have outcomes they are actively pursuing….and less likely to have consistent outcomes useful for comparison with their peers.” 

The White House Office of Social Innovation and Civic Participation would like to change that.  This convening was meant to draw attention to the technical assistance needed by social service agencies when tasked with measuring and reporting and using outcomes.

Hot Tip: Principles of OFTA:

  • Identify the most important measurable outcomes
  • Implement evidence-based practices
  • Use data to inform research-based service delivery

We were asked to speak based on our experience with the Unified Outcomes Project.  We shared our experiences focusing on increasing grantees’ capacity to report outcome measures and utilize this evidence for program improvement, while streamlining the number of tools being used to collect data across cohort members.  Our model emphasizes communities of practice, evaluation coaching, and collaboration between the foundation and 29 grantees to affect evaluation outcomes across grantee contexts:

Lessons Learned:

  • It takes at least 2 years to see measurable outcomes and be able to model the use of this data at the cohort level of shared outcomes.
  • Grantees are experts through lived experience, use their community voice to determine specific strategies, because they have the language and experience to take each other to the next level, so when they are brought together, a learning community organically develops.
  • The beauty of using an evaluation coach visiting organizations on-site to provide technical assistance is that each organization has different needs to make data-informed decision making.

We hope that this initial convening will encourage ongoing discussion and development of strategies in OFTA for evaluation practice and government policy making.  Since it is not a thing unless it has an acronym, let all of us in the evaluation community commit to “OFTA often!”

rempert

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Older posts >>

Archives

To top