AEA365 | A Tip-a-Day by and for Evaluators

We are Kathryn Newcomer, Director of the Trachtenberg School for Public Policy and Administration at The George Washington University (GWU) and 2017 AEA President, and David Bernstein, CEO of DJB Evaluation Consulting and 2017 AEA Conference Program Co-chair. We have been exploring the relationship between evaluation and learning for over 35 years. David was part of Kathy’s first evaluation cohort at GWU, Kathy was David’s doctoral dissertation advisor, and we have been frequent collaborators, co-authors, and AEA co-presenters.

Evaluation is dependent on learning from each other and putting theory into action. Each learning opportunity presents unique challenges and together, as a community, the 2017 AEA Conference in Washington, DC from November 6 to 11, 2017, will allow us to move beyond these challenges to find solutions to improve our programs and create greater good for society as a whole.

The four conference themes are a way to explore the full lifecycle of an evaluation: learning to enhance evaluation practices; learning what works; learning from others (other evaluations, other professions), and learning about evaluation users and uses. Over the next four days, evaluators who have assisted Kathy with planning the 2017 AEA conference will reflect on each of the subthemes, and provide tips to get the most out of the conference and our host city of Washington, DC. Some blogs will include inside knowledge from members of Washington Evaluators (WE), the local DC area affiliate. We are both enthusiastic Past-Presidents of WE.

Rad Resources: The AEA Conference Program is online. You can see a color coded conference overview at the bottom of the page. The top of the page has a very useful search feature. You can search the conference program by session title, track (Topical Interest Group themes and cross-cutting topics including Presidential Strand sessions), time slot, presenter, and session type. Be sure to look for the keynote sessions and keynote discussions featuring terrific speakers reflecting on different aspects of the conference theme.

Hot Tip: There are some great places to visit in DC before and after the conference. Two of our favorites provide an opportunity to “learn from the animals” and to reflect on what you’ve learned in a beautiful environment. David’s daughters are from China, and when they were younger they enjoyed the Panda statue right outside of the Marriott Wardman Park, the 2017 Conference Headquarters, before a visit to see real pandas at the Smithsonian National Zoo. The Zoo is a short uphill half-mile walk from the Marriott Wardman Park. Want a chance to quietly reflect on what you learned at the AEA Conference? Check out the Hillwood Estate, Museum, and Gardens, a short ride or two mile walk from the Marriott. Reflect on what you learned at the conference, and put your evaluation learning into action by sharing what you learned with others.

We’re looking forward to November and the Evaluation 2017 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

·

Greetings everyone of AEA!

I am Natalie DeHart, the Programs Coordinator for AEA. Since joining the team in May, I have been in learning mode, soaking up as much as I can about the profession of evaluation and about our members. I’m passionate about carrying out thoughtful programming to the organization that adds value to our members’ professional lives and I am thrilled that I get to do that every day at AEA. Today, I’m here to talk about biggest thing on all of our minds these days: Evaluation 2017, and I could not be more excited!

One of my favorite aspects of my role is teamwork and there is no bigger team project than an annual conference. One of my responsibilities is working on the Presidential Strand track for Evaluation 2017. I am grateful for the experience to help put this part of the program together and I am excited to see it in action in two short months.  Our President, Kathy Newcomer, and her group of volunteers with whom I have worked have been instrumental in bringing our theme “From Learning to Action” to life. Coordinating their efforts with my staff team has been a thrilling way to dive right in and I already have a few ideas in the works for next year.

Another wonderful thing about Evaluation 2017, is that I will finally be able to meet with members face-to-face. I’ve been getting to know a few of you over the past few months, but there is nothing quite like making an in-person connection.

Feel free to contact me anytime at info@eval.org, and please stop by the information desk at Evaluation 2017 so we can chat! I look forward to seeing you there.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello! I’m Martha A. Brown President of  RJAE Consulting. Lately, an endless stream of conference speakers, blog writers, Indigenous evaluators, and authors have confronted and challenged my “programming” as an evaluator. Traditional evaluation methods place tremendous emphasis on research methods and evaluation theory – but not necessarily on the people we work with and for. At the 2017 Canadian Evaluation Society conference, Nora Roberts told me that the very tools of our profession continue to oppress and silence others. Her statement sent me reeling. Gail Barrington spoke about the value of reflecting upon our work and our methods so we can improve our craft and learn more about ourselves. Indigenous speakers at multiple conferences reminded me that we are all interconnected and that our relationships with ourselves and each other are the most important things in life. All of this can be summed up in one word: love.

Additionally, I research, practice and teach restorative justice, which is grounded in Indigenous values such as interconnectedness, openness, honesty, vulnerability, and respect. I bring these values and restorative practices to my work. However, too many times I have felt like I am “breaking all the rules” that I learned in graduate school as I infuse love into my work and the people I work with.

When I read the invitation to submit a blog on evaluation and labor, the first thing that came to mind was to write about putting love and relationships at the center of our work. What would our work look like if each of us took time at the outset and throughout every evaluation to build trusting relationships with our “stakeholders” and “participants”? Do those of us who are products of Western culture even know how to do this? In a society that values goals, outcomes, and return-on-investment above all else, how can we return to the teachings and the ways of our ancestors and put our relationships at the center of everything we do? We knew this once, but have forgotten.

In AEA, many evaluators are truly committed to changing the world, to improving people’s lives, and to creating more just and equitable ways of doing what we do. But we don’t always know how to live out our goals. That requires us to critically reflect upon what we were taught, how we do our work, and to ask who is being inadvertently silenced, harmed, or oppressed during an evaluation – or in an evaluation classroom. It requires us to love.

Love requires us to engage our whole selves – mind, body, heart and spirit – in our work. We can learn how to do this by studying Indigenous values, practices, and ways of being. I am so grateful to those who helped me wake up, including our own Nicky Bowman.

Rad Resources:

The American Evaluation Association is celebrating Labor Day Week in Evaluation: Honoring the WORK of evaluation. The contributions this week are tributes to the behind the scenes and often underappreciated work evaluators do. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello community of evaluators. I’m Salima Bhimani, Founder and Director of Diversity, Equity and Inclusion at Relational. Relational offers research and evaluation and consulting and education services to organizations, companies and institutions. My focus as an evaluator and researcher over the last two decades have been on addressing bias, discrimination and barriers faced by marginalized people in communities and institutions. Here I share the centrality of getting underneath the language of diversity so that evaluations can reveal how social inequities are designed into institutions and operate within them. 

Case for consideration: Recently I conducted an evaluation for a higher education Institution. They wanted to understand how to make their curriculum and pedagogies more accessible to the linguistic, gender, racial, ethnic and economic diverse constituencies they serve in more than 10 countries. These constituencies all fall under the same religious community. The institution already had a conceptualization of accessibility. Their understanding foregrounded that everyone should be able to obtain their resources and relate to them.  It was clear to me that their approach to accessibility was intimately connected to how they thought about what diversity means. In this circumstance, their benign conception of diversity was obscuring the connection between the social subjectivities of their constituencies and their relative power, voice and positioning in relation to their institution and the broader community.  That is, there was no analysis of the historical and contemporary dynamics of unequal relations between their constituencies that were implicitly and explicitly defining the curricular content and pedagogical approaches. What was required is an awareness of how their approaches and content were already shaped for those unquestionably thought to be the norm.

Hot Tip: Break open taken for granted notions of diversity

  • A benign concept of diversity flattens difference. It undermines and diminishes histories and cultural forces that design inequities within institutions and which relationally shape individual and group identities, positions, interests and needs
  • A more critical conception of diversity understands how people and their experiences are socially and politically constituted in relation to each other, even within a community with a shared identity
  • Such analysis is foundational to a more nuanced conceptualization of what the curriculum and pedagogies need to be and for whom
  • Accessibility then is directly entangled with social realities and the biases, barriers, and inequities experienced differently within social minority groups
  • Accessibility must be framed with a clear view of how social markers of difference intersect to inform experiences of access

Rad Resources:

As I have written before diversity is often used as a ‘safer’ concept within institutions. Yet, those researchers that have examined the limits of diversity as an institutional marker, make an incredibly strong case for why we should understand the function of its uses. We need be cautious and as evaluators ask whether the use of diversity in fact undermines goals towards equity and social justice.

 

The American Evaluation Association is celebrating Labor Day Week in Evaluation: Honoring the WORK of evaluation. The contributions this week are tributes to the behind the scenes and often underappreciated work evaluators do. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Happy Labor Day Week! I am Calista H. Smith, President of C H Smith & Associates, a project management consulting and evaluation firm in Ohio.  C H Smith & Associates has done multiple evaluation projects for the Ohio Department of Education and designed evaluations for other clients related to public policy. In this work, it has been important to understand policymakers and the legislative decision-making process.

Lessons Learned:

  • Legislative processes may influence your evaluation design and timeline. Publicly sponsored projects may have reporting deadlines written into legislation or their funding streams may be subject to annual budgeting reviews.  Projects sponsored by private philanthropy may also be influenced by the legislative cycle as findings may be helpful to craft or change public policy.
  • Policymakers may get data and information from a variety of sources. It was common for a policymaker to have visited a program site or talked extensively with program champions. Program critics may also be vocal to policymakers. External criticism may be based on program perceptions (rooted in experiences or in ideology), or a sense of competition for resources. Your evaluation data will need to be clear and easily accessible to cut through what may be noise.
  • You may need various reports of the same analysis. For one evaluation, we produced a one pager of highlights for quick reference by high level administrators and officials, a 6-page summary of lessons to insert in a public annual report, and a full technical report with more detailed explanation of methodology and data for staffers and stakeholders.

Hot Tips (or Cool Tricks):

  • Spend time refining research questions related to what legislative decision-makers want to or should know regarding the project and related policies.
  • Regardless of the scope of your program evaluation, identify what policies and funding streams impact the program. This understanding helps you to gain clarity on who the stakeholders are and their interests and constraints.
  • In your evaluation design, consider legislative timelines. Think about what data you may be able to reasonably collect, analyze, and report to provide insights to legislators in line with the legislative decision-making process.
  • Encourage your client to think independently from your evaluation about courses of productive action they may take if findings are less favorable than expected. Consider building in extra review time for analysis so that the client can process data and determine how to make lessons actionable or identify questions that may emerge from policymakers about the results or the evaluation approach.

Rad Resources: 

  • The National Conference of State Legislators has a Program Evaluation Society for its state policy staff members. It is helpful to see what materials policy staff members may reference when they would like to implement or review an evaluation.
  • You may map out stakeholder interests, including policymaker’s interest, in your evaluations in a” Power/interest matrix.”:

The American Evaluation Association is celebrating Labor Day Week in Evaluation: Honoring the WORK of evaluation. The contributions this week are tributes to the behind the scenes and often underappreciated work evaluators do. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello! My name is Nora E. Douglas and I am a consultant at CNM Connect where we provide evaluation and capacity building services to nonprofit organizations. One of the common challenges I consistently face is managing projects and keeping them moving towards a final deliverable. I have found the following four tips to be useful in the successful completion of my evaluation projects.

Hot Tip #1: Make a Plan and Follow Through

This tip can be attributed to Stephen Covey’s second habit “Begin with the End in Mind” from his 1989 book, The 7 Habits of Highly Effective People. For a different perspective, you can also delve into the world of the Project Management Professional (PMP)®. You need to know where you want to go and make a specific and clear plan for how to do that.

Rad Resources:

Hot Tip #2: Seek out an evaluation champion.

I am always on the lookout for a “champion”; the person within the organization that I’m evaluating that can create enthusiasm and momentum for the project and ensure success.

Rad Resource:

Hot Tip #3: Identify constraints.

The Theory of Constraints was introduced in 1984 for Eliyahu M. Goldratt in his book titled, The Goal. There is always at least one constraint in a project, and identifying that constraint and restructuring the project around it can assist in completing the project successfully.

Rad Resources:

Hot Tip #4: Address tendencies to procrastinate.

I’m guessing we all have procrastinated at one point or another. Procrastination can leave you feeling guilty and anxious about completing the project on time. Two ways I deal with procrastination are to publically commit to getting something done and creating small steps that lead to big progress. Another way to deal with procrastination is to determine the importance and urgency of tasks and which to tackle first.

Rad Resources:

 

Time management matrix as described in Merrill and Covey 1994 book “First Things First,” showing “quadrant two” items that are important but not urgent and so require greater attention for effective time management (Photo credit: Wikipedia / Wikimedia Commons)

The American Evaluation Association is celebrating Labor Day Week in Evaluation: Honoring the WORK of evaluation. The contributions this week are tributes to the behind the scenes and often underappreciated work evaluators do. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

My name is Samantha Theriault, and I am the Research Assistant at Randi Korn & Associates (RK&A), a research, evaluation and intentional planning company that specializes in museums and informal learning.

Lesson Learned: As a research assistant, I spend much of my time entering and processing data.  Data entry and clean-up is time consuming and challenging, and it makes me feel like a worm turning dirt among the roots of our work. From this point of view, I’m watching projects grow from the ground-up, so it’s even more exciting to see the final product. Preparing quantitative and qualitative data for analysis isn’t glamourous, but it is invigorating and contributes to the success of all our work. Evaluation runs on data – so keeping it organized from the beginning is vital.

Hot Tip: Set aside chunks of time to spend with data. Depending on the size of your project and methods, data entry can take a few hours or several days, or even weeks! Prioritizing data entry – rather than “squeezing it in” between tasks – which increases my comfort with the data set and puts me in a rewarding flow state. With mindfulness during data entry, I notice patterns as they emerge. For example, I recently struggled to interpret a participant’s shorthand on a question about which neighborhood they live in, but noticed others spelled out all the words in their responses. I matched the shorthand to the full neighborhood name (and double-checked with Google!), which I would have missed were I entering data mindlessly or too quickly. Similarly, I might notice other trends, such as reduced responses to a certain question, which I flag for examination later.

Cool Trick: Create a “living” data entry processing handbook.  When I was first learning to process data using SPSS, I created a “cleaning up data files checklist” and add unique tasks and tips to it each time I work on a new data set.  My checklist includes recoding system-missing responses, ensuring that survey responses follow skip logic, and reminders such as “slow down and double-check your work!” Since my colleagues depend on these data sets to do their work, I include their needs in my checklist, too: spell out acronyms, label variables, and delete “working” variable labels I created while collapsing categories into single columns. I also create a log for each digital survey’s lifecycle, documenting any changes to the museum’s exhibition or program during data collection, and quirks to remember when it’s time to process the data for analysis. This detailed approach to record-keeping is especially useful when my colleagues have specific questions about the data during analysis.

Rad Resource: Microsoft OneNote is a password-protected digital notebook that I use to keep track of my data cleaning process (and many other elements of managing data collection and processing). I like that I can repurpose checklists and save relevant files on the same page – it’s like a 4D Moleskine to me!

(click for larger image)

The American Evaluation Association is celebrating Labor Day Week in Evaluation: Honoring the WORK of evaluation. The contributions this week are tributes to the behind the scenes and often underappreciated work evaluators do. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! We are Kellie Hall, from the National Association of County & City Health Officials (NACCHO), and Emmalou Norland, from Cedarloch Research, LLC. We have worked in consort as internal and external evaluators on public health programs. During our time together, evaluation has been growing in popularity within the non-profit sector—and with that, so has the need to engage stakeholder groups from multiple levels (i.e., top executives, program managers, and front-line staff).

Lessons Learned: The importance of stakeholder engagement during evaluation—particularly as a critical component in ensuring the evaluation meets the utility standard—is well known in the field. As familiar as the concept is, however, the complex nature of engaging stakeholders in appropriate ways can be a perplexing challenge. For example, when federal funding dictates not only that a program evaluation must be done but also specifies its design, engaging stakeholders in the planning phase can seem superfluous. Furthermore, stakeholder engagement sessions typically focus on the why behind engagement, rather than the how of engagement with those of varying authoritative powers, divergent priorities, and competing needs. Understanding these contextual factors is crucial to engaging various levels of stakeholders.

Hot Tip: Engage stakeholders in the process of determining how to engage stakeholders!
Many evaluators begin their stakeholder engagement by creating a Stakeholder Engagement Plan. Instead, start one step earlier.

One way to do this is to gather your stakeholders together for a “hack-a-thon,” a process that comes from the technology field and is focused on collaborative problem solving. This highly interactive meeting starts with your stakeholders and ends with solutions tailored to address their needs. During a “hack-a-thon,” each stakeholder group works through the following stages together:

  1. Empathizing with another stakeholder group
  2. Defining a focused need for that other stakeholder group
  3. Ideating solutions to address that need
  4. Deciding on the most effective solution

(Check out an example hack-a-thon setup, including handouts, here.)

Then, you can use the results developed by the stakeholders themselves to create a “Stakeholder Profile” for each group, documenting their power, values, priorities, and engagement needs. This is now the beginning of your Stakeholder Engagement Plan!

Rad Resources: Some great stakeholder planning resources that I’ve referenced in my work include:

If you have a useful stakeholder engagement resource, please share in the comments below.

The American Evaluation Association is celebrating Labor Day Week in Evaluation: Honoring the WORK of evaluation. The contributions this week are tributes to the behind the scenes and often underappreciated work evaluators do. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! I’m Sheila B Robinson, aea365 Lead Curator and sometimes Saturday contributor. I’m composing this post in a somewhat leisurely fashion, enjoying a 3-day weekend as Monday September 4 is Labor Day in the US. As a public school district employee my entire career, I’ve always enjoyed my time off on this day, but I also stop to remember and appreciate all of those who do not get to enjoy Labor Day as a holiday including:

  • hospital personnel
  • law enforcement
  • first responders
  • nursing home personnel
  • group home personnel
  • airline personnel

These are just a few who will report for work on Monday in service to the public along with the millions of retail workers tending to Labor Day vacationers’ insatiable desire for sale prices on home electronics or linens.

Lesson Learned: Since 1894, Labor Day has been celebrated the first Monday in September. Created by the labor movement, this holiday “is dedicated to the social and economic achievements of American workers. It constitutes a yearly national tribute to the contributions workers have made to the strength, prosperity, and well-being of our country” (US Department of Labor).

I thank Michael Quinn Patton for the idea of dedicating a week of aea365 blog posts to Labor Day. In 2016, we set aside two weeks to honor evaluators – evaluation’s workforce. Check out our aea365 archive to read those posts! This year, we are highlighting the work of evaluation for Labor Day. We invited contributors to write a post on the work of evaluation that goes on behind the scenes and is often underappreciated.

This week, our authors will acknowledge the labor of evaluation and help us learn more about:

  • engaging stakeholders
  • the “dirty work” of evaluation: data cleaning
  • keeping the work of evaluation going
  • working with policymakers and the legislative decision-making process.
  • addressing bias, discrimination and barriers faced by marginalized people in communities and institutions
  • infusing love and relationships into evaluation work

Let’s not forget too, that there are undoubtedly many, many evaluators who will be at home on Monday writing evaluation plans, conducting literature reviews, designing protocols, analyzing data, writing reports, and emailing stakeholders.

Here’s wishing everyone a happy Labor Day!

The American Evaluation Association is celebrating Labor Day Week in Evaluation: Honoring the WORK of evaluation. The contributions this week are tributes to the behind the scenes and often underappreciated work evaluators do. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

I’m Josh Joseph, senior officer in planning and evaluation at the Pew Charitable Trusts, here to talk about getting more from networking as AEA’s Annual Conference approaches.

I’m also introverted. Not shy, but not a natural mingler either.  Where some folks get energized by big gatherings, I still remember when meeting people at professional events felt more like an obstacle than an opportunity for career growth.

Much of the networking advice I’ve been given—like developing an elevator speech—hasn’t gone far enough. Below are a few lessons learned, followed by some tips for addressing common challenges that many people, including introverts, regularly face.

Lesson Learned: Content matters. Networking is about sharing and listening for information and ideas that you care about—not making small talk in crowded rooms. Your deeper interests can help anchor you in any conversation. Try not to let networking stereotypes and imagery distract you.

Lesson Learned: Look for common ground with others. Shared interests are like magnets for networking. They tend to make discussions more engaging and useful, and they increase the chances of following up.

These lessons, while important, are limited in helping to pin down your professional interests and increase your comfort.  Below are some tips to get at these frequent concerns.

Hot Tip: Look inside before reaching out.  You’re probably busy at work and may resist this, but trust me. At least a few days before heading to an event that includes networking, set aside time to reflect on two key questions and then jot down your thoughts:

  • What do I hope to learn? (e.g., are there work challenges on which you could use advice?)
  • What do I have to share? (e.g., how might others learn & benefit from your work?)

You’ll have a clearer sense of purpose and will be primed to talk about and learn things that are more relevant and engaging.

Hot Tip:  Focus on what you enjoy. We’re most comfortable and confident when talking about things that interest us and “light us up”.  We know what that feels like away from work, but it’s also true professionally. Interest shows on our faces and in body language and people absolutely respond to it.  Conversations can feel almost effortless. So find what sparks you professionally and tap into it.

Hot Tip: Be ready to listen. Networking isn’t a competitive sport—it’s about give and take.  Engaging topics often emerge by chance, so ask questions, keep an ear out for connections and, above all, strive to be a good listener.

Two final thoughts: not every conversation will be golden and there isn’t a right way to network. It’s worth trying different things. Instead of crowded rooms, you might prefer connecting over breakfast, a cup of coffee, or in quiet hallway. Find what works for you and go with it.

Rad Resource: While there are plenty of networking resources out there, this NYTimes blog—An Introvert’s Guide to Networkingis worth a look.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

<< Latest posts

Older posts >>

Archives

To top