Hi again, Fellow Evaluators! Amanda Jones, LGBT TIG Program Co-Chair, here again to provide some concluding thoughts on incorporating sensitivity to LGBTQIA (lesbian, gay, bisexual, transgender, questioning/queer, intersex, asexual) folks into evaluations. Just as culturally competent evaluators think through all aspects of every evaluation as they relate to race and ethnicity, these same evaluators should also consider how their methods, measures, and reports relate to issues important to the LGBTQIA community.

Hot Tips:

• Learn the meanings of the terms important to LGBTQIA individuals, and advocate for sensitivity to these individuals in all evaluations. Before we can ensure that LGBTQIA folks are well-served by our evaluations, we must first recognize who they are! UC Davis provides a useful glossary to explore.
• Watch your assumptions. Do not assume everyone involved in your evaluation is heterosexual, or that that they identify with the sex they were assigned at birth. Read over your evaluation materials to see if they would address someone who is gay or transgender as well as someone who is heterosexual or cisgender (check that glossary!).
• If you need to measure gender or sexual orientation in your evaluation, consult experts who have thought through the options. Luckily, many have put guidelines online, including the Human Rights Campaign and the Williams Institute at the UCLA School of Law. In addition, you could contact the leadership of the LGBT TIG to see if they might be able to help.

The American Evaluation Association is celebrating LGBT TIG Week with our colleagues in the Lesbian, Gay, Bisexual & Transgender Issues Topical Interest Group. The contributions all this week to aea365 come from our LGBT TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi there! I’m Scout Black, Research Associate at Smith & Lehmann Consulting. For today’s post, I’m tackling the importance of LGBTQ representation and affirmation in research and evaluation, and especially how that impacts young people.

A good deal of my time is spent planning and facilitating our Youth Participatory Action Research (YPAR) interns and research assistants, who are all between the ages of 15 and 19, and I have the wonderful opportunity to mentor and provide guidance to these amazing teens as they conduct research to inform the youth-serving project we are evaluating. In my work, I interact with many young people of many varying identities, including those within the LGBTQ community. It’s very important to me that I create a safe space for them to learn and grow and be accepted for who they are, while also representing to them that as an LGBTQ person myself, people in our community can excel in research and evaluation fields and be out and accepted by our employers. By both setting up a safe, affirming environment for budding researchers and evaluators, where their identities are validated and accepted, and by providing visible representation of an LGBTQ person in the field, young people can see themselves represented and envision themselves as future researchers.

Hot Tips:

• Make your Equal Employment Opportunity language inclusive. By seeing that their potential employer explicitly does not discriminate against applicants based on their sexual orientation, gender identity, or gender expression, young researchers can feel safe being themselves in their workplace.
• Make pronoun introductions routine. During group meetings, focus groups, or any other scenario that involves introductions (e.g. name, job title, etc.), ask people to share their pronouns. This sets the stage for inclusion and shows youth and young adult researchers/participants that their identities are respected and celebrated.
• Be vocal on issues of inclusion and diversity. If disparaging comments are being made about people with marginalized identities, address them head-on. Silence speaks volumes.
• Be yourself. If you’re out, be out. If you’re not comfortable being out, that’s okay too – there’s no one way to be an LGBTQ person. Youth and young adult researchers seeing that they can be themselves in this field can build, diversify, and strengthen the profession.

I hope to see our profession grow and flourish with young talent of varying identities, and hope that these tips may help you in your interactions with budding LGBTQ youth/young adult researchers.

The American Evaluation Association is celebrating LGBT TIG Week with our colleagues in the Lesbian, Gay, Bisexual & Transgender Issues Topical Interest Group. The contributions all this week to aea365 come from our LGBT TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Happy LGBTQ+ Evaluator Week! I’m Joseph Van Matre, an institutional research analyst at the University of California*.

Surveys are an integral part of evaluation, and when designing a survey, gender is often the second question after name. We add it to our surveys without a second thought. But do we need to know?

Evaluators are, by our nature, curious people, but evaluations are not fishing expeditions. Look carefully at the program’s theory of action and your own evaluation plan.

Cool Trick: Ask yourself:

• Is this program targeted at a particular sex or gender (they’re different!)?
• Is there reason to believe that this program will have a differential impact on participants/subjects of different genders or sexes?
• Do you need to collect gender/sex information of everyone involved in the program, or just some participants (e.g. teachers and students?)
• Do you have the time, resources, and mandate to evaluate and report on gender/sex differentials?

While our first instinct is to collect as much data as possible about the programs and interventions that we evaluate, it is our responsibility not to collect and store personal information that we do not need. If you will not or cannot use information on someone’s sex or gender, you do not need to ask in the first place.

Ask for what you need to know. Unless you are evaluating a health-related program, you probably only want to ask about a person’s gender.

It is often best to allow people to identify their gender in an open ended way: I identify my gender as ___________. With large-scale surveys, you can provide male, female and an open response so that you only manually code responses that are not male or female.

Hot Tip:

Some government agencies or funders may require rigid gender reporting. The Integrated Postsecondary Education Data System (IPEDS) requires colleges and universities to report the gender of every student as either male or female. Even unknown or missing is not an option, leading to some very strange reporting outcomes.

While dreadfully cliché, the adage, “say what you mean and mean what you say,” is an important rule when reporting evaluation outcomes related to sex and gender.  Your forethought and planning will make your communication inclusive and accurate.

Cool Trick:

For example, when you ask people to identify their gender on a survey, the phrase, “there were 24 men in our sample” can be replaced by, “27 people in our sample identified themselves as male.”

It is our job to continually educate ourselves about the people and programs we evaluate, and any population is likely to include people who do not fall within the gender binary. Asking for exactly what you need (or not asking at all!) is a simple way to create more inclusive evaluations.

* The views and opinions expressed in this post are those of the author and do not necessarily reflect the views, positions or policy of the University of California.

The American Evaluation Association is celebrating LGBT TIG Week with our colleagues in the Lesbian, Gay, Bisexual & Transgender Issues Topical Interest Group. The contributions all this week to aea365 come from our LGBT TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for

No tags

Hi, I’m Carlos Romero, Co-chair of the LGBT Issues TIG. Being gay makes me a better evaluator.  Let me explain. I specialize in systems evaluation, which is grounded in systems thinking theory. It involves testing hypotheses using four universal patterns of thinking:  distinction making, part-whole system structure, relationships, and perspective.  When evaluation is focused on learning, the driving force behind my work, there is a premium on perspective.  Our human tendency is to see the world through the lens of what we already think is true. Coming out later in life, after a brief stint in conversion therapy, I know firsthand what it’s like to be locked in a perspective.  This blog is personal, but it illustrates how our personal lives can influence our professional lives.

Prior to my ex-communication, a pastor summarized the religious perspective behind the position that homosexuality is a sin, a choice, and can be changed. “We will always be loving, always be learning, but we will never change our position on this issue.”   How can you truly learn if you only consider evidence that supports one position and dismiss any information that might threaten it?  The heated debate on sexual orientation is an extreme example, perhaps.  But how often do we do this in subtler ways in evaluation?

Lessons Learned:

What does it take to change our perspective?  Direct experience can adjust our thinking quickly.  A friend who lost everything in the recession went from being adamantly opposed to government programs of any kind to being grateful for the services his special needs son received at his public school. A dramatic shift to be sure, but that same friend still can’t fathom how I could possibly “choose” to be gay.  Personal experience is powerful, but unreliable as a change agent. We can’t experience every perspective directly, but we can get better at perspective-taking if we understand its nature.
Perspective involves the other universal patterns of thinking: distinction-making, part-whole systems, and relationships. LGBT, for example, is not monolithic nor does it exist in a silo. There are the distinct parts of LGBT and well as distinct parts to our identity such as generational, spiritual, political, etc – all of which relate in diverse ways. I experience the G in LGBT – but that is in combination with being a 47-year old male, Latino, Christian, upper middle class, liberal from New Mexico. I must be cautious about making assumptions about other LGBT perspectives.

Being mindful of perspective in a systems thinking context is what cultural responsiveness looks like in practice. It’s understanding that the same thing might look different from another perspective. Taking perspective is a skill that can be learned and honed; being good at taking perspective is recognizing that you can never be perfect at it.  You must always allow for the new and unknown.  Perspective is one of my most valuable tools. Understanding that an alternative perspective’s validity does not necessarily threaten a differing or opposing perspective – but it might.

The American Evaluation Association is celebrating LGBT TIG Week with our colleagues in the Lesbian, Gay, Bisexual & Transgender Issues Topical Interest Group. The contributions all this week to aea365 come from our LGBT TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi, Everyone! My name is Amanda Jones, and I am one of the program chairs for the LGBT Issues TIG. I have a background in clinical psychology that has proven invaluable in my work as an independent evaluator.

I am writing today about a nagging question I had when I first learned about the LGBT TIG at AEA: Who does the LGBT TIG exist to serve? Lesbian, gay, bisexual, transgender, questioning, intersex, or asexual (LGBTQIA) evaluators, or evaluators interested in addressing LGBTQIA issues through evaluation?

After many informal conversations and email exchanges with other LGBT TIG members, I’ve concluded that this TIG is serving both purposes. It’s a place to connect with other evaluators with similar backgrounds, and a forum for exploring the best ways to address LGBTQIA issues in evaluation. In fact, both purposes are connected to the values AEA supports, including diversity of membership and culturally responsive evaluations.

So, what does this mean for other evaluators reading this post?

Hot Tip:

The leaders of the LGBT TIG would love to hear from you if:

• You identify as LGBTQIA. Many of us also identify that way, and are interested in getting to know – and maybe even working with — other evaluation professionals with similar identities.
• You want to increase the sensitivity of your evaluations to the LGBTQIA community. We won’t necessarily have ready-made answers for every issue, but many of us have thought a great deal about these issues in evaluation and would love to act as a sounding board.

We hope to hear from you soon! 

The American Evaluation Association is celebrating LGBT TIG Week with our colleagues in the Lesbian, Gay, Bisexual & Transgender Issues Topical Interest Group. The contributions all this week to aea365 come from our LGBT TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi, my name is Libby Smith, I am an evaluator at the Applied Research Center at the University of Wisconsin-Stout and program co-chair for the LGBT Issues TIG. When I first took on the task of introducing LGBTQ Week on this blog, I considered that the most common conversations I have about cultural competence in evaluation are related to race and ethnicity.  What about LGBT cultural competence? It rarely comes up specifically in my own work, but I am regularly asked to consult with others on how to ask about gender/sex/sexual orientation in evaluation work (we’ll address this later this week!).

This leaves me with the feeling that people often think their work is done once they have asked these questions in a “correct” way. Obviously, it goes deeper than that, but how are we as evaluators expanding our cultural competence around LGBT issues? And this applies to LGBT evaluators as well (again, we will address this later in the week).

Rad Resources:

If you aren’t familiar with the AEA’s Statement on Cultural Competence, this is a wonderful place to start. The statement is intended to proactively assert the responsibility that we, as evaluators, should address the needs of culturally diverse populations in our work.

The statement doesn’t tell us how to achieve that goal though. We must take it upon ourselves to be open, self-aware, culturally humble, and intentional about educating ourselves.

The health care field has done exceptional work in developing tools to train providers in addressing the unique needs of LGBT individuals in both physical and mental health. While not directly applicable to most evaluation work, we can leverage these resources to expand our understanding of the many ways in which LGBT populations present diverse perspectives.

This week, evaluators from our TIG will share their strategies, experiences, and insights gained from being evaluators with a unique perspective. We will address gender, sexual orientation, and perspective-taking and hopefully change some perspectives in the process!

The American Evaluation Association is celebrating LGBT TIG Week with our colleagues in the Lesbian, Gay, Bisexual & Transgender Issues Topical Interest Group. The contributions all this week to aea365 come from our LGBT TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

cultural competence

Greetings aea365 readers and potent presenters!  Sheila B Robinson here, aea365’s Lead Curator and sometimes Saturday contributor doing double duty as coordinator for our Potent Presentations Initiative, aka p2i. P2i is about helping evaluators improve their presentation skills, whether it’s for the AEA annual conference, any other conference, or any other type of presentation or meeting. Our p2i site is chock full of free tools, guidelines, videos, and checklists to help you develop your presentation’s message, design, and delivery and help you engage your audience with interactive strategies.

Hot Tip: Now is a great time to check out and download some of our free resources on the p2i Presentation Tools & Guidelines page. They’re organized around three primary components of a great presentation: Message, Design, and Delivery. Engaging your audience is an additional key element and we have a resource for that as well.

Hot Tip: Read what some of our aea365 authors have had to say about their experiences using p2i tools and resources.

Cool Trick: Check out what people are saying about p2i on Twitter using our new hashtag, #aeap2i!

Get Involved: Have you used a p2i tool to help with your presentation? Tweet about it (don’t forget the hashtag!), or consider composing an article for aea365 on how you used it and how it worked for you!

Not presenting, but chairing a session? There’s a tool there for you too! Our Session Chairs Checklist offers advice for preparing for your role and supporting presenters.

Finally, check out the Presenter Resources page for Evaluation 2017 for more info!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

p2i · Potent Presentations Initiative

Hi! I’m Kelci Price, Senior Director of Learning & Evaluation at the Colorado Health Foundation. I used to think learning was an event that happened when evaluation results were ready, but I’ve come to realize that learning is actually at the core of good strategy, and it’s the critical link to whether evaluation actually gets used. We evaluators pay a lot of attention to assessing organizational strategy, and we need to pay just as much attention to creating effective learning practices in our organizations to inform strategy.

Lesson Learned: Implement an effective learning practice.

We wanted a learning approach that would help us assess evidence and connect that to our decisions. It needed to be simple, flexible, accommodate multiple types of evidence, and link insights directly to action. We came across a platform called Emergent Learning, and it has become the indispensable core of our learning practice. Emergent Learning isn’t just a set of tools. It’s a practice that deepens our ability to make thinking visible so that we can more effectively test our thinking and evaluate our results.
Lesson Learned: Start small, stay focused.
Don’t start with a huge plan for learning – focus on smaller learning opportunities to begin with. We started by understanding what strategic decisions staff needed to make in the very near future, and we offered to use our learning practice to help them reflect on past work and assess what might be effective approaches moving forward. They loved it! The most successful learning will happen when you focus on questions that are relevant right now in your organization – these may be related to internal processes, culture, funded programs, or strategies. Approaching learning this way keeps it compelling and relevant to current decisions.

Lesson Learned: Learn deliberately.
The most effective learning takes planning and prioritization. You can start by capitalizing on emergent opportunities, but over time you should move towards being planful about how your organization learns. Know what decisions you want to inform, then work backwards to determine what you need to learn, when, and how you’ll do that. Seek to build learning into organizational practices and routines so it’s not an add-on item. Look for opportunities to change the content of meetings, documents, planning processes, etc. to embed better learning.

Rad Resource: Emergent Learning is an incredibly powerful learning framework, and integrates seamlessly with evaluation practices.

Rad Resource: This paper from Fourth Quadrant Partners overviews learning in foundations, FSG talks about learning and evaluation systems, and this piece gets you thinking about foundation learning as central to strategy under conditions of complexity.

The American Evaluation Association is celebrating Nonprofits and Foundations Topical Interest Group (NPFTIG) Week. The contributions all this week to aea365 come from our NPFTIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

emergent learning · organizational learning

Greetings! We are Kelly Hannum, Jara Dean-Coffey, and Jill Casey and as part of the Luminare Group we offer strategy and evaluation services and we regularly work with foundations. Evaluation is part of the work to advance the aims of foundations. Ideally evaluation is part of and flows from a foundation’s strategy. Evaluation provides the means to both understand and to demonstrate value and to do so in a manner that reflects the values of the foundation. The connections between the value created and the values by which that impact is created should be meaningful and explicit. This isn’t a new or particularly revolutionary idea, but the practice of explicitly incorporating values into evaluation work continues to lag. While making values an explicit part of work is important for any kind of organization, it is critical for foundations. Why? Because by their nature foundations are values-driven organizations and that should be explicitly reflected in all facets of their work. Doing so in evaluation work is particularly important because it enables foundations to hold themselves accountable for living their values in meaningful ways and to demonstrate to others that they are living their values. It also sets an example for others to do the same.

Hot Tip: Be intentional and explicit about your organizational or programmatic values

Hot Tip: Incorporate values into frameworks used to guide or describe efforts, such as your Theory of Change

Hot Tip: Reflect on the ways in which how you currently engage in evaluation may or may not be aligned with your organizational values and your grantmaking strategy

Hot Tip: Think about how you have conducted and used evaluation findings in the past and identify what worked, why and how to get more of that (and less of the other stuff). What implicit values are suggested in how you’ve conducted or used evaluation – both in terms of the processes used as well as what stakeholders were involved and how they were involved?

Hot Tip: Clarify how evaluation will be used and regularly communicate that to stakeholders (it also never hurts to be transparent when using evaluation so stakeholders see you doing what you said you would and so they see how data they provided are being used)

Rad Resource: Developing an evaluative mindset in foundations – This two-part post provides more information about our perspective on why having an evaluative mindset and being explicit about it is important in foundations.

Rad Resource: The Role of  Culture in Philanthropic Work – This is a collection of resources from Grantmakers for Effective Organizations

Rad Resource: Raising the Bar  This article discusses how philanthropy can use an equitable-evaluation approach to apply the principles of the AEA statement, present the concept of equitable evaluation alongside an approach for building equitable-evaluation capacity, and apply equitable evaluation capacity building to philanthropy.

The American Evaluation Association is celebrating Nonprofits and Foundations Topical Interest Group (NPFTIG) Week. The contributions all this week to aea365 come from our NPFTIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings, we are Kristina Jamal and Jacqueline Singh. In addition to being NPFTIG members, we serve on the PDTIG leadership team. Kristina is founder of Open Hearts Helping Hands (OH3), a nonprofit that collaborates with student-focused organizations and community members. Jacqueline is an evaluation/program design advisor and founder of Qualitative Advantage, LLC. We started working together to help OH3 move from being a young nonprofit “flying by the seat of its pants” to becoming a viable organization that competes for funds to bring about common outcomes between formal and informal secondary education organizations.

Because foundations and grantors look for promising programs that can get results, we wanted to move beyond logic model linearity to show a complementary and easy-to-understand way of how a nonprofit program is strategic and intentional. From a nonprofit’s perspective, this AEA365 article addresses the utility of conceptual frameworks and models for front-end evaluation activities, measurement, and strategic planning. 

Lesson Learned: Collecting evidence for improvement, decision-making, and accountability continues to intensify. Funders expect recipients to partner with other organizations and provide evidence of program outcomes. Young nonprofits are overwhelmed at the thought of where to begin. Indeed, navigating disciplinary fields, paradigms of inquiry, and complex environments that commingle evaluation with research can be daunting. Conceptual frameworks can reveal program alignment with other operating mechanisms that logic models alone may miss—and, help bridge the relationship evaluation has with strategic planning, measurement, program management, and accountability. They are often used within the context of evaluability assessment (EA) and prospective evaluation synthesis (PES) as exemplified within these links. Similarly, nonprofits can use conceptual frameworks to clarify their purpose, questions and build evaluation capacity.

Program designs are merely abstractions unless conceptualizations are made explicit and understood by stakeholders. Creating conceptual frameworks is developmental and experiential. The process involves document analysis, reading literature, asking questions, describing and defining relationships, capturing or proposing plausible links between components or emerging factors—dependent upon what is to be evaluated. Conceptual frameworks such as the OH3 Conceptual Framework take “context” into account and help nonprofits to expand their view of what logic models capture.

Hot Tip: Do not undervalued or overlook conceptual frameworks. They come in a variety of forms, serve different purposes, and help figure out what is going on. Conceptual frameworks provide an aerial view and are useful for connecting multiple areas of disciplinary work (e.g. research, theory, policy, technology, etc.). They help guide the selection of useful data collection tools and evaluation strategies.

Rad Resources: What we have found to be useful for understanding how to create conceptual frameworks, thinking through overlapping aspects of program design, measurement, and focusing future evaluations are: 1) James Jaccard. & Jacob Jacoby’s Theory Construction and Model-Building Skills, 2) Joseph Maxwell’s Qualitative Research Design: An Interactive Approach, 3) Joseph Wholey’s Exploratory Evaluation approach (EA) in the Handbook of Practical Program Evaluation, and 4) Matthew Miles & Michael Huberman’s Expanded Sourcebook: Qualitative Data Analysis.

The American Evaluation Association is celebrating Nonprofits and Foundations Topical Interest Group (NPFTIG) Week. The contributions all this week to aea365 come from our NPFTIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

logic models