AEA365 | A Tip-a-Day by and for Evaluators

Hello! My name is Miki Tsukamoto and I am a Monitoring and Evaluation Coordinator at the International Federation of Red Cross and Red Crescent Societies (IFRC).

What if video could be used as the “spark” to increase the engagement and interest of communities in your programmes?

Recently, I had an opportunity to be part of a PVE team for the Global Framework for Climate Service’s programme which aimed to deliver and apply “…salient, credible and actionable climate services towards improved health and food security in Malawi and Tanzania.” To ensure better use and acceptance of this PVE for future programming, IFRC piloted the Most Significant Change technique[1](MSC), using the OECD/DAC criteria of relevance/appropriateness, effectiveness, coverage, sustainability and impact as themes for group discussions. Here are some of the lessons learnt:

Lessons learned:

Rad Resources: PVE videos were made at the community level, the country level and the multi-regional level.

Country level PVEs:

(https://www.youtube.com/watch?v=fSXj0IllfvQ&index=3&list=PLrI6tpZ6pQmQZsKuQl6n4ELEdiVqHGFw2)

(https://www.youtube.com/watch?v=mFWCOyIb9mU&index=4&list=PLrI6tpZ6pQmQZsKuQl6n4ELEdiVqHGFw2)

Multi-country PVE:

(https://www.youtube.com/watch?v=HzbcIZbQYbs&index=2&list=PLrI6tpZ6pQmQZsKuQl6n4ELEdiVqHGFw2)

A Red Cross Red Crescent Guide to Community Engagement and Accountability (CEA)

Guide to the “Most Significant Change” Technique by Rick Davies and Jess Dart

[1] http://www.mande.co.uk/docs/MSCGuide.pdf

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Greetings.  We are Keira Gipson, Monitoring and Evaluation Officer at the U.S. Department of State, Bureau of Conflict and Stabilization Operations and Cheyanne Scharbatke-Church, Principal at Besa, a boutique social enterprise that specializes in the evaluation of programming in fragile states.  We wanted to share insights from an evaluability assessment (EA) we conducted as part of an evaluation capacity building exercise.

Hot Tip:  If you are open to a variety of evaluation approaches and learning opportunities, using a return on investment (ROI) lens to analyze your EA data helps maximize evaluation utility.  There are many good EA guidance notes available with criteria for determining a program’s evaluability.  Some use a weighting approach to determine if one should proceed with an evaluation while others use a percentage of criteria met.  We found the evaluation decision depends more on what you want to learn and the resources you’re willing to invest rather than strictly meeting a given number of criteria.

There are a few non-negotiable EA criteria when recommending an evaluation, such as being able to conduct it safely and ethically.  Most, however, have nuanced implications for an evaluation that mere tallying doesn’t capture.   Even the lack of a program design needn’t prevent an evaluation if the program team is willing to retroactively create a theory of change, for example, or pursue a goal-free evaluation.  The significance of the criteria, in other words, depends on an evaluation’s context.

Building on Rick Davies’ work, specifically the idea of EA results representing an “index of difficulty,” we developed a decision flowchart to help work through the costs to a particular evaluation when criteria aren’t met and how those compare to the learning/accountability benefits for specific users that would result from pursuing the question.

Lessons Learned:

  • EAs provide broad capacity building opportunities: An EA process offers exposure to analysis, design, monitoring, and evaluation concepts, making it an excellent introductory capacity building vehicle.
  • Develop a multi-faceted communication strategy: The value in doing an EA versus an evaluation may not be immediately obvious to program staff.  Plan several iterations of what one gets from an EA compared to an evaluation.

Rad Resource:  We developed a version of an EA checklist specifically for those with less evaluation and EA experience.  We built from Rick Davies’ work, spelling out what meeting each criterion means to help those with less experience better understand the concepts.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi from Jane Fields and Tim Sheldon, Research Associates at the Center for Applied Research and Educational Improvement (CAREI) at the University of Minnesota. CAREI is an independent research and evaluation center in the College of Education and Human Development (CEHD) at the University of Minnesota. As part of our work at CAREI, we serve as external evaluators for EngrTEAMS, a five-year, $8 million project funded by the National Science Foundation. EngrTEAMS is designed to increase students’ learning of science content, as well as mathematical concepts related to data analysis and measurement, by using an engineering design-based approach to teacher professional development and curriculum development (https://sites.google.com/a/umn.edu/engrteams/).

Hot Tip: To make evaluation more relatable to clients, connect evaluation to problem-solving in many fields.

Hot Tip: Shared language enhances collaboration.

Bransford & Stein (1984) state that “All areas of study, including biology, psychology, economics, physics, and chemistry, involve a host of core concepts and theories that people have found to be helpful for conceptualizing (defining) and solving important problems” (p. 20).  With EngrTEAMS, we recognized early on that the steps in an evaluation cycle (see figure below from the CDC) are closely aligned to the steps of the engineering design process used in the project (see figure below from EngrTEAMS). Thus, with project staff, we talk about how when we, as evaluators, engage stakeholders and describe the program, we are essentially defining and learning about the problem. Focusing the evaluation design is similar to the planning stage in the engineering design process. Gathering credible evidence, justifying conclusions, and using the lessons learned are similar to the solution phases in the EngrTEAMS design process of trying, testing, and deciding. According to Thomas & McDonagh (2013), “Shared language is critical to collaboration…Developing and nurturing a shared language is an essential element to enhance communication and collaboration” (p. 46). When we were able to demonstrate that the problem-solving process used in evaluation was similar to that used in their engineering design framework, program staff easily saw the connections and were even more receptive to the evaluation side of the work.

The EngrTEAMS Engineering
Design Process

An example Evaluation Cycle developed by the CDC (https://www.cdc.gov/eval/framework/)

 

 

 

 

 

 

 

 

 

 

 

Rad Resources:

Samples of cycles in other fields:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

 

No tags

Hi there!  I’m Liz Zadnik, member of the aea365 curating team and Saturday contributor.  I don’t know about you, but summer has started here on the East Coast of the United States and I could not be more excited.  I’m ready for some sun and time on the beach!

We all have our vacation rituals – we might bring books with us or some magazines to catch up on.  I personally love podcasts.  To be honest, I probably listen to podcasts more than music – I love how they teach and entertain and help me tap into my creativity.  I thought I would share some of my favorite evaluation-themed (and non-evaluation-themed) podcast episodes to really kick off the summer!

Rad Resource: TED Talks are a perennial favorite – and for good reason!  I found data journalist Mona Chalabi’s recent talk so insightful and practical.  I appreciated how she sought to make data meaningful and reflect the lives of communities – something we strive to do as we work with organizations creating change and generate accessible data visualizations that will resonate with folks.  She asks us to ask three questions – one of which is “Can I [see] myself in the data?”  She uses this question as a way to encourage folks to be mindful of axes labels and breakdowns of data points.

TED Talk Honorable Mention: Giorgia Lupi’s talk on data visualization and representation, “Because data are just a tool we use to represent reality.”

Hot Tip:  In my experience, folks often consider data as separate from them and their realities.  To help navigate this, I’ve talked about data as similar to constellations.  The stars are already there and we care connecting them to tell a story – a story that helps us make sense of our world.  

How have you personalized data so folks can relate and connect?

Image-of-night-at-dusk-with-stars-in-the-sky-and-a-silhouette-of-a-tree-in-background

Rad Resource: Another favorite podcast of mine is NPR’s Invisibilia which explores “the invisible forces that control human behavior – ideas, beliefs, assumptions and emotions.”  I’m currently saving up the most recent Season Three episodes for a work trip, but some of my favorites from earlier seasons explore different solutions to “problems” and ways we are and feel connected to one another.

What’s your favorite podcast?  Do you have a favorite episode that you go back to every now and again?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello good people! My name is Robert Perez and I am a research assistant at Hamai Consulting and a Data Analyst at Youth Policy Institute.

My first foray into independent consulting happened was when I was offered the opportunity to work at a small consulting company. Admittedly, I was nervous in the beginning but I knew that this was what I wanted to do; and I had an inspiring leader who would later become my unofficial guide through the world of independent consulting. I still have a long way to go.

Lessons Learned:

Find your passion. I spent my college years exploring a variety of topics within the field of Psychology, with particular focus on the research realm. I knew I was interested in the psychological development of youth. I read anything I could get my hands on until I found what I thought was the most interesting topic: Positive Youth Development. Ask yourself all the thoughtful questions. What are you curious about? What are you good at? In what ways could you use your skills to pursue your passion?

Once you’ve identified your passion, find others who share your interests and passions. I’m a bit of an introvert, so connecting with people is not one of my strong suits. Thankfully, I can still connect with people online! Post or respond to questions in your LinkedIn networks or use good ‘ol Facebook to search for groups that revolve around some of your interests.

Begin demonstrating your skill. Consider starting a blog or a vlog (video blog) where you can share your insights, experiences, or interesting tools. Writing a blog can help you refine your writing skills, build professional relationships, and afford you the opportunity to educate others about your field of expertise. Do a search online to find out what other evaluators are writing about see where you can fill in any gaps. Reach out to other bloggers to build relationships!

Rad Resource:

One of the most important pieces of advice that has been shared with me is this: don’t forget to learn about the business side of things. All of the consultants I’ve talked to tell me how important it is to balance my ideals with managing a business. I frequently turn to Consulting And Evaluation With Nonprofit And Community-Based Organizations by Viola and McMahon as a reference. It offers accessible insights about where to begin if you are considering starting a consulting firm with a focus on nonprofits, from developing your fee structures to what kinds of services to provide.

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Koolamalsi Njoos (greetings colleagues).  I’m Dr. Nicole Bowman-Farrell, the Founder and President of Bowman Performance Consulting (BPC), a consulting firm located in Wisconsin.  As a traditional Mohican and Lunaape – Munsee (AKA Delaware) Indigenous person, the concept of writing an origin story about BPC is steeped in traditional cultures.  If you know who you are and where you come from then those origin stories help shape how you do business.  BPC started in 2001 as a result of standing my ground professionally, ethically, and morally.

Over five centuries ago, the first Europeans reached Turtle Island.  The Stockbridge Munsee Indians (that is the contemporary name of my Tribal community) lived in the East and in Canada.  We were amongst the first to start an entrepreneurial relationship with the Dutch fur traders in the early 1600’s.  As we were forcibly removed across the country, my community survived. Being resilient problem solvers and smart negotiators transfers well to sustaining a business.

I’m at least the fifth generation of entrepreneurs in the Bowman family.  That’s a picture of my great Grandpa Beaumont Bowman in 1917 who was a contracted logger for many communities in WI.  He’s leading the team, looking sharp, and standing at the top of his game!  Knowing my history is so valuable; it helped sustain my business, where many tough decisions are made.

As a Tribal community, we were always on the move and as a business owner you must never be still.  Adapting to changing resources, political environments, and market needs is key to long term success. There are generations of Bowmans that contributed to our strong family reputation:  hard working, generous and kind, and honest and ethical. These are solid lessons that a business owner needs to live by if they want to be respected, impactful, and sought after as a trusted partner/collaborator.

Hot Tip: Traditional elders tell us that there isn’t “good or bad” there just “is.” In everything that happens, remember to find the lesson in it.  That is good advice for your personal life or your profession.  I enjoy my free subscription to White Bison’s Elder Meditation of the Day (by Don Coyhis, Mohican).

Cool Trick:  Tricksters, in Indigenous storytelling, serve an important purpose to the community.  Sometimes by tricking someone you teach them important lessons.  Check out “Trickster:  Native American Trickster Tales – A Graphic Collection”.

Rad Resource: Indianpreneurship is the fusion of traditional teachings with contemporary sustainable business practices.  It is social entrepreneurship at its best:  evidence based and culturally situated while still making a living and making a true difference in the lives of others.

Lesson Learned:  The toughest times of my business (former employee stole nearly $100,000) taught me the most about myself and my business acumen.  There were a lot of tears and sleepless nights. The relationship of money, emotions, and being a woman played into this life lesson.  My relationship with money, charitable giving, profitability, and sustainability now all co-exist in a balanced and peaceful way with each other.

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi! I’m Myia Welsh of Welsh & Company Evaluation Consulting. I’m also the Chair-Elect of the Independent Consulting TIG. I’m going to share an independent consulting origin story that isn’t often talked about, but I suspect is rather common: dire necessity.

Years ago I was laid off. Yep. Lost my job. It was awful. I loved that job and those co-workers. But change is the only constant in this life, right? So, I had to turn that lemon into lemonade. I was really lucky to have been working with an evaluation consulting firm that let the junior staff in on the business development end of the operation, so I knew just a little about how things worked with proposals, contracts, and the like. Let’s be real here: what I didn’t know could fill volumes (and that is probably still the case). Going independent was terrifying, but I already had been doing evaluation consulting, so continuing to do it seemed like the quickest way from unemployment to a paycheck.

Lesson Learned: Clients can come from anywhere. My longest client relationship developed out of my interest in becoming a potential volunteer for an organization that does work I’m really passionate about. I thought I would have to tirelessly respond to Requests For Proposals, but it turned out that current relationships were far more fruitful.

Lesson Learned: There is so much help to start a business. I leaned very heavily on my state Small Business Development Center (SBDC). Most states have SBDC initiatives that include training, resources, and even one on one business coaching. My business coach walked me through my business license forms and my state LLC paperwork. The U.S. Small Business Administration has fantastic help as well: training, regional offices, partner networks, and more. All for free.

Lesson Learned: Everything comes to an end, even if it isn’t how we imagined. As it turned out, my lemonade is pretty good. I bet yours could be too.

Rad Resources: There is support out there in all kinds of forms!

  • The S. Small Business Administration has free online training, regional offices and partner networks to help fledgling businesses.
  • SCORE is a national nonprofit dedicated to helping small businesses get started. They have over 300 chapters across the country and will set you up with a business mentor.
  • There are many podcasts dedicated to starting and running a business. Two of my favorites are StartUp and Being Boss.

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Hello, I am Matt Feldmann, the principal researcher and owner of Goshen Education Consulting, Inc. and the out-going chair for the Independent Consulting TIG.  My company focuses on educational evaluation and data support for clients in Southern Illinois.

I started my company in 2011 based on the unmet needs for educational evaluation in my local community. I was a data analyst for a local college and took on a “side-job” that turned into a couple of jobs…and finally ended up being too much for a moonlighting effort. Before I left the full-time dependable position, I wrote a business plan to convince myself (and my wife) that this was not a fool’s errand. The following are some ideas when writing a business plan for an independent evaluation practice.

 Lessons Learned: The following are some concepts that will inform your internal business plan document.

Practice what you preach. Many of us help our clients with program development including the identification of objectives, logic models, and SMART goals. Your business needs many of the same things including a mission statement, vision, and organizational principles. These ground your organization and provide you with direction for what you are good at, who you are, and who you serve.

Identify your services and your target market. It is important to identify what you can do and who you can serve, and it is equally important to set limits on who you will serve. For example, I try to do most of my work in Southern Illinois (not in Missouri), even though I live about 20 miles from the Illinois-Missouri border. While my market is geographical, it could just as easily be based on your skills, or knowledge of a sector.

Know your competition and your sales strategy. It is likely that others have completed evaluations in your target market. Knowing these firms and individual consultants will inform how you will approach your potential clients and how to develop your sales strategy.

Rad Resources:

Rhonda Abrams, a small business columnist for USA Today, has provided me with a constant source of material for reflection and strategy for my small business. You should check out her book on successful business plans: Successful Business Plan: Secrets & Strategies.

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Dana Powell Russell

Hi, everyone, this is Dana Powell Russell, Ed.D., sharing my path to independent evaluation consulting. Funny thing is, I set a vision for myself at age 15 of ultimately becoming an independent consultant—I’m not sure how that seed was planted, but here is how it has unfolded…

I started my K-12 education career leading museum outreach in the public schools. In the 1-month gap between one museum job and other, I picked up two brief consulting projects and, in today’s lingo, my “side-hustle” was born. In the ensuing 14 years I developed my full-time career in museum and arts education, while consulting part-time on the side with colleagues and former employers.

As an education director, I became passionate about program evaluation, and decided that would be the focus of my future consulting practice. I earned a doctorate in education on nights and weekends to hone my research skills. When a dear colleague my same age passed away very suddenly, it sparked a pivotal “life is too short” moment, and I decided to take the plunge into full-time independent consulting. My springboard included 20 years of nonprofit and education experience, a two-decades-strong colleague network and, as luck would have it, 14 years of part-time consulting experience already under my belt.

A decade later, I’m grateful for a full dance card, which I chalk up to the following Lessons Learned:

  1. Expand your skillset: Every experience in your career is a chance to build your skillset for future consulting. A broad, adaptable skillset is the best insurance for a full consulting plate and repeat business—the broader the better.
  2. Get (and stay) educated: Figure out what expertise you’ll need for your consulting niche, and carve out the time to get it. Beyond core degrees, staying on top of your game is easy now with abundant online courses and webinars.
  3. Get your feet wet: Don’t wait until you can consult full time; start consulting part-time now with bite-sized projects. It’s a great way to find out if consulting is really for you; plus when you decide to go full-time, you’ll already be an experienced consultant!
  4. Be visible: Engage in social media, workgroups, governing boards, and professional associations in your field; attend conferences, make presentations, get out there! Being a presence ensures that you’ll stay top of mind, and you just may find RFPs in your inbox rather than having to seek them out.

Rad Resources:

  • Peter Block’s Flawless Consulting, and the Fieldbook & Companion that go with it contain solid advice for all kinds of consulting, with a practical tone and real-world examples.
  • Gail Barrington’s Consulting Start-Up and Management is geared specifically toward evaluators and researchers like us. Her perspective on the personal characteristics and skills for success in evaluation consulting is spot on.
  • Nolo Press (com) is a great resource for navigating the legal aspects of running a small business, and they offer publications tailored to the laws of individual states throughout the country.

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Independence Day is approaching, which means the Independent Consulting TIG is taking over aea365 and we’ve decided to tell our origin stories. Up first, me. My name is Laura Keene, owner of Keene Insights and this year’s Independent Consulting TIG Chair.

It’s not a very exciting origin story, but it speaks to a key point when it comes to starting a business: You just have to get out there and meet people! In 2012, I left a full-time job with every intention of finding another one. I had some contract work to keep me busy in between, but I realized pretty quickly that I really enjoyed the freelance life. So, being the introvert that I am, I did the next logical thing in my mind: I grabbed a pile of books and began reading all about how to start a business. A month later (possibly longer), I got a call from my “little” brother (he’s 30 and several inches taller than me). He also owns a business, which he started back in 2007. Here’s how that call went:

Little bro: How’s it going with the business?

Me: Great! I’m learning all about contracts, and accounting, and marketing, and business structures.

Little bro: Awesome. Any prospects? Are you finding some good networking opportunities?

Me: Um, well, I haven’t actually left the house yet.

Little bro: What? Silly goose! Get out from behind your books!

He was absolutely right. A week later, I finally mustered up the fortitude to make it to my first event. It was a total miss. Let’s just say no one in that room was going to ever hire an evaluator, but I networked like crazy anyway. Those early conversations were so hard. It took a while and a lot more events to figure out how to articulate my passion, experience, and expertise. But, I got there. Five years and a whole lot of connections later, Keene Insights is thriving.

Lesson Learned:

With few exceptions, in order to get a business off the ground (or even just grow professionally) we have to get out there and build relationships: Relationships with potential clients, plus collaborators, mentors, influencers, and more. Check out this post I wrote on networking for some tips and resources.

Rad Resources:

While you’re making those connections, it doesn’t hurt to spend a little time in some business how-to books. Here are a few of my faves:

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

<< Latest posts

Older posts >>

Archives

To top