AEA365 | A Tip-a-Day by and for Evaluators

CAT | Independent Consulting

Barbara Klugman

Barbara Klugman

Hi, I’m Barbara Klugman. I offer strategy support and conduct evaluations with social justice funders, NGOs, networks and leadership training institutions in South Africa and internationally.  This blog is about the value of establishing an Evaluation Advisory Group when your task requires more skills that you have!

I’m currently the ‘learning and assessment partner’ of Tekano an organisation which runs the Atlantic Fellows Programme for Health Equity South Africa.  I have found this experience challenging, not least because it’s my first experience in long term developmental evaluation, because I have multiple accountabilities – to funder, Tekano board, and staff, as well as to the public good given the country’s desperate need for a strong fellowship programme on social determinants of health; and because my primary expertise is in strategizing and evaluating social justice advocacy initiatives rather than leadership development. Finally, I am a white older woman playing the role of ‘critical friend’ to an initiative whose staff and fellows are mostly black and young and whose advocacy experience is taking place in a vastly different historical moment from when I built such experience.

With these constraints in mind, when I developed my terms of reference, I motivated a budget to allow me to establish an Advisory Group. I sought members to fill three gaps in my expertise (that I was aware of) – an organizational psychologist whose expertise traverses organisational behaviour and monitoring and evaluation; a fellowship evaluation expert; and an expert in community-building and evaluation in the South African context.

I engaged the group to help me review the vast quantity of data and analysis I had produced in order to hone in on priorities for my final 18 months in this role, and to help me clarify how to manage my layers of accountability. It was fantastic!! The group brought totally independent eyes to bear on my questions. They helped me distinguish the nature and hierarchy of my various accountabilities. They confirmed my own conclusions on achievements and challenges thus far while hauling me out of the data, insights and relationships I was buried in.  This allowed me to hone in on powerful questions and ways of asking them, for the second half of my tenure in this role.

I loved every minute of the experience even while it was exhausting and challenging. I would encourage you to build in space for this kind of independent support if you’re conducting developmental evaluations.

Hot tips: You have to be a fairly confident person to open yourself up to criticism. Choose people who are ethical and don’t have oversized egos!

Lesson Learned:

  • Building a formal advisory group into evaluation budget allows you to identify exactly
  • what skills might complement your own, and to compensate people appropriately  for their time
  • My huge thanks to Drs. Jane Reisman, Mark Abrahams and Suki Goodman for their efforts on my advisory group.

·

Hey! I’m Sara Vaca, independent evaluator and part of the AEA365 team, eventual Saturday contributor. Some time ago, I ran into this framework for critical thinking and I thought it was interesting for evaluators.

Rad Resource: The Six Thinking Hats is a system designed by Edward de Bono which describes a tool for group discussion and individual thinking involving six colored hats. “Six Thinking Hats” and the associated idea parallel thinking provide a means for groups to plan thinking processes in a detailed and cohesive way, and in doing so to think together more effectively. 

The premise of the method is that the human brain thinks in a number of distinct ways, which is also the case for evaluators when conducting an evaluation. Six distinct directions in which the brain can be challenged are identified and assigned a color. In each of these directions the brain will identify and bring into conscious thought certain aspects of issues being considered (e.g. gut instinct, pessimistic judgement, neutral facts). None of these directions is a completely natural way of thinking, but rather how some of us already represent the results of our thinking.

The six directions, that in some cases make us think of some paradigms, are:

  • Information White – considering purely what information is available, what are the facts? Very relevant to any evaluator, but probably resonating strongly with positivist paradigm.
  • Emotions Red – intuitive or instinctive gut reactions or statements of emotional feeling. Clearly something that can emerge more easily with some qualitative methods like life stories.
  • Discernment Black – logic applied to identifying reasons to be cautious and conservative. Practical, realistic.
  • Optimistic response Yellow – logic applied to identifying benefits, seeking harmony. Sees the brighter, sunny side of situations, which resonate with Appreciative Inquiry approach or method or designs.
  • Creativity Green – statements of provocation and investigation, seeing where a thought goes. Thinks creatively, outside the box.
  • Managing Blue – what is the subject? what are we thinking about? what is the goal? Can look at the big picture.

 

https://www.storyboardthat.com/storyboards/nathanael-okhuysen/the-six-thinking-hats

Cool Tip: Ask yourself these questions: Which hats you usually wear? And which is the prenominant one? Thinking of this approach could maybe deliberately challenge you to go out of your “natural” tendency, and hence planned for use in a structured way allowing one to develop tactics for thinking about particular issues in your evaluation practice.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings! I’m Seema Mahato, board member (student representative) for The Ohio Program Evaluator’s Group (OPEG) and a member of the Local Arrangements Working Group for Evaluation 2018.

‘Evaluation without Borders’, a pro-bono evaluation consulting program was piloted by Washington Evaluators at the 2017 AEA conference. As a participant of this event during AEA 2017, as a pro-bono consultant, I realized the tremendous value this event holds in terms of materializing AEA’s end goalsGrowth in the effective use of evaluation as a means for enhancement of the public good (with results optimizing use of available resources). This volunteer consulting event provides a platform for evaluation professionals to share their knowledge and expertise with professionals from community-based organizations (non-profits), and thus advances AEA’s commitment towards greater public good. Other professional organizations such as Bankers without Borders, Engineers without Borders USA, and Statistics without Borders live the similar philosophy of sharing technical expertise for the enhancement of public good.

Given the current times of facts versus alternative facts tussle, evaluators and evaluation could play an important role in creating the awareness necessary to distinguish facts from alternative facts as we deliberate Speaking Truth to Power in Cleveland this year. Continuing the Washington Evaluator’s initiative from AEA 2017, OPEG would like to invite evaluators and non-profit professionals to participate in the second iteration of Evaluation without Borders in AEA 2018. This pro-bono consulting event provides an opportunity for conference attendees to reflect on this year’s conference theme and understand how evaluation could facilitate and contribute towards Speaking Truth to Power.

Rad Resources:

Giving back to the community. We would like to invite you to participate in Evaluation without Borders and have created a sign-up form, also available on the OPEG website, to facilitate participation in this event. Please share some details about your evaluation experience and areas of expertise that you would like to volunteer with as a pro-bono consultant. The pro-bono consulting sessions are scheduled for Monday, October 29th, Tuesday October 30th and Wednesday October 31st.  On Wednesday, October 31st the event ends at 2pm so that we can attend the opening plenary.

[For additional information, please contact Seema Mahato at sm618312@ohio.edu]

 

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

· ·

Hello, I’m Martha Brown, and I started my consulting business in 2016 thanks to some encouragement from a group of passionate independent consultants. I was recently inspired by a keynote speaker, Lea Bill, at the 2018 CES Conference. She displayed a logic model that was circular and artistic, and it included the values of the people she worked with. I immediately thought of one my clients, a botanical garden, and how I could use their stated values to deepen their thinking about their programs.

Like Kylie Hutchinson’s data parties, I decided to have a “logic model party” full of big sheets of paper and lots of colored markers. We started by determining outcomes, which had never been done. My job was to ask the right questions to the garden’s Executive Director, the Program Manager, and a very hands-on and knowledgeable board member. This was more difficult than I imagined, as the Program Manager’s thinking constantly defaulted to activities. But after an hour, we had our first working draft of outcomes.

Kids & Schools desired outcomes handwritten chart

Using Bob William’s questions about assumptions, I guided them into digging deeper about why they believed these were reasonable outcomes. From there, we worked backwards and after 3 hours, had a very colorful logic model taped to the wall. It is only the beginning of our work, but in the process of making the first draft, something beautiful sprouted. They decided to approach one school and invite them to participate in a whole-school partnership, which we can evaluate in several ways over three to five years. None of us saw that coming, and it is a great idea. My work with this client went from a very small 2-year contract for assessing student learning in a single program to a potentially long-term contract evaluating a new model in science education. Because I don’t hesitate to share that I explore new approaches and bring them to our work together, my short-staffed client trusts they have a real partner – a consultant and evaluator – who looks out for them. It’s a win-win.

Lessons Learned:

Ask the right questions. Then…

Step back. While the team brainstormed, I took notes on big pieces of paper that led to 7 pages of typed minutes.

It can’t all be done in one session. We have more work to do, but we planted many seeds on fertile soil.

Organic is good. If the seeds we planted grow, I will be conducting a longitudinal evaluation on a new model that may have an impact on how educators think about teaching science.

Hot Tips for Independent Evaluators:

Work with organizations and programs that inspire you. Have fun at your job.

It’s OK to start small. You never know what will come of it.

Get out of the way. Your clients are the experts.

Learn as much as you can about as much as you can – and bring it back to your clients.

 

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Melanie Hwalek

Melanie Hwalek

My name is Melanie Hwalek. I have been practicing evaluation for almost 40 years. I’ve always taken for granted my knowledge of social science research design. Recently, I had a profound experience that made me realize how this knowledge can be used to empower voices from the field.

Hot Tip: Respect the power of our social science research knowledge and use it for the greater good.

I learned this lesson from BABESWORLD, a systems approach to healthy living and substance use disorder prevention. BABESWORLD began 40 years ago as BABES, a K-12 puppet-based storytelling curriculum designed to assist young people develop positive living skills through accurate, non-judgmental information about the use and abuse of alcohol and other drugs. BABESWORLD’s design emanated from the personal recovery of its designer and hours of storytelling from recovering adults. Decades of stories from BABESWOLD participants illustrate how it has profoundly impacted their lives.

My organization was hired to help BABESWORLD become “evidence-based.” Certification as “evidence based” would qualify BABESWORLD for U.S. federal funding. At the contract kick off meeting, the BABESWORLD CEO asked me to read a chapter in Antoine De Saint-Exupery’s The Little Prince. The Little Prince is a fable written in the 1940s where a little prince travels from his home on a star to visit other stars in the galaxy. On each star he visits there is a parable. The parable the CEO asked me to read was about a star where a Turkish astronomer had identified, for the first time, the little prince’s home. In his turban and robe, the Turkish scientist presented his findings at a galaxy-wide conference of astronomers. Nobody believed him. Then, the king of the Turks mandated that all people dress in European clothes or face death. Ten years hence, the Turkish astronomer made the exact same presentation to the exact same conference of astronomers. This time he was dressed in an elegant European suit. The conference hailed him for his discovery. After I read this story, the CEO said to me, “I need you to dress BABESWORLD in European clothes.”  This was a profound realization to me of how my knowledge of quasi-experimental design could be used to translate what BABESWORLD knew from decades of anecdotal evidence (Turkish) into the language of the U.S. scientific community (European). It humbled me to think about our social science research knowledge as “power,” and that we can use this power to give voice to those in the field.

Rad Resources: To learn about BABESWORLD, please visit: www.babesworld.org

Hazel Symonette describes this translational power as boundary spanning, and the evaluator as boundary spanner. Read Symonette’s chapter on culturally responsive evaluation in Hood et. al’s Continuing the Journey to Reposition Culture and Cultural Context in Evaluation Theory and Practice.

 

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! My name is Norma Martínez-Rubin. As an independent evaluation consultant, I specialize in evaluating nonprofits’ and foundations’ disease prevention and health promotion programs and projects. I’m particularly drawn to initiatives designed to address health, economic, and educational inequities. Because of the complexity associated with such efforts, I’ve been part of multidisciplinary evaluation teams in which, after labored discussions of evaluation approaches and data-collection strategies for our evaluation proposal(s), one of us is tasked with developing a budget to accompany a proposal.

I recall an early attempt where my team lead strongly believed we would increase our chances of obtaining a contract award by positioning ourselves as the lowest bidder. At the time, I didn’t voice my reluctance about that believing that team member to be well experienced in such matters. In retrospect, the low-cost approach was not the defining factor in contract selection. Nor would I want it to be now regardless of which side I occupy at an evaluation proposal reviewers’ table.

What changed for me since I erroneously succumbed to the notion that low-cost evaluation services equal winning contracts, is a continual attempt to align my service offerings to prospective client needs. This requires being certain that my core services —not necessarily different from other evaluators’— are presented along with features to enhance a prospective client’s desire to seek my services.  A greater perception of value is the “value added” to an ordinary services offer. Value-added services are not cheap, low-cost menu items, but extraordinary services to support clients’ business-related reasons for entering into a contract. Among those are activities that save time, provide peace of mind, and increase efficiencies. Developing an evaluation proposal budget with this in mind ties one’s technical knowledge with business savvy. It contributes to the client-evaluator satisfaction of having a proposal that’s fair to both parties. 

Hot Tips:

  • Think of what you are willing to do if presented with certain work conditions e.g., access to decision makers, ready access to evaluation project stakeholders or data, staff availability to negotiate evaluation plans, acceptable distance to evaluation sites, reasonable number of drafts required before finalizing a data presentation.  Note: These are things that, if provided, are likely to prevent or reduce scope creep.
  • Create an outline of your core services i.e., what are the technical skills you can demonstrate and enjoy using while concurrently serving a prospective client’s evaluation needs?
  • Create a list of your “value-added” services i.e. what, additional to your core list of evaluation services, would delight your clients? (Think: what would make your prospective clients’ work lives easier?)
  • Quantify, in time and monetary terms, the cost of your core and value added services.

Lesson Learned:

  • Think of ways to bundle your services so when opportunities exist, you are prepared to describe them to prospective clients.
  • Value-added services may be intangibles, too! Don’t undermine your interpersonal skills and relationship-building abilities. 

Got value-added services? Sure you do! Think: What delights  prospective clients? What delights me?

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Hello, my name is Kate Clavijo. I earned my degree in Educational Program Evaluation from the University of Louisville in 2002, and have been working as an evaluator ever since. Over the years, I have held full time evaluator positions and worked for non-profits while doing evaluation consulting on the side. About eight years ago, I left a “traditional” evaluation job and became a full-time consultant. In the early days of full time consulting, I budgeted most of my time for drumming up business and networking with potential clients. This was time well spent, but quite isolating and lonely. My breakthrough moment came after attending an AEA conference and realizing what I missed the most was sharing with evaluators. I like to collaborate and talk about evaluation. The first place I turned to when I returned from the conference was the AEA website. Through the AEA “Find an Evaluator” link, I found two people living in my area and sent them an email asking if they wanted to meet for coffee.

The first lived right around the corner. After several coffees and good conversations about evaluation, she asked for help with one of her projects.

I said yes…and when she moved away to greener pastures she passed on what, to this day, is my favorite client.

I reached out to the second person after admiring her website. We have met for coffee and lunch over the years. During our first meeting, she provided me with inspiration, friendship and some excellent resources related to evaluation consulting.

Recently, she asked “Can you help with conducting an interview?”

I said yes…and was exposed first hand to an evaluation of a complex initiative where networks of people and organizations are changing systems in local communities and I saw a very innovative use of graphic presentation.

Lessons Learned:

Reach out to other evaluators. There is no need for an agenda – friendship and talking about evaluation are the immediate rewards.

Say yes. When an evaluator asks for help, just say yes. You never know what you will learn, who you will meet and how it will shape your own evaluation practice.

Rad Resource:

Find an Evaluator is a free searchable online resource provided by the American Evaluation Association. It includes the contact information and area of expertise of hundreds of evaluators across the country and the world. Most people might want to start by searching for evaluators close to home, but you can also search for evaluators working in a specific niche.

 

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Hi there!  I am Mary Nash of Nash Insights, based in Pittsfield, Massachusetts. As the country celebrates Independence Day, we at the Independent Consulting TIG are reflecting upon breakthrough moments in our careers. Mine occurred in January while I was in Costa Rica on a Live the Life of Your Dreams Yoga Retreat. After several days of yoga, meditation, journaling, walking on the beach, and discussing how to live the life of my dreams, a thought occurred to me that I had never considered in 20+ years of consulting.  It was time to relocate from my home office.

My home office worked really well for me over the years. I am disciplined enough to separate home from work and had a lot of productive years there. It is well equipped and has a view of the beautiful Berkshire Mountains. I don’t have to pay rent for it and it gave me the flexibility I needed when my children were younger. But I realized that I was getting restless working from home and starting to feel secluded. I needed a change and the time was right to make a move.

Yoga retreat class

Image credit: Bella Retreats

When I returned from Costa Rica, I acted on my epiphany right away. I found office space in a restored industrial building, got to work ordering furniture, sent out 50 postcards announcing the move and reminding people of the services I provide, and within six weeks Nash Insights was in its new home. Being a diligent evaluator, I tallied the results of my intervention and they look pretty good. 

Rad Results:

  • I leave the house every day, even if I don’t have meetings I need to attend.
  • I go out to lunch and for coffee more often with colleagues.
  • I attend more community events.
  • I get more done in less time.
  • I have three new clients.
  • I can invite clients to meet in my office rather than always going to them.
  • Friends and family don’t call me during the day as much “just wanting to chat.”

Author's office

Rad Resources:

  • Yliving was a great source for furnishing my industrial office with a mid-century look.
  • Moo makes really nice oversized luxe postcards out of thick cardboard stock, perfect for announcing that you’ve moved.
  • In search of the perfect office plant, I discovered Plantz.com. They sent me a beautiful Lisa Cane plant, and even featured me on their blog! 

Lessons Learned:

  • Just because you have been doing something for 20+ years doesn’t mean you have to do it forever.
  • Moving into a new office has paid off with higher productivity and an increase in business. And if I still want to work at home occasionally, my home office is still there.

 

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Welcome to Independent Consulting TIG Week on AEA365! I’m Myia Welsh, the 2018 IC TIG Chair and owner of Welsh & Company Evaluation Consulting. Our AEA365 week will focus on the theme: breakthrough moments in independent consulting. Those moments, from tiny to enormous, that had a profound impact on our IC TIG members.

My breakthrough moment came on a rainy spring day in North Central West Virginia. I was filling out paperwork at local bank branch. I was opening a business account.  My hand shook when I filled out the form. Was this for real? This seemed so serious. I had my little folder of business artifacts all ready to go: business license, state LLC paperwork, EIN number. All those other structural aspects of business had happened anti-climactically, in online forms or through the mail. The day I set up my bank account in person, it all felt real in a whole new way. Suddenly an external person was talking to me about my business and it’s needs as if it was a real thing. It was. I was really doing it.

Then I deposited the first check, which felt extra real.

And then tax time came… And the realness continued…

I have never been so proud of paying taxes!

Rad Resource:

I get great satisfaction out of tending to taxes. It’s also something that people who don’t do independent work (but seem to want to) ask me about.  It usually goes something like “I wish I could just do it myself, but then I’d have to figure out taxes.” There’s help for that!

Enjoy our week of independent consultant breakthroughs!

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Hi, I’m Sara Vaca, independent consultant (www.SaraVaca.com), helping Sheila curate this blog and eventual Saturday contributor.

After 5 years as a freelance evaluator, I am starting to get more contracts coming from previous jobs or mouth-to-ear, but still, I see myself going through the regular selection process: the evaluation commissioners publish the Terms of Reference where they ask you to prepare a methodological proposal, your estimated budget and, in some occasions, they also ask for references from your previous jobs to have more information about how you work. And, if you have gone through that, I don’t know about you, but I’m often embarrassed to disturb my former colleagues or managers (again!) to ask them for references.

Cool Tick: So, I have thought of this idea I haven’t tried out yet, but I thought it could be cool: What if, after each evaluation, I sent the evaluation commissioners and users I’ve worked for a short survey so they can assess my performance (once), so when future potential employees ask for references I could forward them what they have said about my work. I would use some of the typical questions they usually ask in the templates they send so the content is useful for them. Would that work? This could have several advantages: 1) serve me to learn about my performance in a more structured manner, 2) it would allow me to stop asking them for the same favour over the years, and 3) also it could save some time (as often they are busy and the selection/contract process can be delayed until they have the time to answer).

Hot Tip: If we could do that, we would save our references the trouble of being bothered by this, because I also often doubt about whether to reach out to warn them or not: if I do, chances are they are finally not contacted – while if I don’t say anything, they’ll probably get contacted and reach out to say so later.

Rad Resource: There are many resources out there to do quick online survey, but in my case, I would use www.SurveyMonkey.com (that I use quite a lot for other purposes, such as before conducting a workshop, I contact participants to understand their level in the subject and their expectations so I can better customize the survey – but that is another story).

If you want to comment on the idea (whether you think it could be a good fit or not), it would help me to decide whether to try it – and if I do, I would certainly let you know about it later! Thanks 🙂

 

UPDATE: Patricia H. Mueller let me know that AEA posted an evaluation form for members to use with their clients (on principles and an abbreviated one). The principles one should probably be updated to match the newly revised ones. She uses a revised version of the longer form every other year or so with her clients that she finds helpful for constructive feedback and also for marketing purposes.

 

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Older posts >>

Archives

To top