AEA365 | A Tip-a-Day by and for Evaluators

CAT | Uncategorized

Sara Vaca

Sara Vaca

Hi, I am Sara Vaca (Independent Evaluator and Saturday’s contributor to this blog). One day some months ago I tweeted that this book by Michael Patton (Qualitative Research & Evaluation Methods: Integrating Theory and Practice) was solving two thirds of my evaluation doubts.

“What is the other third?” he replied. And he left me thinking.

As I think about and practice my dear evaluation transdiscipline, I encounter many small and big doubts, and they are often recurrent. So I decided to compile them and share them here.

Lessons Learned (Or To Be Learned):

  1. I don’t have real-life data about it but…

Why do most evaluations use the same methods? That is: documentation review and interviews (I would say in 100% of them); focus group discussions, surveys, case studies and observation are also more or less common. But I’ve hardly ever seen the rest of available methods in the (120+) reports I’ve meta-evaluated.

  1. We (often) care to explain in detail how we are going to answer the impact questions to infer results, but…

Why we only focus in the Impact criteria design? Why not explain the logic behind how we answer if the program was relevant? Or efficient? (So I’m working on this issue).

  1. I know (part of) the great variety of evaluation elements (see a catalogue in the Periodic Table of Evaluation), but…

How many tools do I need to have in my evaluator’s toolkit to be “well equipped”? Is it enough to know they exist? Is it sensible to try to explore new ones on your own?

  1. I know rigor is not linked to the methods used, but…

How could we showcase examples of how better customized evaluation designs led to better results? (probably there is already literature about this?)

  1. I know objectivity does not exist, it is all about transparency, but… is it really? Isn’t our intended systematic way of collecting and analyzing data an attempt to claim credibility (as objectivity)?
  2. How to make visible (my) invisible bias? Should I talk more (or to be more precise, talk, period) about paradigms with my clients? If I am totally honest in describing the limitations in my reports, does it increase its credibility – or quite the opposite?
  3. Not always sure if this is a relevant doubt or not, but…

Should I expect that at some point we agree in the difference between Logic Models and Theories of Change? Or should I let it go?

  1. How to better explain the articulation of the logic of my evaluative thinking in my inception reports?

And these two are not technical:

  1. This one is more of a personal internal conflict: being that evaluation is my main livelihood, to what extent are my ethics and independence guaranteed?
  2. Will the sentence “I’m an evaluator” will need no further explanation one day?

And I have more – but here are the top 10…

I hope to be solving some of them (maybe you can help ;-P). Others may have no answer (yet). But at least it is a relief to share :-).

 

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Nov/18

3

Using Twitter at #Eval18

Jessica Byrnes

Hi, my name is Jessica Byrnes and I’m a member of the AEA marketing team. With Evaluation 2018 in full swing, I’ve been keeping up with the conference on Twitter using #Eval18.

Twitter is a great tool for collaboration, expanding your professional network, and staying on top of the latest conversations in evaluation – whether you’re attending Evaluation 2018 in-person or virtually! Check out some of the ways #Eval18 attendees are utilizing Twitter during the conference.

Hot Tip: Tweet from the Presentations
Sharing quotes, insights, and tidbits you especially enjoyed from presentations is a popular way to utilize Twitter during a conference. If the presenter shows an interesting stat or slide, take a photo and share it with your followers! Pro tip: If you’re taking pictures of speakers, get their Twitter handle and tag them in the photo.


Hot Tip: Ask Questions
Sharing questions is a great way to engage with presenters, connect with others, and find out information. Contribute to the conversation yourself by posing a question on Twitter using the hashtag #Eval18.


Hot Tip: Let Other Attendees (and AEA!) Know What You’re Enjoying
Events like Evaluation 2018 have a lot to offer. Why not reflect on your favorite parts of the conference?

I hope you enjoy the last day of the conference. Feel free to use these tips at other evaluation events you attend!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello dear readers! Sheila B Robinson, aea365 Lead Curator, sometimes Saturday contributor AND Conference Week blogger here (again!) with a few photos of the fabulous artwork here at Evaluation 2018.  No evaluation tips to share today…because sometimes we just have to step back and find beauty and joy in things outside of our work!

These are just a few of the vistas we are being treated to as we traverse the halls and walkways between the Hilton Cleveland Downtown, and the Huntington Convention Center of Cleveland.

The Moss Wall at Hilton front desk

The Moss Wall at Hilton front desk

 

Photo mosaic mural wall leading into Convention Center

Photo mosaic mural wall leading into Convention Center

Silver Orbs (Bubbles?) installation at Convention Center

Silver Orbs (Bubbles?) installation at Convention Center

I regret that I didn’t have the time to stop and find out the proper names of these installations or the artists names. If you happen to see this information  any where, would you please send it to me so that I may update this post? Thank you!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello faithful readers! Sheila B Robinson, aea365 Lead Curator, sometimes Saturday contributor AND Conference Week blogger here with a quick update on the goings-on in Cleveland!

Our series of two-day workshops got off to a great start Monday and continued on Tuesday, while a number of one-day workshops also took place Tuesday. Wednesday will feature a slew of half day workshops in the morning and afternoon and those will lead us right into our opening plenary!

Many of my friends are here taking the pre-conference workshops and it’s wonderful to see the mix of veteran evaluators continuing the learning journey right alongside first time conference attendees.  I’m hearing great feedback about the workshops people are participating in and these conversations spill out into the break areas as people gather for mid-morning coffee and tea, and again in the elevators and sidewalks on the way back to hotel rooms after the day comes to a close. Even the late night conversations over adult beverage seem to gravitate back to evaluation and our collective work and learning. It’s incredibly heartening to see so many people engaged in talking about work that inspires them and that they clearly enjoy.

Have you made it to Cleveland yet? Perhaps you’re coming in for the start of the concurrent sessions on Wednesday afternoon. This year, you’ll be greeted by a pretty fancy registration station and information booth, along with the usual plethora of signs that help us find our way in unfamiliar territory. Many thanks to the AEA staff and volunteers who make this happen!

Lesson Learned: And did you know the while many of us were having fun think-pair-sharing, scratching out ideas on post-it notes, writing on chart paper, numbering off, getting in groups, moving around, drawing pictures, practicing new skills…and otherwise having a grand old time learning and adding to our toolboxes, our AEA Board of Directors, presidents (current, past, and elect), executive director, and operations coordinator were working intently and intensely on key association issues? Let’s give them some love and thanks too, for all they do to support us. Offer up a handshake, a high five or a hug to these hardworking folks!


Registration booths at Evaluation 2018

Directional signs at Evaluation 2018

 

 

 

 

 

 

 

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hey! I’m Sara Vaca, independent evaluator and part of the AEA365 team. I’m not in Cleveland (sad face) this week (as I just came back for the European Evaluation Conference (EES) in Greece), but I wish I was, as I love evaluation conferences.

I am not an expert compared to so many admired evaluators who have been going to AEA conferences since 1990’s, but from my experience with 3 AEAs, 2 EES and 1 SAMEA (South African Evaluation M&E Association), here are things I learned not to do anymore.

Cool Tips: Some things NOT to do! (;-P)

  1. “Don’t talk to other people you don’t know”

We are all shy to start a conversation with someone you’ve never met: you think you are not that interesting, maybe the person is not that interesting and that you don’t have much in common. But I’ve learned that many, many times the person is interesting, she or he finds you interesting too, plus you have plenty of things in common that occasionally even bring other unexpected developments.

  1. “Improvise your program: Don’t look at the program in advance, and 5 minutes before sessions, just choose one randomly”.

Well, chances are you end up missing things that really interested you!

  1. “Study the program in detail and decide in advance what you will be doing at each moment”. Well, some improvisation is always welcome, as unexpected meetings, acquaintances and events happen at some point of the conference that are worth not missing too.

 

  1. “Don’t go out in the evenings or by night”

To be honest, this one has never happened to me (maybe is my Spanish blood), but I know friends who want to keep it calm and they go back to the hotel as the sessions end. Well, each one has to do what is best for them, of course, but mingling after conference hours is a great way to know colleagues in a different circumstances, and of course, of having fun!

  1. “Stay up all night”

Well, let’s not get crazy either, and if you want to enjoy the sessions and have the stamina to go through it all, you also need some reasonable resting time, no matter how fun the night might seem.

  1. “Pretend to go to every single thing you think you should go”

Even if you study the program, allow certain flexibility and try to be reasonable and keep a balance, don’t feel frustrated if too many things happen or simply too many good sessions overlap and you miss plenty that you would have liked to attend.

Please take all these as personal ideas or suggestions, hoping they resonate with you and help you have a better experience if this is your first time. And please feel free to share more “Not to do things” in the comments, or if some of these Don’ts on the contrary work very well for you! :-)…

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Leigh M. Tolley

Leigh M. Tolley

Hi everyone! I’m Leigh M. Tolley, Assistant Professor, Secondary Education at the University of Louisiana at Lafayette, Past Chair of the PreK-12 Educational Evaluation Topical Interest Group (TIG), and member of AEA’s TIG Council Group on Conference Quality and Logistics. This year will be my ninth AEA conference, and I foresee many more in my future!

A big part of why I love coming to the AEA conference each year is catching up with old friends and colleagues and making new friends and connections. I have always found AEA members in all stages of their careers to be friendly, welcoming, and willing to chat with and learn from others. TIGs are an integral part of why I feel AEA is my “professional home.”

Lessons Learned: TIGs help you grow professionally. At my second AEA conference in 2011, I decided to check out a few TIG business meetings and find ways to possibly get involved as a doctoral student. While at the PreK-12 TIG’s business meeting, they asked for volunteers to be Members-at-Large, and I raised my hand. I have been a Member-at-Large or part of the Program Chair and Chair teams—we have elect, current, and past positions for each in our TIG—ever since. Through these roles, I have been able to develop an excellent network, gain experience in leadership, and better understand the proposal submission and review process, which all help me tremendously as a tenure-track academic and evaluation practitioner.

Hot Tip: Business meetings are open to all AEA members, not just current TIG leaders. Don’t let the name fool you: these are non-stuffy, collegial sessions where like-minded folks can come together to share ideas, learn more about what works in their areas of expertise, and discover ways to engage with AEA.

Hot Tip: Take advantage of myriad opportunities to interact with TIG members while at Evaluation 2018! There will be TIG Tables set up during the Poster Exhibit and Meet the Authors session on Wednesday, and TIG Receptions for groups of TIGs and TIG Business Meetings will be held on Thursday evening. Information will be posted by the Registration Desk during the conference, where you can also pick up badge ribbons to represent your TIGs.

Hot Tip: Members of AEA may join up to five TIGs as part of membership. To join, view, or change your TIGs, log on to the AEA main site, then select Preferences and Update TIG Selections from the far right of the top menu.

Rad Resource: Check out the many different TIGs AEA has to offer on the Topical Interest Groups page of the AEA website. There are almost 60 TIGs, including those focused on research methods, special groups, and the theory, practice, and teaching of evaluation itself. Two new TIGs to check out this year are the Graduate Education Diversity Internship (GEDI) TIG, open to GEDI alumni as well as all AEA members interested in the program; and the Health Professions Education Evaluation and Research TIG.

See you soon in Cleveland!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings AEA members, conference-goers, virtual conference goers and AEA365 fans! I’m Sheila B Robinson, your very excited AEA365 Lead Curator and sometimes Saturday contributor. I’m getting ready to spend nearly a week in Cleveland, OH at Evaluation 2018. The annual AEA conference is one of my favorite weeks of the year – time for me to learn more about my field, attend exciting presentations and see fabulous presenters, meet new people, and reconnect with old friends.

To help you get ready for the conference experience, I’ve bundled some handy resources right here:

Rad Resources: 

  1. AEA365 LAWG posts: We had two weeks of LAWG posts this year – one in July and one in September – with the Local Area Working Group (LAWG) working hard to let us know what to do and where to go in Cleveland.
  2. Tips for first timers (and veteran conference goers!): I contributed an AEA365 blog post on this,  Donna Podems wrote a great article on LinkedIn, Navigating AEA 2018, and Dana Wanzer started this conversation during #EvalTwitter chat a few nights back.
  3. Don’t for get to explore and complete your profile in the Evaluation 2018 Conference App and all the great features it has to offer.
  4. Look for special events, such as the Plenary Presentations,  Design Loft, Topical Interest Group (TIG) Business Meetings, AEA 2018 Awards Luncheon, and other Social Events.
  5. And of course, use the Evaluation 2018 Conference Website for all your conference information needs.

Don’t forget to take a break from silly cat videos and share your conference learning and experiences on social media with the hashtag #Eval18.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Andrea Scallon

Andrea Scallon

Hi! I’m Andrea Scallon, the ME&L Officer at Miriam’s Kitchen – working to end chronic homelessness in DC.

In 2015, the Performance Management (PM) department was created with a focus of helping programs determine goals, reporting on progress, and facilitate learning. While there was excitement about how data could help fuel the organization’s mission, there was some hesitancy. How do you help staff view PM as valuable partners in achieving the mission instead of officers policing their work?

Challenge 1: Overcoming the “police” image

Our front-line staff are helping guests experiencing homelessness overcome crises like getting medical care, providing a healthy but delicious meal, and helping them move into a safe, permanent home. Acknowledging the critical work and true impact program staff have on people’s live, goes a long way. Clearly communicate your role is to help goalsetting and reporting truth through data so they can refine their work processes.

Our PM team invests in building strong relationships with all staff. One way we do this is by rolling up our sleeves to help. Once a month we volunteer four hours in another department on whatever they ask. Not only is this helping them but we get insight into programs and understand the demands on others when we ask for data, time, and feedback.

Lesson Learned 1: Create true partnerships and build strong relationships with all staff. Communicate everyone has a piece of the puzzle that is your mission.

Challenge 2: Creating one report for staff to use and communicate progress externally

Some people might be able to look at numbers in a spreadsheet and see trends but creating a visually appealing report will speak to more people. Balancing the needs of program staff and communicating to external stakeholders will be hard. However, having one report to show true progress of the whole organization will be impactful. It took us until 2017 before we had that balance.

Our first report was an easy to digest infographic but it lacked context for external stakeholders. Without the context, the truth of the numbers could be missed. Our 2016 report went in detail on the context with a scorecard for our data. This was great to communicate to our stakeholders; however, we thought we could provide true insights for staff. Our 2017 reports provide insights, context, and digestible charts. Thanks to our grant from Measure4Change, our Quarter 2 report received praise from staff. This format has continued into 2018 and with streamlined monthly internal reports for each program.

2018 Internal Monthly reports

2018 Internal Monthly reports

Lesson Learned 2: Present staff with a visually appealing report with insights and context to show true impact. Just start, once you have a report it is easy to get feedback.

Overall, for the non-profits out there thinking about deepening your evaluation practices, I would say Just start! Learn from the process, outcome, and your coworkers.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi my name is Jayne Corso and I am the Community Manager for AEA. 

Are you excited for Evaluation 2018? We are! I can’t wait to travel to Cleveland to kick-off #Eval18. Before you pack your bags and check into your flight, add one more item to your to-do list — download the Evaluation 2018 mobile app.

The mobile app houses the most up-to-date version of the Evaluation 2018 schedule and it’s a convenient way to meet other attendees, build your agenda, and figure out which room you are supposed to be in. We have made some improvements to the app this year, so I have compiled a few tips to help best utilize the mobile app.

Hot Tip: Build Your Profile

You can access the app by 1) accepting your invitation, 2) searching “AEA” Events in your app store, or 3) use the desktop version. No matter how you decide to access the app, you must create an account by using your registration email address and creating a password. Once this is complete, start building your profile. Here you can add a photo, your contact information, and most importantly, your interests. Choose from our TIG topics to indicate the evaluation topics you want to learn more about or that you are heavily involved with.

Why do this? The app will recommend other attendees to connect with based on your interests and profile information. Now you have instant friends at the conference!

Hot Tip: Search the Schedule

The AEA schedule can be very overwhelming, due to its size. But, the mobile app makes the schedule easier to navigate. You can search the schedule by keyword, timeslot, session type, or session topic. These search functions, truly are a time saver and makes it easier to build your agenda.

Hot Tip: Build Your Agenda Before Arriving in Cleveland

I recommend picking out the sessions you are interested in attending prior to arriving to the conference. You always think you have more time at conferences than you actually do. Doing more pre-planning can help you relax while in Cleveland and not worry about filling your schedule. With the app, you can also add multiple sessions to your agenda for the same timeslot.

Why do this? Adding multiple sessions for each timeslot gives you options! If a session gets cancelled or the topic is not relevant, then you can easily make adjustments and attend another session. That’s the great thing about the AEA conference, there is always something going on and more to learn.

Also, your agenda will jump to the current date/time (in the timezone of the event) so you see what’s up next on your personal agenda.

Hot Tip: Do your Pre-Planning on the Desktop Version of the App

One of the reasons I really like our app, is that it has a desktop version. This is really convenient when planning your agenda. Use the desktop version to read sessions descriptions, learn more about speakers, and/or take notes. The app and the desktop version are automatically synced, so when you are onsite, you can use the app seamlessly. You must be logged into the app to use the agenda building feature on both the desktop and the mobile app.

I hope these tips prove to be helpful! I will be in Cleveland near the registration desk to provide any additional support. I look forward to seeing you at Evaluation 2018.

To learn more about the app, visit the Evaluation 2018 website.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hey! I’m Sara Vaca, independent evaluator and part of the AEA365 team, eventual Saturday contributor. Some time ago, I ran into this framework for critical thinking and I thought it was interesting for evaluators.

Rad Resource: The Six Thinking Hats is a system designed by Edward de Bono which describes a tool for group discussion and individual thinking involving six colored hats. “Six Thinking Hats” and the associated idea parallel thinking provide a means for groups to plan thinking processes in a detailed and cohesive way, and in doing so to think together more effectively. 

The premise of the method is that the human brain thinks in a number of distinct ways, which is also the case for evaluators when conducting an evaluation. Six distinct directions in which the brain can be challenged are identified and assigned a color. In each of these directions the brain will identify and bring into conscious thought certain aspects of issues being considered (e.g. gut instinct, pessimistic judgement, neutral facts). None of these directions is a completely natural way of thinking, but rather how some of us already represent the results of our thinking.

The six directions, that in some cases make us think of some paradigms, are:

  • Information White – considering purely what information is available, what are the facts? Very relevant to any evaluator, but probably resonating strongly with positivist paradigm.
  • Emotions Red – intuitive or instinctive gut reactions or statements of emotional feeling. Clearly something that can emerge more easily with some qualitative methods like life stories.
  • Discernment Black – logic applied to identifying reasons to be cautious and conservative. Practical, realistic.
  • Optimistic response Yellow – logic applied to identifying benefits, seeking harmony. Sees the brighter, sunny side of situations, which resonate with Appreciative Inquiry approach or method or designs.
  • Creativity Green – statements of provocation and investigation, seeing where a thought goes. Thinks creatively, outside the box.
  • Managing Blue – what is the subject? what are we thinking about? what is the goal? Can look at the big picture.

 

https://www.storyboardthat.com/storyboards/nathanael-okhuysen/the-six-thinking-hats

Cool Tip: Ask yourself these questions: Which hats you usually wear? And which is the prenominant one? Thinking of this approach could maybe deliberately challenge you to go out of your “natural” tendency, and hence planned for use in a structured way allowing one to develop tactics for thinking about particular issues in your evaluation practice.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Older posts >>

Archives

To top