AEA365 | A Tip-a-Day by and for Evaluators

We are Leah Neubauer, Dominica McBride and Anthony Heard of the Chicagoland Evaluation Association. Welcome to the Local Arrangements Working Group (LAWG) week. We look forward to Evaluation 2015 in the Windy City.

In the Exemplary Evaluation conference year, our opening Chicago week offers issues relevant to our evaluation practice, environment, and policy. Today’s post notes 2015 Chicago happenings and themes for this week address Chicago people, politics, culturally responsive assessment, and evaluator learning. As CEA President and LAWG co-chair, I’m excited for our posts.

Hot Tip: What’s Important to Know about Chicago?

This is a great question with unlimited, potentially conflicting answers. Many might go on about sports, politics, corruption, weather, or traffic. Chicago celebrates and critiques. For many of us urban-entrenched evaluators, our Chicago-evaluation landscape demands a knowledge base that: 1) respects and challenges history, 2) keeps timely with rapidly changing, current/controversial events, and 3) ensures just implications for our city.

One way to get “Chicago-ready” is to digest some news. A short highlight reel is shared below. This is not an exhaustive or representative 2015 list. It’s designed to be both serious and light-hearted. Ultimately, we hope that you find it aligned with the celebratory and critical nature of Chicago updates.

Hot Tip: What to Visit When You Come to Chicago.

In addition to remaining Chicago news-updated, there are many other Chicago things to plan for. Check out events and museums . Be on the AEA lookout for the local guide!

We’re looking forward to November and the Evaluation 2015 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

·

Hello! I’m Sheila B. Robinson, aea365’s Lead Curator and sometimes Saturday contributor. It’s Independence Day in the US, the anniversary of the day our Declaration of Independence from Great Britain was signed in 1776. As I thought about what topic to tackle for this week’s post, I wondered what insights a word cloud created from the text of the Declaration of Independence might yield. Since I’m familiar with two word cloud generators, I tried each of them.

Here is what I created in Worldle:

Dec of Ind wordle

And here is what I created in Tagxedo:

Dec of Ind Tagxedo

Both generators allow for some creativity with regard to color, and font. Wordle allows you to choose directionality – whether you want all text horizontal, all vertical, or some mix of the two. Tagxedo is best known for its shapes, allowing you to choose from a bank of available shapes that can help you illustrate a point. Tagxedo can also use custom fonts from popular font sites.

Lesson Learned: Word clouds can be fun and have a number of appropriate applications. Use them with caution, however, as they never substitute for analysis, and can potentially detract from important themes. Stuart Henderson and Eden Segal tackle word clouds in a chapter on qualitative data visualization in a recent issue of New Directions for Evaluation.  “Word clouds create a dramatic visual, which likely accounts for their popularity,” the authors posit. “Despite concerns with word clouds, their ease of creation and striking visuals make them a useful tool for evaluators if they are used sparingly and their challenges are acknowledged” (p. 57-58). 

Hot Tip: Henderson and Segal also mention another word cloud generator, TagCrowd, and I decided to give that one a try as well. Here is the result:

Dec ind TagCrowd

While TagCrowd has fewer opportunities for creativity, it does give the user the option to see word counts after each word. I simply checked this option as I created this one.

Rad Resources: A few other aea365 authors – Susan Kistler, Stacy Carruth, Sue Griffey, Jaquelyn Christensen, and Sarajoy Pond – have also shared their perspectives on word cloud generators.

Did you know there is an advanced version of wordle? Learn about this and other Wordle tips, such as how to keep word phrases intact, and how to get numbers to show in Wordle here!

Read the entire article Visualizing qualitative data in evaluation research, by Stuart Henderson and Eden Segal, to understand more about qualitative data visualization.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.


· ·

Hello, I am Matt Feldmann, the principal researcher and owner of Goshen Education Consulting, Inc. and the chair-elect for the Independent Consulting TIG. My company focuses on educational evaluation and data support for clients in Southern Illinois. As an independent consultant, it is imperative to maintain a strong network of clients, colleagues, and advocates with whom you can grow and develop your business. The following are some of the “Soft skills” associated with working with clients that are frequently referenced in business literature.

Lessons Learned:

Form a business strategy and stick to it. Marriage of Skill, Enjoyment, and Resources. Jim Collins refers to this as the “flywheel concept” in his book, “Good to Great”. After you have developed a well-considered business plan there is an additive effect to unyielding discipline to that plan. Learn more about the flywheel concept from Jim Collins here.

Recognize you are in a service industry and focus on your client relationships. Harry Beckwith refocuses attention on the service industry in Selling the Invisible. The key point is that evaluation is not a commodity. From our client’s perspectives, our expertise with complicated evaluation approaches is secondary to our ability to communicate and relate the importance of our work. Jeffrey Gitomer says that you are more likely to receive your next consulting contract from an existing client in his book Little Red Book on Selling.

Practice networking Karma. Recognize that your success is connected to the success of your network of clients, colleagues, and advocates. Your selfless work for others will return to you in unexpected ways. Keith Farazzi says you should not keep score with your networking relationships in his book Never Eat Alone; business development is not a zero sum game. Because we evaluation continues to be in a growth mode, evaluators should reach out to their competitors to learn from one another and to seek ways to develop cooperatively.

Rad Resources:

The following are the author websites for the four books referenced above and that provide excellent understanding for these and several more “soft skills.” You can probably find these on audio CD, mp3, or download from your library and listen to them as you travel among your clients.

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi there! I am Ana García-Femenía, (@anagfemenia), PhD in Sociology and Political Sciences and independent evaluation consultant. I live in Spain and conduct evaluations for various international donors in different countries.

Have you ever been in the middle of an interview in an evaluation and felt some specific emotion with no reason for it? What happens when you stop and ask yourself: Why do I feel this way?

Over the years I have understood how important it is to take care of my emotional well-being, in order to do my job as evaluator. But one could say this holds true for any profession, doesn’t it? Well, yes it does. But it holds especially true for our profession where we as evaluators are confronted with constant decision-making requiring sound judgement and a special degree of calm, cool and collectedness. There is a list of situations and times when we need to exercise our best judgement not only during the evaluation process itself, but with clients and stakeholders in order to make the right decisions, to say ‘no’ when we need to, and to maintain our integrity.

There are no purely rational decisions. Whether negative or positive, our emotions always let us know when we are or not on the right track.

Lesson Learned: Make emotional care part of your self-care.

Two Hot Tips to increase your emotional care:

Meditating: for 15 minutes once a day. Meditation provides peace and serenity and opens your mind and your heart. It helps to improve your relationship with your emotions. It also provides a good preparation for full attention and deep listening.

Dancing: A lot of stress and tension are relieved through dancing. Dancing helps to free our emotions. And it is a great way to socialize, which is very valuable for those of us who work alone!

Rad Resources: A wonderful tool for the initiated in meditation and the uninitiated alike. It provides guided meditation ranging from 5 minutes to one hour.

Online resources for dancing are huge. It also will depend on the music you like!

Here are three links on the benefits of dancing for your emotional health:

And finally, for those interested, a beautiful video by one of the best world tango couples to get inspired!

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Gail Vallance Barrington and I’m an independent consultant. It’s hard to teach novice clients about logic models. This year, instead of presenting a technical monologue, I came up with a story to capture their attention. It’s called My Rich Uncle and it goes like this:

Icebreaker: You suddenly discover that you have a rich and eccentric uncle who lives far away and he wants to give you $30,000 to buy a car. Working in small groups, decide what your uncle’s name is and where he lives. To get the money, you will need to prove to him why you need the car. Use this worksheet and have fun coming up with the answers. [N.B. Do not provide the labels below until the exercise is over.]

Activity:

  1. List three compelling reasons why you need the car and the difference it will make in your life. [Program Purpose, Long-term Outcomes]
  2. Your uncle wants to know that you are a good planner. What will you need to have in place before you get your car? List three or four important things you need. [Inputs]
  3. He wants to know how you will use your car on a regular basis. What will your main uses be? [Activities]
  4. Your uncle is pretty picky, and of course, he lives very far away. He wants to be sure that you actually spent the money on a car and that you actually use it. Every six months, he wants some concrete proof that you have been using it as planned. What evidence can you put in a brown paper envelope and mail to him? [Outputs]
  5. Finally, after three years, you suddenly hear from your uncle. He is coming for a visit! He wants you to show him what changes the car has made to your life. Can you start planning for it now? [Short- and Intermediate-Outcomes]

Reflection: Once the groups have reported their answers, reflect on such topics as the group interaction, breakthrough thinking, general conclusions and the fact that everyone has just developed a logic model.

Next Step: Launch immediately into developing the logic model for their program. Having set the stage in this relaxed and fun way, it is easy to make connections with their own context.

Lessons Learned: I have used this exercise with novice clients, students and webinar participants. Every time the typical problem of differentiating between Outputs and Outcomes seems to melt away. Further, the accessible nature of the story provides clarity about a funder’s need for accountability.

Next time you teach logic models, why don’t you try it? Let me know how it goes.

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Susan Wolfe and I am a Senior Consultant at CNM Connect where I provide evaluation and other capacity building services to non-profit organizations. Two months ago I was the owner of Susan Wolfe and Associates, LLC, my own consulting firm; however, after five successful years I chose to close my doors and become an employee.

People have asked why I made this choice to give up my independence. The main reason is that although I LOVED the work I was doing, I did not so much care for the business side of independent consulting. When I was offered the opportunity to join the non-profit CNM Connect as a consultant I easily accepted it without once looking back.

Lesson Learned: Even if I am successful with something, if I don’t enjoy it then it’s OK to make a change when an opportunity that better fits my interests and needs. When I started as an independent consultant it fit well with my interests and needs, but things changed over time. When I was no longer enjoying my work, I allowed myself to be open to other possibilities, and because of this a more suitable situation presented itself.

Throughout the past five years, I had a love-hate relationship with working alone at home. I liked the convenience of not commuting and having to dress up for work every day. I also liked being able to work with one of my dogs in my lap. On the other hand, after some time I started to feel the absence of co-workers. I missed belonging to an organization with other people but I knew I did not want to grow my consulting business.

Hot Tip: If you like having colleagues that you work with daily while at the same time would like to be an independent consultant, consider forming a company or collective. Working at home alone can be isolating.

Hot Tip: Networking with other evaluators, locally, statewide, and nationally, and teaming up for collaborative proposals and projects is one way to reduce your sense of isolation. I regularly scheduled lunch with colleagues or community partners, and ultimately teamed up with another local evaluator to form an informal local evaluator’s group that gathers bi-monthly for happy hours.

Rad Resource: If you are thinking of transitioning from independent consulting to working for an organization, scan the AEA job listings to see what is available. It’s a great way to look around and see if there is something that would better fit your current needs and interests.

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Bernadette Wright, founder of Meaningful Evidence, LLC. As an entrepreneur, I noticed a proliferation of books, papers, courses, and other sources espousing conflicting ideas of how to succeed in business and entrepreneurship. For evaluators who are building independent consulting businesses, this makes finding the best ideas for our business difficult.

Wright

In my and Dr. Steven E. Wallis’s recent research on entrepreneurship, we found a frequently mentioned problem was a lack of agreement on what entrepreneurship means. Some sources portray entrepreneur as those who start ventures, generally small businesses. Others present entrepreneurship as about being innovative, pursuing opportunity, or filling a need.

Several studies discussed the need to integrate theories of entrepreneurship across disciplines and across levels of analysis (e.g. individual, organizational, and market levels). Studies have also recommended coordinating entrepreneurship theories across research, education, and practice. A 2006 U.S. Association for Small Business and Entrepreneurship (USASBE) white paper titled Are We Teaching Small Business Management to Entrepreneurs and Entrepreneurship to Small Business Managers? recommended making teaching more consistent with the literature. However, the fragmentation of theories makes choosing the best theories to teach and apply difficult.

Integrative Propositional Analysis (IPA) is an emerging method to analyze and integrate theories across studies, strategic plans, and other documents. IPA draws on a long research stream related to systems thinking. This scholarship indicates we can improve a model by understanding its structure and making it more inter-connected. We can use IPA combined with empirical data and feedback from stakeholders to assess plans before we implement them.

IPA is a six-step process. For details, see our white paper on “The Science of Conceptual Systems.” Benefits for business leaders include:

  • Lets you choose theories that are more likely to work in the real world (as compared to in the academic world)
  • Gives you more “structured” knowledge, which research indicates increases managers’ chances of success
  • Supports communication, collaboration, and shared understanding
  • Avoids unanticipated consequences

Rad Resource: “A Revolutionary Method to Advance Entrepreneurship Theories” by myself and Steven Wallis. In review, draft available upon request. Discusses our study on using IPA to integrate and improve entrepreneurship theories.

Rad Resource: Casson, M. Entrepreneurship and the theory of the firm. Journal of Economic Behavior & Organization, 2005. This study creates an integrated theory combining insights from management and economics.

Rad Resource: “Strategic Knowledge Mapping: The Co-Creation of Useful Knowledge,” by Steven Wallis and myself. Developments in Business Simulation and Experiential Learning, 2015. Introduces a gamified version of IPA.

Where do you find business/entrepreneurship information to consider for your business? How might your firm benefit from using theories that are measurably better than your competitors’?

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, I’m Stephen Maack, Owner and Lead Consultant of REAP Change Consultants, founded October, 2002. The Fourth of July is this week. Many become independent evaluation consultants because of the lure of freedom – freedom to choose projects, what you do, how you do it; freedom from the politics of organizations, bad bosses, and organizational hierarchies; freedom to set your own hours and work in your pajamas. It all sounds great.

It’s all true, too – partly. Yes, you can choose which projects to take on but as Max A. Eggert and Elaine van der Zeil put it “Clients tend to be more rigorous and constraining than managers,” and “Every client wants the right solution to his or her need without taking exceptional risks. Thus you have to have controlled creativity….”[1] Yes, work in your pajamas – but you also need to meet deadlines and juggle time for yourself, your family, loved ones, and friends. Try shutting down your computer and walking away at 5 p.m. when the final report is due tomorrow and you must get it done now because you spent time with your spouse earlier! Yes, you might be free of your own organizational politics and can walk away from those of other organizations once you finish a project. However, for good client relations, pay attention to the politics of your clients’ organizations!

So what to do? Take hints from young America. Once the American Revolution resulted in independence, leaders of the young nation set about forming thirteen disparate states into the United States of America and created a new constitution. As you go independent you’ll figure out your own rules for operation, what you will do and how you’ll go about doing it. The young nation worked at developing allies and establishing markets for its goods and services. You’ll have to do that. Find business partners, learn from fellow consultants, explore and start marketing. Each of the states had others with whom to work. You do, too – AEA and the IC TIG.

Go ahead. Declare independence. Just remember that bravery, responsibility, and hard work come with freedom. Like each of those who financed the revolution and founded this country, you are putting your fortune, fame and the welfare of your family on the line. Good luck!

Hot Tips: Join the Independent Consulting TIG for community and advice. Take AEA training workshops.

Rad Resources: Gail Barrington’s Consulting Start-up and Management: A Guide for Evaluators and Applied Researchers.

Elaine Biech’a The Business of Consulting: The Basics and Beyond. Second Edition.

[1] Max A. Eggert and Elaine van der Zeil (2011) The Perfect Consultant: All You Need to Get It Right First Time. The Perfect Series. Arrow Business Books, Random House Kindle edition.

 

The American Evaluation Association is celebrating IC TIG Week with our colleagues in the Independent Consulting Topical Interest Group. The contributions all this week to aea365 come from our IC TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, my name is Jayne Corso and I am the community manager for American Evaluation Association and the voice behind the AEA Facebook page.

If you manage a company Facebook page, you might have noticed a drop off of “likes” recently. Facebook has begun removing memorialized and voluntarily deactivated accounts from Pages’ like counts. This change ensures that data on Facebook is consistent and up-to-date—but could mean a drop for your analytics. Although some Pages might lose “likes,” they could also gain a more accurate way to track their followers. I have compiled a few tips for tracking your analytics and gaining more visibility for your page.

Rad Resource: Take advantage of Facebook “Insights”

Facebook offers Page Insights after at least 30 people have liked your Page. Use this tool to understand how people are engaging with your Page. With this tool, you can see your Page’s growth, learn which posts have the most engagement, find demographic information about your audience, and identify when your audience is using Facebook.  This data is available for free and can easily be customizable for time frame and downloaded to excel.

Rad Resource:  Use Google Analytics to track effectiveness

Tracking your analytics through Google allows you to see how many people are coming to your site from social networks, understand the website pages they are most interested in, and gain a better understanding for how your audience is engaging with your web content.  To find this information, enter your Google analytics account and go to “Acquisitions”. From here you can look at the performance of your social networks as an overview or look more specifically at referrals, activity, and user flow. All of this data allows you to gage the effectiveness of your social campaigns.

Hot Tips: Increase your Facebook likes

Finally here are a few simple tips for increasing the likes on your Facebook Page—hopefully you can make up for any followers you lost when Facebook made their changes.

  • Add the Facebook icon to your website, so visitors know you have a presence on the social network (Place the icon high on the website page, near your navigation)
  • Add the Facebook icon to your email communication or blog to reiterate your presence on Facebook to your subscribers
  • Cross promote your Facebook page on your other social media sites. You may have followers on Twitter that have not liked your Facebook page or didn’t know you had a Page

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Hello.  My name is Sally Thigpen and I work in the National Center for Injury Prevention and Control (NCIPC) at the Centers for Disease Control and Prevention (CDC).  As an evaluator, I often find myself encouraging scientists and stakeholders to think about evaluation from the very beginning of any study or project.  I do whatever I can to be included as early as possible so I can help build evaluation into every step of the process.  The value of inclusion is equally true for our practice partners.  They need to be included as early as possible in our scientific thinking because they are vital to the translation of research to practice.  Practitioners speak to the relevance and utility of the science and the value it has to current programmatic or policy efforts.  In today’s budgetary realities, understanding these practical aspects of uptake helps assure limited dollars have the maximum impact.

The Division of Violence Prevention within NCIPC developed the Rapid Synthesis and Translation Process (RSTP) to systematize this communication loop between the research and the field of practice. This six-step process (in the graphic below) can help users facilitate the negotiation between the science and practical application.

Thigpen

Hot Tips:

  • Before engaging with a group of practitioners, do a gut check with the scientists of record.  Ask questions about what they see as the most valuable aspect of the study for practical application.  What is their biggest apprehension about how the science might be misinterpreted and used in ways it was not intended?  These answers not only help to focus the translation efforts, but also offer a little insight as you begin working with a selected group of practitioners.
  • Work with the same group of practitioners from the beginning to the end of the translation process.  Begin this relationship with similar questions from above.  How do they anticipate using the science?  What is the significant contribution to the field?  What is least valuable?
  • As the translational product moves through development, keep checking in with the group of practitioners and scientists.  Practitioners can guide you on relevance and balancing science with action.  The scientists can guide you in making sure you are keeping scientific integrity along the way.

Lesson Learned: You’re not just a communicator/evaluator/researcher – you’re a negotiator.  In the role of translator, you are often negotiating between the details of pure science and the brevity of the practical world.  This is a critical role and takes finesse.

Rad Resource:  My colleagues and I published an article in a July 2012 special issue of the American Journal of Community Psychology reviewing RSTP’s usefulness in the field, Moving Knowledge into Action: Developing the Rapid Synthesis and Translation Process Within the Interactive Systems Framework.

he American Evaluation Association is celebrating Translational Research Evaluation (TRE) TIG week. All posts this week are contributed by members of the TRE Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Older posts >>

Archives

To top