AEA365 | A Tip-a-Day by and for Evaluators

CAT | Uncategorized

Hello! I’m Deven Wisner of Wisner Analytics, LLC, V.P. of Communications for the Arizona Evaluation Network (AZENet), and avid Tweeter. I’d like to share with you some Hot Tips and a Rad Resource for bringing Twitter into your AEA Local Area Collaborative (or any other organization!) – courtesy of myself, Jenny McCollough Cosgrove, and Nicole Huggett.

When I joined the AZENet board mid-2017, one focus was bolstering our presence in Arizona. As someone who had greatly benefited from connecting with fellow evaluators on Twitter, I thought that would be a great place to start! Of course, that’s easier said than done… because although social media has great potential for furthering an organization’s reach, it takes a significant time commitment. So, how did we manage?

First, we developed a plan, which included a minimum number of posts per day (we started with two) and following eval-focused individuals every day (our goal was 10). After noticing an increase in followers and engagement, we made a more formal commitment to Twitter.

At that point, we addressed sustainability (that time piece I mentioned earlier!) because let’s be honest – it’s one thing to have a great idea but another to keep it going! So, we decided to make social media a team effort. That’s when we had the idea to implement posting guidelines, create a schedule, and start using Hootsuite. A combination of these things helped us manage our “voice” across multiple users, avoid burnout, and make scheduling ahead during your “week” more manageable.

Hot Tips:

  1. Establish post guidelines – If you are going to have several individuals posting, you want to be sure everyone has an idea of what is acceptable for the organization and what isn’t (e.g., avoiding politics, promoting organizations, etc.). However, we encourage a wide variety of disciplines to be covered because we want our membership to be well-represented!
  2. Create a schedule – Social media burnout is real. Scrolling through Facebook at night before bed is different from having daily minimums and sharing quality content. Instead, share the work and let different perspectives shine! We created a schedule in OneNote that our board members sign up for. Depending on what events are going on, we include special things that need to be posted in addition to ad-hoc content.

Rad Resource:

Hootsuite – Just do it. There is a free version that allows you to post across platforms. Although that isn’t the purpose of this post, organizations usually need to be active on more than just Twitter. Hootsuite will allow you to schedule ahead, follow hashtags, and even require that posts be approved prior to releasing them.

image of Hootsuite

Have questions about incorporating Twitter into your AEA LAC? Let me know!

The American Evaluation Association is celebrating #EvalTwitter week. All posts this week are contributed by evaluators engaging, networking, and collaborating through Twitter. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hey there, I’m Echo Rivera, owner of Creative Research Communications, LLC and advisory team member of AEA’s Potent Presentations Initiative (p2i).

I love Twitter, and have been tweeting since 2012. I’m here to help your tweets stand out from the crowd. And by “crowd,” I mean the estimated 500 million tweets sent per day. The easiest way to do that? Add a great visual.

Hot Tip: The easiest way to do this is to add a gif to your tweet. Gifs are magical little visuals because they hit the sweet spot of (a) being a visual, (b) having movement, and (c) showing an emotion.

When you want to tweet about something you created (e.g., report, blog post), then I recommend you create your own visual. Spend some time creating your own design style or brand, so you establish consistency across your images. Create a template so all you have to do is change the wording and/or a photo.

Case Study: Here is the twitter template I use for my blog posts. I created a canvas that follows the Twitter dimensions (which change over time, so I just Google it first). I use fonts in a similar way, place the same type of info (e.g., title, photo attribution) in the same place, and alternate between my 4 branding colors (which are transparent shapes so the text is easy to read). I use photos of real people, and I try to match the emotion to the title as much as I can.

Rad Resource: You probably already have the software you need to make visuals: PowerPoint or Keynote! The above visuals were made using Adobe Illustrator, and you could use something just as specialized. But I learned how to make visuals using shapes in Keynote. Simply (1) choose a 16:9 slide size (or custom size, if you want); (2) use shapes (or icons you find online) and multiple text boxes to create the design you want; and (3) export that slide as an image. That’s how I made this and it took me about 2 minutes (and if you want this slide to copy, see below!):

The slide dimensions don’t adhere to the Twitter specifications exactly, but because it’s widescreen it still fits well. Here is what it looked like when I tweeted it. Which of the following tweets do you think was the most eye catching (hint: it’s this tweet)?

Rad Resources:

The American Evaluation Association is celebrating #EvalTwitter week. All posts this week are contributed by evaluators engaging, networking, and collaborating through Twitter. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi, I’m Libby Smith from ARC Evaluation. I’m going to talk about using Twitter to better understand the field of evaluation. For the past four or five years I have encouraged our evaluation students to use Twitter, but it’s only been in the last six months that I have really taken my own advice!

I used to worry about two things: 1) adding another social media platform to my list of things to do and 2) sending tweets out into the world! What did I have to say that other people would care about? When one of my former students turned independent consultant and dataviz influencer, Deven Wisner, started tweeting up a storm, I knew I couldn’t sit on the sidelines anymore.

I have long seen the potential in using Twitter to enhance understanding and to build connection to our professional community. As practitioners, it can be challenging to stay up to date on the published literature and the trends in our field. Following the leaders in our field and even your peers will provide you access to all sorts of hot tips, cool tricks, and rad resources.

Hot Tip: Developing a well curated list of accounts to follow is key. A good place to start is the Evaluators list compiled by Sheila B. Robinson. You can either follow that feed or use that list to start building your own. You can’t follow too many people, but keep your list focused on your professional interests. Social media can turn into a time suck, so make sure your feed is filled with valuable information that will help you grow as a professional. You will quickly realize the size of the field and the scope of the impact that evaluators are making in the real world.

evaluators on twitter list & number of subscribers

Rad Resources: So, now you are following all the cool evaluators and leaders in our field, learning about cutting edge data-viz techniques, improving your presentation skills, getting inspired to make a difference, and finding awesome evaluation jobs. Adding your voice to this mix can seem overwhelming! I recommend event-based tweeting as a great place to start. I started tweeting while attending the AEA annual conference, but you don’t have to wait for a big event. For example, you could tweet from the next professional development session you attend! I recommend including an interesting photo that adds context and meaning to your tweet. Joining the chorus of voices from evaluation will both improve your understanding of our field and help you connect with other evaluators from your specific evaluation niche!! I’m happy to give you more hot tips and cool tricks, you can tweet at me or send me a DM. Happy Tweeting!!!

The American Evaluation Association is celebrating #EvalTwitter week. All posts this week are contributed by evaluators engaging, networking, and collaborating through Twitter. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello! I’m Dana Wanzer and this week we’re discussing Twitter! You can find me on Twitter at @danawanzer.

Together with Libby Smith, Deven Wisner, John BurrettJenny McCullough Cosgrove, and Echo Rivera, we’re going to show the benefits of joining Twitter for evaluators and the field of evaluation. You may know of EvalTalk, the LinkedIn group, and many of the AEA TIGs have Facebook groups. But did you know there is a rich and vibrant community of evaluators who talk daily via Twitter?

Hot Tip: Participating in Twitter as an evaluator is great for professional development! Evaluators from around the world share evaluation reports, tips and tricks, and lessons learned. This has helped me stay up to date with the latest news and newest techniques that I can apply to my practices. You can stay connected in the conversation by following the hashtags #eval and #evaluation.

Cool Trick: The networking that happens on Twitter is amazing. Carolyn Camman and Elizabeth Grim have both gotten jobs through their connections on Twitter. Elizabeth followed a notable evaluator on Twitter for months, retweeting and favoriting each other’s content. “Later that year, when we were both at AEA, they sent me a Twitter message asking to meet with me. After those interactions, we ended up collaborating on a panel presentation and I was asked to do some independent consulting work. Without Twitter, I’m not sure we would have ever connected.”

Sheila Robinson even collaborated with Kim Firth Leonard for two years before meeting in person for the first time at the 2015 AEA Conference to present a skill building workshop on survey design!

image of tweet between evaluators

Lesson Learned: The evaluation community on Twitter is like no other. It extends the conversations past our yearly conferences, but it also opens doors for networking. Deven Wisner stated that because of Twitter, “for #eval17, it made connecting in-person SO much easier. Conversations were a lot more free flowing because the relationship was established before I even arrived at the conference.”

image of tweet from evaluator

Rad Resources: Various evaluators have written blog posts about how to get started on Twitter and other benefits they’ve experienced.

Stay tuned for upcoming AEA365 posts on using Twitter for networking and at conferences, as an evaluation methodology, for AEA local affiliates, and how to participate in our very first #evalTwitter chat!

The American Evaluation Association is celebrating #EvalTwitter week. All posts this week are contributed by evaluators engaging, networking, and collaborating through Twitter. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Greetings, AEA Members. We are Anne Vo (Chair; University of Southern California), Brandon Coffee-Borden (Community Science), Jennifer Greene (University of Illinois, Urbana-Champaign), Maurice Samuels (MacArthur Foundation), and Nicole Vicinanza (JBS International) — your 2018 Nominations Working Group. We have the pleasure of announcing the Call for AEA Board Nominations and encouraging you to consider getting involved in our beloved organization by serving on the AEA Board.

Opportunities to Serve:

The 2018 slate calls for nominations for President-elect and 3 Board Members-at-Large.

Time Commitment:

The President serves a three-year term in the roles of President-elect (2019), President (2020), and Past President/Secretary (2021). The President presides over all Board meetings.

Board Members-at-Large will serve three-year terms, starting January 1, 2019, and are expected to attend three Board meetings per year.

In addition to meeting attendance, each Board member and the President actively participate in the work of the Board, engage in ongoing email communications, and liaise with Board Task Forces, Working Groups, or other Board-focused volunteer groups.

Qualifications:

Particularly strong candidates for these leadership positions are individuals who have:

  • Been active members of AEA for a minimum of three years,
  • Served in leadership roles within AEA units such as TIGs, Working Groups, Task Forces, Local Affiliates, and others,
  • Contributed to and served the evaluation profession in other ways.

Note that candidates who would like to be considered for the post of President-elect must be AEA members who are based in the United States. Prior service on the Board is preferred, though not required. All candidates’ availability and accessibility are also considered.

Diversity:

AEA and the Leadership Team are committed to actively cultivating meaningful diversity in the governance of the association. Particular attention is paid to meaningful balance among individuals representing the following dimensions of diversity:

  • Gender (including nondiscrimination based on sexual representation and orientation or gender identity)
  • Racial/ethnic representation
  • Disciplinary heterogeneity
  • Practitioner/academic balance
  • Geographic heterogeneity (within the United States)
  • International representation and perspectives
  • Heterogeneity of areas of application

Learning More:

AEA relies on members to provide leadership, and values a large and diverse pool of member leaders! To learn more…

  • Access the Nominations Working Group’s Coffee Break Webinar for information about:
    • What is involved in running for and serving on the AEA Board of Directors
    • What the nominating committee looks for in potential nominees
    • How you can become involved in AEA leadership
    • Paths to serving on the AEA Board
  • Reach out to current Board Members to learn about their experiences.

Navigate to the AEA’s Resource Page to access materials that describe how to prepare a nomination packet for yourself or others. Contact us! We are also glad to help answer questions about the nomination process. If we do not have the answers, we can definitely offer suggestions for who to ask.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Christina Peterson. Shortly after starting my first semester as a PhD student at the University of Tennessee, I had an opportunity to participate in an evaluation project as a data collector. It was just the type of experience I felt I needed to establish myself as an emerging evaluator in my community. Instead, it ended up being a lesson on establishing my ethical boundaries. Fortunately, the faculty in my department had provided the Rad Resources I needed to navigate this situation early in my PhD career.

Rad Resources:

Researcher Journal. Reflexivity is a critical evaluator competency. During my first semester, we were required to start a reflexivity journal to explore our growth as evaluators. The first exercise was to write about our personal ethics in conducting evaluation. I found clarity in the key words from this statement: self-determination, transparency, and autonomy. I described how it was important to me that people were respected and had a voice in the evaluation protocol. Because of the journal, I was also able to reflect back on the two major themes that emerged from my entries about this situation: integrity and social justice.

AEA Guiding Principles. The concerns I saw emerging in my journal are addressed explicitly in the Guiding Principles of the American Evaluation Association. Two principles provided the direction I needed to make a decision about moving forward with the data collection: integrity/honesty and respect for people. One of my concerns about the project was that the poorly constructed survey items would provide misleading information about the population. Furthermore, since there was no clear purpose for the data collection and the survey protocol did not include informed consent, I was not confident that this work would maximize benefit and reduce unnecessary harm to the community.

Mentorship. Although I felt confident that I needed to let go of this opportunity, there was a lingering feeling of self-doubt. Who was I to question the survey protocol of a professor? What did I know about conducting field research? The AEA Guiding Principles provided direction, but is that how evaluation really works in practice? For the answer, I turned to a faculty member who I knew had talked openly in class about making similar ethical choices as a novice researcher. She reassured me that this protocol was not business as usual in field research and we discussed the courses of action I could consider.

Later that evening, I notified the lead researcher that I could not continue my participation in the data collection and, in line with the AEA Guiding Principles and my personal ethics, I was transparent about my concerns. These Rad Resources were essential to my development as an ethical evaluator.

reflective pool of water in brick pavers

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Cynthia Williams

Cynthia Williams

Hi, I’m Cynthia Williams, copy editor and independent contractor with Style Sheets Editorial Services. I help all kinds of experts—academics, consultants, book authors, nonprofit communicators, and advocates—refine their prose so that they speak clearly to their audience. Everyone (including blog-writing copy editors) needs an editor.

Below are a few of the ways copy editors’ work goes beyond helping with punctuation or correct grammar.

  1. Consistency: Did you say 34% of all grantees in the executive summary and 4% in section 2? Does your introduction mention coverage of program participant suggestions for improvement, but your Recommendations section writer forgot to include that? Are your subheads and sub-subheads uniform? Copy editors can make sure information in reports and proposals is consistent in content and formatting.
  2. Accessibility: Your editor can help you sense when a report begins to resemble alphabet soup (too many acronyms) or when turns of phrase may be too foreign and jargony. A sentence can be grammatically correct and still not be the best way to present information to your audience. Help in this area can come in handy with grant-proposal writing, when the audience may not be familiar with your organization and you need to describe your mission, methods, and plan as simply as possible.
  3. Cohesion: “Voice” isn’t a term just for fiction writers. When a three-person team drafts a report, and then gets feedback from the team lead, separately written sections can seem disjointed. A copy editor can help smooth these out and help your report or proposal reflect the tone of the organization.
  4. Style manual adherence: Researchers can’t be expected to keep up with the latest changes to The Chicago Manual of Style or the Publication Manual of the American Psychological Association. That’s why editors pore over these tomes and keep track of rules on capitalization, citation and reference formats, and other topics exciting only to us.

Too, your organization may have a house style, and your editor can ensure that your document is formatted like your organization’s other reports and that it follows special phrasing, capitalization, spellings, etc.

If your organization doesn’t have a style guide but would like to create guidelines for its writers and researchers, an editor can help you create one!

Hot Tip:

In a sentence packed with too many elements (e.g., a “which” clause, parentheses and/or dashes, and multiple phrases beginning with “of,” “from,” or “to”), your excellent point may get lost in the clutter.

The solution might be to break up that sentence into two or three separate sentences.   

Rad Resources:

Need an extra pair of eyes? graphic

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I am Carla Hillerns from the Office of Survey Research at the University of Massachusetts Medical School’s Center for Health Policy and Research. In 2015, my colleague and I shared a post about how to avoid using double-barreled survey questions. Today I’d like to tackle another pesky survey design problem – leading questions. Just as we don’t want lawyers asking leading questions during a direct examination of a witness, it’s also important to avoid leading survey respondents.

By definition, a leading question “guides” the respondent towards a particular answer. Poorly designed questions can create bias since they may generate answers that do not reflect the respondent’s true perspective. Using neutral phrasing helps uncover accurate information. Here are a few examples of leading questions as well as more neutral alternatives.

Leading questions alternative wording, reason for change table

Hot Tips for Avoiding Leading Questions:

  1. Before deciding to create a survey, ask yourself what you (or the survey sponsor) are trying to accomplish through the research. Are you hoping that respondents will answer a certain way, which will support a particular argument or decision? Exploring the underlying goals of the survey may help you expose potential biases.
  2. Ask colleagues to review a working draft of the survey to identify leading questions. As noted above, you may be too close to the subject matter and introduce your opinions through the question wording. A colleague’s “fresh set of eyes” can be an effective way to tease out poorly phrased questions.
  3. Test the survey. Using cognitive interviews is another way to detect leading questions. This type of interview allows researchers to view the question from the perspective of the respondent (see this AEA365 post for more information).

Rad Resource: My go-to resource for tips on writing good questions continues to be Internet, phone, mail, and mixed-mode surveys: The tailored design method by Dillman, Smith & Christian.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello! We are Rebecca Reznik-Zellen and Lisa Palmer, librarians at the University of Massachusetts Medical School’s Lamar Soutter Library. Have you ever received a suspicious email from an open access publisher or journal that you are unfamiliar with? Have they invited you to publish with them for a low publication fee or promised fast peer-review and publication timeframes? If so, you may have been solicited by a so-called “predatory” publisher. We want to share some tips for evaluating journals and avoiding “predatory” publishers when you are ready to publish your evaluation research.

Lessons Learned:

In the wake of the Open Access movement, opportunistic publishers have emerged that charge publication fees without providing editorial and publishing services associated with legitimate journals. Predatory publishers exploit authors who may be inexperienced or who are under pressure to publish; in doing so, they corrupt the scientific record with low-quality science.

It’s important to remember that not all open access journals are predatory. In fact, legitimate open access journals (such as the Journal of Multidisciplinary Evaluation) conduct peer-review and follow established publishing standards. Some open access journals are very important and influential in their fields. Also, many open access journals don’t even charge authors article processing fees. So how do you tell the good from the bad?

Hot Tips:

Predatory publishers engage in questionable practices to solicit and process content, such as aggressively soliciting article submissions; promising rapid publication; eliminating or automating peer review; not following publication standards (such as COPE); not submitting content to major indexing and abstracting databases (such as MEDLINE or Scopus); not disclosing all fees; and misrepresenting editorial boards.

A 2017 study published in BMC Medicine by Shamseer, et al., identified 13 attributes that distinguish a predatory journal from a legitimate one, including:

  • Overly broad scope (includes biomedical and non-biomedical subjects)
  • Spelling and grammatical errors on website
  • Non-professional contact email address (e.g., @gmail.com or @yahoo.com)
  • Poor quality images
  • Homepage language targets authors
  • Questionable journal metrics, such as the Index Copernicus Value, are promoted as quality indicators
  • Accepts or requests manuscripts by email
  • Lacks information about manuscript handling
  • Promises rapid publication
  • Unusually low article processing charges (<$200), and special time-limited offers
  • No retraction policy
  • No information on whether or how content will be archived
  • Journal retains copyright or does not mention copyright

To make sure that you will be publishing with a legitimate journal, open or toll access, always evaluate the publication venue directly prior to submitting your manuscript.

Rad Resources:

These resources can help you distinguish an ethical publisher from an unethical one.

You may want to check in with a local librarian for other resources. Good luck with your publishing efforts.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I am Gene Shackman, manager of the Free Resources for Evaluation Methods site, and author of the Beginners Guide to Program Evaluation. In this blog entry I talk about online resources that can help people learn about program evaluation. Just to mention, I have not taken many of the classes listed below, so I can’t speak for their content or quality. If you do take any, please let me know.

Rad Resources:

There are a wide variety of free online classes and class material about evaluation and methods. One place to start is the Public Health Learning Network. The PHLN lists regional centers, which have public health classes, some of which are about evaluation.  For example the Northwest Center for Public Health Practice has classes on epidemiology and program evaluation. I think many of these classes offer certificates of completion. Another place to start is the Khan Academy, with classes on statistics and probability and other topics. I start with these because I have looked into their classes and they are generally good.

Some other websites link to classes elsewhere, for example, Edx, links to statistics classes from many universities all over the world. While the classes are free, for many classes it costs $49 to add a certificate.

One example of a university with free classes is Carnegie Mellon Open Learning Initiative, which offers several classes in statistics. You can sign on and keep track of your classes and scores. The University of Minnesota also has an online section. They have a class on biostatistics and one on epidemiology. The self-paced versions are free, or you can get continuing education credit for $10.

There are a number of other people or organizations that have online classes, some of which are free. Udacity has some free classes, some about data. Look near the bottom of their page. Bob Dick has an Action Research and Evaluation class. The Global Health Learning Center, partially funded by USAID has a section on monitoring and evaluation.

I keep a list of these and other classes on my site about evaluation resources.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

<< Latest posts

Older posts >>

Archives

To top