AEA365 | A Tip-a-Day by and for Evaluators

TAG | journals

My name is Christina Peterson. Shortly after starting my first semester as a PhD student at the University of Tennessee, I had an opportunity to participate in an evaluation project as a data collector. It was just the type of experience I felt I needed to establish myself as an emerging evaluator in my community. Instead, it ended up being a lesson on establishing my ethical boundaries. Fortunately, the faculty in my department had provided the Rad Resources I needed to navigate this situation early in my PhD career.

Rad Resources:

Researcher Journal. Reflexivity is a critical evaluator competency. During my first semester, we were required to start a reflexivity journal to explore our growth as evaluators. The first exercise was to write about our personal ethics in conducting evaluation. I found clarity in the key words from this statement: self-determination, transparency, and autonomy. I described how it was important to me that people were respected and had a voice in the evaluation protocol. Because of the journal, I was also able to reflect back on the two major themes that emerged from my entries about this situation: integrity and social justice.

AEA Guiding Principles. The concerns I saw emerging in my journal are addressed explicitly in the Guiding Principles of the American Evaluation Association. Two principles provided the direction I needed to make a decision about moving forward with the data collection: integrity/honesty and respect for people. One of my concerns about the project was that the poorly constructed survey items would provide misleading information about the population. Furthermore, since there was no clear purpose for the data collection and the survey protocol did not include informed consent, I was not confident that this work would maximize benefit and reduce unnecessary harm to the community.

Mentorship. Although I felt confident that I needed to let go of this opportunity, there was a lingering feeling of self-doubt. Who was I to question the survey protocol of a professor? What did I know about conducting field research? The AEA Guiding Principles provided direction, but is that how evaluation really works in practice? For the answer, I turned to a faculty member who I knew had talked openly in class about making similar ethical choices as a novice researcher. She reassured me that this protocol was not business as usual in field research and we discussed the courses of action I could consider.

Later that evening, I notified the lead researcher that I could not continue my participation in the data collection and, in line with the AEA Guiding Principles and my personal ethics, I was transparent about my concerns. These Rad Resources were essential to my development as an ethical evaluator.

reflective pool of water in brick pavers

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hello! We are Rebecca Reznik-Zellen and Lisa Palmer, librarians at the University of Massachusetts Medical School’s Lamar Soutter Library. Have you ever received a suspicious email from an open access publisher or journal that you are unfamiliar with? Have they invited you to publish with them for a low publication fee or promised fast peer-review and publication timeframes? If so, you may have been solicited by a so-called “predatory” publisher. We want to share some tips for evaluating journals and avoiding “predatory” publishers when you are ready to publish your evaluation research.

Lessons Learned:

In the wake of the Open Access movement, opportunistic publishers have emerged that charge publication fees without providing editorial and publishing services associated with legitimate journals. Predatory publishers exploit authors who may be inexperienced or who are under pressure to publish; in doing so, they corrupt the scientific record with low-quality science.

It’s important to remember that not all open access journals are predatory. In fact, legitimate open access journals (such as the Journal of Multidisciplinary Evaluation) conduct peer-review and follow established publishing standards. Some open access journals are very important and influential in their fields. Also, many open access journals don’t even charge authors article processing fees. So how do you tell the good from the bad?

Hot Tips:

Predatory publishers engage in questionable practices to solicit and process content, such as aggressively soliciting article submissions; promising rapid publication; eliminating or automating peer review; not following publication standards (such as COPE); not submitting content to major indexing and abstracting databases (such as MEDLINE or Scopus); not disclosing all fees; and misrepresenting editorial boards.

A 2017 study published in BMC Medicine by Shamseer, et al., identified 13 attributes that distinguish a predatory journal from a legitimate one, including:

  • Overly broad scope (includes biomedical and non-biomedical subjects)
  • Spelling and grammatical errors on website
  • Non-professional contact email address (e.g., @gmail.com or @yahoo.com)
  • Poor quality images
  • Homepage language targets authors
  • Questionable journal metrics, such as the Index Copernicus Value, are promoted as quality indicators
  • Accepts or requests manuscripts by email
  • Lacks information about manuscript handling
  • Promises rapid publication
  • Unusually low article processing charges (<$200), and special time-limited offers
  • No retraction policy
  • No information on whether or how content will be archived
  • Journal retains copyright or does not mention copyright

To make sure that you will be publishing with a legitimate journal, open or toll access, always evaluate the publication venue directly prior to submitting your manuscript.

Rad Resources:

These resources can help you distinguish an ethical publisher from an unethical one.

You may want to check in with a local librarian for other resources. Good luck with your publishing efforts.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, we’re Abhik Roy and Kristin A. Hobson, students and Doctoral Associates (we know what you’re thinking…wow…they must be rich) in the Interdisciplinary Ph.D. in Evaluation (IDPE) at Western Michigan University (WMU), and Dr. Chris L. S. Coryn, Professor of Evaluation, Measurement, and Research and Director of the IDPE (our boss…please tell him to pay us more). Recently, Abhik formulized a Scriven number and we wrote a paper on it entitled “What’s in a Scriven Number?”

Lesson Learned: What’s so important about a Scriven number? Since the article appeared, evaluators are asking each other “what’s your Scriven number?” Perhaps you’re new to the field of evaluation and have no idea what this means or the significance. Dr. Michael Scriven is widely considered the father of modern evaluation. His influence theoretically and in application within the field of evaluation has ben quite significant as his numerous manuscripts total over 400. In addition, Dr. Scriven is a past president of the American Educational Research Association and the American Evaluation Association. He is also an editor and co-founder of the Journal of MultiDisciplinary Evaluation.

Cool Trick: Determining your Scriven number. You may be asking, what’s a Scriven number? Well that’s what we’re here to explain. To put it simply, a Scriven number is a measure of collaborative distance, using both direct and indirect authorship, a person is from Dr. Scriven. Ok maybe that wasn’t so simple. Let’s try explaining this in a different way. A Scriven number is how far you, as an author of a published paper, are away from Dr. Scriven. In other words, Dr. Scriven has a Scriven number of zero, a person who has written a paper with Dr. Scriven has a Scriven number of one, a person who has written a paper with another person who wrote a paper with Dr. Scriven has a Scriven number of two, and so on. For example, using the paper Cook, Scriven, Coryn, and Evergreen (2010), Cook, Coryn, and Evergreen have a Scriven number of one. Now anyone who has published with Cook, Coryn, or Evergreen receives a Scriven number of two, unless the person has published with Dr. Scriven directly, then the person has a Scriven number of one. If a person has multiple Scriven numbers, his or her Scriven number is the lower number.

Rad Resources:

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Jan/11

1

Susan Kistler on Joining AEA

Happy New Year! My name is Susan Kistler and I am the Executive Director for the American Evaluation Association. Today I’m writing to the aea365 readers who are not members of AEA, to encourage you to join the association for 2011.

Hot Tip: Membership in AEA is only $80/year ($30 for students), and easily pays for itself. You can join online.

Why join?

Hot Tip – AEA members have access to four of the leading evaluation journals: Members receive hardcopy and electronic subscriptions to AEA’s own journals, The American Journal of Evaluation and New Directions for Evaluation, as well as electronic access to Evaluation Review and Evaluation and the Health Professions. You’ll receive not only the latest articles, but also 20+ years of archival content. AEA365 contributors regularly recommend journal readings (see 12/29, 12/26, 12/24 from the past week alone!).

Hot Tip – AEA members engage with the field’s thought leaders: For one week each month AEA hosts a leader in the field on its members-only Thought Leaders Forum. The discussant shares insights, explores issues, and answers your questions. You can lurk in the background and take it all in, or dive in and discuss. Coming soon? Michael Scriven in January and Melvin Hall in February.

Hot Tip – AEA members promote their business: Whether you are an independent consultant or part of a 100 person firm, AEA membership gives you a free listing of your evaluation-related services in AEA’s Find-an-Evaluator database. Directly accessible from the homepage, the FAE listings are among the most highly used sections of the AEA website.

Hot Tip – AEA members attend free coffee-break webinars: Each month, we host 3-4 20-minute webinars that demonstrate a tool or provide insight into a methodology or theory. Topics in the works for 2010 include one each on collaborative, participatory, and empowerment evaluation, an exploration of photo journaling, and tips for strengthening your evaluation reports. Membership also gives you access to the recorded archives!

Hot Tip – AEA members have a voice in their association: As an AEA member, you’ll regularly have opportunities to provide input to comments on legislation, to make your voice heard to the Board, and to take advantage of AEA volunteer opportunities.

Hot Tip – AEA members stay abreast of happenings in the field: Each month, AEA members receive an electronic calendar and newsletter. The newsletter’s Policywatch column keeps you up to date on issues in the US Federal government. The Tech Talk column helps you make the most of the online resources available for evaluators. Other articles provide insight into the people and events shaping the field, announce new books, and ensure that you know what is happening within your association.

Hot Tip – AEA members save money: AEA full members receive discounts on the AEA Annual Conference ($80 off), workshops ($50 off per full-day workshop), the AEA/CDC Summer Evaluation Training Institute ($50 off registration), and on books ordered directly from our publishing partners (20% off from SAGE, Jossey-Bass/Wiley, Guildford, and Lyceum).

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org.

· · ·

My name is Susan Kistler. I am AEA’s Executive Director, and I contribute each Saturday’s aea365 post. This week, I’d like to share information about online journals of potential interest to evaluators.

Rad Resource – Survey Practice Journal: On September 11, I wrote about resources from the  American Association of Public Opinion Research (AAPOR) for calculating sample size and response rates. This past week, Josue De La Rosa at the US Census Bureau sent me a link to an article on mobile device use for online surveys to add to AEA’s weekly Headlines and Resources list. The article came from Survey Practice, a free online peer-reviewed journal from AAPOR. I don’t know how I had missed this great resource, but they have been publishing online monthly since August of 2008 and cover a range of topics spanning the gamut of survey research.

Rad Resource – Journal of MultiDisciplinary Evaluation (JMDE): Published twice each year, JMDE is a free peer-reviewed online journal that publishes articles, news, and columns spanning the breadth and depth of the field. Recent articles include Moving to Genuine: Credible Cultural Competence and Stakeholder Believability (Evergreen and Cullen, 2010); What Do All These Numbers Mean? Data Visualization as an Innovative Methodology to Make Program Decisions (Porter, Koch and Henderson, 2010); and Integrating Evaluation and Agent-Based Modeling: Rationale and an Example for Adopting Evidence-Based Practices (Morell, Hilscher, Magura, and Ford, 2010).

Rad Resource – AEA Online Journal Access: As part of their membership, AEA members receive free online access through the AEA website to current and archival content from four journals – AEA’s own American Journal of Evaluation (AJE) and New Directions for Evaluation (NDE), as well as Evaluation and the Health Professions (EHP) and Evaluation Review (ER). To get a feel for the content in each journal – check out our table of contents alerts archive.

Hot Tip – Join AEA: First, let me admit my bias and conflict of interest here. As the American Evaluation Association’s Executive Director, of course I think you should join AEA if you are not already a member! However, strictly from the standpoint of access to journal content, AEA’s $80 per year membership dues ($30 for full-time students) provides you access not only to current content, but also to over 20 years of back content for four of the major journals in the field. To receive thousands of articles at your fingertips, learn more about joining AEA.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. This is a shortly monthly series highlighting contributions from AEA’s Graduate Education Diversity Internship (GEDI) Program and its interns. You can learn more about the GEDI program by visiting its webpage. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Hello, I am Kimberly Kay Lopez, I have a community-based evaluation and research practice based in Houston, Texas. My work is concentrated in participatory evaluation methods used for the evaluation of youth programs and services.

Hot Tip: Using Photovoice in an Empowerment Evaluation: When working with youth, I have used the Empowerment Evaluation model many times. I have found that using Photovoice and journal writing within the Empowerment Evaluation model yields a variety of rich evaluation data. The goals of the Photovoice process enhance the evaluation process. The Photovoice process allows participants to document issues, engage in dialogue, and impact policy. I first integrated Photovoice with the Empowerment Evaluation model when evaluating the long-term impact of a multi-year teen pregnancy prevention program among urban Latino youth, I utilized the Photovoice process as a tool to “take stock” within the Empowerment Evaluation model. Youth were given cameras to capture the impact of the program. Youth were also given journals and guided writing assignments to express the impact that the teen pregnancy prevention program had on them. We also held traditional Empowerment Evaluation discussion groups. The youth and I analyzed visual data, journal data and discussion group data to develop the final evaluation report.

Rad Resource: PhotoVoice.org: PhotoVoice is an international organization that works with vulnerable populations. They offer several publications including a manual for using Photovoice. The methodology series gives further instruction on working with specific populations such as refugees. http://www.photovoice.org/shop/info/methodology-series

Rad Resource: PhotoVoice Manual: A comprehensive Photovoice Manual developed by Prairie Women’s Health Centre of Excellence. www.pwhce.ca/photovoice/pdf/Photovoice_Manual.pdf

Hot Tip: Using Journal Writing in Evaluations: I have found when working with middle school students, some students may be reluctant to participate in a discussion group. Offering youth writing opportunities allows those students uncomfortable in a group discussion a way to contribute to the evaluation process, on their terms.

Rad Resource: Guide on Engaging Youth: The National Clearinghouse for Families and Youth has a great guide on engaging youth in writing. www.ncfy.com/publications/pdf/lbd_write.pdf

There are all kinds of ways to get creative with data collection-digital storytelling, video cameras, blogs, tweets, text messages! Get creative! Use your imagination! Have fun!

Want to learn more from Kimberly? She’ll be on the program this November at Evaluation 2010, The American Evaluation Association’s Annual Conference in San Antonio, Texas.

· · ·

Happy Saturday! My name is Susan Kistler and I am the Executive Director for the American Evaluation Association. I contribute each Saturday’s post to the aea365 blog.

AEA members receive as part of their membership, free online access to four journals – AEA’s own American Journal of Evaluation (AJE) and New Directions for Evaluation (NDE), as well as Evaluation and the Health Professions (EHP) and Evaluation Review (ER).

Hot Tip: Subscribe to AEA Contents Alerts – whether you are a member or not*, it is an easy way to stay abreast of what is happening in the field. AEA has aggregated the Table of Contents alerts from these four journals so that you need to subscribe only once, rather than four times. You will receive the table of contents from each of the four journals via email or RSS (you choose!) as an issue becomes available online. See an example, and subscribe, at http://www.eval.org/Publications/journalfeed.asp.

Hot Tip: For AEA members and others with access to these journals, be sure to search the journal archives. Each journal’s online access includes over 20 years of back content. The journal archives are a ripe repository of research into the field, case examples with lessons learned, and information about what works, what doesn’t, and why.

Cool Trick: AJE, NDE, and ER, are all on the SAGE Journals Online platform. When you find an article of interest online in any of these three journals, you also have free access to any other articles in ANY journal published by SAGE that is referenced in that article – just click through on the reference link.

*If you aren’t a member of the American Evaluation Association, consider joining today. Our 6,000th member will join at some point this month and receive a great surprise! Learn more about AEA at http://eval.org/.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Archives

To top