AEA365 | A Tip-a-Day by and for Evaluators

Feb/11

26

Susan Kistler on Seeking Your Input Regarding the Pros and Cons of Evaluator Certification and Credentialing

I’m Susan Kistler, the American Evaluation Association’s Executive Director, and I contribute each Saturday’s aea365 post. The AEA Board of Directors has adopted a Question of the Quarter to generate discussion with and among members around issue critical to the field.

This quarter’s question is:

  • What are the pros and cons of AEA instituting a certification and/or credentialing program?

Hot Tip: If you are an AEA member, sign up for this month’s Thought Leaders Discussion. François Dumaine and Keiko Kuji-Shikatani helped to shepherd the Canadian Evaluation Society’s Credentialed Evaluator program. They’ll be discussing issues related to evaluator competencies, certification, and credentialing.

Hot Tip: What do you think? Share your thoughts related to the pros and cons of evaluator certification and credentialing via the comments section below. If you are reading this article in email, click on the title to return to the aea365 website and scroll down on this entry to find the comments box.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

14 comments

  • Janet Smith · March 7, 2011 at 6:28 pm

    I’ve been thinking about this question a lot and am concerned that often credentialing is an exclusionary pursuit, rather than an inclusionary and clarifying set of practices. For example, credentialing often excludes those who cannot pay for it. I work for a non-profit that only in recent years has found the funding to pay for an internal evaluator (me). They would be hard pressed to fund a regular credentialing process for me as well.

    Having also been an external evaluator prior to my current job, I would not have been able to pay for additional credentialing during my doctoral studies a few years back (which were focused in part on research methodologies, which dovetail with evaluation, but do not result in an evaluation credential). I did a variety of evaluation work with area non-profits, for whom an additional fee from me to pay for credentialing would have been unrealistic. I don’t think the issue of cost is the definitive issue in the credentialing debate, but it is important to consider that it may impact smaller evaluations that smaller non-profit organizations seek to carry out — do you want to leave this group out?

    There is also the issue of making sure that credentialing, should it come to pass, be inclusive of evaluators whose native language is not English, especially those particularly suited to evaluate programs that serve immigrants. A test in English is a test of English, and I would suspect credentialing will entail some sort of “test” of skills. I would hope that a multi-language approach would be a consideration in your thinking. Again, I am worried about who gets left out.

    Those are my thoughts at the moment. Thanks for opening up the conversation to all readers. I look forward to hearing your concluding thoughts and about the actions you decide to take on this issue.

    Reply

  • Leslie Cooksy · March 3, 2011 at 11:54 am

    Hello everyone, I am Leslie Cooksy, last year’s AEA President and current member of the Board. I really appreciate the thoughtful discussion that has been going on. I hope you will pose some of the questions that have come up here to Francois Dumaine and Keiko Kuji-Shikatani from the Canadian Evaluation Society in next week’s Thought Leaders forum (http://www.eval.org/thought_leaders.asp). For example, they could speak to why CES decided to pursue credentialing instead of certification, what CES’ plans are for evaluating their “professional designations” program, and the ways that they handled the diversity of evaluation practices in setting credential standards. I hope we will also talk about how AEA is different from CES and what that might mean for our ability to pursue a similar program here.

    As Susan mentioned in the initial post, the question about certification and credentialing is the Board’s “Question of the Quarter”, one of several outreach strategies for deciding what policy issues warrant deeper investigation by the Board. The next step is for me to summarize your comments and next week’s Thought Leaders discussion and present the summary to the Board at its June meeting. I hope to share an abbreviated version of the summary with everyone in one of the coming newsletters.

    Thank you again!

    Reply

  • Judy Woods · March 2, 2011 at 7:44 pm

    For a graduate student and novice evaluator, certification, credentialing and even licensing in evaluation generate interesting discussion.

    It seems that credentialing, by Worthen’s definition (Worthen, 1999), could serve as an entrance requirement for evaluation practice. Entry to practice requirements could be met through successful completion of designated courses and field experiences within a graduate or university program.

    Jacob and Boisvert (2010) describe certification as a more rigorous process. Is there an organizational body prepared to certify evaluators for practice?

    Reply

  • Jim Altschuld · March 2, 2011 at 3:54 pm

    Lawrence, AEA did do this in the mid to late 1990’s and I was fortunate to have chaired the task force. A lot of great people worked on it with me and I certainly learned a lot. A straw poll was taken of AEA members and they split on the idea. Additionally some in the leadership at the time were opposed not due to the idea or concept but in terms of legal issues and the possibility of law suits. Most of what happened was in a number of articles in AJE in 1999. If you have a chance please read them and feel free to raise questions and of course others are encouraged to do the same.

    Reply

  • Lawrence Smith, JR · March 2, 2011 at 9:31 am

    I strongly support efforts to develop some type of evaluator certification because we need to have a way to determine minimal qualifications of those individuals that organizations want to employ as evaluators. In a recent meeting with one of our senior recruiters, I was asked if a professional certification or designation existed for M&E professionals since recruiters often cannot easily determine whether an individual seeking a position as an evaluator has the necessary competencies and experience to do the job. As you know, project management professionals can obtain certification through a PMP training program. Recruiters often specify PMP certification as an indicator of competence and external validation of skills. Of course we all know that many superb project managers don’t have a PMP certification but who knows what will be required in the future? I have seen many Federal job requirements that include a PMP certification as a desired pre-requisite.

    I think the AEA should create a task force to develop recommendations for an evaluator or M&E certification. Inputs could also be solicited from the broader AEA membership (such as you did in your email) and once a draft proposal is prepared, it could be voted on at the next meeting. Perhaps there may be a way to get someone to fund this activity? If AEA establishes a professional evaluator certification, it could also be a source of future revenue.

    I wish you the best of success and send my best wishes.

    Reply

  • Jim Altschuld · March 1, 2011 at 5:20 pm

    Just a couple of quick comments from what others have written.

    Credentialling in a somewhat restricted sense does not attest to quality whereas certification attends to it more. To be credentialled you have to have had certain training and/or experiences. It is more of a starting point but one that would push the field of evaluation to consider what might be key features of training and experience that an evaluator should have.

    To me it is better to have evaluators do this than having an external force suggesting what should be the elements of the credential. That way some knowledge of the history of the field, alternative models of evaluation, alternative methodologies, etc. could be essentials for the credential.

    While I admire what CES has done, a troubling concern for me was that the impetus seemed to come more from the government than strictly from the field. Although that may be somewhat of an overstatement. Again I am positive about CES actions but wonder if the government was the main pusher would it value something like history, yet we in a field would place more importance on that idea.

    Self report is more than having the funds to submit for the credential but is honestly a part of it. In the US, the Certified Government Financial Analyst was in its initial phase if memory serves correctly, self report that then had to be judged against a set of criteria. It was more involved than self report.

    Hope that this is helpful.

    Reply

  • Ann Kanof · March 1, 2011 at 3:49 pm

    I think it is a great idea but would certainly depend on what is needed to become certified. As an independent contractor, it would great to be able to say I am certified to give me some credibility. The cost of this would be crucial, though.

    Reply

  • Elena Harman · March 1, 2011 at 10:41 am

    With the concerns about cost and governance in mind, I am strongly in favor of certification and credentialing for two reasons:
    First, as someone brand new to evaluation, it has taken me quite some time to figure out what education and experience I need to seek out and it still feels a little bit amorphous. I would personally benefit greatly from the structure and standardization of evaluation competencies provided by a C&C process.

    Second, as a funder, it makes me sad to watch the variable quality of the evaluation reports we receive. When organizations are hiring an external evaluator, there is little to no guidance (short of providing a list of “preferred evaluators” or reviewing the responses to their RFP) we can provide to ensure an organization unfamiliar with evaluation selects a high-quality evaluator. I think a credentialing system would be a step to ensure organizations without the expertise to judge evaluators were receiving quality work products.

    Thanks!

    Reply

  • Gabriel Della-Piana · February 28, 2011 at 3:23 pm

    Perhaps a prior question should be, “What is credentialing and certification (C& C) an answer to”. Of course the question has been asked. But a couple of thoughts.

    Experience of the American Psychological Association (APA) is relevant here. At one time APA had ethical standards for test use independent of the test standards that had an enforcement mechanism in spite of the fact that the the Standards for Educational and Psychological Testing themselves had a statement indicating that there is no enforcement mechanism. They later made sure that this was not intended — there is NO enforcement mechanism for those standards.

    Is there an intent in C & C to have enforcement with respect to the evaluation standards resulting in anything from a colleague sent to inform you of best practice all the way to expulsion from AEA? Though I would not exclude the value of C & C, my sense is that what is needed more is principled examples of good practice in the varied specific contexts of use (see the contexts represented in the Joint Committee on Standards) for people who do training (university, private, professional development) and their own professional monitoring to use.

    I was once asked to meet with a small group of orthodontists who met once a month with one member bringing in a tough case and inviting an outside expert to comment and participate in their discussion. This self-monitoring would be another alternative.

    Reply

  • Sara · February 28, 2011 at 3:10 pm

    I’m writing from Canada where I participated in the CES debates on certification. I have also had concerns about this process.

    Since the process involves self-reporting about skills and competencies, I fear that it doesn’t have much reliability. The payment of a fee might also mean that those who get the credentials are those who can afford to pay for them. That’s not really something that should be encouraged.

    Another major question that this raises for me is “Who gets to decide what makes a good evaluator?”
    I am concerned that by agreeing on a strict skill set, we could be reducing our diversity and not recognizing the multitude of skills (including “soft” skills) that make a good evaluator. This may further marginalize evaluators who learned skills “on the ground” and who have a wealth of practical knowledge.

    There are many intangible factors that make for a good evaluator, and I just don’t believe that creating an artificial merit system will capture that. It may give some evaluators a competitive edge in getting jobs or contracts, but that doesn’t contribute to the field or to discourse about evaluation.

    I am also curious whether/how the CES intends to evaluate its own credentialling program?

    Thank you for raising this interesting topic!

    Reply

  • Tanya Ostrogorsky · February 28, 2011 at 2:53 pm

    I am stepping into this conversation a bit late as I know this has been in discussion for several years. So, forgive me if I am completing missing something.

    If the purpose is to provide “buyers” of evaluation services some level of assurance as to the proficiency of the evaluator, I am all for that. HOWEVER, the bigger question becomes who sets that standard and from what philosophical perspective? How would evaluators demonstrate proficiency–or is this a knowledge based credential? Is this a one-time credential or would there be continuing education requirements?

    I also wonder what organization (AEA?) would be contracted to manage this credentially process and the continuing education credit process. I know that other professions have credentials and required CE, but I am wondering if the “definition” of evaluation is so broad within the US that we can even come to a shared understanding as to even the lowest level of proficiency or knowledge an evaluator must have.

    In reflection, my response poses more questions that answers. Without some concepts or parameters to respond to, this is all I have for now.

    THANKS for asking!
    Tanya

    Reply

  • Jim Altschuld · February 28, 2011 at 2:52 pm

    It would be no surprise to anyone who knows my history that I am highly supportive of a credentialling system and also the certification process now being implemented in Canada. Additionally it is worthwhile to point out in response to Jennifer Zipoy’s comment that credentially and certification are some what different (see Altschuld, 1999).

    The two comments posted so far note concerns about cost and indeed doing anything must be sensitive to this issue.

    That being said, the arguments of being proactive for credentialling or certification are very strong as laid out in Canada, by me in previous publications, and as given by CES members over 16 years ago. The amount I proposed for a system was 200 USD. Since that was 12 years ago the costs are most likely low. It was just a guess-timate on my part.

    Once the initial fee for entry was completed the maintennace cost per year was considerably less and with the numerous skill workshops offered by AEA, the Evaluator’s Institute, and local affiliates it is possible to envision a reasonable way to proceed.

    Without taking up a lot of space I do encourage commenters to go back and reviews the arguments in support of moving in some way ahead so that evaluators would at least be credentialled and/or certified.

    Just some thoughts and I hope that they are helpful.

    Reply

  • Jennifer Zipoy · February 28, 2011 at 12:58 pm

    I like the idea of skill building through credential and certificate programs – as a relatively new evaluator, I would love to develop my expertise. i don’t think you can mandate it though… it would be nice to have relatively inexpensive programs as well – thousands of dollars in tuition is unaffordable for me.

    Reply

  • not-for-profit consultant · February 28, 2011 at 10:25 am

    I work as an independent consultant with many small non-for-profit human service agencies. Without exception funding is extremely tight for them and therefore any services they contract out for – like evaluation assistance – need to be provided as economically as possible. The services I provide for them, are done at very low rate. My business simply couldn’t afford the costs asscoiated with travel, classes, etc that would be associated with a credentialing process.

    While I certainly see merit in the credentialing process, I would find it very diifuclt to be able to afford the associated costs, either as a personal business expense or as a cost passed on to my clients. I am as busy as ever since my clients are not able to afford the services of larger, evaluation firms. I benefit greatly from the insights and services of AEA and fear a credientialing process would be difficult for the small non-for-profit niche that I serve.

    Reply

Leave a Reply

<<

>>

Archives

To top