Greetings! I’m Ann Emery from Innovation Network in Washington, DC. I also tweet and blog about my adventures as a nonprofit and foundations evaluator.

Lessons Learned: Are you a recent graduate or novice evaluator? If so, you’ve probably already made tremendous professional and personal growth. Congratulations! However, you’ve probably also learned that evaluation is challenging! As I followed the #evalAHA hashtag on Twitter during the AEA conference, I started reflecting on my own aha moments. When faced with new evaluation challenges, celebrating my past aha moments gives me fuel and pushes me forward.

Hot Tips: Here are the aha moments I experienced during my first few years in evaluation:

• Evaluation isn’t research. This realization is especially common for those of us who entered evaluation from the social sciences. Hallie Preskill’s graphic about research and evaluation and Jane Davidson’s article about Unlearning Some of our Social Scientist Habits have been invaluable during my transition from research into evaluation.
• Data will always be missing and messy. Despite your best precautions, you’ll need to budget plenty of time for data cleaning. One of my favorite resources is Missing Data: A Gentle Introduction by Patrick McKnight, Katherine McKnight, Souraya Sidani, and Aurelio José Figueredo.
• Evaluation takes time. The evaluation planning and the process itself can take months (if not years!) and the resulting programmatic and policy changes can take years (if not decades!). I remind myself that evaluation is a marathon – not a sprint – and I rejoice in all victories, no matter how small.
• Qualitative methods rock. Qualitative data are meaningful and useful for program staff, and qualitative approaches are often a better fit for newer or not-yet-evaluated programs than quantitative approaches.
• Randomized control trials are no longer the gold standard. Or appropriate for some programs.
• Jargon’s unacceptable – well, most of the time. Just when I figured out how to really, truly banish jargon from my reports, I started working on an evaluation project for economists, and they loved reading statistical details. That old saying “The only constant is change, continuing change, inevitable change” certainly rings true in my own evaluation practices. I re-visit and re-question my assumptions, approaches, and techniques with every new project.

Lesson Learned: Have you had similar aha moments? What types of evaluations, events, and experiences have prompted your own insights?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

No tags

My name is Susan Kistler. I am the American Evaluation Association’s Executive Director and aea365’s regular Saturday Contributor. Today, we’re talking all things LinkedIn!

Rad Resource – AEA’s LinkedIn Group: AEA’s LinkedIn Community has over 9000 subscribers from around the world and is open to anyone with an interest in evaluation. It’s free (always worth noting) and takes 30 seconds to join if you are already on LinkedIn and a few minutes if you need to make an account. Current popular discussions include ones focusing Theory of Change, Logic Modeling, and Webinars for Capacity Development.

Lessons Learned – why bother? You can post questions to AEA’s LinkedIn group and take advantage of the collective knowledge of those 9000 subscribers. LinkedIn also provides a way to build your professional network, connecting with colleagues with common interests. The group’s ‘jobs’ tab is a place to look for open positions (although I would still recommend searching AEA’s job listings first as they are more extensive).

Rad Resource – LinkedIn’s Endorsement Feature: On October 1, I received an email:

“Patricia Rogers has endorsed you!”

“Why thank you Patricia!” thought I – and then I didn’t think a whole lot more about it.

We were deep into conference preparations and I didn’t click through on the “See endorsements” button that came with that initial missive. By the time I returned from the conference, there were a number of emails waiting, telling me that someone had endorsed my skills. Now I was intrigued, and appreciative, and humbled. And I clicked through.

Here is what I found – at least today’s version of it:

Hot Tip – Adding Endorsements: When signed in and you view the profiles of most people in your LinkedIn network, you can both see and add to their endorsements (unless they have turned this section off), and you can see the endorsements of people outside your network (but not add to them).

Lessons Learned From Endorsements: The endorsements features is a bit like a light, appreciative, 360 evaluation. I learned about how others view me (I wouldn’t have even put Volunteer Management on my personal skills list, but it is indeed a rewarding part of my work). For those I know well, it enables me to provide public kudos and support their work and career – as well as to learn about aspects of their work that were unknown to me.  For those I know less well, it offers insight into their perceived capacity as well as the scope of their personal networks, at least on this platform. In the future, as you seek contracts and positions, potential funders and employers may be looking at your LinkedIn endorsements.

Hot Tips for Leveraging Endorsements from Entrepreneur Magazine’s Daily Dose Blog:

• Endorse others first and endorse fairly
• Keep it easy for your inner circle
• No mass emails

The above reflect my own opinions and not necessarily that of AEA. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Linkedin

Jean King and Laura Pejsa, Minnesota Evaluation Studies Institute (MESI), here, with broad smiles on our faces. We are the proud coaches who are wrapping up this week of posts written by our creative student consultants about ways to evaluate a conference (using exit surveys of Ignite sessions, network visualizing, Twitter, and video clips).  Progressive educators long ago documented the value of experiential learning–“learning by doing”–and our experiences during this year’s AEA conference again provide support for the idea as a means of teaching evaluation. Thoughts about how to use a conference setting to engage evaluation students follow.

Hot Tips:

• Create an evaluation team. Our experience at MESI confirms the value of having students collaborate on projects. Not only do they learn how to do evaluation tasks, but they also learn how to collaborate, an important skill set for evaluators, regardless of their eventual practice.
• Encourage innovation. Our charge was to think broadly about conference evaluation. At our first meeting, students brainstormed many possible ways to collect data at the conference, no holds barred, the more creative, the better.  As we sought to be “cutting edge,” technology played a role in each of the four methods selected.
• Make assignments and hold people accountable. Social psychology explains the merit of interdependence when working on a task. We divided into four work groups, each of which operated independently, touching base with us as needed. Work groups knew they were responsible for putting their process together and being ready at the conference. As coaches, we did not micromanage.
• Make the process fun. University of Minnesota students take evaluation seriously, but their conference evaluation work generated a great deal of laughter. In one sense it was high-stakes evaluation work (we knew people would use the results), but without the pressure of a full-scale program evaluation.

Lessons Learned:

• Students can learn the evaluation process by collecting data at a conference or other event. Unlike programs, short-term events offer an evaluation venue with multiple data-collection opportunities and fewer complexities than a full-scale educational or social program.
• A week-long conference offers numerous opportunities to engage in creative data collection. It is a comparatively low-stakes operation since most conference organizers opt for the traditional post-conference “happiness” survey, and any data gathered systematically will likely be of value.
• Innovative data collection can generate conversation at an evaluation conference.  Many people interacted with the students as they collected data. Most were willing to engage in the process.
• Minnesota evaluation students really are above average. Garrison Keillor made this observation about Minnesota’s children in general, but this work provided additional positive evidence.

We’re learning all this week from the University of Minnesota Innovative Evaluation Team from Evaluation 2012. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluator.

No tags

Greetings, I’m Ann Emery from Innovation Network in Washington, DC. I also tweet and blog about evaluation.

If you recently completed your graduate or undergraduate degree, congratulations! You’ve already learned some of the key elements of evaluation. However, evaluation involves more than theory, methods, and statistical formulas.

Hot Tips: Here are 10 things you probably didn’t learn in school

1. Facilitating group discussions. Managing multiple viewpoints will be key to your success. Learn to facilitate staff meetings, board meetings, client meetings, and retreats.
2. Teaching. If you’ve already taught college-level courses, you’re ahead of the game. If not, don’t despair. For starters, learn to communicate well with non-evaluators (read John Gargani’s post about jargon) and then adapt to new formats (read Stephanie Evergreen’s post about webinars).
3. Adapting to external factors. Universities offer controlled, organized environments. Maybe your program offered a semester-by-semester layout of your courses or a handbook with thesis guidelines. In evaluation, you’ll have to adapt to poor data quality and turbulent political environments.
4. Managing people and projects. Get started by reading the book Managing to Change the World and the blog Ask a Manager, both by Alison Green.
5. Writing non-APA format reports. Evaluation clients want, and need, a variety of formats – dashboards, memos, fact sheets, brochures, handouts, and verbal presentations. Visit the Data Visualization and Reporting website.
6. Giving potent presentations. Check out AEA’s Potent Presentations Initiative to polish your messaging, design, and delivery skills.
7. Negotiating (and compromising). You’ll need to determine deliverables, deadlines, and budgets.
8. Communicating and resolving conflict within teams. Most grad school programs offer a mix of solo and group work, but in evaluation, you’ll be working within teams, work groups, task forces, and committees on every project. You’ll also be co-authoring papers, co-presenting at conferences, co-directing new initiatives, and co-chairing committees.
9. There’s a lot more to evaluation than data analysis. Evaluation is a cycle that involves planning; data collection; analysis and reflection; and action and improvement.
10. Getting out from behind your desk. In grad school, you could disappear into the library on a Saturday morning with your coffee cup and textbook and emerge on Saturday night with a completed paper. In evaluation, there may be full days (or weeks!) when you’re away from your desk because you’ll be talking to stakeholders and convening work groups. Gathering input from stakeholders takes time, but designing an evaluation that meets their needs is well worth the effort!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi, I’m Jennifer V. Miller. For my entire career, I’ve been in some sort of consultative role – either internally as a human resource generalist and training manager in corporate America, or for my consulting company SkillSource.

When you are a consultant your primary role is to assess, then make recommendations for improvement. It’s my observation that people will not take action on your recommendations if they don’t trust you. What follows is my take on trust-building with your customers. “Customers” in this context is anybody who is asking for your professional recommendation. For evaluators this affects the entire process from initial consultation to customer utilization of your final recommendations.

Lesson Learned:

Customers use several measuring sticks for gauging whether or not they trust the advice they’re getting from their consultant. For one, they’re checking out what direction your moral compass points. They’re watching to see if you act with integrity.

Here’s something I learned a long time ago: in your customer’s eyes, integrity is only the start of building a trusting customer-consultant relationship. You see, it’s not enough to behave ethically to be seen as trustworthy.  You also need to understand your customers’ unique trust filters, which they apply in addition to their perceptions of your moral compass.

Hot Tip:

A customer’s personality is reflected in their trust filters.  For example, some folks are naturally more people-focused; others are more detail-oriented. Some people are hard-charging “get it done” types. Your customers are viewing all of your actions through the filter of these personality preferences. If, as a consultant, your actions don’t match up with their natural priorities, then your recommendations may not be fully trusted. Four typical trust filters are:

• Quality – does your work standard meeting that of your customer’s?
• Getting Results – do you deliver results in the timeframe the customer expects?
• Sociability – are interpersonal considerations as important to you as task-related issues?
• Dependability – can the customer depend on you to deliver what you promise?

Your customers are using all four of these filters . . . but most likely, they are relying more heavily on one of them – based on their personality. Pay attention and respond accordingly.

Rad Resources:

• Leading with Trust blog – written by Randy Conley, Trust Practice Leader for The Ken Blanchard Companies.
• Learn about concepts of the Trust Touchstone in the upcoming book The Character-Based Leader.
• Read more articles on trust-building at The People Equation blog.
• Know AEA’s Guiding Principles for Evaluators.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

ethics · integrity · leadership · trust · truthworthiness

Hello, I am Ayesha Tillman, a fourth year Ph.D student in Educational Psychology at the University of Illinois Urbana-Champaign (Illinois). I received my Bachelor of Science in Psychology and my Master of Arts in Research Psychology and never dreamed of working in evaluation. After receiving my MA, but prior to coming to Illinois, I worked as an education research associate in the Research and Evaluation Section of the Arizona Department of Education for two years.

Lessons Learned:

• There are many paths to a professional evaluation career. There are many educational paths and even more opportunities for a career in evaluation. Evaluation positions can be found in academia, private consulting firms, educational research companies and government agencies.
•  Government agencies are a great place to work! While you likely won’t make as much as working for a private evaluation firm or an educational research company, there are lots of perks to working for the government. These agencies often have great healthcare packages, ample vacation/sick time, job security and usually support professional development.

Hot Tips:

• Know what skills they are looking for. Government employed evaluators are usually expected to be able to write reports, give oral presentations, have comprehensive qualitative and quantitative research skills, be able to develop surveys, and analyze data using SPSS, MS Access or SAS.
• Know the policies. In addition to the previous, you will also be expected to have some knowledge of policy or legislation as it relates to the agency you are applying for.  For example, if you are applying for a state Department of Education, you should be familiar with the No Child Left Behind Act of 2001.
• Know what to look for. Government agencies may not always post positions that require evaluation skills as “Evaluator”. Know what type of positions to look for. They may be titled: research associate or program specialist. Check the position description in the job posting. Don’t pass up a job because it doesn’t have ‘_JGevaluator’ in the title.
• Know where to look. The AEA Career Center is a great place to start looking for jobs. You will also want to look at the specific government agency’s website as well. Some government agencies that hire evaluation positions are: Departments of Education (federal and state), the Center for Disease Control and Prevention, and Government Accountability Offices.

Rad Resources:

• AEA Job and RFP Database
• U.S. Department of Education Employment Opportunities
• U.S. Government Accountability Office Careers
• USAJOBS.gov the Official Federal Jobsite
• U.S. Office of Personnel Management for most civilian jobs
• For jobs, networking opportunities, and RFPs in the D.C. area, join the AEA affiliate Washington Evaluators and sign up for the email list

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

careers · employment · job hunting · Jobs · opportunities

Greetings, I am Chelsea Heaven, a recently graduated master’s level biostatistics and public health student. As with many of my fellow recent graduates, I am always on the lookout for as many opportunities as possible to kick-start my young career in statistics and evaluation. A colleague of mine who I work with at my campus’ statistical consulting center recommended becoming involved with Statistics Without Borders (SWB), and I could not be more grateful to them.

My experience with SWB has been invaluable for my professional development. SWB is an all-volunteer organization run as part of the American Statistical Association. Involvement in SWB has exposed me to real data that is used in the daily operation of health care facilities in East Africa, and given me the opportunity to formulate recommendation reports to NGO’s who would use this data for health policy decisions.

The intention for me writing this blog post is reach out to other graduate students involved in evaluation and share how involvement with SWB could be beneficial for their evaluation portfolios, and prepare them for “the real world” of program evaluation.

Lessons Learned:

• It’s important to gain experience with real-world data.Gain experience with the real-world, messy data that is all too often present in evaluation projects.
  • Real data (i.e. data that does not come from textbooks) is often messy, comes in many different formats, and requires substantial effort and teamwork before it can be interpreted. Working with SWB projects allows one to have direct experience with messy data in a team environment, which is invaluable preparation for evaluation projects that happen in “the real world”.
• It’s important to network.  Network and collaborate with evaluation professionals from around the world.
  • Collaboration is the name of the game nowadays in research and evaluation projects, and oftentimes these collaborations happen over e-mail and conference calls. When you are placed in a project at SWB, you are put on a team with 4-5 diverse professionals who you must effectively coordinate with to produce high-quality work. Therefore, SWB gives students highly beneficial experience working in a long-distance collaborative team environment – which will prepare them for future collaborations in their professional careers.

Hot Tip:

• List your volunteer experience on your LinkedIn profile.

Rad Resources:

• Join Idealist. SWB is a member organization of Idealist. Consider joining Idealist, and then joining the SWB site.
• Join SWB. Anybody interested applying statistical and evaluation knowledge to the international community is welcome to volunteer at SWB.
• Volunteer as a web-based statistical mentor for students in a developing country.
• Volunteer at Peoples-uni. Courses of particular interest would be Evaluation of Interventions, Biostatistics, Public Health Concepts for Policy Makers, and Health Economics all with evaluation and research components.

mentoring · Networking · pro bono · volunteer

Hello, I am Gisele Tchamba, a PhD candidate in the Interdisciplinary PhD in Evaluation program (IDPE) at Western Michigan University. During the three years I spent in this program I came to understand that knowing the difference between evaluation and research matters. I found myself defining evaluation and explaining the difference to people in various disciplines.

Lessons learned:

• Know the difference. To become good evaluators, students with social science background must learn to distinguish between evaluation and research. I came to realize that practicing evaluation does not preclude doing pure research. On the contrary, the methods are interconnected but the aim is different. Therefore after I have learnt and understood the difference between the two, I developed a personal method to lighten the burden of explaining the difference by getting two imaginary hats (researcher and evaluator), that I switch on and off as I do evaluation/research or explain to people what evaluation is and how it differs from research.
• The burden of explaining. Many people in academia vaguely know the meaning of evaluation. Those who think they do mistake evaluation for assessment in education. Whenever I meet with people whose understanding of evaluation is limited to educational assessment, I use Scriven’s definition and emphasis words like “value, merit, and worth”. This usually brings forth expression like “oooh” that is usually followed by their desire to learn more. So I give Coryn’s definition “Evaluation is driven by the need to make informed, defensible, decisions and the desire to impose classifications regarding quality or value…..” Then I explain that social science research is a truth seeking activity aimed at contributing to existing knowledge or generating new knowledge, or for application to some specific problem related to human action and interaction and does not have the burden or rendering judgment of merit or worth.

Hot Tip – Distinguishing between evaluation and social science research:

1. Theoretical and practical experiences are helpful ways to distinguish between the two disciplines. The IDPE curriculum prepares students to do both evaluation and research, e.g. research methods courses that are required for the mastering of evaluation methodology are actually social science research methods.
2. Extensive reading of evaluation literature helps to see the difference. I also like Trochim’s definition of evaluation found in his evaluation of Scriven minimalist theory, “Evaluation is a profession that uses formal methodologies to provide useful empirical evidence about public entities (such as programs, products, performance) in decision making contexts that are inherently political and involve multiple often conflicting stakeholders, where resources are seldom sufficient, and where time-pressures are salient.”

Resources:

• Mathison’s chapter – What is the difference between evaluation and research and why do we care?
• Social Research Methods
• Evaluation Exchange

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation · research

My name is Xiaomei Song, a PhD Candidate in Assessment and Evaluation, Queen’s University, Kingston, Canada. In the past four years, I have been working on my doctoral degree and a part of my dissertation is to evaluate a large-scale high-stakes testing system from the stakeholder perspective. I have learned a number of lessons along this academic inquiry.

Lessons learned:

• Be prepared for questions. Get yourself prepared for the questions like: “what do test takers know about the test?” in the meetings and team work. Psychometricians and evaluators live in different worlds. For sure they use different mindsets to approach and interpret phenomena. Traditionally accepted conceptualizations of important terms in testing such as validity and fairness rest on the assumption of objectivity and neutrality.  Well, that is not what evaluators believe in.
• Identify key informants and legitimate stakeholders. In the Encyclopedia of Evaluation, Jennifer Greene emphasizes that since not all stakeholders are involved in the whole process of the testing, it is important to identify key informants and legitimate stakeholders. In her article, From products to process: an ecological approach to bias detection, Janna Fox shares that some stakeholders such as test takers and instructors may discuss about their perceptions associated with test design, item development, administration, scoring, use, impact and consequences.  However, Catriona Scott in her article, Stakeholder perceptions of test impact found that stakeholders, such as administrators and parents, may provide meaningful information only in limited aspects.
• Don’t take knowledge and experience for granted. Don’t take for granted about stakeholders’ knowledge and experience. They may be titled as classroom instructors, but they can also be program administrators, test administrators, item writers, or even test takers. Test stakeholders are often separated into three groups in testing standards, guidelines, and codes of practice: test developers, test users, and test takers (e.g., AERA, APA, & NCME, 1999; JCTP, 2004). Apparently, a clear delineation of these roles may not always exist in reality. Survey questionnaire and interview questions need to take this into consideration accordingly.

Resources:

• The American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education jointly developed Standards for educational and psychological testing.
• The Joint Committee on Testing Practices’ Code of Fair Testing Practices in Education that provides guidance in the following:
• Developing and selecting appropriate tests
• Administering and scoring tests
• Reporting and interpreting test results
• Informing test takers

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

standardized testing

Ah the joys of a truly beautiful spring day. I am Susan Kistler, the American Evaluation Association’s Executive Director and aea365 Saturday contributor. I’m hoping you’ll join us in Atlanta in a few short weeks for the AEA/CDC Summer Evaluation Institute – but even if you can’t, scroll to the bottom and learn about a great blog that’s worth a read.

Hot Tip: Attend the 2012 AEA/CDC Summer Evaluation Institute to fill your evaluation toolbox, and network with colleagues. We’ll be in Atlanta from June 3-6.

Lesson Learned: Because the Institute is co-convened by the Centers for Disease Control and Prevention, some people think that you have to be working in public health to participate. Not so! The Institute is full of professional development workshops appropriate for beginning to intermediate level evaluators, as well as those called upon to perform evaluation as part of their broader duties.

Lesson Learned: Because the event is called an Institute, some people think that it isn’t for practitioners. On the contrary, the event is filled with sessions focusing on concrete skills, from convening focus groups to developing surveys, from evaluating media campaigns to project management, from planning your evaluation to attending to culture and context.

Hot Tip: And, on a completely different note, I want to suggest that those of you who are new evaluators (and perhaps even those who have been at it for a while) consider subscribing to Karen Anderson’s blog On Top of the Box Evaluation. You gotta love anyone with a countdown to the Institute on her homepage, but Karen also has some wonderful insights into working to be a better person and a better professional and the intersection of the two. Karen is a (relatively) new professional, a graduate of AEA’s GEDI program, and an all around wonder woman. Oh, and did I mention that Karen is our (relatively) new Diversity Programs Intern? So much wonderfulness in one person makes her blog a must-read. Recent posts have included “What Evaluation Hat Are You Wearing” and a series on the ROI of Building Trust: Are You an Evaluation Trust Agent.”

Clipped from: ontopoftheboxeval.wordpress.com (share this clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blogs · institute