My name is Deshonna Collier-Goubil and I am a young scholar (newly minted PhD) who has had the wonderful opportunity to collaborate with seasoned evaluators and practitioners throughout my graduate education. Most recently I collaborated with a group of practitioners and evaluators to contribute a chapter in a newly released volume specifically written for young scholars. Our book chapter highlights the importance of collaborating with practitioners.

Lessons Learned: Collaboration has many benefits for both evaluators and practitioners. For the evaluator, receiving assistance from practitioners may help to decrease barriers to rich data, the quality of research can be improved with collaboration, and the overall research process can be improved by adding the input and assistance of frontline workers. For practitioners, evaluators can aide in obtaining research funding, clarifying research goals and expectations, and can highlight the need for institutional change or can put sound research behind an excelling program.

Research collaborations can be transformative for both the evaluator and the practitioner. One should approach a collaboration with deliberation, willing to both teach and learn. An array of cognitive, technical, and interpersonal skills are needed to develop and maintain effective collaboration. Having a firm grasp on communication, trust, honesty, respect, commitment, and flexibility can make or break a collaborative relationship.

Keep in mind, however, that just as benefits exist barriers may also arise during a collaborative effort. Evaluators should try to be as open and honest as possible with practitioners in initial negotiations to attempt to eliminate issues popping up in the future. Discussing division of labor, purpose of the collaboration, timelines for completion, how research will be conducted, data ownership, and how results will be communicated and disseminated for example can eliminate misunderstandings about these things in the future. Other barriers to be cognizant of are blurred roles, divergent perspectives, differences in degree of institutional support, competing and conflicting goals, and communicating difficult results. Despite experiencing barriers, evaluators should embrace adversity and persist in the collaborative relationship. Overcoming these barriers can strengthen the collaboration.

Overall, in a model collaboration, evaluators and practitioners develop shared goals, with consensus on a few key practice and research standards. The investment of time, resources, effort, flexibility, and the willingness to think outside of the box are required. Members of the collaboration learn to enter each other’s world and appreciate the others perspective. This is where the true learning begins.

Would you like to discuss evaluator-practitioner collaboration more with Deshonna and her colleagues? She’ll be contributing to a roundtable on the topic this November at Evaluation 2010, AEA’s Annual Conference.

collaboration, practitioners

My name is Norma Martinez-Rubin and I am an independent program evaluator, occasional trainer, and public health practitioner. Having chosen program evaluation as a second-career focus (www.EvaluationFocused.com) the necessity to ramp up, jump in, and practice the art and science of evaluation as best possible led me to revisit a career strategy to which I was introduced a few decades ago: build your skills by working with a preceptor or mentor. But how does one find that mentor? Where does one start? How do we enlist their participation in our development? Since I joined AEA in 2006, I’ve explored some of the AEA’s many offerings, actively participated, and remained involved to varying degrees. I’ve come to realize that one doesn’t necessarily need a formal arrangement to be mentored. The “right” mentor will be the person who willfully takes a few minutes to guide you to a resource, listens to you, or simply exchanges a thing or two that they’ve learned and thus, frees you to take that (or not) and make it your own. You know that query, “What would you do if you couldn’t fail?” It’s a reminder of the requisite optimism and willingness to test, try, and move on for more —-successful or not, there’s always more!

I realize that there are readers of this blog who are interested in a mentor-mentee arrangement that is more formal than that. Becky Melzer (www.evaluationedge.com) and I reported a bit on that in 2009. (See our posts in the AEA e-Library.) In the absence of any formal mentor-mentee arrangement, here are some tips to get you in a frame of mind to seek out colleagues with whom you can exchange pearls of wisdom:

Hot Tip: Mix business with pleasure . . . Attend a TIG’s business meeting while at the next AEA conference. You’ll get a feel for the group’s sense of collegiality, identify with who you’ll remain in contact through the year, and determine how you might contribute. In a volunteer-run structure, seldom is a person’s time and energy turned away. What fuels a potential mentor-mentee relationship, however subtle, is a continual interest in finding ways for mutual benefit.

Hot Tip: S-T-R-E-T-C-H Consider initiating a mini-project with others. For example, seek assistance to jointly present or contribute toward an AEA conference presentation. Among the benefits of working collaboratively are the small lessons (e.g., meeting timelines, negotiating, asking for help) you learn about yourself in relation to another.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

aea, mentoring

I’m Tamara Bertrand Jones and I evaluate programs and services in Student Affairs at Florida State University. I’ve always been intrigued by the ways that different people develop the skills they need in order to be successful in their chosen profession. Fundamental to the success of budding evaluators is the important interaction with successful evaluation role models.

AEA has many programs that promote the exchange of knowledge between novice evaluators and more experienced professionals, including the Pipeline Program, the Graduate Education Diversity Internship Program, and the Multi-Ethnic Issues in Evaluation TIG sponsored mentor lunch held during the annual conference. The programs provide formal ways to link mentors and mentees.

Developing mentoring relationships with other evaluators not only helps to provide the evaluation exposure needed, but can also be a source of personal and professional support for both the mentor and the mentee. For graduate students or new evaluators not already engaged in a mentoring relationship, here are some tips that to help you take the first step in developing a mutually beneficial relationship.

Hot Tip: Develop a list of characteristics you desire in a mentor. Do you want someone who has the practical evaluation skills, communication skills, and/or professional standing you can aspire to?

Hot Tip: Seek mentors through relationships developed by attending professional association meetings and conferences, professional development workshops, and other gatherings where evaluation is a focus. These connections serve as a means to hear about opportunities in the field. In addition, these networks facilitate sharing, collaboration, professional visibility, and skill development.

Hot Tip: After you have developed your list of characteristics and have identified potential mentors, reach out to them. Send an introductory email; schedule a meeting to establish a personal connection. At the meeting, discuss your future goals and career plans, and how you envision the relationship benefitting both you and the mentor. Remember that mentoring relationships work both ways.

Hot Tip: Be clear about expectations of your mentor. Communicate to confirm that his/her expectations align with yours. Do not expect one mentor to meet all of your needs. If, after you have discussed expectations and one mentor cannot provide everything on the list, continue to pursue the relationship with your newly revised expectations. Take the time to find additional mentors that can meet your remaining needs.

Hot Tip: Be open to mentors from a different gender or ethnicity. Just as one mentor cannot meet all of your needs, mentors of different ethnic backgrounds or genders bring different aspects to the relationship based on their lived experiences. These experiences may be different from your own and can add a depth of perspective that you may not have previously considered. This heightened awareness can only serve to improve your evaluation skills.

mentoring, novice

My name is Bob Spencer and I work for the Riverside, California Department of Mental Health. I serve as an evaluator for a program that aims to keep children with mental health and substance abuse problems in their natural home with their family. My goal for this blog post is to share some of my experiences from my recent job search. Finding a job in this economy is an unenviable position, but with enough persistence and tenacity, it is certainly not impossible.

Ted Williams, widely regarded as one of the best baseball players ever, owns one of the most famous quotations in the game. He said, “Baseball is the only field of endeavor in which a man can succeed three times out of ten and be considered a good performer.”

“Teddy Ballgame” never had to look for a job.

As anyone who has gone through an extended job search can attest, receiving a call back on just one of ten applications can be promising, let alone three. If baseball is known as the “Game of Failure,” what does that make job hunting?

Here are some of the things I focused on when looking for my first “grown-up job.” Hopefully they will help you as well.

Hot Tip: Be Persistent!  Hiring managers are extremely busy—filling open positions is but one of their many responsibilities. If you submit an application or resume and do not hear back within several days, do not hesitate to follow up with a polite e-mail or phone call. Applicants are sometimes afraid to seem to “pushy,” but a friendly reminder can often help your chances of getting an interview, and may help you stand out from the other candidates. I actually looked forward to that follow up call, because it gave me the chance to connect personally with my potential employer instead of being just another name on a page.

Hot Tip: Give Them No Reason NOT to Hire You! Double-negative notwithstanding, this was my philosophy for any job interview I got. A well-crafted resume and cover letter are essential for “getting you in the door,” but it is up to you to make a lasting impression and convince the hiring manager that you are the right person for the job. Be prepared to talk about your experience instead of just your coursework and research history. Be true to yourself—the worst thing you can do is misrepresent yourself to your future employer. In my experience, adaptability is key. If you do not have experience with a certain skill or resource, emphasize that you are willing and able to learn anything they need you to learn. Always have questions for your interviewer! Have several questions, prepare them ahead of time, and write them down if you need help remembering. Your interviewers will be pleased that you are taking a genuine interest in the position, and will certainly not mind the role reversal of having you interview them for a few minutes.

Hot Tip: Most importantly, Don’t Take It Personally! Ted Williams hit 521 career home runs, but he also struck out 709 times. Job hunting is undoubtedly a frustrating experience. However, the key is to not take it personally when you get turned down, and instead continue looking for your next opportunity. As in baseball, patience and persistence are the tools for success in finding a job. Don’t get discouraged if you strike out. Instead, step up to the plate again, knock the dirt off your cleats, and swing for the fences!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

interviews, Jobs

My name is John LaVelle.  I am a student at Claremont Graduate University and served as the Research and Jobs Coordinator for the evaluation program for two and a half years.  Today I will be sharing three tips and one resource.

Well, it’s graduation time of year, and congratulations to all the new graduates!  When I was working as the Jobs Coordinator for my school, I would often help our students identify opportunities for them to practice their craft.  What I learned was that there are a lot of organizations looking for individuals with evaluation skills!  But the job search can be intimidating and a bit overwhelming at times; where should a person start looking?!

Hot Tip: The American Evaluation Association’s Job Bank.  This is a wonderful resource that advertises a LOT of evaluation opportunities.  The bank can be instructed to look for opportunities by state, or you can just browse!  As of yesterday there were 90 opportunities in a range of sectors.  http://www.eval.org/programs/careercenter.asp

Hot Tip: Association for Institutional Research.  It seems that evaluation shares some interesting characteristics with Institutional Research, and IR might be a fruitful avenue for some young evaluators to follow. This is a link to the jobs page of the Association for Institutional Research. http://www.airweb.org/?page=2

Hot Tip: USAJobs. This is the official website of the US government!  There are a lot of departments that employ evaluators, from the Department of Justice to the Department of Transportation to the Government Accountability Office to name a few.  A tip in this area is that they may use the term “program analyst” or “specialist” or “assistant.” http://www.usajobs.gov/

Rad Resource: Virginia Tech’s Career Services website.  Most colleges and universities employ career specialists to help students and graduates with cover letters and resumes.  I have to say that the resources offered by Virginia Tech are excellent, and I very much appreciate all the excellent advice on the essential elements for all cover letters.  In addition, they have examples of letters of inquiry, letters of inquiry about internships, and follow-up communication. http://bit.ly/VTcoverletters

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Jobs

Hello!  My name is Stefanie Leite and I am a Research Assistant for Advanced Empirical Solutions and an Independent Evaluation Consultant.  As program evaluators, we seem to find ourselves in the role of interviewer more often than not.  However, job seeking transfers our role to interviewee.  While it may take a different set of skills, I daresay that as evaluators, we make the best interviewees!  This is because we tend to be sympathetic toward interviewers (having been in their shoes), and hyper-sensitive to answering questions in ways that provide the exact bits of information that the interviewers seek.

Hot Tip: For the purpose of this blog, I’d like to share with you some interview questions I have come across as an interviewee for jobs in the evaluation field, with the intention of helping those of you out there who are job seekers to prepare.

1. What computer skills do you have and how have you applied them in research and evaluation?
2. What steps do you take to ensure the integrity of the data?
3. What experience do you have in quantitative/qualitative data analysis?
4. Describe a problem you encountered in analyzing data and how you handled it.
5. Describe your experience in [project management/developing data collection instruments/report writing/presenting results and recommendations to stakeholders].
6. [The interviewer reads a case study of a program.]  How would you evaluate this program?
7. Describe an accomplishment that you are especially proud of.
8. Describe a mistake you made.  How did you handle it?  What did you learn from it?
9. What is the most challenging aspect of program evaluation for you?  What is the most rewarding aspect?
10. Describe your ideal work environment.
11. Describe your ideal supervisor.
12. Describe your experience working in a team.  What do you like most about working in a team? What do you like least?
13. Why are you interested in working for us?
14. In what ways would this position contribute to your long-term career goals?
15. In what capacities would you like to grow professionally?

One more thing—at the end of the interview, the interviewer always asks, “Do you have any questions for me?”  An excellent strategy is to refer to your own list of questions that you prepared prior to the interview.

Read Resource: These two books by Tony Beshara have been indispensible in helping me prepare for job interviews: The Job Search Solution (2006), and Acing the Interview (2008).

Best wishes to you all in securing the job of your dreams!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

interviews, Jobs

Hello from Oregon! My name is Tanya Ostrogorsky. I am the Assistant Dean for Evaluation Science at the Oregon Health & Science University School of Nursing where I am responsible for the evaluation of 13 academic programs across 6 campuses. Although I am sure that sounds just thrilling to many of you, I writing to you today to discuss how getting connected with your local AEA affiliate is a mutually beneficial experience. As with many evaluators, my background includes primarily research methods and data analysis courses from within psychology and education programs. I have only had one formal course in program evaluation in all of my graduate work. So, where did I learn most of what I know about evaluation? From attending Oregon Program Evaluators Network (OPEN) annual meetings, trainings, and working with connections from within that organization.

Early in my doctoral program (2001) I attended my first OPEN event. I was ‘recruited’ to work the registration table in order to get into the conference for free. That first event was on Appreciative Inquiry by Dr. Hallie Preskill. It was at that moment that I realized that I really didn’t know much about the different philosophies and approaches to evaluation. Each year that I have been involved with OPEN I have been exposed to major thought leaders in our field at OPEN conferences (e.g., Jennifer Greene, Michael Scriven, Michael Quinn-Patton, Melvin Mark, Deb Rog). I have also had the opportunity to meet local evaluators as we all work together to better our practice. Over the past 10 years I went from volunteering at the registration table to being the President of OPEN for several years.

I encourage each and every one of you who are practicing evaluation in the US to engage with a local affiliate group. I know that it made a significant impact on my professional pathway and I believe that it will with you as well.

Hot Tip: There are 26 local affiliates across the country. Click here to see a list.

Rad Resource: Oregon Program Evaluators Network (OPEN) website – We have a searchable membership directory, links to past events with handouts, and resource links, as well as a Council member list, calendar, and easy event registration.

For the past several years, OPEN has offered programming that is focused on skill development including such topics as: context and evaluation, logic modeling; qualitative data analysis; and visual displays of quantitative Information.

We are in progress of planning our October 2010 event. If you are interested in being added to our mailing list you can sign up at http://oregoneval.org/. Click Join OPEN. To be on our announcements list it is free. Membership is $30; $15 for students.

Local Affiliate, Oregon

My name is Derrick Gervin and I currently work as a Lead Evaluator at The Evaluation Group (TEG) in Atlanta, Georgia. I work with school systems and nonprofit organizations to improve student achievement. After completing my first six months as a full-time evaluator, I would like to share some tips with other newcomers to evaluation.

Hot Tips:

• Look inside: Identify your strengths and how they may be used in the evaluation. I realized early on that the field of evaluation was too diverse for me to know everything so I chose to pull from my strengths as a social work practitioner.

• Relationship building: The more you know about your client and their work, the better you can meet their evaluation needs. I’ve taken to doing a Google search of both the client’s organization and key people in the organization – going beyond just their website helped me to uncover valuable information to assist in my work. Also, I take advantage of opportunities to interact with clients during special events (i.e., career fairs, book festivals, and trainings).

• Build trust and be accessible: Make commitments and keep them. Ask clients for their input. Set aside time to be available to clients and return calls and emails as soon as possible. I have monthly evaluation meetings to discuss successes and challenges. Also, I spend as much time as possible on site meeting with project staff and observing processes.

• Get Organized: Find an organizing system that works for you. Also, plan to do as much project management as direct evaluation work. Especially, when projects are at the beginning stage. I’ve found a need to take continuing education classes in time management and the use of Microsoft Excel. I’m constantly searching for ways to maximize my time and work more efficiently.

• Conceptualization: Explore techniques to assist in conceptualizing planned work and expected outcomes. I regularly visit AEA365 for helpful data visualization tips and conceptualization resources. I really like DoView for creating logic models.

• Professional Development: Take advantage of opportunities to increase evaluation knowledge and skills. Know your limitations and consult with mentors and other evaluators in the field. I’ve found my co-workers to be a great source for answering and/or talking through challenging evaluation related issues. I participate in monthly lunch and learn sessions, as well as, group conference calls where we discuss and receive feedback on our evaluation projects.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

reflection, sharing information, stakeholders, time management, trust

My name is John LaVelle, I am an advanced graduate student at Claremont Graduate University.  When I worked as the Jobs Coordinator for my department, I would encourage the students to develop a personal statement about evaluation.  This is important because when they would go to interviews, they would often be asked to describe their understanding of evaluation and explain it to people that may or may not have an background in evaluation.  This exercise eventually became an important element in the Evaluation Procedures course.

Hot Tip: Develop a personal statement of what evaluation means to you and how it can and should be practiced in dynamic, fluid, and political organizational and community environments and how it differs from basic research. In other words, if a client asked you to explain your understanding of evaluation, your approach to evaluation, how you would work with stakeholders, and so on, what would you tell him or her? In your statement, explain what processes you think are important for designing and implementing an evaluation, and how you would approach determining an evaluation’s design and data collection methods.

What might your personal statement of evaluation look like?

interviews, Jobs, teaching