My name is William Rickards; I am currently senior research associate in the Office of Program Accreditation and Evaluation at USC Rossier School of Education. My career has largely been focused in higher education, although I have worked in program evaluation in delinquency prevention, youth services, and in a range of educational and social services.

Lessons Learned-How I’m using video: Over the last few years I have been particularly interested in the use of video-recording for interviews; in my case, this has usually meant interview studies with students and graduates. Use is primarily as data collection, although I use select segments for reporting to faculty; I do the taping on my own, often with portable equipment.

Two examples:

• In evaluating the use of an undergraduate learning e-portfolio, I interviewed graduates regarding their use of the portfolio to monitor and assess their own development
• In an evaluation for a graduate teacher education program for teachers in international schools, I interviewed the teachers on their paths into international school work to understand how to best meet their needs

Hot Tips—Considerations when using video in evaluation include:

• The video as a particularly rich artifact presents potential challenges in terms of analysis: How will the transcript be handled? How much depth will be included in the text?
• At another level, the video record offers a unique opportunity—and often a stark one—from which to study and hone one’s own skills as an interviewer.
• Additionally, the video artifact can provide material that can be used in reporting, depending on clearances, in presentations, websites, or project videos.

Hot Tips—Taming the technology

• The biggest consideration with the technology (particularly in field settings) will be the microphone. External mics—that plug into the camera—are usually best, even if they must often be purchased separately.
• Data storage and transfer need to be studied in relation to individual situations, equipment, and comfort levels.
• Power will always be a consideration—as in battery life and access to a power supply.

Rad Resources

The ethics of informed consent and participation are always a concern, but video complicates this because of participant identity recorded visually. For example, it is standard practice to de-identify data that are being stored for analysis, but this is difficult with video records. These factors need to be considered in the consent and video release forms.

• The American Anthropology Association’s statement on Ethnography and Institutional Review Boards may be helpful in this regard.
• The AEA Guiding Principles are also helpful.
• There is considerable information available on YouTube and Vimeo under ethnographic interviewing that can be useful for evaluation applications (e.g., http://vimeo.com/1269848 ).

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I am Gregory D. Greenman II, the Evaluation and Assessment Coordinator at the Center for Community and Learning Partnerships at Wentworth Institute of Technology (WIT). I am also the Massachusetts Campus Compact AmeriCorps*VISTA at WIT. For the past six months, I have been working on evaluating and assessing the Center’s civic engagement programs and the impact of our community partnerships. I’ve learned a number of lessons about gathering information from college students.

Lessons Learned:

• Tailoring the delivery of the instrument to the student and to the event is a must!
  • Programs that are single events easily lend themselves to paper surveys at the end of the day.
  • Online surveys work best for semester-long projects where students only come to the office a couple times.
  • Peers are often the best interviewers of students. (This means that the interviewer will have to be trained, but adding to the skills and experiences of a student is never a bad thing.)
  • Focus groups are great, but finding a time where everyone can meet is sometimes impossible.
  • Students can be great allies to evaluators; use them.
    • Teaching students about the importance of evaluation and assessment will help rally them to the cause. We increased the response rate from 6% to 76% in just one semester by teaching student leaders the importance of the survey data.
    • Informing students about the importance of evaluation can be just as important as getting data. College students want their voices to be heard and to impact future programming.
    • A little prodding is necessary.
      • Our typical student is balancing their coursework, one or two jobs, and a social life. Things frequently get lost in the shuffle! Occasional reminders are not bad, but one has to tread the line between reminding and nagging.
      • If you have any sort of deadline for the information, subtract two weeks from the time you need the data and make that your published deadline – but do not close the survey. Students will hand in surveys well after that date.

I hope this gives everyone a few ideas on how to gather data from students without resorting to the old tricks of raffles, prizes, and stipends. Tailoring your methods and involving students in the process is not only cheaper, but might even yield better data because you’re not incentivising.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

higher education · students

My name is Vanora Mitchell and I am a professional independent evaluator working in Washington, DC.

How do we evaluate online learning? About a year ago, a school that I volunteer with asked me to help them to evaluate their new online learning initiatives. They were in the process of developing online courses and wanted to know how to identify success and if the evaluation process was different than for the traditional classroom. Here are three resources I found particularly useful as I did my background research:

Rad Resource: Evaluating Online Learning: Challenges and Strategies for Success: Written in 2008 by WestEd for the U.S. Department of Education, this 80 page report was my go-to guide. It had concrete examples, came from a reliable source, and was research-based.

Rad Resource: E-Learning Concepts and Techniques: This online collaborative ebook was developed in 2006 by a class at Bloomsburg University of Pennsylvania’s Department of Instructional Technology. The entire book is useful and Chapter 9 is devoted to E-Learning Evaluation.

Rad Resource: eLearn Magazine: Focusing mostly on the online classroom context, this magazine is web-based and free and full of articles that helped me to understand more about electronic learning. I read a number of background articles, but here are a few that were directly evaluation related:

• Online Learning Indicators
• Tutorial: ffectively Evaluating Online Learning Programs
• Best Practices: Measuring success: raid on Deerfield revisited
• And..here is a link to where you can readily search and find all of their articles tagged “Program Evaluation”

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi, we are Gita Upreti, Assistant Professor at the University of Texas at El Paso and Carl Liaupsin, Associate Professor at the University of Arizona in Tucson. Much of our work involves implementing broad academic and behavioral changes in educational systems. As such, we’ve had a front row seat to observe the explosion of educational data confronting stakeholders. Parents, school staff, school and district level administrators, state departments of education, and federal agencies are all expected to create and consume data. This is a really unique paradigm in evaluation.

In our work with schools, we noticed that similar measures of effectiveness could be used by various stakeholders for varying purposes. A construct that we call stakeholder utility has helped us work with clients to develop efficient measures that will be useful across a range of stakeholders. For example, students, teachers, administrators, school trainers and researchers may all have a stake in, and use, student achievement data but not all stakeholders will be impacted in the same way by those data. So, not only could the same data be used differently by each stakeholder, but based on the individual’s role and purpose for using the data, the level of utility for the data could also change.

It may be possible to affect stakeholder utility and perhaps to maximize it for each stakeholder group by mapping, across four dimensions, how the stakeholder is connected to the data in question, the purpose for measurement, and the professional or personal rewards which might exist as a result of the use of those data. Here are some questions to ask in considering these dimensions:

We are very interested in hearing from folks from other fields and disciplines for whom this model might be useful, and devising ways to measure and monitor the influence of these factors on how data are generated and used by a variety of stakeholders.

Upreti, G., Liaupsin, C., & Koonce, D. (2010). Stakeholder utility: Perspectives on School-wide Data for Measurement, Feedback, and Evaluation. Education and Treatment of Children, Volume 33, Number 4, November 2010, pp. 497-51

A tip of the nib to Holly Lewandowski : http://www.evaluationforchangeinc.com/

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

education · Evaluation · fidelity measures · measure

My name is Laura Plybon. I am currently the Director of Assessment and Instructional Design for Drury University College of Graduate and Continuing Studies in Springfield, Missouri.  In addition to conducting academic assessments for the graduate programs at Drury, I also develop and implement assessment initiatives to improve instructional practices for those adult students who attend classes at our seven campus sites located across southwest Missouri. I also work closely with Tony Bowers, Director of Drury University’s Law Enforcement Academy, on academy assessment initiatives. It has been through this unique partnership that I have seen the value of using evidence-based academic assessment tools in predicting cadet persistence, academic achievement, and academy and career success.

There exists a small — but strong — body of theoretical and applied academic assessment police research.  I have found the theoretical perspectives to be refreshingly practical and applicable.  Hoekstra and Van Sluijs’ (2003) model (Figure 1) provides an excellent police assessment framework by considering the dual importance of personality and related psychological traits and cognitive ability and skills in influencing behavioral competencies of police cadets and officers.

Figure 1. Model from Hoekstra & Van Sluijs (2003)

One must have communication and critical thinking competencies to succeed in the field of law enforcement. Consider Holgersson, Gottschalk, and Dean’s (2008) model below. Cadets must have solid professional knowledge of the multiple components of the criminal justice system and critical thinking competencies to effectively perform in each domain.  Strong reading, writing, and communication skills are furthermore beneficial to the many other aspects of law enforcement, including police interviewing, report writing, and testifying in court.

Hot Tip: Evidence-based academic assessment tools have a place in professional programs, including law enforcement academies.  They are useful in retention initiatives and can provide guidance as to what student support interventions are most needed.

Hot Tip: Use academic assessments in coordination with personality assessments for police academy cadets to understand how psychological traits and academic skills of the cadets interact to influence academy behavior.

Hot Tip: Emphasize reading and writing skills across the curriculum as part of the valued-added educational assessment process of professional programs, especially law enforcement academies.

Rad Resources

Chappell, A.T. (2008). Police academy training: comparing across curricula. Policing: An International Journal of Police Strategies & Management, 31(1), 36-56.

De Fruyt, F., Bockstaele, M., Taris, R., & Van Hiel, A. (2006). Police interview competencies: assessment and associated traits. European Journal of Personality, 20, 567-584.

Henson, B., Reuns, B.W., Klahm, C.F., Frank, J. (2010). Do good recruits make good cops? Problems predicting and measuring academy and street-level success. Police Quarterly, 13(1), 5-26.

Holgersson, S., Gottschalk, P., & Dean, G. (2008). Knowledge management in law enforcement: knowledge views for patrolling police officers. International Journal of Police Science and Management, 10(1), 76-88.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

academic achievement · instructional design · law enforcement

My name is Doug Koch, I am an assistant professor in the Department of Industrial and Engineering Technology at Southeast Missouri State University.

As a university faculty member I find it both ironic and concerning that faculty evaluation instruments often have some very basic issues. Being knee-deep in the promotion and tenure process, I thought it might be appropriate to offer a few tips to new faculty. One thing I have found after using and looking at different universities’ evaluations is that they are often poorly constructed instruments that don’t always aid in documenting your teaching effectiveness. Additionally, if you are not using the evaluation correctly, the instrument may not provide the correct data you need, or at the very least, you end up spending a lot of time explaining the data and why it yielded the results it did. The reliability and validity of student evaluations is an ongoing dispute. So, without even considering the actual soundness of the instruments from a testing theory perspective, there are some basic things you can do or look out for when you are evaluating your performance as a faculty member or other profession.

We currently use a departmental evaluation and also the IDEA Student Ratings of Instruction. Each of them has their own unique nuances and there are a few things I have run into. For the departmental evaluations, some questions were worded in a manner in which a strongly agree is positive and a few questions were negative. If the raw data is printed in a table or displayed in a graph, as below, it looks a little out of place. Even with bold explanations of the two drops in the graph, some reviewers wanted clarification.

Yes, this is easily resolved with reversing the assigned weights for the analysis. That raised flags for others.

Lessons Learned:

• Make sure the evaluations are consistent.
• I would recommend using two different evaluations when possible. We use both a department evaluation and the IDEA. Think of it as using alternative forms to address reliability or error measurement concerns. If there are large discrepancies between the two, you can justify why.
• For the IDEA and similar evaluations, make sure you study how the evaluation works and how you need to use it. You select the objectives that you consider essential, as selecting too many objectives can have ill effects on your scores (see this video).
• Review the instructions on the IDEA website and meet with the group at your institution responsible for the assessment in order to be sure that you are getting the results that you can use to improve your teaching.
• Inform your students of the objectives of the class throughout so that they realize what they are evaluating you on.

IDEA

I am Dr. Jill Ostrow, an Assistant Professor of Teaching in the Department of Learning, Teaching, and Curriculum at the University of Missouri. I coordinate and teach a yearlong online capstone graduate course titled, Classroom Research. The first half of the course is devoted to learning about Classroom Research: developing the question, collecting data, and beginning to write the literature review. The second half of the course is mainly devoted to writing the paper. The students write the paper in sections and receive many comments on each draft they submit. Their final paper is assessed on a rubric that was developed long before I arrived at the university, and as all rubrics, has been modified, updated, and tweaked in the years since it’s creation. I have found the following useful when using such a rubric with my graduate students:

Hot Tip: Make sure to rewrite the highest section (if you use points) of the rubric word-for-word directly into the instructions for each given section of the paper. That way, the student will know what to expect right at the start of the writing process.

Hot Tip: After the student has written the final draft of each section of the paper, send along just that section of the rubric. I cut and paste the individual sections right into a Word Doc. Ask the student to do a self-assessment using that section of the rubric. Once you receive the students’ self-assessment, compare yours against it. Often, I find this is where confusions and misconceptions hide between student and teacher.

Hot Tip: Often with rubrics, students fall into the middle two categories. I often highlight words and/or phrases of one box in a scoring category and words and/or phrases from another.  If relying on points, this can become difficult to score, but again, this is where negotiation between student and teacher is important.

Hot Tip: On the final assessment, it is important to write comments and not just fill out the rubric. But it is also useful to note some of the comments the student wrote on the self-assessments if you found them to be thoughtful and constructive.

Rubrics · teaching

My name is Brian Silvey, and I am an Assistant Professor of Music Education and the Director of the Symphonic Band at the University of Missouri.

I constantly struggle with how to best assess my students. One thing I do know is that using a point system does not seem to work for me. What feedback really comes from, “Hey, you got an 89” versus “Hey, you got a 90?”  In most cases, the only significant difference is the grade (Don’t get me wrong. I realize that the difference between these numeric values is very big when it comes to final grades, GPA, and even scholarships). We never hear stories of the ultimate success of the 3.9 student over the one who graduated with a mere 3.8. “Oh, my life would have been so much more fulfilling if I had just gotten a 92 on my final exam!”  I doubt an extra tenth of a point ever enabled someone to cure a disease or better society.

My point is to encourage many of you to think about gauging student success in terms of measurable student outcomes that are interesting and embody meaningful aspects of the discipline.  In music, no one cares much if you can play all of the notes if they don’t sound beautiful.  For some students, we have to limit the amount of music that they play in an effort to get just a few notes to sound beautiful.

Hot Tip: Don’t be afraid to do the same with your students.  To think that everyone should know the same exact information at the same exact time while using the same exact testing procedures is not very helpful towards gauging students’ mastery or competency of overarching principles.

Hot Tip:  When designing your next assessment, allow students to select from a variety of questions.  What we are really interested in when assessing students’ knowledge is the depth of their understanding of key principles.  Not everyone will understand all of the principles equally well (or should).  Allowing students to expand upon what they do know, in a profound and meaningful way, will give you the opportunity to see how well they understand the important aspects of the subject matter.

Assessment

Our names are Wendy Viola, Lindsey Patterson, Mary Gray, and Ashley Boal and we are doctoral students in the Applied Social and Community Psychology program at Portland State University.  This winter, we took a course in Program Evaluation from Dr. Katherine McDonald.  We’d like to share three aspects of the seminar that we felt made it so useful and informative for us.

1. Classroom Environment. The format of the course encouraged open and interactive dialogue among the students and the instructor. The atmosphere was conversational and informal, allowing students the space to work through sticky issues and raise honest questions without fear of judgment. Regular course activities allowed us to consider creative approaches to program evaluation and develop activities that we brought to class for other students. For example, Dr. McDonald incorporated program evaluation activities, such as Patton’s activities to break the ice with stakeholders, and Stufflebeam’s (2001) “Program Evaluation Self-Assessment Instrument,” into our classroom activities.

Hot Tip: Engage students by facilitating an open and interactive environment that fosters discussion and creativity.

1. Course Content. The course covered both evaluation practice and theory, including the historical and philosophical underpinnings of evaluation theories. Because gaining expertise in the theory and practice of program evaluation in a 10-week course is not possible, Dr. McDonald provided us with a tremendous amount of resources for us to peruse on our own time and refer back to as necessary, as we begin working on evaluations more independently.

Hot Tip:  Provide students with templates, examples, and additional references about any activities or topics covered in order to allow them access to resources they will need once the course is over.

1. Applications. One of the most valuable aspects of the course was its emphasis on the application of theory to the real world.  During the course, we developed and received extensive feedback on logic models, data collection and analysis matrices, and written and oral evaluation proposals. Additionally, we participated in a “career day” in which Dr. McDonald arranged a panel of evaluators who work in a variety of contexts to meet with our class to discuss careers in evaluation.

Hot Tip: Allow students to practice skills they will need in the real world and expose them to the diverse career opportunities in the world of program evaluation.

Our seminar only scratched the surface of program evaluation, but these features of the course provided us with a strong foundation in the field, and elicited excitement about our futures in evaluation.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

application · classroom environment · interactive · resources · self-assessment · theory-practice relationship

Greetings. I’m Joi Moore, Associate Professor in the School of Information Science & Learning Technologies at the University of Missouri.   I also serve as a faculty member for the Information Experience Lab. Teaching an online application development course, such as Flash animation, can easily become overwhelming for students and the instructors.  Students must acquire the core authoring techniques, while determining whether their animation projects are easy to use.  Learning occurs from constant trial and error, along with feedback from peers and the instructor. Developing a functioning project based on requirements is meaningless if the user has difficulty with interactions.

Hot Tip: By following a systematic evaluation process, students can obtain timely feedback throughout the development lifecycle of their animation project.  The following activities outline the overall process and the value for each activity.

1. Design Plan and Storyboard Review

During this activity, students obtain feedback on a visual prototype of their project.  They can make changes to the design based on feedback from peers and the instructor.  The result is a design plan and storyboard that can be used as a development guide for the project.

2. Version One Peer Evaluations

After students include all of the project requirements, they present their project to class peers for evaluation.  Students post a link to discussion board, and peers use a usability heuristic checklist to provide feedback.  The benefits of this activity are two-fold: 1) the feedback improves the project, and 2) students gain ideas from viewing other projects.

3. Version One Revisions

After all of the peer evaluations and instructor feedback, students can make changes to their project.  This will eliminate the occurrence of common usability issues discovered from your peers.  The goal is to eliminate simple issues and errors that can cause the target audience to have a negative attitude toward the project.

4. Usability Testing

Students perform usability testing with 3 members of the target audience.  All testing is performed with the same version of the animation project, which allows students to compare the results.  This phase of testing is very valuable, because the target audience might reveal issues that were not discovered during the peer evaluations.  Although students have gained experience in noticing usability and interaction issues, it is easy for evaluators to miss interaction issues when using the project from a different perspective than the target audience.

5. Final Version Revisions

After all of the usability testing, students can make final changes to the project before submission for a grade. The final product represents several iterations of evaluation that inform the student about interaction design.

By following the systematic evaluation process, students gain informal evaluation learning experiences.  In addition, the process assists the instructor with creating an online learning community of support and feedback.

systematic evaluation · usability testing