My name is Laura Plybon. I am currently the Director of Assessment and Instructional Design for Drury University College of Graduate and Continuing Studies in Springfield, Missouri.  In addition to conducting academic assessments for the graduate programs at Drury, I also develop and implement assessment initiatives to improve instructional practices for those adult students who attend classes at our seven campus sites located across southwest Missouri. I also work closely with Tony Bowers, Director of Drury University’s Law Enforcement Academy, on academy assessment initiatives. It has been through this unique partnership that I have seen the value of using evidence-based academic assessment tools in predicting cadet persistence, academic achievement, and academy and career success.

There exists a small — but strong — body of theoretical and applied academic assessment police research.  I have found the theoretical perspectives to be refreshingly practical and applicable.  Hoekstra and Van Sluijs’ (2003) model (Figure 1) provides an excellent police assessment framework by considering the dual importance of personality and related psychological traits and cognitive ability and skills in influencing behavioral competencies of police cadets and officers.

Figure 1. Model from Hoekstra & Van Sluijs (2003)

One must have communication and critical thinking competencies to succeed in the field of law enforcement. Consider Holgersson, Gottschalk, and Dean’s (2008) model below. Cadets must have solid professional knowledge of the multiple components of the criminal justice system and critical thinking competencies to effectively perform in each domain.  Strong reading, writing, and communication skills are furthermore beneficial to the many other aspects of law enforcement, including police interviewing, report writing, and testifying in court.

Hot Tip: Evidence-based academic assessment tools have a place in professional programs, including law enforcement academies.  They are useful in retention initiatives and can provide guidance as to what student support interventions are most needed.

Hot Tip: Use academic assessments in coordination with personality assessments for police academy cadets to understand how psychological traits and academic skills of the cadets interact to influence academy behavior.

Hot Tip: Emphasize reading and writing skills across the curriculum as part of the valued-added educational assessment process of professional programs, especially law enforcement academies.

Rad Resources

Chappell, A.T. (2008). Police academy training: comparing across curricula. Policing: An International Journal of Police Strategies & Management, 31(1), 36-56.

De Fruyt, F., Bockstaele, M., Taris, R., & Van Hiel, A. (2006). Police interview competencies: assessment and associated traits. European Journal of Personality, 20, 567-584.

Henson, B., Reuns, B.W., Klahm, C.F., Frank, J. (2010). Do good recruits make good cops? Problems predicting and measuring academy and street-level success. Police Quarterly, 13(1), 5-26.

Holgersson, S., Gottschalk, P., & Dean, G. (2008). Knowledge management in law enforcement: knowledge views for patrolling police officers. International Journal of Police Science and Management, 10(1), 76-88.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

academic achievement · instructional design · law enforcement

My name is Anne Vo and I am a doctoral student in the Graduate School of Education at UCLA. I served as a session scribe at Evaluation 2010 and attended session number 273, Truth, Beauty, and Justice: Thirty Years Later. I chose this session because I was interested in learning about Ernie House’s influence on the way in which social justice is addressed in evaluation.

Lessons Learned: We heard a number of reflections rooted in four panelists’ efforts to understand House’s 1980 publication, Evaluating with Validity. Presenters in this session included Timothy Cash, Mel Mark, Laura Peck, and Beth Weitzman. Below are some take-away messages from their presentations.

• Tim uses the evidence-based policy and practice lens to interpret House’s idea of justice. From this perspective, it seems that one must be fair in order to be just. And, in today’s context, fairness/justice is best accomplished through three primary means: 1) conducting formal evaluation, which enables change; 2) coming to terms with the idea that change is incremental; and 3) having hope for moral progress.
• Mel focuses on the idea that justice is best served by obtaining truth; that is, the most correct and valid answer that one can derive through evaluation. And, that there are different forms of justice – those dealing with process and others pertaining to outcome. He also indicates that the crux of many challenges in present day evaluations are those dealing with how to balance values-based discussions about ends versus those of means and that attention to contextual variables is key when trying to reach reconciliation.
• In an effort to understand how House’s ideas of truth, beauty, and justice interacted with different validity types, Laura and her colleagues conducted a study where they applied his evaluation typology to 75 evaluation studies that were published in the American Journal of Evaluation between 1980 and 2010. Results of their coding process suggest that truth and beauty were more heavily emphasized in the literature compared to justice. While social justice might be most important to House, there is insufficient evidence to believe that it figures centrally in the literature.
• Beth and her colleagues suggest that attaining social justice means democratizing evaluation, but the process of democratization cannot be reduced to packaging evaluation into “toolkits” so that “program insiders” can conduct evaluations on their own and without any methodological backing. Rather, evaluators ought to situate themselves as facilitators, consultants, and guides in the ongoing evaluation and social decision-making processes so that more attention and respect can be paid to local knowledge and as a means to ensure justice and fairness.

Great Resource: Additional information on Ernie House’s (1980) book, Evaluating with Validity, can be found at: http://www.infoagepub.com/products/Evaluating-with-Validity

At AEA’s 2010 Annual Conference, session scribes took notes at over 30 sessions and we’ll be sharing their work throughout the winter on aea365. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

scribing · social justice · validity

Our names are Margaret Braun and Shannon Myrick and we are Research Analysts for the Oregon Department of Corrections and the Oregon Youth Authority. We would like to share a few words about our experience in process evaluations of multiple drug courts.

For those of you not familiar with the drug court model, visit the National Drug Court Institute’s website (www.ndci.org) for all things “drug court”. In short, drug courts are designed to preserve public safety through intensive supervision of individuals who have committed crimes while also engaging in the use of drugs or alcohol. The basic premise is that if the addiction can be treated, future criminality will be reduced.

Hot Tip: Engage in solid process evaluations

Process evaluations are an important component of evaluating drug courts across multiple jurisdictions, mostly because each jurisdiction has its own personality with respect to policies, procedures, stakeholders, and level of support from other key agencies such as law enforcement. Solid process evaluations bring us out of the land of “Black Box Evaluations” and can provide important and timely information to drug courts on how they are doing and why they see specific outcomes. Additionally, process evaluations facilitate interpretation of outcome evaluations across multiple drug courts.

Hot Tip: Bring everybody to the table in the beginning

Drug courts have several stakeholders including district attorneys, public defenders, treatment staff, parole and probation officers, Judges, and other court staff. It’s important to bring those stakeholders to the table to discuss evaluation efforts, as each stakeholder maintains a unique perspective in a drug court effort. For example, District Attorneys may feel very strongly about preserving public safety and as a result, may want stiffer sanctions placed on drug court participants. Treatment providers may have greater insight into a participant’s life (e.g., tough living situation) that other team members may not, allowing for increased effectiveness in supervision. By having every team member at the table, the evaluator can more effectively understand the operation of the drug court.

Rad Resource: Develop a solid observation tool and GO TO COURT!!

Our team developed a tool for observational research that allowed us to systematically track each team member’s involvement, what happened to participants in court, how the drug court team addressed the needs of each participant, and how much time the court spent working with participants. This type of tool provides plenty of standardized data that can be interpreted across jurisdictions and that can help identify key process variables that may lead to increased or decreased effectiveness in public safety outcomes.

For more information about drug court evaluation, visit www.ndci.org.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

drug courts · process evaluation · research analysts