AEA365 | A Tip-a-Day by and for Evaluators

TAG | instrument

My name is Lija Greenseid. I am a Senior Evaluator, with Professional Data Analysts, Inc. in Minneapolis, MN. We conduct evaluations of stop-smoking programs. Smokers generally have lower education and literacy levels than the general population. Therefore, we want to make sure the materials we develop are understandable to smokers.

Rad Resource: Use a “readability calculator” to check the reading-level of your written materials. I have used this with program registration forms, survey instruments, consent statements, and other materials. Not surprisingly, the first drafts of my materials are often written at a level only grad students (and evaluators) can understand. With a critical eye and a few tweaks I can often rewrite my materials so that they are at an eighth-grade reading level, much more accessible to the people with whom I want to communicate.

A good Readability Calculator can be found here:

http://www.editcentral.com/gwt1/EditCentral.html

It provides you with both a reading ease score, and a number of different measures of the US school grade level of the text.

This blog posting is rated at a high-school reading level. Do you agree?

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· · · · · · ·

Hello! I am Maryann Durland and I own an independent consulting firm, Durland Consulting and have been using Network Analysis (NA) since the early 1990’s and hence my focus on our evaluation network. My post today is about the methodology of NA in evaluation applications.  First what is it, second how do you do it, and three what does it look like.

Hot Tip: Network analysis is the methodology for studying relationships among and between members of a set(s).  A set can be people, references, roads and towns, organizations, and so on. Relationships defined for a set can be at three levels –  individuals, subgroups or the whole set, and from a variety of contexts, such as friendships, co-membership in groups, related to and how, work with, readers of the same book, etc. To apply network analysis requires three components:

  1. Define the Network and the Relationship: In some applications the network is self defining – members of an extended family.
  2. Measures Used to Analyze the Data: The choice of measures is usually based on a theory about the relationship.
  3. The Sociogram: The sociogram illustrates the network and also allows us to see the position of individuals within the network to further understand the analysis and which may also indicate further data analysis.
generation shape color
great circle orange
grand square pink
parent triangle blue
child box x green

As an example, in family money exchanges, the data might indicate that dad loans to more people, but daughter 2 is engaged in larger loans. In the sociogram we see how location is important.  The sociogram nodes’ size is related to the amount of money and shape and color are by generation.


We see that dad and mom have similar locations in the network and, except for son 2, they connect to different individuals, suggesting further analysis. Though daughter 2 is connected less as measured by outdegree, she is involved in larger loan amounts. Son 2 is in a pivotal location in the network, bridging mom and dad’s subgroups, indicating that subgroup membership might be another level of analysis.

This provides a small glimpse into network analysis. What I like about network analysis is that it forces you to focus on how we assume behavior will play out in our initiatives like what do we assume mentors will do in a mentoring relationship? And it allows us to explore the complexity of our programs and initiatives.  It requires thinking about the systems within which our initiatives are situated and it is fun.

Rad Resource: Want to learn more about Social Network Analysis? An introductory text is available online at http://ow.ly/1rlcQ

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

family borrowing3.jpg

· · ·

Greetings colleagues!  I am Elizabeth Harris, Ph.D., Vice President of Evaluation, Management and Training Associates, Inc. (EMT).  The focus of this blog is a free resource for measuring youth resiliency that we developed out of necessity.  For over 25 years, we have focused our evaluation, technical assistance, and training work on the prevention of substance abuse and other behavioral health needs; on policies and programs promoting the positive social-emotional and behavioral development of children and youth; on family service needs; and on related fields of public health.

The evaluation of youth initiatives represents a critical aspect of our work.  Unfortunately, as we sought to evaluate one of the largest national federally-funded youth initiatives, available instruments reflected the prevailing thinking that the only aspects worth measuring were attitudes of despair and hopelessness and illegal and risky behaviors.   Available measures did not reflect the reality of programmatic objectives at the local level, many of whom provided activities grounded in youth development theory, seeking to strengthen existing assets.

Resource:  In order to honor the evaluation tradition of measuring what programs actually intend to impact, we developed an instrument of youth resiliency, the Individual Protective Factors Index (IPFI).  The IPFI is a 71-item questionnaire which provides a single measure that captures the various protective factors that have been identified as contributing to individual resiliency in youth between the ages of 10-16 years who may be at risk for developing substance use and other problems. For the national study, we combined the IPFI with federal GPRA measures in order to also measure the risky behaviors and attitudes that were required by the funding agency.

The instrument has been used extensively and is the product of extensive conceptual development and empirical testing, including norming and validation studies on 2,416 youth in 15 states nationwide.

The IPFI is available free of charge to our colleagues and can be downloaded at http://www.emt.org/ipfi.html.  A Spanish language version is also available.  Our only request is that you share the results of your evaluation with us.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

·

My name is David McDonald and I am the Director of a small Australia-based consultancy Social Research & Evaluation Pty Ltd, and a part-time academic at the National Centre for Epidemiology and Population Health, The Australian National University, in Canberra, ACT, Australia. My consultancy work is largely policy analysis and policy & program evaluation in the alcohol, tobacco and other drugs (ATOD) field, and my academic work is in Integration and Implementation Sciences.

Rad resource: People who use and/or experience problems with alcohol, tobacco and other drugs are typically the objects of stigma and discrimination in the community, and too often receive second-rate services in ATOD treatment and other human service agencies. For this reason, it is important that such services conduct regular review of service user satisfaction with the services provided. The eight-item Client Satisfaction Questionnaire (CSQ-8) is a simple and effective evaluation instrument for this purpose. It has sound psychometric properties and is easy to administer. It produces a single satisfaction score from the eight questions. I have used it in a system-wide ATOD service user satisfaction survey in which I used the numerical scores derived from the CSQ-8 as part of a larger client satisfaction survey instrument. I have also used it in an evaluation in which I asked drug treatment clients to complete the instrument, and then I used their responses as a basis for discussing their experiences and assessments of the treatment program. This produced a productive mix of qualitative and quantitative evaluation data.

[Update: The link to the instrument that originally appeared here has been removed by the blog administrators. Unfortunately, the link had been made available by a third party inappropriately. The CSQ Scales, including the CSQ-8, are copyright and cannot be used without the written permission and payment of use fees to Tamalpais Matrix Systems, LLC. Information about use of the CSQ-8 and the other CSQ Scales can be found at:  www.csqscales.com. Please accept our apologies to CSQScales and to our readers for the error.]

The instrument’s psychometric properties and information about its origin are provided in Fischer, J & Corcoran, K 2007, Measures for clinical practice and research: a sourcebook, 4th ed, vol. 2, Oxford University Press, Oxford, pp. 155-6. Its use is discussed and illustrated in ‘Workbook 6: client satisfaction evaluations’, part of the Evaluation of psychoactive substance use disorder treatment workbook series published in 2000 by the World Health Organization in conjunction with the United Nations International Drug Control Programme and the European Monitoring Centre on Drugs and Drug Addiction.

The CSQ-8 is suitable for clients of most human services, not only those specialising in treating people with substance use disorders.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

·

My name is Daniel Brandão and I am an evaluator from Brazil. I work at Instituto Fonte as a consultant and we have a participatory approach to evaluation. I used to work with evaluation of programs focused in youngsters under social exclusion situations where methods oriented by arts were very useful. In 2007 to face the challenge of creating a good environment to have in-depth conversations with youngsters in conflict with the law we developed the FRAMES Method. Inspired by comics, FRAMES is a set of 27 drawings used to trigger dialogues with these youngsters.

Rad resource: You can get the complete set of drawings at: http://tinyurl.com/ygqafsk. We didn’t produce any article or so about FRAMES. But it is in our 2010 to-do list! A good resource about arts based research practice is the book “When Method Meets Art” (Levy, P. Guilford Press)

FRAMES

  • Can be adjusted for multiple possibilities of applications.
  • Can be used in evaluations or as an educational tool.
  • Can be applied Individually or with groups.
  • With both genders
  • With youngsters over 12 year old
  • Used by psychologists, social service assistants, non formal educators, social scientists and others.
  • Application time lasted from 20 minutes to 5 hours.

The linked document is written in Portugese, but here’s a link to an online translation program: http://www.freetranslation.com/

Regards!

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· · · ·

Archives

To top