Hello, my name is Lisa Garbrecht. As a Research Associate at EVALCORP Research & Consulting, I work on numerous projects requiring data collection from youth. As you may know, it is not always easy to obtain high quality data (i.e., sufficient numbers of completed surveys, academic data, etc.) when relying on schools to help facilitate the data collection process. Below are a few tips that have proved useful!

Hot Tip #1:  Take time up front to identify the right data and methods. With the limited time and resources faced by schools and school-based programs today, it is important to collaborate with clients early on to identify priority needs and ensure that data are collected efficiently. Would a brief post-survey suffice instead of a comprehensive pre-post? Be strategic and include only items that really matter. Phrase items clearly and simply to ensure they are easily understood. Show schools you respect their time by only asking for the most vital information to inform the evaluation. 

Hot Tip #2:  Partnerships are key. Working together, evaluators and clients can build mutually beneficial relationships with schools to overcome their resistance to providing data. By showing school personnel and stakeholders how the findings may be of use and providing them with the necessary tools and databases, schools are more willing to collect and provide data in a timely manner. Communicate regularly with clients and schools, providing contact information so that you can answer their questions and offer assistance as needed.

Hot Tip #3:  Look at the data before it is too late. Whenever possible, do not wait until the end of the data collection process to analyze what’s coming in. Running the data early on allows you to identify problems with the tool or data collection process and make changes. Monitor data quality on at least a quarterly basis. This allows you to provide clients and schools with formative information that can serve to strengthen their programs and their motivation for assisting with ongoing data collection.

Hot Tip #4:  A little incentive goes a long way. Use incentives with project staff, school personnel and/or students as allowed. For instance, EVALCORP uses an award system for rewarding site staff members who consistently collect accurate, legible and complete survey data with a small gift card and certificate of appreciation. Pizza parties or other youth-friendly activities are other alternatives for showing clients/schools your thanks. If tangible incentives are not possible, be sure to let those involved know the value of their input and how much you appreciate their time. Oftentimes, a “Thank you and I really appreciate your help” goes a long, long way!

This aea365 Tip-a-Day contribution comes from the American Evaluation Association. If you want to learn more from Lisa, check out the sessions sponsored by the PreK-12 Educational Evaluation TIG on the program for Evaluation 2010, November 10-13 in San Antonio. If you would like to contribute an aea365 Tip, please send an email to aea365@eval.org.

Data, partnership, quality

My name is David Merves and I work for Evergreen Educational Consulting LLC (EEC), an evaluation consulting firm. A part of our practice involves evaluating federally funded, Special Education grants and programs that are intended to improve education and services for children and youth at risk of school failure or in need of special education (e.g., IDEA, Section 504).

The advent of online learning, distance learning and use of technology for the classroom has transformed how our clients work and the strategies and processes they employ. In order to perform our evaluative functions EEC is committed to staying abreast of technology issues/changes as they relate to Special Education. How can we evaluate that which we do not understand?

Hot Tip - iNACOL – International Association for K-12 Online Learning: iNACOL has excellent resources for collaboration, advocacy, and research to enhance quality K-12 online teaching and learning for all students.

Hot Tip – ISTE – International Society for Technology in Education : ISTE is an association for educators and education leaders engaged in improving teaching and learning by advancing the effective use of technology in Pre K-12 and teacher education.

Rad Resources: If you wish to dive deeper into technology waters here are three additional resources, which, go beyond Pre K-12 and bridge the transition to the working world:

1. Christopher Dede- Harvard Graduate School of Education He states, education needs to prepare today’s students for tomorrow’s jobs, which may not, currently, have been invented/created. He uses Web 2.0 as a jumping off point to discuss “thinking skills” for teachers and students utilizing tools, such as, wikis, blogs and discussion forums. He goes further and discusses creativity skills using video/photo sharing websites, mashups and online writer workshops.
2. Elliott Masie is an internationally recognized futurist, analyst, researcher and organizer on the critical topics of workforce learning, business collaboration and emerging technologies. He has a free e newsletter, as well, as fee-based services.
3. The New Division of Labor: How Computers Are Creating the Next Job Market (book), Frank Levy (Author), Richard J. Murnane (Author). Their premise is, if educational curriculums aren’t changed to reflect the market’s demand for sophisticated thinking and communication, students may graduate without the skills they need.

Google the above names for further materials and references.

These resources have enhanced our abilities as evaluators to work in the changing and increasingly technological-educational world. We as evaluators need to stay current with the methods and processes being adapted for use within Special Education. One example is the promising research being demonstrated in Second Life (a MUVE- Multi-user Virtual Environment), with students on the Autism Spectrum at the Center for BrainHealth at the University of Texas at Dallas.

The American Evaluation Association is celebrating Disabilities and Other Vulnerable Populations (DOVP) Week with our colleagues in the DOVP AEA Topical Interest Group. The contributions all this week to aea365 come from our DOVP members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting DOVP resources. You can also learn more from the DOVP TIG via their many sessions at Evaluation 2010 this November in San Antonio.

special education, technology

My name is Marijata Daniel-Echols. I am Director of Research at the HighScope Educational Research Foundation. HighScope is best known for our work in preschool curriculum development and for the Perry Preschool Study. Our Research Department conducts evaluations of early childhood programs (e.g. state funded preschool initiatives) and general research on early childhood theory and practice. Because our work is focused on turning data into actual policy and changes in practice, we spend a great deal of time working with program partners who do not necessarily have a strong background in research or evaluation methods.

Lesson Learned: Researcher-program partnerships can be both a point of strength and a challenge. Program evaluation is often seen by administrators as threatening or focused solely on accountability as opposed to informing a cycle of ongoing program improvement. Evaluators often hold unrealistic expectations of the capacity of their partners to meet rigorous evaluation design and data collection standards given real world constraints.

When researcher-program (or evaluator-client) partnerships are successful, they can result in clear, relevant, and useful data. In my 10 years of experience conducting research in partnership with programs, there are a couple of basic tenets to follow:

• Having clear expectations of what everyone has to gain, lose, and must contribute to the evaluation process is essential. Acknowledge the trepidations and anticipations on both sides and keep them in mind during the evaluation planning, implementation, and dissemination phases of your work.

• Involve your partner in the evaluation process from the beginning. Early understanding and buy-in increases the likelihood that they will work hard to protect the integrity of your design. Solicit from them what they hope to learn from the evaluation process and to what uses they plan on putting the resulting report(s).

• Have your partner work with you to delineate their theory of change and create a logic model that guides your methods and instrumentation. That process will make clear to everyone what questions the evaluation will and will not be able to answer and what information has to be collected.

• Share preliminary findings with your partner and ask for their interpretation of the findings. In addition to creating a sense of ownership and understanding of the data, you will gain useful contextual insights that will help you draw conclusions and make suggestions for improvement.

• Work with your partner to create an evaluation report dissemination plan. Keep in mind that there are probably several different types of stakeholders with different data reporting needs.

• Finally, recognize that providing relevant, useful evaluation data leads naturally into consultation – ongoing support from you will increase the likelihood that the data is in fact used to make real changes in policy and practice.

This contribution is from the aea365 Tip-a-Day alerts, by and for evaluators, from the American Evaluation Association. If you’d like to learn more from Marijata, consider attending her session at the AEA Annual Conference this November in San Antonio. Search the conference program to find Marijata’s session or any of over 600 to be presented.

Childhood, partnership

My name is Andrea Velasquez, and I am a doctoral student at Brigham Young University. For the last four years, I have been an instructor of an undergraduate class that teaches pre-service teachers how to use technology effectively in elementary and secondary education settings. One of the principal frameworks that we use to teach pre-service teachers how to distinguish between all the facets of designing effective instruction with technology is TPACK, or Technological Pedagogical Content Knowledge (Mishra & Koehler, 2006). This framework states that in any effective technology mediated instruction, technology, pedagogy and content are three components that not only co-exist, but also interact and have an impact on each other. The research examining TPACK can be useful to the field of evaluation by applying it to evaluations of technology-mediated instruction. Distinguishing between these three components- technology, instructional content, and instructional strategies- can help evaluators identify appropriate questions and alleviate the complexity of evaluating e-learning.

Hot Tip: When designing an evaluation of technology-mediated instruction, after determining context and stakeholders, consider technology, pedagogy and content as evaluands. Then, identify criteria and questions for judging each evaluand. Before continuing the evaluation, also identify criteria and questions that take into account how each component impacts the others. These questions should address the compatibility between these components. For example, if an online high school uses video technologies to communicate with students, an evaluation of such a program should take into account the video technologies, the strategies the teacher uses to teach the class (i.e. group work, field experiences, presentations), and the content of the instruction he is teaching. Besides addressing each of these three components, the evaluation should address the relationships that exist between each of these components at each stage of the evaluation process. This approach to evaluation ensures a more holistic evaluation of the technology use in relation to the context and the needs of the students and stakeholders.

Rad Resource: This site is a resource that is maintained by the developers of the TPACK framework. It has updated research articles and many other resources for understanding the practical applications of the framework http://www.tpck.org/tpck/index.php?title=Main_Page

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Want to learn more from Andrea? She’ll be presenting as part of the Evaluation 2010 Conference Program, November 10-13 in San Antonio.

instructional design, pedagogy, TPACK

My name is Debra Thrower, and I am a social worker and a former university administrator. I have served as the Project Director for federally funded, community and faith-based after-school programs. Additionally, I have external evaluation experience, as well as meta-evaluation experience.  I would like to share what I have learned and monitored over a period of years for current and future evaluators.

Are evaluation centers really collaborating with low-income communities in hope of closing the academic achievement gap? In order to address the question fairly, evaluators would first have to be knowledgeable of the No Child Left Behind Act of 2001 and its goals.  Funding opportunities are available to the following eligible applicants;  1) School districts, 2) Educational consortia, 3) Non-profit agencies, 4) City or county government agencies, 5) Faith-based organizations, 6) Institutions of higher education, and 6) For-profit organizations. These funds are available through the state department of education systems.

Hot Tip: I highly recommend that the evaluator(s) be involved during the outset of the Pre-K-12 educational program initiative for low-income communities.

Hot Tip: Be aware that many programs are limited to no more than 5% of the total funding for evaluation efforts.  While program evaluators are welcomed, there may be a program evaluation unit, as well as a compliance unit, which are both separate from the tasks of the program evaluator(s) or local evaluator(s).

Hot Tip: During the evaluation outline to funding source(s), pay close attention to actual participants through well-designed focus groups or other means in order to properly represent students and their families.

Through advanced technology and other collaborative efforts, our systemic structures have greatly improved. However, our abilities to connect on a human level with students and their families in hopes of closing the academic achievement gap remains to be a problem in our society.

achievement gap, education

My name is Sarah Hug and I am a Research Associate with the Alliance for Technology, Learning and Society (ATLAS) at the University of Colorado at Boulder. I am going to give a few tips regarding “pipeline evaluation” for programs aimed at increasing enrollment and diversity in science, technology, engineering, and math (STEM) fields.

Pipeline programs aim to change the career trajectories of young people in the long term. These goals for youth are often beyond the scope of the evaluation and program timelines- for example, an academic science program targeting middle school students will not be able to collect career data for at least five years, when students have graduated high school. What should  an evaluator study in the meantime?

1. Student aspirations and interest: Evaluators can focus on student interest in the STEM fields, and their changing or continuing aspirations for STEM careers.
2. Student knowledge of the fields: Knowledge about careers in technical areas is essential for advancement in STEM, particularly for underrepresented and under-resourced students. Evaluators can focus efforts on program participants’ change in career awareness at all academic stages. Some elements of career awareness evaluators might measure include:  knowledge of the depth and breadth of science careers, knowledge regarding what scientists do, and familiarity with the level of education needed to attain specific careers.
3. Students’ “next- step behaviors”: Students’ college and career readiness can be influenced by early academic experiences. Discover what students might do at each level of the academic pipeline to further their STEM career readiness. This is often context and even school district specific-for example, are there clubs or camps students might join to advance their careers? What high school course choices could indicate students’ preparation for STEM careers? It is particularly important to consult program directors and school partners to gather ideas for measuring “next-step behaviors”. Comparing your program findings to local or national data for similar groups is essential for showing program impact.

Rad Resource: The National Center for Educational Statistics http://nces.ed.gov/ provides national data on student enrollment, graduation, and academic behavior at all levels of the education pipeline. Check out “The Condition of Education 2010” report, and data exploration tools to understand how students are progressing at different stages of the pipeline.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

careers, Geographic information systems, pipeline

My name is Susan Menkes, and I’m an Applied Developmental Psychology doctoral student at Claremont Graduate University (CGU) and received my Masters in Evaluation from CGU, as well.  My work has focused on applying developmental research to understand effective interview strategies with younger children when evaluating programs serving children and youth.  Below are some tips on how to construct developmentally sensitive questions to increase children’s comprehension and subsequent responses:

Hot Tip: Wh-Questions: Particular question formats, such as wh-question words, are understood before others; words such as, what, which, where, and who are easier to understand than how, when, and why. Understanding which question format is easiest for young children to comprehend is critical to constructing effective questions to elicit the most descriptive output from children.

Hot Tip: Negative Phrases: When trying to elicit evaluative feedback from children, an interviewer may need to phrase the question with a negative (What do you not like about this program/service?).  However, re-phrasing the question to incorporate a semantic negative that is not syntactically negative (What can be done to make this program/service better?), can be useful in eliciting more descriptive detail compared to questions that are syntactically negative.

Hot Tip: Linguistic Complexity: Children may find it particularly difficult to understand linguistically complex questions; simply constructed questions containing one verb are easier for children to comprehend (e.g., ”What do you like to play with your friends when you are outside?” versus “What do you like to play outside with your friends?”). Additional nuances in the basic syntactic words used, even in simple questions, can impact children’s comprehension.

• Pronouns and passive constructions can decrease comprehension.
• Understanding person pronouns (e.g., I, you, and he/she) is difficult for young children; third person pronouns are typically learned later than first and second person forms.
• Passive constructions (e.g., The rabbit is following the cat) are more difficult to comprehend than active constructions (e.g., The rabbit follows the cat).

While including young children throughout the evaluation may not be feasible, they certainly can provide valuable feedback to program managers related to the strengths, weaknesses, or suggested improvements in a program. To maximize the amount of descriptive output children provide to questions, it is critical to construct developmentally sensitive questions that young children can comprehend.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

developmental, qualitative

My name is Ted Dwyer and I’m the Manager of Evaluations in Hillsborough County Public Schools. I have worked in multiple school districts and have served as the reviewer of external research projects in several districts. Today, I would like to share some of what I have learned and observed about FERPA and some thoughts from the perspective of an evaluator.

To protect the rights of parents and students in educational settings, the federal government has put in place the Family Educational Rights and Privacy Act (FERPA). FERPA’s main intent is to provide a clear delineation of the parent’s rights to have access to their student’s educational records. While codifying parental rights, FERPA also sets out some very specific guidance and general directions to educational institutions.

Hot Tip: Two reasons that evaluators should be concerned about FERPA are:

1. If FERPA is violated, the district/university can lose its federal funding and they essentially cannot work with you for 5 years.
2. FERPA requires parental consent (or adult consent for non-minor students) in order for an educational institution to provide any individual information on a student.

On its face this law can create major headaches for evaluators. However, there are several ways to easily work within FERPA.

1.  The easiest way to ensure compliance with FERPA is to get parental consent and make sure that it specifies:

a. What records can be disclosed (discipline data, achievement data, grades, etc.

b. How the records will be used (evaluation, etc.)

c. Who will receive the information

d. How long the information will be kept (usually until project completion)

Bonus Tip: Make sure that the student is identified in the way the institution keeps its records (student identifier – often a number)

2.  There is a clause in FERPA for “research” but you have to convince the institution that you are conducting the study “…for, or on behalf of…” the institution and that you are “…developing, validating, or administering predictive tests, administering student aid programs, and improving instruction…”. For many this seems like a “no brainer” for the “place” to put an evaluation; however, it is up to the institution to determine if your project meets the criteria – which can be an arduous process rife with institutional politics. Further, some institutions interpret this in relation to who funds the evaluation.

FERPA affects any educational organization that receives federal funds. As responsibility for ensuring that FERPA is followed falls upon the institution, your experience will depend on which institution you are going into, the institution’s policies, and how they interpret the state and federal law (often based on the legal advice of school attorneys). Find out what the policies of the institution are and follow them.

education, FERPA, Schools

Hello, I’m William Moore and I am the Director of Research and Measurement at the Institute for Research and Reform in Education (www.irre.org).  IRRE partners with schools, districts, and states to transform public schools into more engaging, rigorous, and caring places for students to learn and teachers to teach. My colleagues and I have spent the last decade focused on creating new measurement systems for districts and schools. We have created new approaches to making data actionable for educators, and have provided intensive supports for our clients so that they can better use and take action based on data.

As I write this there is great interest (and federal funding available!) to help our schools become more data-driven. My experience is that most schools and youth-serving organizations do not have a history of systematically using data to inform and improve practice. We have learned that it takes a concerted and sustained effort to create the conditions to support a culture of data use in organizations and intensive support for practitioners to become effective users of data.

Two major challenges educators face in doing this work are: how can accurate and timely data be made available to all who need it when they need it; and how then to use these data in ways that will directly benefit students. Today, I am going to share some of our lessons learned and resources to help you help your client’s establish a culture of data use.

Hot Tip: We have found in our field work with schools that four conditions must be in place for an organization to have a good chance at creating a culture of data use.  Focusing your technical assistance toward these four conditions will be a great place to start.  We call these conditions ‘building blocks.’ The four building blocks are:

1. System leadership — the capacity of system leaders to plan, implement and strengthen data use strategies. This includes a) setting and articulating clear expectations for data use; b) personifying commitment to the use of data; c) providing timely and effective supports to others in the system around data access and use; d) monitoring and reporting progress; e) recognizing others’ accomplishments in using data and intervening to remove barriers to effective data use.
2. Data Governance — The intentional actions of an organization to ensure the accuracy and timeliness of its data by creating policies, procedures and practice guidelines that reduce the chance for data-related errors and increase the likelihood of data use. Data is an organizational resource and it must be carefully managed, audited, and controlled – youth serving organizations make very important decisions about children based often on poorly governed data resources.
3. Enhanced data and technology systems – Using data in new ways requires new ways of managing and organizing data. Organizations are beginning to implement data warehouses and designing new reports of youth outcomes that provide actionable information for youth workers and teachers. One organization that has partnered with many schools to create this enhanced technology for data use is Mizuni, Inc. Here is an example of one report designed by IRRE that is used by teachers, advocates, parents and students. http://bit.ly/examplereport
4. Skill Building and Structures – For professionals unaccustomed to using data to guide and direct their practice decisions an intensive training sequence and supports are needed. Training in data-driven dialogue that takes professionals from learning about what data are available for their use to accessing that data and then using a process or protocol to dialogue with their colleagues around the data and finally to making decisions based on the data in a collaborative, informed fashion is necessary and requires practice, practice and more practice.

Even if an organization has supportive leadership; accurate, credible, and timely data; a data system that generates actionable reports based on this data for all who need it; and staff who have been trained to access and use these reports, the organization will need to examine how it can use time and space to be sure the opportunities to engage in dialogue exist. For example, if teachers do not have common planning time around a shared set of students this reduces the impact that the best data systems and training will have. Organizations who expect staff to be data-driven must create frequent time and space in the workday for this to occur.

Helping your clients create the foundation for effective data use pays off when external evaluators look at their progress.   As evidence comes more and more to drive program investments, organizations whose cultures and individual staff value data and use it well will be more likely to work well with evaluators, to have better information about their work and to do their work better.

Rad Resources: To learn more about providing supports to schools around effective data use: http://www.irre.org/publications/doing-what-matters

To learn more about data governance take a look at the work of the Data Quality Campaign: http://bit.ly/dataqualitycampaign

To learn more about Mizuni, Inc., and their data warehousing and reporting solutions for schools and other youth serving organizations: http://www.mizuni.com

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

culture, Data

Hello, my name is Scott Cody and I’m Deputy Director of the What Works Clearinghouse (WWC). I’m also the Associate Director of Human Services Research at Mathematica Policy Research. I’d like to share with you an important resource for researchers, educators, and policy makers. Evaluators know how important study design is to the validity of study results. Valid research can educate the public and empower them to make better decisions about everything from healthcare to education.

Resource: The WWC, founded by the U.S. Department of Education’s Institute of Education Sciences, is a central source for comprehensive reviews of education research. Each WWC review takes a thorough look at the research on a particular topic or education program, product, or practice. Our goal is to identify well-designed studies and summarize those studies’ findings for decision-makers. To do this, we measure each study against the WWC research standards. These standards apply to the study methodology, the strength of the study’s data, and the adequacy of the study’s statistical procedures. We then summarize the findings of all studies that meet WWC standards, and develop an overall rating of effectiveness. In this way, WWC reports tell educators what the highest-quality research says about the effectiveness of individual education interventions. The WWC may be accessed online at http://ies.ed.gov/ncee/wwc/.

Want to learn more about the WWC?  Join Scott for an AEA Coffee Break Webinar, Thursday, May 13, 2:00-2:20 PM EST. He’ll demonstrate how to get the most out of the WWC website and locate important information for decision-making in education. Sign up at http://ow.ly/1IkXX

Want to learn more about the WWC’s standards for research?  Join Neil Seftor, Deputy Director of the WWC for an AEA Coffee Break Webinar Thursday, May 20, 2:00-2:20 PM EST. He’ll cover the design and reporting requirements researchers must demonstrate to meet WWC standards. Sign up at http://ow.ly/1IkXX

education, standards, webinar