I’m Chad Green, Program Analyst at Loudoun County Public Schools in Ashburn, VA. For over seven years I’ve served as an internal evaluator of instructional initiatives sponsored by central office administrators.

Do you have an interest in understanding school-based professional development from a sociocultural learning perspective?  Read on! Years ago I evaluated two school-wide improvement initiatives using an integrated conceptual framework.  The first component was Learning Forward’s original context standards which today serve as its first three standards for professional learning. The purpose of this framework was to constrain the data to essential long-term staff development outcomes.  The second component (Honig, 2008) operationalized the first one into six overlapping sociocultural learning practices, two for each context standard (see below).

(Share Clip)

Framework for High-Quality, School-Based Professional Development

I.   Skillful leadership is evidenced when school and central office staff:

1. Model high quality teaching and learning practices
2. Boundary span to connect staff with new sources of expertise

II.  Professional learning communities are evidenced when school and central office staff:

1. Interact at a high level of collaborative inquiry
2. Engage in joint work on authentic tasks that are meaningful and sustained over time

III. Dedicated resources are evidenced when school and central office staff:

1. Provide access to ongoing, job-embedded learning opportunities that increase the level of participation in shared work practices (i.e., from novice to expert)
2. Develop common conceptual and practical tools (e.g., principles, frameworks, routines, language, protocols, templates, materials)

Lesson Learned:  The patterns that emerged from the data were surprising on two levels.  At a superficial level they revealed a continuum of leadership approaches to program implementation ranging from a top-down, hierarchical structure on one end to a more subtle, heterarchical structure on the other. Coincidentally, these leadership structures aligned with the level of diversity (i.e., complexity) of the school’s student populations.  At a deeper level, the findings suggested a connection between each school’s sources of power and knowledge (i.e., truth).  In the top-down structure, tacit knowledge was concentrated in the principal and specialist roles (i.e., authority) whereas in the heterarchical setting knowledge was more explicit in the form of online repositories of co-created tools and resources.

Hot Tip:  Since then, I have learned that I am much more effective when I help central office administrators integrate their prepackaged conceptual frameworks (i.e., programs) into coherent strategic thinking portfolios which facilitate increased experimentation and interconnectedness system-wide.

Rad Resource: Check out Honig’s journal article on district central office as learning organizations.

Final Word: Both schools’ staff development programs were equally effective in the short run with respect to implementation and outcomes.  Which school structure do you think will be more sustainable in the long run?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

education · Professional Development

My name is Anne Marshall, Director of Research and Evaluation for the Center for Collaborative Education. PreK-12 education and educational evaluation are a-buzz with phrases like “21st century skills,” “college and career ready,” and, of course, “Common Core State Standards (CCSS).” With these new changes, evaluators may be feeling the same initiative fatigue that many educators feel.  How do we ensure we have the knowledge needed to evaluate programs arising from this latest wave of education reform?  Fortunately, many useful resources exist to get us up to speed and informed.

Rad Resources: Getting Started: What underlies new frameworks and standards is a shift to learning content along with real-world skills that will allow students to apply knowledge in a world requiring constant innovation and problem solving.

Fablevision and the Partnership for 21st Century Skills demonstrate the thinking behind 21st century skills and college and career readiness in a short animated video, Above and Beyond.

Partnership for 21st Century Skills also provides resources, including P21 Common Core Toolkit.  While this resource is designed to help schools and districts implement CCSS, its descriptions of practices and how they align to 21st century skills and content standards can be invaluable for identifying best practices in programs we evaluate.

Look at CCSS website for materials specific to CCSS.

Hot Tip: Look up specific math or ELA standards by grade level and topic with a free Common Core Standards app from Mastery Connect.  The number of webinars on CCSS grows daily and a great one-stop-shopping source is iTunes U.  Their collection includes webinars by the National Governors Association & Council of Chief State School Officers, ASCD, and states’ departments of education.

Rad Resources: Because outcomes in PreK-12 evaluation often include data from state assessment systems, the two assessment consortia, Smarter Balanced and PARCC, will soon be dominant forces in our work.  Full implementation of the assessments is scheduled to begin in the 2014-2015 school year.  Websites for each of the consortia provide updates, sample items, etc. that can help us in thinking ahead about future evaluation work.

Lesson Learned: In talking to PreK-12 educators about the shift to 21st century skills and adoption of CCSS, I have most often encountered reactions of frustration and anxiety – key symptoms of initiative fatigue.  It is yet one more change or addition to what they must accomplish in their work and another change to how their students and programs will be evaluated.  Being mindful of this can strengthen evaluation work. In evaluating this new wave of education reform, we must pay attention not just to large-scale measurable outcomes on key standards, but also to systems and supports in place to assist this transition. 

(Share Clip)

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

21st century · common core standards · education

Hi! I’m Silvana Bialosiewicz, an advanced doctoral student at Claremont Graduate University (CGU) and Senior Research Associate at the Claremont Evaluation Center. My goal as an applied researcher is to help develop and disseminate “best-practices” for high-quality evaluation of programs that serve children. Today I’d like to share some strategies for collecting valid and reliable data from young children.

Research on youth-program evaluation and child development reveal that:

• Children less than nine years old possess limited abilities to accurately self-report, especially by way of written surveys
• Previously validated measures are not always appropriate for diverse samples of children

Therefore, a critical step in the process of designing evaluations of youth programs is the development and/or choosing of measures that are sensitive to children’s language skills, reading and writing abilities, and life experiences.

Hot Tip: Consider using alternatives to written surveys, such as interviews, when collecting data from children less than nine years old. If written surveys are used, be cognizant of young children’s inability to understand complex questions and accurately recall past experiences. Surveys for young children should be orally administered, use simple language, and use response options that children can easily understand.

Hot Tip: Do not assume that a measure, which has been demonstrated to be valid in a previous study, is appropriate for your participants, especially when the program serves a diverse population of children. The majority of psychological measures for children have been developed and normed on samples of high SES Caucasian children and cannot be assumed to be valid and reliably for diverse samples of children (i.e. English Language Learners, ethnic and cultural minorities, children with physical or sensory disabilities).

Hot Tip: Pilot test your measures, even previously validated measures, before launching full scale data collection to ensure developmental and contextual appropriateness.

Rad Resources: Researching with Children & Young People by Tisdall, Davis, & Gallagher and Through the Eyes of the Child: Obtaining Self-Reports from Children by La Greca are two great books for anyone looking to expand their knowledge on this topic.

Other AEA365 posts on this topic:

Susan Menkes on Constructing Developmentally Sensitive Questions 

Tiffany Berry on Using Developmental Psychology to Promote the Whole Child in Educational Evaluations

Krista Collins and Chad Green on Designing Evaluations with the Whole Child in Mind

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Children · Data · surveys

Hello. We are Susan Shebby and Sheila Arens, evaluators from Mid-continent Research for Education and Learning (McREL). In this post, we want to share our experience with a district working to implement a federal grant in two schools.

When the grant was awarded, it received a great deal of attention throughout the district and community. The funding was unprecedented, and the potential opportunities available to students and teachers were remarkable. However, initial excitement waned once the work began. Schools were slow to implement planned interventions and teachers were frustrated by grant goals. Moreover, no champions for the program emerged at the district level so the intervention was largely forgotten by the district—except during reporting and budget periods. Well before the end of the grant period, district- and school-level administrators lost sight of the grant goals and looked to the next grant or funding source for inspiration. Grant activities supported by teachers and community partners were terminated after the final grant performance period.

Lessons Learned:

The district requested a “case study” about development and implementation of large-scale grant initiatives that would yield recommendations for future initiatives in the district. Four themes repeatedly emerged as areas for improvement: communication and collaboration, leadership, consistency of policies and procedures, and sustainability. Evaluators help clients plan for implementation and sustainability by incorporating the following strategies into planning and delivery discussions.

• Communication and collaboration. Create a systematic process for collaboration and communication with staff at all levels of the district, as well as with outside partners. This communication should occur regularly and frequently from the inception of the grant to build awareness of grant activities and successes.
• Leadership. Create structures that support consistency in leadership for grant initiatives, provide clear reporting structures, and build the leadership capacity of existing personnel. Multiple changes in leadership—especially at the beginning of a grant award—were perceived as particularly harmful given the limited time available to demonstrate grant impact.
• Consistency of policies and procedures. Raise awareness of existing policies and procedures, and create structures that support coherence between these policies and procedures and grant initiatives. This might include involving key implementers and policymakers early in the planning process (i.e., during the grant writing process) to ensure they perceive grant goals as important and attainable.
• Sustainability. Create structures early in the grant cycle towards supporting programs after grant funding has concluded. In a five-year grant, meaningful discussions about sustainability should occur by Year 3 at the latest.

Rad Resources: These resources may be helpful as you work to support sustainability of initiatives.

• Sustainability: Does Evaluation Have a Role? is a suggested framework for how to support sustainability with evaluation
• Evaluation’s Role in Supporting Initiative Sustainability is about the role that evaluation can play in helping foundations support the sustainability of initiatives.

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

sustainability

Hi! I am Kelly Murphy, a Doctoral Candidate in Applied Developmental Psychology at Claremont Graduate University and a new Member-At-Large in the PreK-12 Educational Evaluation TIG. Over the past five years I have had the pleasure of working on an evaluation of a large multi-site out-of-school time (OST) program that serves over 15,000 K-12 students. Today I’m going to share some of the variables that I’ve come across that have improved my ability to sensitively measure program impact.

Hot Tip #1: We all know that sufficient participation in OST programs is essential for students to achieve desired outcomes, but what is the best way to measure student participation?  While cumulative days attended is the most commonly used approach, I have consistently found very interesting effects when I include other indices of program participation such as duration of participation (number of months attended) and intensity of participation (ratio of days attended to days enrolled) in my analyses. By including multiple indices of program participation we can get a clearer picture of students’ attendance patterns and enhance our understanding of how participation relates to student outcomes.

Hot Tip #2: As OST programs are beginning to offer a wider array of activities to students (e.g., tutoring, performing arts, sports, and leadership) it is important to understand how participation in these different activities relates to outcomes.

By measuring attendance by activity type we can learn whether participation in different activities leads to differential outcomes in students and this information can help us better align our outcome measures to the specific contexts of our programs.

Hot Tip #3: Multi-site OST programs usually serve a fairly large and heterogeneous population of students that have the potential to “dilute” program effects. To overcome this issue it is important to disaggregate data by important student and site characteristics. Characteristics that we have found to be key moderators of program effect are school level, district association (i.e., public or charter), grade level, and the reason students joined the program (i.e., self-joined or other joined).

Rad Resource #1: The Harvard Family Research Project has free publications and resources for OST program evaluators.

(Share Clip)

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

impact · OST

Hi! This is Shelly Engelman and Brandi Campbell with The Findings Group, LLC, a private evaluation firm in Atlanta, GA.

Evaluators typically implement pre/post surveys to assess programmatic impact on participants.  However, pre/post surveys are plagued by challenges:

1. Participants have difficulty responding to the “pre” survey items because they have little knowledge of the program content and choose to leave many items blank.

2. Participants feel overburdened with the “post” survey because they answered similar items on the “pre” survey and do not fill-out the “post” survey.

3. A participant is not present for either the “pre” or “post” survey, resulting in an incomplete data set for that individual.

4. Participants gain insights into program content and see it differently than at the beginning. Known as the Response Shift Bias, participants may overestimate their initial attitudes due to lack of knowledge at baseline; after the program, their deeper understanding affects their responses on the “post” survey.

Lesson Learned: Retrospective Results – Complete and Stable

Retrospective surveys ask participants to compare their attitudes before the program to after.  Because a participant completes a retrospective survey in one sitting, responses are more complete.  Not only is there a higher completion percentage with this method, but it also has been found to reduce the Response Shift Bias in participants.

Lesson Learned: The Utility of Retrospective Results

In several of our projects, the retrospective survey had advantages over the pre/post survey.  It yielded more complete datasets and higher response rates. On the other hand, because students complete the survey after the program, they may not accurately remember their attitudes before the program.  This is especially prevalent if the program occurs over several months.  Additionally, younger participants may have trouble navigating the retrospective survey format and may require additional assistance.

Contribute to the Practice of Retrospective Surveying

We appreciate that the evaluation community has more to learn about appropriate uses for retrospective surveys. To more fully understand the differences in true pre/post vs. retrospective pre/post approaches, The Findings Group is conducting pre surveys followed by retrospective pre/post surveys on a handful of programs.  We expect to measure the differences, if any, between the two “pre” response sets.  We invite you to do the same and share your results.  We could put together a panel presentation at AEA 2014!

Hot Tips: Implementing a Retrospective Survey

It is simple to rewrite pre-post survey items for a retrospective survey.

Pre/post survey: I am confident in my ability to solve computer science problems.

Retrospective pre-survey: Before this workshop, I was confident in my ability to solve computer science problems.

Retrospective post-survey: After this workshop, I am confident in my ability to solve computer science problems.

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PK12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

retrospective pretest · survey

Hello! My name is Tiffany Berry and I am a research associate professor at Claremont Graduate University, associate director of the Claremont Evaluation Center, and the chair for the PreK-12 Educational Evaluation TIG. Welcome to our 3^rd annual sponsored AEA365 week, in honor of Teacher Appreciation Week.

For those who are unfamiliar with our TIG, here is a brief summary of our TIG’s mission, vision and values. Please visit our website for the complete version, along with additional information about our TIG.

(Share Clip)

Mission: Raise the quality of educational evaluation.

Vision: Foster a close community of educational evaluators, become more responsive to context in education, and maintain high standards for educational evaluation practice.

Values: Relevant, responsive, high quality educational evaluation that reflects our beliefs in social justice, equity, and educating the whole child.

We asked our TIG members to identify important topics in educational evaluation and this week’s posts will reflect several of those topics. Today, I’ll begin our week by sharing a Rad Resource for educational evaluators—our TIG!

Our TIG has a Facebook Page, a Linkedin Page, and a Twitter Account (@PK12EvalTIG) where we share evaluation news, research, Rad Resources and Hot Tips year round!

Hot Tip #1: Connect with us on our social networking sites to stay up to date with current news in education and evaluation.  We also encourage our members to contribute articles, reports or headlines that impact educational evaluation.

Recent posts to our Facebook page include articles and discussion on:

• using student surveys for teacher evaluation
• multiple stakeholder perspectives
• Common Core State Standards

Recent discussions on our LinkedIn page include:

• sharing the types of work we do as educational evaluators
• our take-aways from the Evaluation 2012 conference
• our core values and how we apply them in our work.

Hot Tip #2: Our social networking sites can connect you with other educational evaluators who you can network, share ideas, and collaborate with.

education · educational evaluation

Hi there. I’m Corey Newhouse, the Founder and Principal of Public Profit, based in Oakland, California. We help public service organizations measure and manage what matters.

Lessons Learned – how we’re using video: Public Profit uses video in our evaluation of the Partnerships for Learning (PFL) initiative of the National Equity Project.

We use video as an elicitation technique in our teacher interviews. That’s a fancy way to say that we take video of PFL teachers in the fall, and then show the footage again to them during a follow-up interview in the spring. When teachers can see themselves in the classroom, they are able to be very specific about the ways in which their practice has changed while receiving coaching from PFL, and helps outsiders (including us!) better understand the changes they describe.

We work with a professional videographer to get high quality footage, and usually edit the video in-house. We have also taken our own video using inexpensive handheld cameras. If you chose to shoot your own footage, invest in a lapel microphone, as it makes a big difference in the quality of the footage.

Hot Tips – integrating video in evaluation: Enhancing subjects’ ability to recall their performance improves the quality of our interview data, particularly when we’re interested in such a complex phenomena as teaching.

Plus, we have video of teachers’ practice to share, improving the explanatory value of our reports.  Speaking of, here’s a clip of teachers’ practice, along with their reflections on their work with students. If you can’t view the video below in your browser or email, you can view it directly at: http://www.youtube.com/watch?v=Huw9IYY8hO4.

Lessons Learned – when and when not to use video: Take time to negotiate with your client – and their clients, if needed – about when videotaping will take place, and how it will be used. This will help to put your subjects at ease and assure that the shoot day runs more smoothly. A one-page description of your video project is a really useful conversation starter.

Hot Tip – taming the technology: It pays to plan when taking video! Collaborate with your team (including a videographer, if you have one) on a moment-to-moment calendar for your shoot day, including what kind of footage will be taken and when. While shooting, make notes to yourself about when particularly important moments were taped so that you can find the footage easily during editing.

Rad Resources: Video in Qualitative Research by Heath, Hindmarsh, and Luff, is an exceptionally helpful guide to those of us new to using video in evaluation.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Kim Kelly, PhD, from the Psychology Department at the University of Southern California where I teach undergraduate courses in statistics, research methods, psychobiology and human development. I have been involved in the evaluation of STEM (Science, Technology, Engineering and Mathematics) curriculum and professional development programs since 2002. These courses and projects are focused on improved student learning and span informal science settings, elementary, secondary, and post-secondary levels. I have come to appreciate, as I’m sure many of you do, the enormous influence of national curriculum efforts such as Common Core Standards and New Generation Science Standards as well as policy efforts to streamline and consolidate the funded STEM education portfolio across federal funding agencies.

Rad Resources: I really recommend National Research Council publication A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas to understand what is motivating the design of the Next Generation Science Standards

Like the Common Core Standards for Mathematics and English Language Arts they serve as a blueprint for states to follow in aligning their STEM education standards in the coming decade.

The National Science and Technology Council Committee committee on STEM Education has been initially charged to inventory Federal STEM education activities and develop a 5-year strategic Federal STEM education plan. In their most recent progress report, they discuss activities focused on evaluation guidance and common metrics and evidence standards for inclusion in the strategic plan.

The report also states that “an evaluation interagency working group will be created to support agency efforts to develop and carry out evaluation strategies.” One such group has already formed among evaluators of three climate change education programs funded by the National Science Foundation, the National Aeronautics and Space Administration, and the National Oceanic and Atmospheric Administration. This tri-agency evaluation working group has formulated a common logic model for the collective portfolio of climate education projects and is currently seeking feedback from the AEA membership as well as program officers of the agencies in identifying next steps in evolving a common evaluation framework consistent with the emerging federal strategic plan. Contact Committee Chair Ann Martin at ann.m.martin@nasa.gov to learn more and get involved in this timely effort.

Hot Tip: The Potent Presentations Initiative (p2i) is an AEA-sponsored effort to help evaluators improve their presentation skills. As you get ready to prepare a presentation for Evaluation 2013, visit the p2i website for ideas.

The American Evaluation Association is celebrating STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to aea365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

common core standards · STEM

My name is Disa Cornish, PhD. I am the Program Evaluation Manager at the Center for Social & Behavioral Research (CSBR) at the University of Northern Iowa (UNI). I coordinate the Iowa STEM Monitoring Project. The purpose of the Monitoring Project is to systematically observe a series of defined metrics and sources to examine changes regarding STEM education and economic development in Iowa. In particular, I work alongside the Governor’s STEM Advisory Council, a group of stakeholders in STEM education and economic development from across the state. The STEM Monitoring Project includes four primary components:

1) The Iowa STEM Indicators System (ISIS) to track publicly available data related K-16 STEM education and the STEM workforce pipeline;

2) A statewide survey of public attitudes toward STEM, to be conducted annually;

3) The statewide STEM student interest inventory added to the annual Iowa Assessments; and

4) Regional/Scale-Up Program process and outcomes data collection and analysis.

Lessons Learned: Collaboration is key. Evaluation of large-scale projects involves a lot of (rapidly) moving parts. When conducting evaluation of a statewide initiative, there are many strands to keep track of in terms of methods, sources of data, analysis, and dissemination strategies. The Iowa STEM Monitoring Project is a collaborative effort between partners at three different universities. We are all responsible for portions of the Monitoring Project and we are successful because of frequent, high-quality communication. In addition, I reached out to evaluators of other state STEM initiatives about their work. Having a network of supportive colleagues who were grappling with some of the same issues was very helpful.

Know the field. In order to know what indicators would be helpful in a statewide STEM monitoring project, we needed to know what national indicators were already being measured and tracked. What were other states doing

Rad Resource: With the rapid evolution of STEM evaluation (and STEM education programming), it’s important to stay current. STEMConnector.org is a fantastic resource. Their tagline is “the one stop shop for STEM information” and it’s quite true. There are state-by-state guides to STEM initiatives and programs, and news from the world of STEM education.

Change the Equation is an organization that works with the business community to improve STEM education. Their site has a wealth of information related to STEM education, including state-specific data. I especially like their Design Principles for Effective STEM Philanthropy and their Design Principles Rubric.

Hot Tip:

At Evaluation 2013, a new type of session will be offered. Birds of a Feather Gatherings (aka idea exchanges or networking tables) are a chance for attendees to share ideas and learn from one another. There is no formal presentation, but there is a designated facilitator to get the conversation started.

The American Evaluation Association is celebrating STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to aea365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

STEM