Greetings from Alaska. I’m kas aruskevich, principal of Evaluation Research Associates (ERA), I work in rural Alaska with a great team of evaluators, associates, and local intermediaries. In the unique Alaskan context in which we work, telling the story through video helps us to show the context of people, place, and situations. Video clips, compiled into a video report, can be used as evidence of accomplishment as well as to educate an audience (often the funder) holistically about a project. Shorter impact videos can also motivate participants, giving the evaluation an effect beyond reporting.

Most of us have used written interview quotes in our evaluation reports. As example, below is a quote from an interview with a Gaalee’ya STEM project student:

Uvana atiga Nanuuraq (my name is Nanuuraq) I’m from a place called Noatak, my name is Brett James Kirk, 18 years old, incoming freshman at the University here in Fairbanks. So far what I know about STEM seems great. I really agree with how they’re incorporating the indigenous ways with the western ways here because we have a chance to talk about the similarities and differences between the two. And I’m looking forward to all the other meetings throughout the school year.

Compare the 40 second video clip below of the text quoted above. If the video does not show in your browser or email reader, go to https://vimeo.com/62366707 to view it on Vimeo

Gaalee’ya-AEA from kas aruskevich on Vimeo.

Lessons Learned – Generally:

1. Good audio is EXTREMLY important.
2. Shooting footage is easy, editing the video is challenging.
3. Editing is time consuming. One minute of finished video may take 8 or more hours of editing – and that’s after clips are selected and cut to approximate size.
4. Take good pictures. It easy to put motion to a photograph and use it as background to an audio quote taken from an interview.

A good evaluative video starts with data collection in the form of video and photos that gives evidence of accomplishment and provides visual description.

Lessons Learned – Taming the technology:
For the majority of video reports I work with a local videographer who has also mentored me in both camera use (Cannon 7D) and audio (Zoom H4n 4-Track Recorder). After three years of video production, I primarily stick to photographs and video editing (Final Cut Pro 7). I’ve produced video reports 20 minutes in length and less, however now I prefer to produce supplemental impact videos that are 3 minutes and less. Remember it’s technology, and with technology comes glitches.

Rad Resources to explore:

• Find a videographer near you, see if you can watch a shoot and then the editing process
• View impact videos (my alma matter): http://www.uhfoundation.org/impact-video
• Get good equipment: http://www.bhphotovideo.com

But most important, know how to conduct an appropriate evaluation, be reciprocal, gather good evidence, and report out. The rest is technology.

We’re focusing on video use in evaluation all this week, learning from colleagues using video in different aspects of their practice. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I am Caryn Mohr, a Research Scientist at Wilder Research in St. Paul. I’m one of more than 40 researchers in the office who conduct primary research. My own work focuses on education programs addressing opportunity and achievement gaps. Our office also manages Minnesota Compass, a nationally recognized community indicators project. Opportunities to collaborate with Compass staff have shown me the power of connecting primary research to community indicators data.

Those of us who conduct primary research gather new data first-hand. We use a variety of methods to evaluate the impact of individual programs and test research hypotheses. My own work ranges from case studies to long-term, quasi-experimental studies of education programs. We administer surveys, conduct interviews, convene focus groups, and employ a variety of methods that give us deep and direct knowledge of study participants’ experiences. We work closely with individual programs and organizations to help them understand their impact.

My colleagues at Minnesota Compass help us see the big picture. Compass provides a common framework for measuring and tracking state and local progress on a range of topics, including education, the economy, health, housing, and other important social issues. In each area, an advisory committee of stakeholders identified key indicators which are monitored over time to understand the health and progress of our community.

Lessons learned:

• See the big picture. Considering results of individual program evaluations in the context of community indicators provides a broader perspective and meaningful context to stakeholders. For example, results of a recent STEM (science, technology, engineering, and math) education program evaluation can be considered in the context of indicators of progress along the STEM cradle-to-career continuum. This context can help program staff consider their goals in relation to benchmarks and gaps pertinent to the continuum. Connections to indicators afford exploration of questions such as: How do program goals relate to research-based benchmarks and community needs? Are resources being targeted effectively?
• Identify themes. Considering study results in the context of community indicators can help researchers identify meaningful themes across individual program evaluations. In education, our community indicators show stark achievement gaps. It is important to consider what our first-hand knowledge of individual programs tells us about addressing these gaps, and how this relates to research literature and community needs. Moving the needle on indicators requires understanding how programs work on the ground. Likewise, effectively targeting program resources necessitates an understanding of community needs.

Hot tip:

Explore connections to community indicators to provide meaningful context to individual program evaluations.

Rad Resources:

• The Community Indicators Consortium provides links to community indicators projects and resources.
• Minnesota Compass Project Manager Craig Helmstetter offers tips and lessons for indicators work in this AEA365 post.
  

  (Share Clip)

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

community · indicators

Hello! We are Tarek Azzam (Claremont Graduate University) and Matt Keene (Environmental Protection Agency). We are members of the External Review Panel for ECLIPS.

Whoa! Even though it’s oh-so-tempting to try, you don’t need the perfect string of words to define “systems-oriented evaluation.”

Lesson Learned: It’s in the roots of evaluation 

At its core a systems approach to evaluation encourages the evaluator to consider the physical, political, and structural issues that surround a program, and to examine how these factors help or hinder the success of a program. This examination and reflection process is incorporated in the work and writings of lead evaluation scholars such as Lee Cronbach, Robert Stake, and Jennifer Greene. The presence of systems thinking also can be seen in our standards (specifically standards F3, A4, and A7).

And it’s also something different, because it requires the evaluator to recognize that the program is part of systems that have their own dynamics. It forces the evaluation to examine issues that go beyond the process and outcomes of a single program.

Lesson Learned: How to become a systems-oriented evaluator

1)      Adopt some habits of systems thinkers…

2)     Know the domains of Social Ecology and use them to understand the leverage points of boundaries, relationships, and perspectives. Donella Meadows says that boundaries are problem dependent and messy. Don’t make the world linear for your mathematical or administrative convenience.

3)     Delve into the dynamics of systems to find the regions of organized, adaptive, and unorganized patterns.

4)     Find leverage points (places to intervene where small tweaks can lead to big changes). Here are some leverage points from Meadows you can use to influence relevant systems:]

5)     Let systems thinking do fuzzy things to your logic model. A fuzzy logic model takes into account the dynamic nature of the systems surrounding a program. It gives a visual image of the complexities that can affect processes and outcomes.

In the ECLIPS, all of our logic models used to look like this:

But after applying systems thinking, we made them into fuzzy logic models. See how different they look.

Try creating a fuzzy logic model to find and depict the system’s complexity, making your logic model more useful to people for a longer time.

That’s all for now

This concludes ECLIPS week on aea365! Don’t expect to learn about systems alone or in a short period of time. It may well be a journey into a new way of thinking about evaluation. Get involved in an existing community of practice about systems or form your own group. ECLIPS members are happy to share what they are doing.

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! We are Karen Peterman (Karen Peterman Consulting, Co.) and Marah Moore (Director of i2i institute and ECLIPS coPI). We are going to talk about some of the implications of a systems approach for the role of the evaluator.

In a recent Thought Leaders Forum, Michael Patton reminded us that evaluation is a transdisciplinary field. Evaluators need expertise in evaluation theory, practice, methods and use, as well knowledge of theories of change and how to generalize what they have learned about patterns in effective interventions. A systems approach to evaluation can enhance evaluators’ work across each of these areas of expertise.

I (Karen) found the ECLIPS’ biggest impact on me was in how I view my overall role as an evaluator. And I (Marah) found that the evaluator’s role was revisited throughout the process of addressing each new systems concept. Here are some ideas we came away with:

Lesson Learned: Use your “systems-based evaluation expertise” to add value for clients.

Systems concepts can provide a valuable lens through which to view project evaluation findings. For example, you can help clients consider their individual projects within the context of academic literature, their institution’s larger mission, and/or their funders’ goals.

To guide these discussions, check out ZIPPER, A System-Based Evaluation Mnemonic and the Systems Archetypes.

Lesson Learned: Use a systems approach to push evaluation beyond the traditional stages.

The ECLIPS graphic below shows a traditional and a systems view of the stages of evaluation.

The systems view encourages the evaluator to work at the intersection of the traditional stages. The evaluator asks clients and participants to help shape:

• the evolution of the evaluation plan,
• the data collection procedures, and
• the interpretation of results.

Bringing together a systems orientation and participatory evaluation leads to an evolution in evaluation practice.

Hot Tip: Break down the silo approach to evaluation.

At AEA last fall, a number of presenters stated that the purpose of their work was to improve education. The systems perspective provides methods to start thinking about and achieving that goal on a broader scale. It enhances the thinking and work of evaluators by providing a framework and tools to move beyond the evaluation of the immediate project to start challenging clients (gently!) to think about system-level change and how their projects really can make a big-picture difference.

Read the final ECLIPS blog tomorrow where Tarek and Matt give an overview of systems- oriented evaluation and look at some fuzzy logic models.

Rad Resources:

• Karen’s presentation at the 2012 AEA conference:Blurring the Lines: The Process of Applying Systems Thinking to an Ongoing Evaluation.
• ECLIPS on the InSites website

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, we are David Reider (Education Design, Inc.), Ginger Fitzhugh (Evaluation and Research Associates), and Alyssa Na’im (Education Development Center, Inc.). As ECLIPS members, we are incorporating systems concepts into STEM education evaluations related to the National Science Foundation program, Innovative Technology Experiences for Students and Teachers (ITEST).

Keying off the iceberg diagram (see Monday’s post), we go deeper into a system to find leverage points for change by considering:

• boundaries (demarcations that define regions or entities)
• relationships (connections and exchanges between project parts or people)
• perspectives (the paradigms held by various parties and the purposes they seek)

Hot Tip: Don’t disregard the simple in a complex setting.

I (David) am evaluating a project using science probes and models in K-12 classrooms in four states. Although there were vastly different support structures in the sites, one of the lessons I learned in my evaluation was quite simple. The more frequently teachers posted to the online learning platform (thus reaching beyond the boundaries of their classroom), the higher the quality of their classroom projects. This was a case of leveraging small actions toward larger gains.

Hot Tip: Ask about boundaries, relationships and perspectives.

Questions That Matter has terrific examples of evaluation and interview questions that relate to boundaries, relationships and perspectives. I (Ginger) added several of these questions to our interview protocols for project leaders. For example, we added, “What, if any, unanticipated outcomes (positive or negative) have happened in the project thus far?” We learned that parents were interested in obtaining the project equipment to use with their children. This spurred the project team to consider how to make the materials more widely available.

Hot Tip: Consider how program goals can be leveraged.

I (Alyssa) am now paying more attention to acknowledging and identifying the boundaries, perspectives, and relationships for both program implementation and evaluation. Programs express their perspective through their statement of purpose, e.g., improving the nation’s STEM workforce development capacity. Systems thinking helps us understand why their strategies to accomplish this purpose may overlap or diverge and to see possible leverage points for change.

Hot Tip: Don’t lead with systems language; find ways to include it.

It’s not always necessary to say, “This is a systems idea.” Rather, use familiar language to explain how what you are doing adds value to the evaluation and the project.

Join us again tomorrow as we move away from the Iceberg diagram to consider the role of evaluators.

Rad Resources

• Ginger’s 2012 AEA presentation, Challenges and Successes Associated with Introducing Systems Concepts to an Existing Evaluation, is about applying boundaries, relationships, and perspectives to evaluations.
• David’s 2012 AEA presentation, Leverage Points: Places to Intervene in a System.
• ITEST LRC website

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! We are Lloyd Bingman (Brighter Day Consulting, LLC and evaluator of an NSF STEM project) and Pat Jessup (InSites associate and coPI for ECLIPS). We’re here to talk about the “Patterns” portion of theiceberg diagram displayed on Monday of ECLIPS week.

Both of us evaluate programs that are part of complex social systems that have multiple and dynamic patterns. An application of complexity theory provides us with a way to understand and distinguish among three patterns of system dynamics:

• Organized patterns: With high agreement and high certainty in the system, the patterns appear fairly predictable.
• Self-organizing: With a middle range of agreement and certainty, the patterns are adaptive.
• Unorganized patterns: With low agreement among key players and low certainty, patterns may not be present.

Creating a Robot Diagram to Understand Coordinator’s Role in a Complex System

When we talked about systems dynamics in the ECLIPS, I (Lloyd) discovered a systems lens for evaluating a project I was working on. I envisioned a key project leader superimposed on the diagram of systems dynamics. The leader is attempting to connect all of the project parts with their different dominant patterns of movement.

In this illustration, the “robot” figure is the IT Apprenticeship Coordinator. The Coordinator brings all the pieces together to meet program goals. The numbers on the robot correspond to different program goals.

At the bottom left (#1), the goal of presenting the apprenticeship program is controlled and organized; the Coordinator has direct control of presentations. The other goals on the left side (#3, 5) tend to be fairly predictable but key players do not agree as much on these goals as on the #1 goal.

On the right side of the robot, the activities related to the three goals are more self-organizing or unorganized. The  college recruitment process goal (#2), bottom right side, is less certain than #1 because the college competes with other colleges at recruitment events. Moving up the right side, the approaches to reaching the goals (#4, 6) are increasingly unorganized.

At the head of the diagram (#7), the Coordinator is constantly planning, assessing, and implementing activities to ensure project success.

Insights Gained from the Robot Diagram

This image provided new insight into the relationships among the parts and where changes could lead to a more networked flow of information. For example, by strengthening the relationship with the state liaison (#4), I (Lloyd) was able to gain data on the state workers’ experiences with the apprenticeship program.

Join us tomorrow when Dave, Ginger, and Alyssa talk about identifying leverage points for changing a system.

Rad Resources

• Using Complexity Science Concepts in Evaluation
• Designing initiative evaluation: A systems-oriented framework  

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! I’m David Hata, an independent consultant who serves as an external evaluator for a number of NSF-funded Advanced Technological Education (ATE) projects and centers.

I’m here to share my experience helping a STEM project see itself as part of a system that creates value. For some clients, success is doing a lot of activities. The clients may not be thinking about what value those activities create.

My early evaluations focused on evaluating individual activities using methodologies such as described in Kirkpatrick’ book Evaluating Training Programs: Four Levels. With my recent participation in ECLIPS and the Synergy Project, I have started to think of evaluation more holistically in terms of systems and value creation. I now view NSF ATE projects and centers as systems that create value at multiple places within their structure. By recognizing these value-creation systems, I am discovering more ways that evaluation can help fulfill the mission of the ATE program at NSF—to increase the number of and strengthen skills of technicians for the workforce through the implementation of workforce development initiatives.

Lesson Learned: Finding Value in the Interconnections  

A useful conceptual framework for assessing value creation has been developed by Etienne Wenger and colleagues.

They outline five types of value: immediate, potential, applied, realized, and reframing. I used the first four types of value to help develop a road map for The Southwest Center for Microsystems Education (SCME), an ATE regional center at the University of New Mexico in Albuquerque. Working with my client to create the road map helped them develop a systems view of their project. The road map shows the connections between grant activities and a career pathway that produces advanced technicians for the U.S. workforce.

The diagram below shows how SCME activities fit into a career pathway from high school to community college to workforce.

Center activities can be viewed as value investments:

• immediate value: knowledge, skills, and materials gained by each participant;
• potential value: what each teacher plans to do with their new knowledge, skills, and materials;
• applied value: what changed in classroom instruction and student learning;
• realized value: number of microsystem technicians produced based on graduate data.

The evaluation measures the value created by these investments at different points in the system. The links in the diagram emphasize the nature of the exchanges. Understanding the links between the boxes is as important as defining the activities and outcomes in the boxes.

Check out the following resource and join us tomorrow as my fellow ECLIPS members examine system patterns via a Robot diagram.

Rad Resource: Donella Meadows’ book, Thinking in Systems.

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings! We are Beverly Parsons (InSites and ECLIPS principal investigator) and Veronica Thomas (Howard University and ECLIPS external advisor). Today we introduce you to an exploratory research project, Evaluation Communities of Learning, Inquiry, and Practice about Systems (ECLIPS). It’s funded through a National Science Foundation grant to InSites.

The ECLIPS Community of Practice is 15 evaluators involved in STEM (science, technology, engineering, and math) education evaluations. During 18 months of webinars and annual meetings, ECLIPS members discussed and applied system concepts—especially systems dynamics and complex adaptive systems concepts—to their work. This week we share examples of our learning through this exploratory project.

Lessons Learned: A Systems-Oriented Evaluation?

We’re using a definition of a system from Meadows, Thinking in Systems: “A system is an interconnected set of elements that is coherently organized in a way that achieves something.”What the system achieves may or may not be what we want.

ECLIPS members identified the systems of relevance to their work. They looked at patterns within the systems, paying particular attention to culture and social justice – two areas that I (Veronica) kept in the forefront of the ECLIPS work.

In this type of systems-oriented evaluation, we pay attention not only to activities and results but also to patterns; norms, infrastructure, and policies; and paradigms. You’ll hear more this week about how ECLIPS members have used the iceberg diagram to go deeper in their thinking about systems.

Hot Tip:

Use a systems lens in your evaluation practice to:

• ask different kinds of evaluative questions, including questions that address social justice concerns (e.g., questions about access and opportunity)
• look for patterns that give clues about appropriate theories of change
• find leverage points (where small changes can create large effects),
• consider different roles as an evaluator (e.g., to include social change agent)

This Week with ECLIPS

Tuesday: Dave describes a systems view of a project in the form of a “road map” that shows where interconnections create value.

Wednesday: Lloyd and Pat provide an example of seeing and understanding patterns in systems.

Thursday: David, Ginger, and Alyssa discuss working with boundaries, relationships, and perspectives as leverage points for change.

Friday: Karen and Marah address how a systems orientation influences an evaluator’s role.

Saturday: Tarek and Matt give a wrap-up of systems concepts and connections to fuzzy logic models.

At another time, we’ll share our learning about how to use the systems concepts in more powerful ways than we could do in this exploratory project.

Rad Resources

• ZIPPER, A System-Based Evaluation Mnemonic
• Using Complexity Science Concepts in Evaluation
• Designing initiative evaluation: A systems-oriented framework
• Thomas, V.G. (2011). “Cultural issues in evaluation,” American Journal of Evaluation, 34 (4), 578-582.

The American Evaluation Association is celebrating this week with our colleagues involved in ECLIPS—Evaluation Communities of Learning, Inquiry, and Practice about Systems—and the AEA Systems in Evaluation Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Kim Kelly, PhD, from the Psychology Department at the University of Southern California where I teach undergraduate courses in statistics, research methods, psychobiology and human development. I have been involved in the evaluation of STEM (Science, Technology, Engineering and Mathematics) curriculum and professional development programs since 2002. These courses and projects are focused on improved student learning and span informal science settings, elementary, secondary, and post-secondary levels. I have come to appreciate, as I’m sure many of you do, the enormous influence of national curriculum efforts such as Common Core Standards and New Generation Science Standards as well as policy efforts to streamline and consolidate the funded STEM education portfolio across federal funding agencies.

Rad Resources: I really recommend National Research Council publication A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas to understand what is motivating the design of the Next Generation Science Standards

Like the Common Core Standards for Mathematics and English Language Arts they serve as a blueprint for states to follow in aligning their STEM education standards in the coming decade.

The National Science and Technology Council Committee committee on STEM Education has been initially charged to inventory Federal STEM education activities and develop a 5-year strategic Federal STEM education plan. In their most recent progress report, they discuss activities focused on evaluation guidance and common metrics and evidence standards for inclusion in the strategic plan.

The report also states that “an evaluation interagency working group will be created to support agency efforts to develop and carry out evaluation strategies.” One such group has already formed among evaluators of three climate change education programs funded by the National Science Foundation, the National Aeronautics and Space Administration, and the National Oceanic and Atmospheric Administration. This tri-agency evaluation working group has formulated a common logic model for the collective portfolio of climate education projects and is currently seeking feedback from the AEA membership as well as program officers of the agencies in identifying next steps in evolving a common evaluation framework consistent with the emerging federal strategic plan. Contact Committee Chair Ann Martin at ann.m.martin@nasa.gov to learn more and get involved in this timely effort.

Hot Tip: The Potent Presentations Initiative (p2i) is an AEA-sponsored effort to help evaluators improve their presentation skills. As you get ready to prepare a presentation for Evaluation 2013, visit the p2i website for ideas.

The American Evaluation Association is celebrating STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to aea365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

common core standards · STEM

My name is Disa Cornish, PhD. I am the Program Evaluation Manager at the Center for Social & Behavioral Research (CSBR) at the University of Northern Iowa (UNI). I coordinate the Iowa STEM Monitoring Project. The purpose of the Monitoring Project is to systematically observe a series of defined metrics and sources to examine changes regarding STEM education and economic development in Iowa. In particular, I work alongside the Governor’s STEM Advisory Council, a group of stakeholders in STEM education and economic development from across the state. The STEM Monitoring Project includes four primary components:

1) The Iowa STEM Indicators System (ISIS) to track publicly available data related K-16 STEM education and the STEM workforce pipeline;

2) A statewide survey of public attitudes toward STEM, to be conducted annually;

3) The statewide STEM student interest inventory added to the annual Iowa Assessments; and

4) Regional/Scale-Up Program process and outcomes data collection and analysis.

Lessons Learned: Collaboration is key. Evaluation of large-scale projects involves a lot of (rapidly) moving parts. When conducting evaluation of a statewide initiative, there are many strands to keep track of in terms of methods, sources of data, analysis, and dissemination strategies. The Iowa STEM Monitoring Project is a collaborative effort between partners at three different universities. We are all responsible for portions of the Monitoring Project and we are successful because of frequent, high-quality communication. In addition, I reached out to evaluators of other state STEM initiatives about their work. Having a network of supportive colleagues who were grappling with some of the same issues was very helpful.

Know the field. In order to know what indicators would be helpful in a statewide STEM monitoring project, we needed to know what national indicators were already being measured and tracked. What were other states doing

Rad Resource: With the rapid evolution of STEM evaluation (and STEM education programming), it’s important to stay current. STEMConnector.org is a fantastic resource. Their tagline is “the one stop shop for STEM information” and it’s quite true. There are state-by-state guides to STEM initiatives and programs, and news from the world of STEM education.

Change the Equation is an organization that works with the business community to improve STEM education. Their site has a wealth of information related to STEM education, including state-specific data. I especially like their Design Principles for Effective STEM Philanthropy and their Design Principles Rubric.

Hot Tip:

At Evaluation 2013, a new type of session will be offered. Birds of a Feather Gatherings (aka idea exchanges or networking tables) are a chance for attendees to share ideas and learn from one another. There is no formal presentation, but there is a designated facilitator to get the conversation started.

The American Evaluation Association is celebrating STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to aea365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

STEM