My name is Linda Meyer and I am an applied developmental psychology graduate student at Claremont Graduate University. I am going to provide you with a few basic tips to keep in mind as you are planning an educational evaluation with English Language Learners (ELLs) in PK-12.

Hot Tip: Find out how students were labeled as English Learners. Most students get labeled through a similar process: a home survey sent to parents upon enrollment, followed by a standardized test on English language development, and a score on that test that falls below what has been designated “proficient.” However, policies are often applied inconsistently and inappropriately. These tests may be inappropriate for young students and the school’s criteria may over- or under-identify English Learners. These students then receive inappropriate instruction that negatively impact their academic performance and development. Lawsuits against various states (see Arizona) demonstrate the widespread problems that stem from inappropriate identification and instruction. Before you evaluate the quality of instruction, make sure students are correctly classified and that the instruction matches students’ abilities.

Hot Tip: Consider English Learners’ ages and language exposure. Children at the elementary level are more likely to develop proficiency in a second language than older students because language acquisition ability trails off at about age 12. Students who fail to sufficiently develop English as a second language after a period of years are designated Long-term English Learners and are at higher risk of dropping out of school, among other things. Students who are not exposed often enough to their second language are less likely to become proficient in that language. Consider how often students hear and use English each day. Higher exposure at earlier ages increases the likelihood that they will become English proficient.

Hot Tip: Match your methods to the instruction type. Some instruction types emphasize learning English as quickly as possible, sometimes at the cost of grade level content instruction time. Others emphasize keeping up with the grade level content in whichever language is more proficient, with some moving students toward English-only instruction over time and others moving students toward proficiency in both languages for all content. Each has its strengths and weaknesses, so match your evaluation questions and methods to the instruction type to obtain the most relevant data.

Rad Resource: A comprehensive national study on English Learners’ academic achievement that investigated various instruction types.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

educational evaluation · English Language Learners

My name is Susan Kistler and I am the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor.

Last week, I saw the new Star Trek movie (yes, my inner geek shines through once again). Part of its premise was that a team had been asleep for years, only to awake and wreak havoc on the Star Trek crew. You would never do that, you would draw on the past to improve the future of course because evaluators are a good lot. That is why today I’m sharing information about the Wayback Machine.

Rad Resource – The Wayback Machine: The Wayback Machine allows you to look at the history of a website over time. You visit the Wayback webpage, enter the URL for the site in which you are interested, and you can see how the site looked historically, with most of the formatting and the links intact.

Clipped from: archive.org (share this clip)

Here’s a screenshot of the AEA website in February of this year:

And in August of 2004:

And in December of 1998:

If you are were exploring it on the Wayback Machine website, you’d be able to try out most of the links and read the content of each page.

Why is it useful? This is a great way to do background research on your evaluand, to understand issues of explore change over time as reflected in their web presence. If your evaluation includes an examination of your evaluand’s communications, the Wayback machine can also help you see how those communications played out online.

There are a few caveats – formatting is better for some sites than for others, and you can’t see a snapshot of every day in history for a site, only at times when the web crawler used by the Internet Archive visited the site.

The Wayback Machine takes all of 2 minutes to use and explore. And, its even a bit of fun, in particular if you are an historian at heart.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association.

No tags

My name is Tanya Ostrogorsky, Assistant Vice Provost for Assessment and Evaluation at Oregon Health & Science University, and I’ve been involved with Oregon Program Evaluators Network since 2002.  I ‘grew up’ studying research methods and data analysis and looking back I was a functioning as an evaluator before I knew what that meant. It wasn’t until my doctoral program that I took my first program evaluation course and attended an OPEN conference. Since then I’ve held leadership positions on seven different occasions including a long stretch as OPEN President during a difficult time in the organization’s history.

The purpose of this post is not to tell you about my trajectory as a local affiliate leader, but to share lessons learned through my observations about the role and function of the local affiliates in supporting AEAs mission. I also want to remind us how critical the local affiliates are to the development of local talent as well as the national leadership pipeline. Finally, I want to highlight the under-realized sources of energy, excitement, and real diversity that are in our midst.

Recently, 126 conference attendees ranging from students to newly minted graduates to early careerist to long-timers gathered to hear about the Top 10 Trends in Evaluation with Dr. Stewart Donaldson.  My first reaction to that day was a strong sense of pride in watching a local affiliate consistently deliver significant professional development opportunities for 16 years. My second reaction, as I scanned the room, was on the diverse and exciting mix of attendees that represent our past, our present, and our future.

So, what’s my point? Just as AEA needs to leverage and develop the local affiliates, past local affiliate leaders need to ensure the next generations of evaluators are provided the organizational history and encouragement to pick up where we left off. In both cases, we have a professional responsibility to support and encourage our peers in taking the next step in their leadership development. We need to offer encouragement and harness their energy. Yes, they will stumble and they will re-create the wheel, but so did we.

Lesson learned: We must leverage the talents and energy of the local affiliates to develop the leadership pipeline needs. My hope is that AEA can bring their focus to the power of local affiliates to create a strong organizational legacy. At the same time, it is local affiliate leadership responsibility to ensure that we do our part and have a strong community to support AEA.

Hot Tip: Local/regional AEA affiliates offer many opportunities to build our evaluation community. Find yours here and take the next step!

(Share Clip)

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · Local Affiliate

I am Elizabeth O’Neill, Program Evaluator for Oregon’s State Unit on Aging and President-Elect for the Oregon Program Evaluators Network. I found myself on this unlikely route as an evaluator starting as a nonprofit program manager. As I witnessed the amazing dedication for producing community-based work, I wanted to know that the effort was substantiated. By examining institutional beliefs that a program was “helping” intended recipients, I found my way as a program evaluator and performance auditor for state government.  I wanted to share my thoughts on the seemingly oxymoronic angle I take to convince colleagues that we do not need evaluation, at least not for every part of service delivery.

In the last few years, I have found tremendous enthusiasm in the government sector for demonstrating progress towards protecting our most vulnerable citizens. As evaluation moves closer to program design, I now develop logic models as the grant is written rather than when the final report is due. Much of my work involves leading stakeholders in conversations to operationalize their hypotheses about theories of change. I draw extensively from a previous OPEN conference keynote presenter, Michael Quinn Patton, and his work on utilization-focused evaluation strategies to ensure evaluation is intended use by intended users. So you think I would thrilled to hear the oft-mentioned workgroup battle cry that “we need more metrics.”  Instead, I have found this idea to warrant more naval-gazing and less meaningful action.  I have noticed how metrics can be developed to quantify that work got done, rather than to measure the impact of our work.

Lesson Learned: The excitement about using metrics stems from wanting to substantiate our efforts and to feel accomplished with our day-to-day to activities. While process outcomes can be useful to monitor, the emphasis has to remain on long-term client outcomes.

Lesson Learned: As metrics become common parlance, evaluators can help move performance measurement to performance management so the data can reveal strategies for continuous improvement. I really like OPEN’s founder Mike Hendricks’ work in this area.

Lesson Learned: As we experience this exciting cultural shift to relying more and more on evaluation results, we need to have cogent ways to separate program monitoring, quality assurance and program evaluation.  There are times when measuring the number of times a workgroup convened may be needed for specific grant requirements, but we can’t lose sight of why the workgroup was convened in the first place.

Rad Resource: Stewart Donaldson with the Claremont Graduate Institute spoke at OPEN’s annual conference this year with spectacular response. Program Theory-Driven Evaluation Science: Strategies and Applications by Dr. Donaldson is a great book for evaluating program impact.

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

metrics · Theory of Change · theory-driven · utilization-focused evaluation

My name is Kelly Smith, and I’m an evaluator, policy analyst and economist at ECONorthwest, a consulting firm in Portland, Oregon. I’ve been a devoted member of OPEN (Oregon Program Evaluators Network), an AEA affiliate, since I was introduced to it in graduate school in 2007. OPEN works hard to provide our members with a variety of learning opportunities, including brown bag lunch talks, workshops, networking events, conferences, and a book club. We do this mostly by tapping into our own members’ expertise. Until recently, I’d been instrumental in planning these events, and I’d certainly been an enthusiastic participant, but I hadn’t yet felt comfortable leading one.

This past year, I overcame my reticence and agreed to give a talk about cost analysis, a topic that seems to raise fear and confusion, if not hackles, among many evaluators. The event was free and the room was packed, far exceeding our expectations. Evidently, people were hungry for knowledge about this subject. Some attendees even expressed surprise that we didn’t charge for such a valuable class (lesson learned!). Having sensed a large untapped demand, we decided to offer two sessions about cost analysis at our annual conference in the spring, and both were well attended. We got great feedback about the usefulness of this topic.

These sessions were my first experiences “teaching”, and I found it both intellectually challenging and valuable (and if I’m honest, a bit nerve-wracking). I had to study the topic to refresh my knowledge, practice public speaking and presentation skills, and think on my feet. I’m confident that I got more out the experience than the attendees.

Get involved! If you’re a member of your local affiliate, don’t hesitate to step up and contribute your own knowledge! There is a real demand for learning opportunities, and certainly not enough supply. Your colleagues will benefit from your willingness to share, your affiliate will become stronger and more active, and you’ll come out ahead, too.

It’s hard to understate the value of learning from each other as evaluators. Not only do we grow as professionals when we contribute to our colleagues and our affiliate, but we expand the value and reach of the evaluation profession as a whole.

Rad Resource: If you are interested in learning more about cost analysis in evaluation, you’ll find a copy of the slides used at the conference sessions here.

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

cost analysis

My name is Adrienne Zell and I work as an internal evaluator for the Oregon Clinical and Translational Research Institute, an organization that provides services to biomedical researchers at Oregon Health and Science University. I also volunteer as the executive director of a small nonprofit, Impactivism, which provides evaluation advice to community organizations. I have been a member of OPEN for over 10 years, and involved with the events committee for the past three years.

Many years ago, I was lent a collection of essays entitled, Yoga for People Who Can’t Be Bothered to Do It. Geoff Dyer’s essays are first-rate — humorous, amorous, and reflective – but it is his brilliant title that has stuck with me. Although OPEN members and volunteers have diverse roles within the field of evaluation, a common theme in our events and conversations has been the effort involved in convincing organizational leadership, staff, and stakeholders that evaluation is worth doing and that they should have a direct role.

This past year, one of our members, Chari Smith, successfully organized an OPEN event and a conference workshop designed to planfully connect evaluators and nonprofit staff and engage them in thinking about reasons why organizations may not “do” evaluation. As evaluators, we rarely can remove all identified barriers. But we can work to understand their complexity and re-focus on opportunities. Participation in OPEN, along with my experience as both an external and an internal evaluator, has inspired a list of tips on addressing evaluation gridlock in organizations and just “doing” it.

Hot Tip #1: Highlight current capacity. Most organizations are already practicing evaluation; they just aren’t using the term. They may collect data on clients, distribute feedback forms, maintain resource guides, or engage in other evaluation-related activities. Identifying and leveraging current accomplishments inspires confidence and makes evaluation seem less forbidding.

Hot Tip #2: Appeal to accountability. Program leaders, by definition, should be held accountable for program impact. The most recent issue of New Directions for Evaluation compares and contrasts the fields of performance management and evaluation. Program managers should regularly request and utilize both kinds of information when making decisions. Elements of these comparisons can be shared with program leadership, increasing understanding about the differences, commonalities, and benefits.

Hot Tip #3: Show them the money. Provide examples of how rigorous impact evaluation can result in stronger grant applications and increased funding. A recent EvalTalk post solicited such an example, and members were responsive. In addition, return on investment (ROI) and other cost analyses (see tomorrow’s post by OPEN member Kelly Smith) can demonstrate savings, inform resource allocation, and target areas for future investment.  A single ROI figure can “go viral” and motivate further evaluation work.

(Share Clip)

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

accountability · impact evaluation

Hi. We are Erin Stack and Lindsey Patterson. Erin is a Community Psychology doctoral student at Portland State University (PSU) and the current student liaison for the Oregon Program Evaluation Network (OPEN). Lindsey is a former OPEN student liaison and a soon-to-be graduate of the PSU Community Psychology program. Lindsey currently works at the Center for Improvement of Child and Family Services. Since Lindsey just transitioned into the workplace and Erin is about to, we thought it would be helpful to other transitioning students and new evaluators to share what we’ve learned about effectively transitioning from student to professional.

Hot Tip # 1: Identify and work with mentors. At PSU, we work closely with faculty advisors in a mutually beneficial capacity.  We have also found it helpful to develop relationships with other students and professionals within and outside of our program. These mentorships have resulted in research methods book clubs to help stay up-to-date on current statistical trends, publication and internship opportunities, and skills to navigate the ups and downs of graduate school and career preparation.

Hot Tip # 2: Network. Relatedly, we have found it helpful to always expand our professional networks by attending happy hours and social events sponsored by a variety of organizations (including OPEN). As a student, networking can be intimidating, but with an increasingly competitive job market, it is important to build relationships with hard-working contributors in the field that could one day be colleagues or collaborators.

Hot Tip # 3: Attend conferences. Conferences provide opportunities to learn about current happenings in the field and about career trajectories in evaluation, and to develop meaningful professional relationships through networking. Conferences have the built-in perk of providing the topics of conversation for you! For example, at the OPEN conference this past March in Portland, Oregon, Stewart Donaldson kicked us off with what he believed were the future trends in Program Evaluation. This lends us an exciting and easy topic to debate and to continue generating ideas with other conference attendees.

Hot Tip # 4: Get involved in local associations. We both volunteer for OPEN. This has been helpful for cultivating relationships with evaluators in the area, identifying diverse career options related to Program Evaluation, and work with a group of individuals who share a similar passion.

Rad Resource:  Early Career Listservs and Topical Interest Groups. Some organizations have dedicated listservs and/or topical interest groups that graduate students and new evaluators can join.  For example, the Society for Community Research and Action (SCRA) has created a listserv for early career individuals.  These resources can provide additional connections, job postings, and new or innovative ideas related to career opportunities.

(Share Clip)

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Local Affiliate · TIG

My name is Kim Firth Leonard, and I have the honor of authoring the first post on the aea365 blog for Oregon Program Evaluators Network (OPEN) week. I have been an AEA member since 2008, and am currently President of OPEN, a local affiliate of AEA founded in 1997. I work as Assessment Research Coordinator at Marylhurst University in Portland Oregon and do contract work in program evaluation via Leonard Research and Evaluation, LLC.

This week’s posts were by OPEN members who have played important volunteer and leadership roles for OPEN. The posts demonstrate the value of our local network by sharing lessons we’ve gathered in reflecting on our work together as evaluators and as volunteers with OPEN.

I have learned much about evaluation and about building learning communities through OPEN. The bulk of the work done by OPEN’s volunteer Council and Committees is in organizing and supporting local events. OPEN’s mission is to provide a regional, interdisciplinary forum for professional development, networking, and exchange of practical, methodological, and theoretical knowledge in the field of evaluation. It is through these events that we build learning communities, and in doing so strengthen our work individually, and as a field.  

Get Involved: Whether you have a local affiliate or just an informal network of other evaluators in your area, you too can host, lead, contribute to, or benefit from local evaluation events.

• Host: Events don’t have to be massive undertakings to be successful. Small, informal gatherings can be just as valuable as large conferences. “Have an idea? Go for it” is practically our events committee motto.
• Lead: Local events are great places to practice your presentation and training skills. Discussion groups, like OPEN’s new-ish Book Club are low-pressure and offer opportunities to discuss emerging topics.
• Contribute: Volunteer to help organize events for unique networking opportunities. Learning event planning skills is icing on the cake.
• Benefit: It’s all about learning together. Valuable learning about one another and the field can happen at any get together – so attend local events whenever you’re able.

Lesson Learned: OPEN has always been welcoming to community members who don’t identify as evaluators, exactly, but do related work or want to learn more about evaluation. In the last year or so we’ve been emphasizing this openness (ha!) and we’ve found that collaborating with and learning from others in related fields greatly enriches our evaluation learning community. Sessions at our recent conference intended to create opportunity to learn from and with others in our community, including non-profit leaders, were well received.

Rad Resource: Materials from our 2013 conference are available on our website.

Rad Resource: Your own learning community is at your local affiliate or among other local AEA members.

(Share Clip)

The American Evaluation Association is celebrating Oregon Program Evaluators Network (OPEN) Affiliate Week. The contributions all this week to aea365 come from OPEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · Local Affiliate

My name is Susan Kistler and I am the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor.

Rad Resource – Juice Analytics Blog: The staff at Juice Analytics puts out one of my favorite blogs focusing on data use, analysis, and visualization. Their tag line alone “Your data is meant for action” is enough to get my vote.

Lessons Learned – Everyday Visualizations and Everyday Evaluations: This week, Ken Hilburn wrote about “Everyday Visualizations” – the laundry on the line that tells you that rain isn’t predicted for today, or the indicators on your laptop that tell you when your battery is about to die.

When I teach evaluation, I usually begin by noting that we are all evaluators. We collect data, analyze that information, and make decisions based on it. We read movie reviews and choose a show to see, peruse the meat counter at the grocery and perform a quick cost-benefit analysis and put down the tenderloin, and select a mate based on what might be seen as extensive interviews and evidentiary analysis. Professional evaluators merely increase the scope and systematization of the processes and apply them in situations with broader implications.

Lessons Learned – Identifying Life’s Key Indicators: My family lives at a boarding school where my husband chairs the science department, and (barring the extremely unexpected) from which my oldest daughter Emily will graduate this afternoon

As I sat through a baccalaureate service last night, my mind wandered to the mental calculus of the success criteria for the situation. She completed the program. Check. Did well, graduating cum laude. Check. Got into college to study computer engineering. Check. These are all indeed successes that have made me very proud.

Yet success over the past 18 years is far more difficult to quantify. A new tradition here is that, as a child of a faculty member, my husband will hand my daughter her diploma. To plan ahead, he asked “hug or handshake” – hug of course she exclaimed, excited at the prospect. I sat next to the headmaster’s wife at the service. She noted that Emily looked beautiful at the prom, no small feat for a child who had seen major health challenges. Earlier in the day, a friend had commented that Emily positively beams when standing beside her longtime boyfriend. Yes, said I, she is in love.

Ultimately, are these not what matter? Success is made manifest in health and happiness, confidence that you are loved and the capacity to love with others.

As I change my own work situation, I am setting goals for myself, looking at what should be my own key indicators of success, personally and professionally, in the coming years. If you have experience to share, I’d love to learn from you.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I am Paula Egelson from the Southern Regional Education Board. Much of the work I have done over the past 15 years has included formative assessment. Formative assessments focus on assessing students to improve learning and instruction. Below are snapshots of teachers who use formative assessment effectively.

“Lydia” is a Head Start teacher in a large northeastern city who teaches in a four-year-old classroom. Half of Lydia’s students have individual education plans and a majority of her students speak Spanish as a first language.

Lydia assesses her students frequently to learn whether students “get it” and to help guide instruction. The different developmental trajectories of her students means there is much individualized assessment. She uses flashcards to assess number recognition or letter sound pronunciation. Lydia does lots of informal questioning to determine whether students understand concepts. Lydia remarks, “I have to know my students.  I need to know how far I can take them.”

“Sutton” teaches a self-contained 5th grade honors class at a rural minority middle school. The science program Sutton uses includes labs. The students do interactive science note-booking. This includes developing focus questions, making predictions, observations, and reflections, vocabulary, and providing evidence collaboratively. In lab groups students are assessed on rubric concerning their engagement. Sutton checks a sample of student notebooks at night to assess understanding and mastery. He then has conferences with students about their notebooks the next day.

For math, Sutton’s students are placed in cooperative groups for instruction. Students must explain how they get their answers and learn different ways to reflect. Math instruction is in the morning; however, students play math games and use the Smartboard in the afternoon to address misconceptions. His students are allowed to redo work and correct mistakes. In addition, students self-assess by learning to read graphs about their own academic progress.

Denise teaches physical science at a minority high school. District policies encouraged Denise to try formative assessment. Denise uses the results of chapter pretests to guide her instruction. Some of Denise’s students have meager science vocabularies and struggle with the math. Denise often asks her students, “What do you think you know?” Once students respond, Denise knows where to start teaching or remediating struggling students. There are also benchmark tests and project rubrics used formatively.

Hot Tip: Formative assessment takes on many forms at all grade levels, and any evaluation of school improvement should include finding ways to capture formative assessment activities.

RAD Resource: See Improving Formative Assessment Practice to Empower Student Learning by E. Carolin Wylie, et al. for many examples of how to incorporate and improve formative assessment activities.

The American Evaluation Association is celebrating Consortium for Research on Educational Assessment and Teaching (CREATE) week. The contributions all this week to aea365 come from members of CREATE. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

education · formative assessment