AEA365 | A Tip-a-Day by and for Evaluators

CAT | Uncategorized

Hello AEA members! Sheila B Robinson, aea365’s Lead Curator and sometimes Saturday contributor with an important message about the AEA elections. The  ballot is now open for our 2019 President-elect and Members-at-Large of AEA’s Board of Directors.

Unfortunately, AEA member voting has been historically low. Let’s change that this year!

Hot Tip: Don’t recognize some of the names? Get to know your candidates and cast your vote! Check out candidate bios here so you can familiarize yourself with their backgrounds and their visions for AEA. These candidates are accomplished leaders not only in AEA but in their own fields as well. Do your due diligence and give them each a few minutes of your time to learn about what they have to offer our association. They’ve worked  hard to get themselves on the ballot through a rigorous nomination process.

Many of us consider AEA to be our professional home and leadership is important to who we are as an association and what we have the potential to become.

Rad Resources: Our 2019 nominees (in alphabetical order in each category):

Image credit: tzu kwan valino via Flickr

Image credit: tzu kwan valino via Flickr

President-elect (we vote for 1):
  • Thomas Grayson
  • Aimee White
Member-at-Large (we vote for up to 3):
  • Lisa Aponte-Soto
  • Thomas Archibald
  • David Bernstein
  • Hanife Cakici
  • Wanda Casillas
  •  Tom Kelly

Rad Resources: Our candidates’ bios. Each has taken considerable time and effort to prepare their statements. Won’t you give them your time to read them? Once you’ve done that, voting takes just seconds to complete.

Hot Tip: Please check your recent email from elections@vote-now.com with information on the election and a link to your ballot. You may vote at any time between now and 11:59 PM Eastern Time on Monday August 6. Reminder notices have been sent to those who have not yet cast their ballot.

Email info@eval.org or call the AEA office at any time with questions or concerns related to the ballot or any of your member benefits.

Get Involved: Play a role in this year’s election and make your voice heard. Let’s ROCK.THIS.VOTE!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, my name is Jan Noga. I’m the owner of Pathfinder Evaluation and Consulting in Cincinnati. I’m excited to see the conference here in Ohio; although, living in the opposite corner of the state, it’s going to take me longer to drive to Cleveland than it would to fly to DC!

The conference is exciting. But it can also be daunting, for those new to evaluation and the annual meeting, and stale, for those veterans of many years of conference attendance. So, novice or veteran, here’s a strategy for negotiating the conference using systems thinking.

Systems thinking is a way of understanding the world that emphasizes the whole and how its individual parts contribute. It is concerned with:

  • The interrelationships among the system’s parts and their relationship to the whole,
  • The multiple perspectives that are part of the system, drive what happens within the system, and influence how actors and elements experience the system
  • The boundaries that define the system. 

Cool Trick: Evaluation 2018 from a systems thinking point of view:

  • Interrelationships: Think of the conference as many dances being danced at the same time. If you’re new, what dance do you pick? Go to the orientation provided by AEA to learn the basic steps. Find a TIG that interests you and attend the business meeting. Attend a workshop to get a start on some familiar faces to “dance” with during the conference. You’ll be weaving your own pattern in no time. But you’re not new, you say? The conference is getting old? Change the dance! Get out of your routine – if you usually go to paper sessions, go to think tanks. Skip some sessions and get coffee with someone you’ve always wanted to talk with. Weave a new pattern for the conference.
  • Perspective: Novice or veteran, each of us brings a perspective around timing, quality, and “appropriate” type of session, among others, that will influence how you engage with the conference and whether it meets your expectations. Challenge your perspectives. Understand where you are coming from but seek out other perspectives and try to understand them as well. You’ll be surprised what you learn about yourself and your fellow evaluators.
  • Boundaries: NO ONE CAN DO IT ALL. There are boundaries of scheduling and energy that we all struggle with. You don’t have to attend a session during every time block if you don’t want to. There are also boundaries of interest and purpose – what are you here for? Rethink the boundaries you create for yourself to redefine how you engage.

Rad Resource: Want to learn more about systems thinking? There are always a number of workshops and sessions sponsored by the Systems in Evaluation TIG at the conference that you can take advantage of. Or, try the TIG’s website: www.systemsinevaluation.com.

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

·

In the spirit of my six year old’s joy of watching kids YouTube, Hey Guys! This is Michelle Davis, from Cleveland, Ohio!  I am the Assistant Director of Tremont West Development Corporation and am excited that your conference is visiting our neighborhood.  Tremont West is a nonprofit serving the 7,500+ residents of the Tremont, Duck Island and Irishtown Bend neighborhoods of Cleveland and our mission is to serve Tremont by organizing an inclusive community, building a unified neighborhood, and promoting a unique destination.

Lessons Learned: The pride and joy of our work is our outreach and engagement with the community.  We have systematically divided the neighborhood into 11 different areas we call block club areas.  On staff, we have two community organizers who work with resident leaders to plan agendas and host monthly meetings.  Since I began working at Tremont West fifteen years ago, these loosely run gatherings have formed into formalized groups with official bylaws, co-chairs, secretaries and attendance requirements for voting on decisions that are made at their meetings.  Our community engagement is essential to our work and it assists in in planning for the future of Tremont.  Our vision is that Tremont is the ideal urban village, led by a growing population of dedicated people, filled with desirable amenities and essential services, and welcoming to all.

Hot Tips: Tremont is a fun place to visit and explore.  We claim to have more historic churches in one square mile than anywhere in the country!  Lincoln Park serves as our town square and place of many gatherings throughout the spring, summer and fall!  Here are a few places to check out during your visit:

  • A Christmas Story House & Museum – explore the property where Ralphie shot his eye out and the Bumpuses’ Hounds ate the Christmas Turkey! There is a great lunch spot, The Rowley Inn, right on the corner across from the house!
  • Tremont Tap House – Cleveland’s first gastropub. If you love beer, this is the place for you!  Plus, its right down Historic Scranton Road from a theatre, The Liminis, which is home to a small avant-garde company called convergence-continuum.
  • Professor Avenue – is home to boutique shopping at Banyan Tree and Evie Lou as well as famous chefs including Dante Boccuzzi and Rocco Whalen of Fahrenheit.
  • Sokolowski’s University Inn – a cafeteria style, James Beard Award winning, restaurant, serving comfort food including their famous pierogis. Now on the Towpath Trail, a regional asset, off road walking and cycling trail that you can take to past Akron, Ohio to New Philidelphia, Ohio 110 miles south of Cleveland.

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Hello evaluators and philanthropists. I’m Myron Marshall, a National Urban Fellow and soon to be MPA graduate. Over the last 9 months I have grown, learning from new experiences. Specifically, I have been introduced to the profession of evaluation and the philanthropic community. Through my fellowship program, I have also experienced a new city, Cleveland.

Rad Resource: The Greater Cleveland area wears a gorgeous Emerald Necklace, a large system of nature preserves that surrounds the city. Exploring the Cleveland’s Metroparks was one of the first things that I did when I arrived to Cleveland to complete a 9 month mentorship assignment at the Cleveland Foundation (TCF) through National Urban Fellows, a program designed to get underrepresented minorities and women into leadership roles within the nonprofit and public sector. During my time at TCF, I’ve experienced the same growth as many of the trees that make up the Emerald Necklace.

TCF is a tremendous asset to the Forest City. Each year, TCF gives between $90 – $100 million to hundreds of nonprofit organizations located within Greater Cleveland to achieve its vison of making Cleveland a great and global American city. To guide its daily interactions and decision making, TCF has established the core values. My capstone, a requirement of the National Urban Fellow’s MPA program, highlights their value of learning, especially it’s factor in driving TCF’s decision making.

Organizations that emphasize learning and properly manage the knowledge gained to influence decision making will excel in this information era. Organizational learning is supported by systematic methods of data collection, knowledge management, and use of evaluation. Evaluation in philanthropy is no longer solely used for accountability purposes, but for learning. As such, funders are structuring time, finances, and staff to accommodate and use evaluation and learning for strategic decision making. The gathering of data and knowledge to support and influence decision making is called strategic learning.

Lessons Learned and another Rad Resource: During my mentorship at TCF, I assessed the Grantmaking department’s readiness to support strategic learning and evaluation. The assessment, appropriately titled the Readiness for Organizational Learning and Evaluation (ROLE), determined that learning from evaluation can be supported and Program is prepared to engage in other kinds of learning practices.

To further test if the Grantmaking department was ready to participate in practices of learning, a focus group was held, providing ROLE data for a collective interpretation by Grantmaking staff. Additionally, I facilitated a learning activity from an FSG guidebook, using the ROLE data to collectively come up with solutions, improving their capacity to support strategic learning. My work has planted the seed for strategic learning capacity to grow.

I believe that if the Cleveland Foundation is to effectively fulfill its mission, to enhance the lives of all residents of Greater Cleveland, now and for generations to come, it must develop, sustain, and support a culture of learning. A culture of learning and supporting strategic learning creates fertile ground for which knowledge can grow and produce fruit of effective grantmaking.

 

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

Greetings! I’m Lana Rucks, Principal Consultant with The Rucks Group. I serve as the current President of the Ohio Program Evaluators’ Group (OPEG) and Chair of the Cleveland Local Arrangement Working Group (LAWG). OPEG and the Cleveland LAWG are looking forward to welcoming you to Cleveland this fall!

In the blogs this week, we are thrilled to introduce you to our beloved city, which used to be referred to as the “mistake on the lake” but is now the city that rocks! What you will come to know is that Cleveland is rich in history and entertainment and is populated by a friendly and diverse group of people. The deep roots in Cleveland also extend into the evaluation space: OPEG was established in 1980 and pre-dates AEA! With regard to the philanthropic community, Cleveland is home to the nation’s first community foundation, The Cleveland Foundation.  Throughout this week, the contributors have woven together general information about evaluation with interesting evaluation work that is occurring in Cleveland.

Towards that end, I would like to share about a new emerging effort to build evaluation capacity for local foundations, particularly focused on small- and medium-sized foundations. Under the leadership of Kathleen Dean of the Saint Luke’s Foundation and Christine Mitton of Sisters of Charity Foundation of Cleveland and in partnership with Philanthropy Ohio, they have created a forum for individuals who are on the front lines of grantmaking and grants management as well as individuals responsible for evaluation, learning, and outcomes measurement. The purpose of the group is to engage in dialogue to advance evaluation practice among foundations in Northeast Ohio. This innovative approach involves discussion topics that vary but essentially revolve around sharing best practices and challenges towards increasing the use of evaluation practice within their work. It is exciting that this forum is serving to promulgate evaluation within the region.

Rad Resource: A recent discussion to facilitate the discussion used the guide Facilitating Intentional Group Learning published by FSG guide as a backdrop for the discussion. The report contains a plethora of information on advancing learning within foundations.

Hot Tip: Like with any city, there are references and language that are unique to Cleveland. One such word is “Halle’s.” Invariably, a reference to Halle’s will come up in a conversation with a native Clevelander. So that you’ll know what people are talking about, here’s some background information: Halle’s is a reference to the Halle Brothers Co., which was a department store in Cleveland for nearly 100 years. Its former flagship location is one of the oldest buildings in Cleveland. Since closing in the mid-1980’s the building was renovated and currently serves as office space for area businesses. Here’s another interesting fact: Halle Berry, the movie actress, is the department store’s namesake.

We’re looking forward to the fall and the Evaluation 2018 conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

 

Kelly Laurie

Kelly Laurie

Hi I am Kelly Laurie and I am the Event Services Manager for AEA.

Evaluation 2018 is upon us! Our registration launch is right around the corner and the AEA team is anxious to go LIVE! Don’t let our July registration opening time frame fool you – conference planning starts months and months in advance and we’ve been diligently working away.

Generally, we’ve already started planning the future year conference while the current year program is still yet to be completed. We take lessons learned, new ideas, and survey responses to implement improvements into future year programming with the goal to make the AEA Annual Conference better year after year.

One new offering in 2018, based upon feedback and comments, is childcare. A lot of research and thought has gone behind this decision and we are happy to be able to offer this new initiative. The team is also very excited to bring Evaluation 2018 to Cleveland – it’s a great city with a lot to offer and we hope you are all looking forward experiencing the city as we are!

 

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

Greetings, evaluators! I am Jacqueline Craven, the doctoral program coordinator responsible at Delta State University for aligning goals and outcomes with Council for the Accreditation of Educator Preparation (CAEP) standards. I write to those of you working with teacher education program personnel undertaking the same challenge. While professors and assessment professionals in universities across the nation continue efforts toward aligning key program assessments with still new/relatively unfamiliar CAEP accreditation standards in education, a quick refresher on the concepts of measurement can serve as a timely reminder for those responsible for these assessments.

Many will recall this information but may not immediately connect how it relates to rubrics. By understanding the levels on which we measure constructs such as math knowledge or linguistic performance, leaders can better determine whether rubrics reflect accurate and appropriate types of measurement. Where there are discrepancies, we can revise and streamline our rubrics to measure attributes of student work using the nominal, ordinal, and interval/ratio scales.

As an example, let’s examine some common language found in rubrics. Descriptive terms such as “good”, “excellent”, and “superior” seemingly accompany higher scores, but how exactly do they differ? Inevitably and even with the best intentions, various raters perceive and define each differently and would thereby score student work differently. So, what does this mean for improving rubrics?

Hot Tip:

Define in precise terms what exemplifies each level. Rather than use broad descriptive terms for processes or outcomes, clarify the characteristics of what constitutes “excellent” into tangible/actionable terms, which will then compose each rubric’s scale. Doing this for all prompts on rubrics will challenge assignment and assessment authors to specify in detail what is (and what isn’t) ideal. Pauline Dickinson and Jeffery Adams elaborate further on rubric creation best practices in their Values in Evaluation – The Use of Rubrics article published in the December 2017.

By completing this step thoroughly, achieving inter-rater reliability will be much easier as the increased level of specificity will result in a greater shared understanding and interpretation among individuals interacting with each rubric. Additionally, the explicit definitions and discussion generated from rubric revisions will also assist with informing students of how they will be assessed, which is another required component of CAEP standards. In fact, recent evidence revealed by Julie Elizabeth Francis in her 2018 article indicates higher student performance after engaging with and discussing the rubrics used to assess their work.

Rad Resources:

To begin, some brief yet helpful videos on the scales of measurement:

https://www.youtube.com/watch?v=OXTdii-b9Co

https://www.youtube.com/watch?v=A5zlhbmBghI

For quick-reference, see the CAEP Evaluation Framework for EPP-created Assessments to evaluate your own rubrics. Items 2-6 pertain directly to the measurement concepts above. And now the fun begins! Which rubric will you revise first?

 

The American Evaluation Association is celebrating Consortium for Research on Educational Assessment and Teaching (CREATE) week. The contributions all this week to aea365 come from members of CREATE. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings, colleagues! My name is Steven Bingham. As a professor of education at High Point University in High Point, North Carolina, and as a former researcher in a federally funded regional research and development laboratory, I approach the dissertation process with a bias for learning “what works”. In the 30 dissertations that I either have chaired or served as committee member in the last six years, the majority of those have used program evaluation, in part or whole, as a methodological approach.

Hot Tips:

Beyond ensuring successful dissertation defense, I have become increasingly interested in impacts of what I call “program-evaluation competency.” My hope is that program graduated candidates and districts that employ them value and apply what I have taught. Notably, our course of study requires that all candidates complete at least one program evaluation prior to conducting their dissertation research.

I did, then, what any curious professor would do: I surveyed a sample of matriculated practitioners. Here’s what I learned: On the upside, 100 percent of responding practitioners reported systemically using the tools of program evaluation in assessing program merits. Eighty percent reported appropriate application of program evaluation, resulting in program scale-up or modification.

On the downside, one respondent stated that a multimillion-dollar reading program was abandoned without evaluating it at all. Of mixed blessing is the respondent who reported that evaluation of a one-to-one instructional technology program had failed to consider human variables. The result was erosion of teacher confidence and reduced program effectiveness. Despite evidence, district leaders deemed the finding unacceptable.

Not surprisingly, my findings suggest that program evaluation may be a credible approach to school and district improvement. Readers whose work involves determining the merits of educational programs may also recognize a familiar fly in the ointment: for program evaluation to be of value, invested leaders must be willing to embrace the results even when politically inexpedient.

Lessons Learned:

As a professor teaching program evaluation to school and district practitioners, my greatest and most affirming lesson learned is that, if taught as part of a doctoral program, particularly in dissertation research, program evaluation seems to have a better-than-even chance of being used and useful for public school districts and their students. In the pursuit of education as evidence-based practice, that is good news.  

Rad Resources: Here is a source the describes the benefits of a dissertation that has a program evaluation focus.

 

The American Evaluation Association is celebrating Consortium for Research on Educational Assessment and Teaching (CREATE) week. The contributions all this week to aea365 come from members of CREATE. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hi!  My name is Daina Lieberman, English teacher and International Baccalaureate (IB) Middle Years Programme (MYC) Coordinator at South Lakes High School in Fairfax County Public Schools, Virginia.  I am also a recent graduate of the Ed.D. in Educational Administration and Policy Studies program at The George Washington University.  Today I’d like to provide some tips on Project-Based Learning.

Hot Tips:

As an IB MYP Coordinator, I work with teachers in my building to create, implement, and assess performance-based assessments in all subject areas, including PBLs.  Project-Based Learning, or PBL, has become an important method of teaching and assessment in schools.  Instead of students being taught a unit and then creating a project, students are asked an open-ended, driving question that requires them to research and learn information to solve a problem.  Their final work may vary in form and content, but students need to collaborate, think critically and creatively, and conduct research, and demonstrate their understanding.

PBL sets up situations that allow students to solve real-world problems and create authentic solutions.  As adults, we solve our problems in the same way—if we want to buy our first house, we conduct research, ask professionals for help, take action, reflect, make adjustments, and hopefully purchase a home successfully.  Teachers need to guide students throughout their inquiry phase to ensure they are learning appropriate and factual content relevant to solving the problem and answering the driving question.

PBL is a great way to enable English language learners, special ed students, advanced students, and all other students to demonstrate their learning in ways teachers can assess and students can enjoy.  This type of assessment can be used with students at any level, including undergraduate and graduate.

Be sure when assessing PBL work that your rubric is assessing student learning, not behavior or completion.  Check in with other teachers who have conducted PBL units and look at various rubrics before creating one; ask a colleague to look it over to ensure you are assessing what you want to assess.  You can also work with your students and have them help you create a rubric to assess their work.

Have fun!

Rad Resources:

For a great definition of performance-based assessments, check out Patricia Hilliard’s article on edutopia called Performance-Based Assessment: Reviewing the Basics or this booklet from Stanford School Redesign Network called What is Performance-Based Assessment? which includes research and examples of PBAs.

Check out this page on Edutopia for articles and videos on Project-Based Learning and this Research Spotlight on Project-Based Learning by the NEA.  Resources and Tools for PBL Start to Finish on edutopia is another great page with even more resources and links to help you get started.

For more information on developing performance-based assessments and rubrics, read Doug Wren’s AEA blog post on the topic and have a look at Ross Cooper’s blog post on Project-Based Learning Professional Development (part 2): Student Created Rubrics on ASCD Edge.

The American Evaluation Association is celebrating Consortium for Research on Educational Assessment and Teaching (CREATE) week. The contributions all this week to aea365 come from members of CREATE. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hello. I am Sean Owen, Associate Research Professor and Assessment Manager at the Research and Curriculum Unit (RCU) at Mississippi State University. Founded in 1965, the RCU n. The RCU benefits K-12 and higher education by developing curricula and assessments, providing training and learning opportunities for educators, researching and evaluating programs, supporting and promoting career and technical education (CTE), and leading education innovations. One of my roles at the RCU focuses on providing our stakeholders practical strategies to inform their practices better and to guide future processes.

Lessons Learned:

Technology can be your best partner. We all agree the rate of development of technological advancements has increased at a rapid pace in recent years. Muhammad Yunus said, “While technology is important, it’s what we do with it that truly matters.” In our group, we have started leveraging Google Forms with a combination of add-ons to increase the efficiency of our classroom observations. One of the keys to implementing effective observation systems is a failure to use observation rubrics that are concise, focused, easy to use, and digestible.

Hot Tip:

Explore add-ons for Google tools. There is a vast community of developers creating Add-Ons for Google’s Suite of Tools. As Alice Keeler has shown in recent months, anyone can create add-ons for the Google Suite to best suit the processes of their work. As we continue to forge ahead in a climate of tight budgets, evaluators using cost-effective and efficient practices have become even more important. To learn more about developing Add-ons for the Google Suite, navigate to the Google Developer site.

Rad Resources:

The American Evaluation Association is celebrating Consortium for Research on Educational Assessment and Teaching (CREATE) week. The contributions all this week to aea365 come from members of CREATE. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Older posts >>

Archives

To top