AEA365 | A Tip-a-Day by and for Evaluators

TAG | computer science

Hi, my name is Lisa Kaczmarczyk; I am a computer scientist with my own project evaluation consultancy and I’m also an adjunct computer science (CS) faculty at Harvey Mudd College. I work primarily with CS and engineering teachers and faculty who face unique challenges when creating their computing curriculum and evaluation procedures. A recent conversation I had with a frustrated school Principal exemplified two of the problems I often encounter in this setting: enthusiasm but lack of formal CS training, and isolation from other CS teachers. K-12 CS Evaluators need to be prepared to deal with this situation.

The Principal explained to me that he wanted one of his teachers to develop a new computing curriculum for grades K-8. I was asked to help them develop CS based assessment metrics for each grade. Unfortunately, neither one of them had a computer science background or professional experience. As a result, they were having a very hard time identifying objectives that were based on age appropriate computational principles.

Unfortunately, this situation is not unusual in the US because CS teaching certification varies widely and is often hard to come by. Frequently, CS teachers have their primary certification in another area of instruction. In addition, whether or not they have a CS teaching credential, new computer science teachers often have no one to talk to. They feel isolated.

Like many of their peers, this Principal and teacher needed resources to build off of and a community to share and vet their classroom ideas and experiences. An evaluator coming on the scene needs to have resources at hand to help teachers develop their understanding about what computer science objectives are and are not.

Rad Resources:

There are several good curricular resources, guidelines and references available, each with their own very active community of teachers. The resources contain varying levels of specificity, but they all have online communities that include both new and experienced CS teachers. Without endorsing any one standard over the other, here are a few to peruse and start a conversation about classroom objectives with:

From the Computer Science Teachers Association (CSTA) http://www.csta.acm.org/Curriculum/sub/K12Standards.html;

From code.org https://code.org/educate/curriculum

From the Scratch community http://scratched.gse.harvard.edu/guide/

Hot Tip: Resources alone only go so far. Teachers and administrators need support to form local support communities. Provide them with the emails or URLs to connect to their state level CS teacher meetups, professional organizations (such as CSTA) or faculty at local community colleges who might be interested in creating bridge programs. In most cases, there are other teachers willing to share computational goals and objectives they are trying in their classrooms along with members of their professional network in computing academia and industry.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

We are Courtney Blackwell, Heather King, and Jeanne Century from Outlier Research & Evaluation at the University of Chicago. For the last 3.5 years, we have been researching and evaluating computer science education efforts.

Computer Science (CS) is becoming a buzzword in education, with educators, policymakers, and industry developers promoting CS as key to developing 21st Century skills and a pathway to employment. While CS is not new to education, the spotlight on it is. In 2014, over 50 U.S. school districts, including the seven largest, pledged to make CS education available to all students.

Like all buzzwords, most people have their own vague idea of what CS means, but even experts working within CS education do not, yet, have a clear, agreed-upon definition. If evaluators are going to be able to accurately measure the effects of CS education efforts on teaching and learning, and accumulate knowledge and understanding, we need to have a clear definition of what “CS education” is. Until CS educators create shared definitions themselves, we, as evaluators, can do our part by ensuring our logic models, strategies, and measures clearly and specifically describe the innovation — computer science education — so that our work can inform others and further the field.

Lessons Learned: Evaluating an ill-defined intervention is not an uncommon problem. In the case of CS, however, the capacity to articulate that definition is limited by the state of the field. As evaluators, we have to find alternatives. In our evaluation of the Code.org’s computer science education efforts, we ask students to provide their own definition of CS at the beginning of our questionnaires. Then, we provide a specific definition for them to use for the remainder of the questionnaire. This way, we capture student interpretations of CS and maintain the ability to confidently compare CS attitudes and experiences across students. Similarly, we begin interviews with teachers, school leaders, and district leaders by asking, “How do you define computer science education?”

Hot Tip: Always ask participants to define what they mean by computer science.

Rad Resources #1: A recent survey by the Computer Science Teacher’s Association (CSTA) found that high school leaders don’t share a common definition of CS education. This suggests that school leaders may promote their schools as providing “computer science” when in fact they are providing activities that would fail to be considered CS at the college and professional levels.

Rad Resources #2: Check out LeadCS.org, a new website about to be launched, for definitions of key terms in Computer Science education. The website offers a range of tools for K-12 school and district leaders and their partners who seek to begin or improve CS education programs.

The American Evaluation Association is celebrating STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to aea365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Welcome to STEM TIG week! My name is Jason Ravitz and I conduct research and manage evaluations of educational outreach projects at Google. This week our blogs focus on Computer Science (CS) education which may officially be counted as a STEM field by act of Congress.

The CS First project is one curriculum that is available for use by schools, camps and after-school programs. In one case, Google has funded Boys and Girls Clubs of America to deploy AmeriCorps VISTAs to build capacity and use this curriculum in summer camps and afterschool clubs. The goals of CS First include increasing confidence and providing a sense of belonging in technology for underrepresented students.

This has been exciting work, but I’ve found there are challenges when evaluating informal STEM and CS programs. Not least is having to make teachers or volunteers test for pre-post content learning. This feels to everyone like it defeats the purpose, that is to have fun and not feel like school.

Cool Trick: Try to make assessment instructional and fun. We chose 5 basic level assessment items for a pre-test and asked volunteers to acquiesce to try these one time and report how it went. Meanwhile, to make these assessments less burdensome I came up with a way to punctuate each question with a fun activity to illustrate what kids would be learning. We had various ideas, like playing “Simon says…” as a way of demonstrating commands and loops. These would work even better with clickers and maybe as part of the curriculum. There are more vetted activities with accompanying research at CS unplugged. This is a “cool trick” because it can make assessment a learning activity that feels less like school. Even if our ideas weren’t generally used with the pre-tests, they showed we were listening to concerns about over-testing and its potential impact on what should be a fun club climate.

Hot Tip: Plan ahead with the curriculum provider. We are coordinating with the curriculum developer to incorporate information from their assessments and produce reports. This is non-trivial, but will be very beneficial for our evaluation. Among its assessments CS First has a way of scoring students’ code. The system they use, that you can try online, is Dr. Scratch, as developed by researchers in Spain. We hope our external pre-post tests can quickly be used to validate the embedded assessments (or refine them) so our external assessments can be retired.

We all want to hear what challenges others are facing and to hear your solutions. Thanks to Kimberle Kelly and AEA’s STEM Education and Training TIG for help organizing this week of blogs. Please join us at our conference sessions and tell us what you think!

The American Evaluation Association is celebrating STEM Education and Training TIG Week with our colleagues in the STEM Education and Training Topical Interest Group. The contributions all this week to aea365 come from our STEM TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Archives

To top