AEA365 | A Tip-a-Day by and for Evaluators

TAG | google

Jason Ravitz (Google) and David Fetterman (Fetterman & Associates, past-president of AEA, and founder of empowerment evaluation) have been using empowerment evaluation in various educational settings, including a graduate school program and with Kathy Haynie (Haynie Research and Evaluation) and Tom McKlin (The Findings Group), in our work with two computer science education evaluation learning communities.

Empowerment evaluation is the use of evaluation concepts, techniques, and findings to foster improvement and self-determination.  This approach aims to increase the likelihood that programs will achieve results by increasing the capacity of program stakeholders to plan, implement, and evaluate their own programs.

3-Step Approach.  One empowerment evaluation approach involves helping a group: 1) establish their mission; 2) take stock of their current status; and 3) plan for the future.  Additional tools include an evaluation dashboard to help communities monitor their own progress.

CS/STEM Learning Communities. The “Evaluation Wrecking Crew” includes over 60 CS education evaluators across the country. A second group (with some overlap) is the NSF-funded Computer Science Outcomes Networked Improvement Community (CSONIC).  

We have joined forces to: 1) build a CS/STEM repository of evaluation instruments and approaches; 2) build a common hub for the community, with the assistance of Oak Ridge Associated Universities; and 3) educate the CS community about the value and role of evaluation to improve the quality of CS and STEM education.  We meet biweekly using Zoom video-conferencing software.  

Kathy Haynie (Haynie Research and Evaluation) Remotely Facilitates Bi-monthly Meetings

Kathy Haynie (Haynie Research and Evaluation) Remotely Facilitates Bi-monthly Meetings

Online Spreadsheet. Jason designed a 3-step online spreadsheet, using Google Sheets, to facilitate the empowerment evaluation process used in both the Evaluation Wrecking Crew and CSONIC workshops.

Mission. Our collaborative process allowed workshop members to remotely record their views about the mission or purpose of the group.  Later comments were transformed into a mission statement (using Google Docs).

Taking Stock. A second sheet in the spreadsheet was devoted to “brainstorming” a list of the group’s most important activities. Members prioritized the list by “voting” for the most important activities to evaluate as a group.

A third sheet was populated with the list of the prioritized activities. The online workshop participants used a 1 (low) to 10 (high) scale to rate their performance on the “taking stock” sheet.  We used the results to facilitate a dialogue about the ratings using videoconferencing software and referencing participants’ ratings.

Planning for the Future.  We used a fourth sheet to help the group record plans for the future, specifying goals, strategies, and evidence.

Evaluation Dashboard.  A final sheet was devoted to the dashboard to help us monitor our own performance.  It included: goals, strategies, and evidence.

Computer Science Education Evaluators Conducting an Empowerment Evaluation Online

Computer Science Education Evaluators Conducting an Empowerment Evaluation Online

Rad Resources:

Free Template.  This spreadsheet is available (free) to use to facilitate your own empowerment evaluation exercise remotely: tinyurl.com/eeblank.

Other free tools we have used include Google Forms to help graduate students evaluate their own as well as their peers’ work.  We used these data to assess students’ performance, and in the process, make mid-course corrections concerning our instruction.  Finally, we used Google Evaluation Worksheets to help them refine their proposals:  /tinyurl.com/evalworksheet-google Additional resources can be found here (https://tinyurl.com/empowermentevaluationresources).

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi my name is Jayne Corso and I am the social media manager for AEA. Today, I wanted to dive into the type of content that should be shared across social media channels. Social media content should be split 20/80: 20 percent brand promotions and strong calls to action, and 80 percent should be dedicated to content that really interests your audience and engages them in conversations. So how can you master this equation? I have put together a few tips and tools to help you out!

Rad Resource: Google Alerts

Google Alerts are a great way to follow key terms, people, or organizations that’s are making headlines. You can get emails when new results for a topic show up in Google Search. For example, you can get information about news, products, or mentions of your keywords. You can set up your google alerts here: https://www.google.com/alerts#

Hot Tip: Repurpose Your Content

If your organization produces a publication, this can be a great resource for social media. Repurpose articles or blog posts by sharing this content on your social media channels. You can even look back at past issues and start a discussion around an archived topic.

Hot Tips: Share Topics, Not Actions

We often want to use social media to promote our webinars, tools, or services. Although social media is a great outlet for these messages, your audience will not be interested in your content if you keep asking them to make a purchase. Try another approach—present your promotional content as topics instead of actions. Create a conversation around the topic of your webinar or ask your audience how they overcome challenges. These types of online conversations can lead to more overall engagement.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Building Evaluation Capacity. David and I have teamed up to apply empowerment evaluation concepts and principles to build evaluation capacity at Google and beyond. We are using rubrics to focus learning, and student ratings to identify areas that are strong or merit attention. We are using a 3-step approach to empowerment evaluation and an evaluation planning worksheet (building on my graduate school courses with Nick Smith) to help our colleagues assess their program’s performance.

The worksheet has 4 parts:

  • describe the program to be evaluated
  • define the evaluation context (purpose and audience)
  • plan the evaluation (questions, data sources, procedures)
  • create an evaluation management plan

With little or no evaluation background needed, teams dive into the worksheet to focus on their program’s purpose and goals before setting up metrics. Laying out the evaluation plan is often illuminating — leading to refined program logic, alternative (and more meaningful) programmatic plans, and more useful ideas about how to measure processes and outcomes.

Beyond Google. We are also sharing our work with nonprofits and higher education. Through the Computer Science Outreach Program Evaluation Network (CS OPEN) Google is supporting evaluation for 12 nonprofits through a partnership with the National Girls Collaborative Project.

David and I are also co-teaching at Pacifica Graduate Institute. David highlights the 3-step approach to empowerment evaluation, including: 1) mission; 2) taking stock; and 3) planning for the future. I follow-up with our worksheet to answer questions such as:

What is the overall program purpose?

Who are the audiences for the evaluation?

How will the result be utilized and by whom?

Rubrics and Technology for Peer Review and Self-assessment. Students in our course are developing evaluation proposals that can help them conduct evaluations, solicit funding, and/or guide their doctoral dissertations. The class meets face-to-face, but includes a virtual classroom strategy that has worked well in the past. Students use rubrics to guide their self- and peer-feedback to refine and improve their work and understanding. This improves the proposals, guides instruction, and models our focus on empowerment and capacity building.

Computer screen snapshot of proposal posted online (using Doctopus) with our rubrics (in Goobrics) above to rate or evaluate the proposal.

Computer screen snapshot of proposal posted online (using Doctopus) with our rubrics (in Goobrics) above to rate or evaluate the proposal.

Rad Resources: We are using our evaluation rubric with tools that require Chrome and free extensions:

This is a partial version of the rubrics used in the empowerment evaluation at Pacifica Graduate Institute

This is a partial version of the rubrics used in the empowerment evaluation at Pacifica Graduate Institute.

Doctopus: A tool for teachers to manage, organize, and assess student projects in Google Drive.

DF3

Goobrics: This rubrics based assessment tool works with Doctopus, allowing teachers to evaluate student’s work in Google Drive.

DF4

Goobrics for Students: Allows students to use a rubric to assess peers’ documents.

DF5

Google Forms: Enables students to self-assess their work and their peers’ work using an online survey.

DF6

DF7

Please contact us for additional information!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

·

Hi! I’m Laura Sefton, Project Analyst in the University of Massachusetts Medical School’s Center for Health Policy and Research. The school’s Lamar Soutter Library gives me access to a database of peer-reviewed journals and resources like Science Citation Index and PubMed to inform the literature searches and survey instrument research that I conduct for many evaluation projects. I have found Google Scholar to be an complementary source for access to materials that might not be available in the library’s databases. Below are a few tips and tricks for getting the most out of this resource.

Google Scholar is free to access and available anywhere you have an internet connection. Keywords, author names, and/or titles can all be used as search terms. Results include scholarly works as well as grey literature, including peer-reviewed and non-peer-reviewed journal articles, conference presentations, technical reports, and theses. Google searches across the web and pulls its results from publishers (free and subscription-based), professional associations, university repositories, and anywhere materials are available.

Hot Tip: Use the options listed in the left sidebar to narrow your search to specific time periods or to sort results by date. Results are, by default, shown in order of their relevance to your search terms, and the top articles may not be the most recent.

Hot Tip: Use the down arrow located to the right of the search box to refine your search. Options include searching by exact wording or without certain words.

Rad Resources: Below each search result are two features that can enhance your searches.

  • The Cited by function denotes how many times the article has been cited in a journal. Frequently cited papers will be closer to the top of your search results. Click on the hyperlink to see those articles, which may or may not be related to your topic of interest, depending on why they were cited.
  • To see other articles that are similar to the search result, click on Related articles.

Cool Trick: The Cite function, which also appears below each search result, shows the result in several citation formats, including MLA, APA, and Chicago, which can be copied and pasted directly into your report. You can also import the reference directly into a bibliography manager via the Cite function’s pop-up box. You’ll want to review each entry, however, since it may not import complete citation information.

Hot Tip: Set up email alerts to receive notifications when new articles related to your search terms are available. This can be particularly helpful if you are interested in the latest information.

Rad Resources: See previous posts by Molly Higgins and Len Levin, my colleagues in the Lamar Soutter Library, for additional literature resources.

Happy hunting!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · ·

Hi, I’m Paul Bakker. I run an independent evaluation consultancy, Social Impact Squared.  As an independent consultant, I don’t have a call centre to conduct population based surveys. However, a colleague (Brian Cugleman of Alterspark) introduced me to Google Consumer Surveys. I would like to share with you my research and trials into how Google Consumer Surveys could be used for evaluations.

Rad Resource: Google Consumer Surveys embeds your survey questions into a network of websites, and people answer your questions in exchange for access to the websites’ content.   You can target specific populations by:

  • The location of respondents’ IP addresses,
  • Demographics such as age or gender,
  • A custom screening question.

Google Consumer Surveys allows you to ask up to 10 questions to any one group of respondents for a relatively low cost – 10¢ per complete for 1 question surveys, and $1.10 to $3.50 per complete for 2 – 10 question surveys.

Lesson Learned: Google Consumer Surveys’ accuracy and reliability is comparable to other surveying methods. You can view their paper on how Google Consumer Surveys compare to other internet surveys here, and you can view how well their predictions on the 2012 U.S. Presidential race compare to other internet and phone surveys here.

Hot Tip: You can use Google Consumer Surveys to help answer evaluation questions around the relevance and effectiveness of population level programs and policies.

As a trial, I ran a couple of 1 question surveys:

  • The first found that only about 29% of Canadians remember reading a Material Data Safety Sheet outside of a training session. That metric could be used as a possible performance measure for Canada’s Workplace Hazardous Materials Information System.
  • The second found that 70% of recipients of Canada’s Universal Child Care Benefit (UCCB) felt that their child would continue to receive no or the same child care if they did not receive the benefit. While there may have been some issues with the survey’s response options, the results could inform an evaluation of the UCCB’s performance.

You can view the overall results and break them down by demographics by clicking on the links above.

survey-wxau5xt2anryq-question-1 survey-wz5nv34j6khxs-question-2

We’re celebrating 2-for-1 Week here at aea365. With tremendous interest in the blog lately, we’ve had many authors eager to share their evaluation wisdom, so for one special week, readers will be treated to two blog posts per day! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Ann Martin, an evaluator working with a team of science educators and outreach professionals in Hampton, VA. I frequently employ Google Formsfor evaluation and even other projects. Forms are free, simple, intuitive for users, and get the job done.

In the past, though, Google Forms had several notable limitations. If you found that Forms didn’t meet your needs in the past, you might not be aware of great new features that represent significant improvements. It’s worth taking another look at this free resource!

Hot Tip: Customize the visual look, feel, and branding of your survey! In September 2014 new Forms functionality allowed survey designers to add background and header images and to customize fonts and other display options. Before, theme options were limited. You can use this functionality to make your survey more readable and inviting. A custom header image with a logo may make your users feel more comfortable responding, or can make your survey a seamless part of a website in which you embed it. You can also embed images and videos within the body of the survey itself, which is handy for quizzes or assessments.

Figure 1 – Customization options range from a header image, page and form background, and fonts.

Figure 1 – Customization options range from a header image, page and form background, and fonts.

Cool Trick: Google Forms now support more complex survey design and administration options, including progress bars, data validation, logic/path branching, and randomizing the order of options in multiple choice questions. It’s also easier now to set up your survey’s questions. For instance, if you have a long list of options to include in a question, you can now copy-and-paste in a list from a word processor or spreadsheet table and automatically populate. (I wish that option had existed a few years back, when I created a drop-down with 200 alphabetized options!)

Cool Trick: New Add-ons enable even more behind-the-scenes functionality. The latest Add-ons includenifty widgets like Form Notifications, which will send automatic emails to your survey respondents, and Form Publisher, which will use survey responses to fill in a new document from a template.

Figure 2. Example Add-ons for Google Forms (screencapture from the Google Drive Add-ons Store).

Figure 2. Example Add-ons for Google Forms (screencapture from the Google Drive Add-ons Store).

Rad Resource: The Google Drive blog shares updates to Forms functionality so that you can always be aware of new features. I’m also more than happy to share tips if you contact me.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org .aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hello, my name is Dan McDonnell and I am a Community Manager at the American Evaluation Association (AEA). In the fast-paced world of social media, things are always changing. Just as soon as you stop to take a breath, Facebook has tweaked its algorithm again. Or Twitter has updated its design. Recently, there have been a slew of changes that hit just about every major social network, and I thought I’d take the opportunity to give a few quick hits on what’s new, and how and how it affects evaluation professionals.

Hot Tip: Facebook Design Changes & More

Do you run a Facebook fan page? Facebook took a cue from Twitter who recently gave a facelift to the layout of fan pages. As of June 5th, the size of fan page  cover photo images has been adjusted to now require 851 x 315 pixels. In addition, Facebook removed what used to be called ‘page tabs’, replacing them with a simple menu of the major sections of your page – Timeline, About, Photos, Likes and More.

If the above changes weren’t enough, Facebook also gave users more freedom to customize the leftmost column of their fan page. Want your page ‘Reviews’ to be front and center? You can do that now! The entire left sidebar can be ordered entirely to your liking.

Hot Tip: Google + Authorship Limited

Remember that post I made a few months ago about Google Authorship? Well, it turns out, Google changed the game when it rolled out some major changes this week. While Google Authorship still exists, the biggest benefits have now been removed. Unfortunately, pictures are no longer supported in Google search results, nor will the author’s Google + circle information be shared. Now, if Authorship is correctly implemented into a blog post, only the name of the author will be added to the search result. Bummer.

Hot Tip: LinkedIn Premium Gets an Update

LinkedIn has become an essential tool for job seekers these days, as well as an excellent way to network. For the power users of the world, LinkedIn has a service called LinkedIn Premium, which I would highly recommend to anyone actively in job search mode. It’s a bit pricey though at $23.99 or $47.99 a month options, but with the addition of LinkedIn Premium Spotlight, a starter package that runs at just $7.99 a month, evaluation professionals can enjoy many enhanced benefits of LinkedIn without breaking the bank.

With Premium Spotlight, LinkedIn will make your profile stand out more among search listings, offer you suggestions for keywords to include in your profile to make yourself more visible for hiring managers and more. Check it out!

Hot Tip: Twitter Changes Fonts

Ok, so unless you’re a Helvetica purist, this one isn’t too big of a deal. Back on May 30th, Twitter angered (or delighted, depending on who you ask) font geeks around the world by changing the default typeface of Tweets from Helvetica Neue to Gotham. Some users have reported that the new font makes it more difficult to read, while others have embraced it fully. What’s your take?

Twitter Font Change

Twitter Font Change

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Dan McDonnell on Google + | Dan McDonnell on Twitter

 

· · · ·

David Fetterman“Ok, glass.” That’s how you activate Google Glass. I’m David Fetterman and that’s me to the right wearing Google Glass. I’m an empowerment evaluation synergist and consultant, busy father and spouse, and owner of Fetterman & Associates.

Rad Resource – Google Glass: Google Glass is a voice and gesture activated pair of glasses that lets you connect with the world through the internet. You can take a picture, record a video, send a message, listen to music, or make a telephone or video call – all hands free.

Hot Tips – Redefining Communications: Google Glass is not just another expensive (currently about $1500) gadget. It can free us up to do what we do best – think, communicate, facilitate, and, in our case, assess. Here is a brief example.

I said “Ok, Glass,” then “make a call to Kimberly James.” She is a Planning and Evaluation Research Officer I am working with at the W.K. Kellogg Foundation.

Kimberly asked how the evaluation capacity building webinar is coming along. Via Glass, I took a screenshot and mailed it to her so we can discuss it. When a colleague is mentioned, with a few swipes of my finger on the frame, I find a picture on the web, and miraculously remember who we are talking about.

Mid-conversation, Kimberly needed to step away briefly. While on hold, I sent a note to colleagues in Arkansas to ask them to check on the data collection for our tobacco prevention empowerment evaluation.

Kimberly returned to the call and we discussed a recent survey. With a simple request, the display of our results appeared, reminding me what the patterns look like.

Did I mention that I did all of these things while making lunch, picking up my son’s clothes off the floor, letting the dogs out, and emptying the dishwasher?

Later in the day, with a tap on the frame, I confirmed our scope of work with Linh Nguyen, the Vice President of Learning and Impact at the Foundation, while dropping my son off for piano lessons.

Later in the week I plan to use Google Hangout to videoconference with another colleague using Glass. When she connects during a project site visit, she will be able to take pictures and stream video of her walk around the facilities, bringing me closer to the “hum and buzz” of site activities.

Lessons Learned:

Respect people’s privacy – do not wear Google Glass where it is not wanted, will put people off, or will disrupt activities. Do not take pictures without permission. Remove it when you enter a bathroom.

Rad Resources

Hot Tip: Stay tuned for Part II tomorrow when I will cover using Google Glass as an evaluation tool.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Enhanced by Zemanta

Hi, I’m Audrey Rorrer and I’m an evaluator for the Center for Education Innovation in the College of Computing and Informatics at the University of North Carolina at Charlotte, where several projects I evaluate operate at multiple locations across the country.  Multisite evaluations are loaded with challenges, such as data collection integrity, evaluation training for local project leaders, and the cost of resources. My go-to resource has become Google because it’s cost-effective both in terms of efficiency and budget (it’s free). I’ve used it as a data collection tool and resource dissemination tool.

Lessons Learned:

Data Collection and Storage:

  • Google Forms works like a survey reporting tool with a spreadsheet of data behind it, for ease in collecting and analyzing project information.
  • Google Forms can be sent as an email so that the recipients can respond to items directly within the email.
  • Google documents, spreadsheets, and forms can be shared with any collaborators, whether or not they have a gmail account.
  • Google Drive is a convenient storage source in ‘the cloud.’

Resource Dissemination:

  • Google Sites provides easy to use website templates that enable fast website building for people without web development skills.
  • Google Groups is a way to create membership wikis, for project management and online collaboration.

Rad Resource: Go to www.google.com and search for products. Then scroll down the page to check out the business & office options, and the social options.

For a demonstration of how I’ve used google tools in multisite evaluation, join the AEA Coffee Break Webinar on February 27, “Doing it virtually: Online tools for managing multisite evaluation.” You can register here.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

My name is Dan McDonnell and I am the Community Manager for the American Evaluation Association.

In my last post, we touched on how evaluators can use Google + Authorship to add photos to their blog posts that will appear in Google search results. Besides making search results prettier, these photos can help you with grow your personal brand, as well as increase the likelihood that someone clicks through to read your post from Google. In this post, I will outline a few more tips and tricks for making Authorship work for you.

Hot Tip: Choose Your Photo Carefully

The photo that displays in search results comes directly from your Google + Profile Picture. As such, you’ll want to ensure that you choose a photo that shows your entire face – no avant-guard snapshots! In addition, Google will not display mascots or logos, so a real, human photo of yourself is the best way to go.

Hot Tip: Easy Authorship with Email Verification

The method I described in my last post demonstrated how to set up authorship on individual posts with a rel=author tag. If you have an email that is on the same web domain as your blog (for instance, if I had dmcdonnell@aea365.org) , it’s as simple as adding that email address to your contact information in your Google + Profile. Once you’ve done that, any post you make will simply need to link back to your Google + profile, no rel=author tag needed.

Hot Tip: Even Easier Authorship with WordPress

If your blog is hosted on WordPress, the process is even more automatic. Sign into WordPress.com with your Google + login information, and any WordPress post you make will automatically ‘author-ized.’ Hopefully, Google makes this a more integrated and automated procedure for other blogging platforms down the line.

Hot Tip: Guide to Troubleshooting Authorship

Having trouble getting Google + Authorship to work? Searchengineland.com has created a really nifty guide to troubleshooting issues – see below.

How To Troubleshoot Authorship Issues: Step by Step Flowchart

Do you use Google + frequently? If not, why not?

Dan McDonnell on Google +

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Older posts >>

Archives

To top