Greetings from the University of Chicago!  We are Sarah Rand, Amy Cassata, Maurice Samuels and Sandra Holt from Outlier Research and Evaluation at the Center for Elementary Mathematics and Science Education. Our group recently evaluated two Chicago-based elementary education programs: Purple Asparagus, a nutrition education program and Science, Engineering, and Technology for Students, Educators, and Parents (SETSEP), a science and engineering program for students and their parents.

Purple Asparagus App

Purple Asparagus App

Purple Asparagus was interested in the program’s impact on children’s willingness to try new foods, attitudes about eating fruits and vegetables, and actual eating habits.  A major challenge we faced was how to measure changes in the eating habits and attitudes of 6- and 7-year-olds, an age that is considered too young to complete a traditional survey. In the SETSEP evaluation, our challenge was finding an efficient way to collect student outcome data from 1^st-3^rd grade students that minimized the amount of time taken away from program activities. We also needed a method of data collection that would capture students’ attention and accurately represent their perception of an engineer.

Lesson Learned: It was because of these dilemmas that we explored the potential to create iPad apps to measure student outcomes for each program evaluation. We worked iteratively with an app developer to create engaging, child-friendly platforms for students to share their experiences and feelings.

Click here to see a video of the Purple Asparagus App in action.

Click here to see a video of the SETSEP app in action.

SETSEP App

SETSEP App

The administration of the iPad app was a great success. We collected data from almost 200 participating students. Students remained focused on the survey for the five to ten minutes it took to complete. Most students had previous experience using an iPad and were familiar with the touch screen. Student response to the app was very positive and many students commented that taking the survey was a fun experience!

Rad Resource: We worked with Matt Jankowiak from Region Apps to program these apps.

Cool Tricks: We bought these iPad cases and these headphones, which were great for students.

Lessons Learned: App development takes time. We were new to app development, so the learning curve was steep at first. There are many pieces to consider including ease of use for young children, audio recordings, and collecting images for the app. It’s also a good idea to test the app with students before administration.

Note: Survey questions for the engineering app were developed by Engineering is Elementary at the Museum of Science, Boston and were used with permission.

The American Evaluation Association is celebrating Chicagoland Evaluation Association (CEA) Affiliate Week. The contributions all this week to aea365 come from CEA members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

iPad apps · survey

My name is Dan McDonnell, and I am a Community Manager at the American Evaluation Association.

There is much, much more to Twitter than first meets the eye. It’s not just a platform where you can share what you had for dinner in 140 characters. It’s a fully customizable RSS feed. It’s a potent networking tool. It’s an incredible resource for research and evaluation.

In today’s post, we’ll show you how to start tapping into the power of Twitter lists to make your social media experience more thorough and efficient.

Hot Tip: Create Lists to Make Social Media RSS Feeds

If you’re following hundreds or even thousands of people on Twitter, it can be difficult to parse through every single tweet being sent in real-time, particularly if there are only a handful of interesting users with whom you’d like to keep up. Twitter Lists will filter all of the tweets coming in based on users you select, which is very easily done.

Start by selecting the Gear icon on the top bar on Twitter. Select ‘Lists’, then click ‘Create list,’ where you’ll enter the name (for instance, “Evaluation Influencers”) and a description. You can opt to keep your list public – recommended, as it encourages community by allowing others to subscribe to your list.

Once you’ve created your list, search for profiles of the users you’d like to add (or, just monitor your feed or @replies until you see people worth including!), call up their profile, and click the icon that looks like a human face. Select ‘Add or remove from lists’,  and check the box next to the appropriate list. Simple!

You can view your lists by visiting your Twitter profile page and selecting ‘Lists’ from the sidebar. No more need to wade through an endless flow of tweets to get at the meat! Now, for an even easier way to view your Twitter lists.

Hot Tip:  Creating a Social Media Dashboard

If you’re using Hootsuite or another third-party Twitter listening or monitoring client to organize your social media experience, you’re likely already aware of the power of custom feeds. These feeds let you automate Twitter searches, hashtags or even pull in a Twitter list to help you create a social media dashboard.

Once you’ve signed up for Hootsuite, click ‘’Add a Stream’ and select the type of feed you’d like to add. To add the list we created above, click the tab marked ‘Lists,’ select your Twitter profile from the drop down, and click the button next to the list you’d like to add. Once you finalize with the Add Stream Button, the list will be added in its own feed to your home Dashboard.

You can also add in numerous other useful feeds. Hootsuite starts you off with @Mentions, Sent Tweets, Retweets and a few other default tabs, but that’s just the beginning. When adding a stream, use the keyword or search function to add a keyword or hashtag you want to monitor, like ‘Evaluation’ or #Eval. You’re not limited to Twitter either- you can add in feeds Facebook, Google +, LinkedIn, WordPress and more.  However you choose to customize, you’re ultimately creating for yourself a powerful tool to centralize and simplify your social media activity. Cheers!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Dan McDonnell, and I am a Community Manager at the American Evaluation Association.

Evaluation 2013 ( #eval13 in Twitter hashtag speak) is less than two weeks away, and already, social media is abuzz with conversation around the conference.  Twitter is a truly incredible tool that can greatly enhance your conference experience. You can connect with speakers and other attendees before you arrive in DC, engage in dialogue, capture your key learnings and observations throughout the week — and then evaluate the key trends and topics discussed when you return home.  Here’s how to get started.

Hot Tip: Tweet on the #eval13 conference hashtag to engage with fellow attendees. The hashtag has quickly become one of the most iconic and powerful Twitter features. When you tweet with the Evaluation 2013 hashtag (#eval13), your tweet is saved to a threaded conversation alongside every other tweet that uses the same #eval13 tag. This allows you to create and participate in a meta-discussion about the event itself, whether it be live-tweeting observations from the sessions you attend, retweeting or favoriting great quotes or takeaways from speakers or attendees or expanding your network through meeting new people via Twitter. For starters, check out this list of Evaluation 2013 attendees who are on Twitter.

Hot Tip: Create content and add value. Heard a particularly noteworthy quote or idea from a session you’re at? Tweet it. Want to thank a great speaker or give kudos to a colleague or friend for their contributions to the event? Tweet them with an @ reply. Want to share a photo or your observations on Evaluation2013 overall with the world? Tweet! The more you tweet, the more likely your knowledge will be shared with a wider audience, and you’ll increase the chance that you’ll connect with interesting evaluators attending the conference.

Hot Tip: On that note, make sure your Twitter profile picture looks like you! I’ve been approached more than once at an event and asked, “Hey, are you @Dan_McD ?” based on my tweets and photo. It’s an incredibly rewarding experience to connect with new people you may have never met otherwise, and allows you to do the digital equivalent of exchanging business cards with a one-click follow.  But how do you keep up with the numerous tweets on the conference hashtag, not to mention your own personal @mentions?

Rad Resource: 

Hootsuite

Hootsuite, alongside other free third-party social media monitoring platforms (including SproutSocial) make listening to social media conversations and responding a cinch. You can set up feeds to see who is Tweeting @ you, the latest tweets on the #eval13 hashtag, and if anyone has retweeted your content. With just a few clicks, you can respond in real time or schedule a tweet to go out at a later time to spread the wealth. There is a great Hootsuite app on both iOS and Google Play, so regardless of your mobile device, it is easier than ever to  get the most out of Twitter while on the go at the Evaluation2013 conference. Happy tweeting.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Dan McDonnell, and I am a Community Manager at the American Evaluation Association.

For many people (myself included), the smartphone has become an extension of ourselves. We use our ‘third limb’ for nearly everything these days: from accessing news on our RSS feeds, to checking banking statements, to finding the perfect sushi spot within walking distance from the office.

With more and more people accessing the web from their mobile devices every day, evaluating your website or blog to measure mobile traffic and ensure a positive user experience is of utmost importance. By creating an engaging experience on mobile and removing barriers to entry, you make strides toward winning the battle for attention in a world filled with endless distractions and legions of multi-taskers.

Hot Tip:  Google Analytics – Mobile Tab

The first place to start is with the free web measurement tool, Google Analytics.  Once you’ve logged into your profile, under ‘Traffic Sources’ on the left sidebar you’ll see the ‘Mobile’ tab. Using the information under ‘Overview’ will help you better understand how many of your visitors are accessing the website on a desktop, mobile or tablet device.

Pages per visit and time spent on page will give you greater insight as to the behavior on the site. Often, you’ll find that desktop users spend the most time on a particular webpage, followed by tablet users, followed by mobile users. Knowledge of how visitors engage with specific pages on your website can help you craft a better user experience, and identify where fresh or optimized content can lead to improvements and increases in traffic.

Quick Metrics Guide:

• Pages per visit: How many pages are visitors viewing during a single visit to the site (mobile vs. desktop vs. tablet)
• Bounce rate: Are mobile users leaving the site without viewing other pages?
• Time on site: How long do visitors spend on your mobile or tablet site? What about on your desktop site?

Rad Resources: DudaMobile

What if you have no mobile website? DudaMobile can help you there.  Their baseline plan is free, and allows you to create and configure a handful of mobile pages for your site.  You’ll be able to view a mocked-up version of your mobile site as you adjust the layout and options to deliver the best user experience. Add buttons to your page, tweak the colors – no knowledge of coding is necessary.

Once your new mobile site is live, continue to measure your traffic and engagement. Don’t be surprised if you see a spike in activity!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Greetings, I am Zoraya Cruz-Bonilla from Binghamton University in New York. As a data research assistant, I have come across all sorts of data both big and small. Regardless of the size, each project solicits the expertise of multiple evaluators; therefore, it is not unusual for more than one person to work on the same spreadsheet. You probably already know that, by default, Excel only allows one user at a time. However, what you might not know is that it is possible to assign multiple users to work on the same spreadsheet at the same time and track changes. The advantage of tracking changes is that you can get a chronological list of all the edits made, including a comparison of the new value with the old value. For projects that call for a lot of documenting and organization, this is one feature you should start using.

Hot Tip:

1. In Excel 2010, click on the Review tab. Select “Track Changes”.
2. Check the box to turn on “Track Changes While Editing.” This selection will enable you to share the workbook in order to allow multiple users to make edits at the same time.

   Next, I recommend selecting “All” and “Everyone” from the first two dropdown menus. This will instruct Excel to save a running record of all saved changes. If you need changes marked on the spreadsheet, then you should check the box, “Highlight Changes on Screen.”

3. “List changes on a new sheet” is only available after edits have been made and saved. Therefore, you will need to select “Track Changes” a second time to activate this option. Once selected, Excel will generate a new tab—titled “History”—with a list of saved changes. The new tab can be filtered to display edits to a particular range of cells based on date, type of edit, and/or author.

Lessons Learned:

• If you close or save the document, the “History” tab will disappear.
• Edits can be accepted or rejected.
• Rather than updating the “History” list whenever the saved button is selected, use the advanced features under “Share Workbook” to auto-save at a specified interval.

Rad Resources:
As you consider whether sharing a workbook is appropriate for the task, it is important to know the ins and outs of this feature. On that note, I suggest that you check on the following link: http://bit.ly/16jxi0w. Also, for using this feature with Excel 2013 I suggest checking out this link: http://bit.ly/12Y4BDY.

AEA is celebrating GSNE Week with our colleagues in the Graduate Student and New Evaluators AEA Topical Interest Group. The contributions all this week to aea365 come from our GSNE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! My name is Matthew Von Hendy, I am a professional research librarian and recently started my own business, Green Heron Information Services, a small research consulting company that provides information services to evaluators and other clients.

For the past three months, I have been conducting informational interviews with professional evaluators to understand how they find and use information and research. Over the course of that time, I have spoken to a dozen evaluators working in all types of settings including government agencies, academic institutions, non-profit organizations as well as independent consultants.

Every evaluator I spoke with was highly-motivated to conduct their own research. Research resources used most often included: Google, professional association websites, peer to peer conversations, other search engines, self-created collections of articles and documents, local public/university libraries and databases.

Hot Tip: Check your Google results. If you use Google frequently, it will filter your results based on your previous searches, meaning you may not be seeing important resources. Other search engines such as Duck Duck Go or Blekko can provide a way to make sure you are not missing something crucial.

Hot Tip: Three resources for finding free full text. Besides Google Scholar, other search tools such as Scirus and Mendley can be useful tools to locate free full text. They will frequently find PDFs that Google Scholar does not.

Evaluators all expressed concerns about limited budgets but indicated a willingness to pay for research or information in specific instances such as: big projects with tight deadlines which involved searching, literature reviews or writing annotated bibliographies, research projects on a small scale, research projects that arise on a case by case basis, professional development, and benchmarking processes.

I look forward to meeting everyone at the upcoming AEA conference this October. I have grown to really like Washington DC and love to show the area off to visitors. As a member of the Washington Evaluators and the Local Arrangements Working Group I hope to get the chance to meet you all in person this fall.

Hot Tip About DC: Many people are surprised to find out that DC is a good place for biking and hiking.  The Capital Bikeshare program make it easy, inexpensive and convenient to get around on two wheels even if you are just visiting for a couple of days. The National Zoo, the National Mall and Rock Creek Trail are relatively close by the conference and can be great places to take a walk or a hike.

We’re thinking forward to October and the Evaluation 2013 annual conference all this week with our colleagues in the Local Arrangements Working Group (LAWG). Registration is now open! Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to contribute to aea365? Review the contribution guidelines and send your draft post to aea365@eval.org.

No tags

Hello, I am Kerry Bruce, Director of Results and Measurement at Pact and I’m currently based in Madagascar and support Pact programs in Africa, Asia and Eastern Europe. In 2012 we started to roll out the use of mobile technology in our programs including evaluation.  This third post in the series on how to get started with mobile phone technology in evaluation focuses on training and taking your data collection to the field.

Hot Tips:

• When you are training people to use mobile phone technology for evaluation, here are some things that you’ll need to consider:
  • Are the people you are training familiar with mobile phones or will they need significant training and mentoring?  What type of phone will be easiest for them to use?
  • Make sure that people are VERY familiar with the survey instrument and VERY familiar with the phone operation before they get to the field.  Otherwise they spend the entire interview looking at the phone and never look at the person they are interviewing.
  • Make sure respondents understand that data are being entered on the phone, but that data are encrypted and will be stored confidentially.
• When you go to the field to collect data using mobile technology:
  • Take along a few paper copies of the survey instrument, both as a back-up in case you run out of power or lose the phone, but also to refer to during questionnaire administration.
  • If a question is not working or needs to be changed this can be done even as the survey is being conducted.  A change is made on the central server and data collectors can be notified via SMS or phone to upload the newest version of the survey.  This is best done at the end of the day.  This is very handy for dealing with troublesome questions that made it through pre-testing and piloting.

Lesson Learned:

• Mobile technology will not improve your survey design or instrument.  It will improve the timeliness and accuracy of your data collection, but it won’t magically insert a missing variable.
• The structure of field supervision changes with mobile technology and happens best from a central location with a strong internet connection.  Supervisors check the data as it comes in and can call data collectors directly with feedback while they are still in the field if there is an issue.

Rad Resources:

Online mobile technology training for a variety of uses is available from TechChange.

A great training resource is available from the CLEAR Initiative.

More information about using mobile phones for data collection is available from the BetterEvaluation page on Mobile Data Collection.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Data Collection · technology

I’m Susan Kistler, the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor. After over three years, this is my last day as your ongoing Saturday author. I’ll be moving to once a month for at least the near future. The other Saturdays will be filled by Sheila Robinson, our intrepid aea365 curator, Stephanie Evergreen, in her role as AEA’s Potent Presentations Initiative (p2i) coordinator, and others from AEA’s new staff team. I am excited to see so many people invested in aea365.

Data Visualization and Reporting (DVR) is near and dear to my heart. The very first post that I wrote for aea365 was on Data Visualization back on January 2, 2010. I am honored that I get to complete up my weekly run by wrapping up the Data Visualization and Reporting week.

Rad Resource – Sarah Rand’s Previous Posts: Earlier this week, Ann Emery included Sarah Rand in her DataViz Hall of Fame. Sarah is part of a team that is breaking new ground, pushing into online reporting and infographic use for research and evaluation. Sarah has written two previous posts for aea365:

• Sarah Rand on Online Evaluation Reports
• Sarah Rand on Using Visual.ly’s Marketplace Service for Infographics

Rad Resource – Interview with Sarah Rand: As I explored the online report from Sarah’s team, I wanted to learn more about what it took to produce such a report as well as the drivers behind, and the benefits or drawbacks of, their innovations in reporting. In March of 2013, I interviewed Sarah. The Take 5 Tech video below provides a very brief overview of the report and then a five minute summary of lessons from Sarah.

Lessons Learned: For me, three things stood out from Sarah’s interview:

1. The composition of the evaluation team may need to adapt to incorporate designers and programmers in order to move into new reporting formats.
2. There is a need for more knowledge about, and perhaps more tools themselves, for assessing the impact of new reporting formats.
3. For this group, innovations in reporting have drawn attention to their findings and also to their firm as well, including securing additional grant funding based.

Get Involved: I would like to expand this project into a short series focused on innovators and innovation in data visualization and reporting, in particular for evaluation. This week has given me some great ideas, but if you know of evaluators who are doing leading edge work in this area that you would recommend. please share via the comments on the aea365 blog.

aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. AEA is celebrating Data Visualization and Reporting Week. The contributions all this week to aea365 come from members of AEA’s Data Visualization and Reporting Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

No tags

Hi, I am Miranda Yates, Director of Program Evaluation and Planning (PEP) at Good Shepherd Services (GSS), a youth and family development, multi-service agency in New York City. My PEP colleagues, Barbara Alcantara, Nora Casey, and Cathleen Mitchell, and I have developed a visually engaging report template that keeps staff focused on the key outputs and outcomes identified in their logic models.

We use Excel to create quarterly and annual reports with three sections, all of which combine text boxes and a variety of charts. The first page presents participant demographic and background information. The second page focuses on outputs. Subsequent pages provide short-term, intermediate, and long-term outcomes data.

For more details on how we develop logic models and create reports staff can use, see the slides from our recent webinar.

Hot Tips:

• Context: On the first page of the report,include a brief introduction that helps orient staff to the purpose of the report, reporting period, and data sources. Identify a contact person for questions and suggestions.
• Color: Be intentional in your use of color. For example, each program area has a theme color which we use consistently. See also paper by Maureen Stone (2006) on choosing colors so as to clarify and not confuse.
• Points of Comparison: Help staff interpret output and outcome performance by providing points of comparison such as an internal or funder target, prior year performance, and system averages. This information really helps to get conversations going and further sets results in context.
• Details: Include an appendix at the end of the report with extra details that staff might want in trying to interpret the results. For programs with multiple locations, we break key information down by location. For smaller programs, we provide key information by participant.

Rad Resources:

The PEP Team are big fans of Andy Kirk’s Visualising Data blog. We were fortunate to attend a recent one-day training where we were inspired to learn that one of the pioneers of data analysis and visualization was none other than Florence Nightingale!

aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. AEA is celebrating Data Visualization and Reporting Week. The contributions all this week to aea365 come from members of AEA’s Data Visualization and Reporting Topical Interest Group. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

excel

Hello, I am Kerry Bruce, the Director of Results and Measurement at Pact.  I’m currently based in Madagascar and support Pact programs in Africa, Asia and Eastern Europe. I am part of Pact’s central technical team that provides monitoring and evaluation support to more than 20 country offices and more than 70 projects around the world.  In 2012 we started to roll out the use of mobile technology in our programs including evaluation.  This is the second post on mobile technology and focuses on choosing your platform. Check out my first post on this topic: Getting Started with Mobile Phones.

Hot Tips:

• Look at a wide range of available platforms and ask yourself:
  • What is my budget for phones?  Some platforms work better than others with entry level (vs. Android) smartphones.
  • What is my budget for the data collection?  Will my data collection reoccur frequently (on-going evaluation) or is this a one time event?  Each platform has a different pricing structure and each lends itself to different types of data collection.
  • Platform operators will promise you the moon – but will their platform deliver?  Test basic issues such as skip logic, ease of set-up and use, how data download and dashboards work before you buy.  Most platforms have a trial version that you can use and some allow small data collection projects for free.
  • Will I need help to set up my survey, or do I have the skill set to set it up in house?  Some platforms offer survey set up and technical support (useful for complicated data collection exercises) and some are all do-it-yourself.
  • What language will the survey be in and can the platform support it?  This is especially important for non-Latin alphabets.
  • Get a reference.  All these platforms should be able to provide you with a reference from someone who has used them before and can tell you what is good and what needs work.

Lesson Learned: Evaluate two or more platforms before you decide which one to use. 

• Some have recurrent or annual costs and others only charge for the data that you collect.  Others are free up to a certain level of data collection.
• Each platform has its strengths (and weaknesses) – you’ll need to understand what you need it to do and shop around until you find it.
• Just because a platform can not do something today does not mean they won’t be able to do it tomorrow, check back and give feedback.  This technology is rapidly adapting.

Rad Resources: Here is a list of some of the mobile technology platforms that are commercially available today.

*These are platforms I have used.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

technology