AEA365 | A Tip-a-Day by and for Evaluators

TAG | multisite

Greetings, I am Brian Molina, a graduate student at in Western Michigan University’s Industrial/Organizational Behavior Management doctoral program. I have conducted single subject research and implemented performance improvement projects across many different settings and organizations.

Lessons Learned:

  • Single subject research is used to evaluate program effectiveness across large groups. The belief that single subject research can only be used with one person at a time is a common (yet understandable) misconception. The number of people in the analysis depends upon the frame of reference we are interested in evaluating. For example, we may easily choose to evaluate the performance of a single individual, a single team, a single department, a single organization…and so on.
  • The methodology is friendly toward organizations with limited resources or experience in evaluation.  Evaluation can be as intimidating! For organizations that are new to the process, interpreting and understanding large group statistical analyses may be difficult. Single subject research typically results in data that show behavior change over time, which can easily be interpreted by researchers and clients alike. Easier data collection and analyses make it more likely that organizations will begin and continue evaluation of their activities.
  • Single subject designs allow for more maximum flexibility in implementing program changes. Conducting research is rarely an orderly process that goes precisely according to plan. Single subject methodology accommodates this unpredictability well. Changes in behavior are rapidly observable during the course of program implementation, not simply at the conclusion of sometimes-lengthy data collection. This allows leaders to make on-the-go changes to the intervention that best serve the client, without contaminating the results of an experimental evaluation.

Rad Resources:

Single Subjects example

The American Evaluation Association is hosting the Disabilities and Underrepresented Populations TIG (DUP) Week. The contributions all week are focused on engaging DUP in your evaluation efforts. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

Hi, I’m Audrey Rorrer and I’m an evaluator for the Center for Education Innovation in the College of Computing and Informatics at the University of North Carolina at Charlotte, where several projects I evaluate operate at multiple locations across the country.  Multisite evaluations are loaded with challenges, such as data collection integrity, evaluation training for local project leaders, and the cost of resources. My go-to resource has become Google because it’s cost-effective both in terms of efficiency and budget (it’s free). I’ve used it as a data collection tool and resource dissemination tool.

Lessons Learned:

Data Collection and Storage:

  • Google Forms works like a survey reporting tool with a spreadsheet of data behind it, for ease in collecting and analyzing project information.
  • Google Forms can be sent as an email so that the recipients can respond to items directly within the email.
  • Google documents, spreadsheets, and forms can be shared with any collaborators, whether or not they have a gmail account.
  • Google Drive is a convenient storage source in ‘the cloud.’

Resource Dissemination:

  • Google Sites provides easy to use website templates that enable fast website building for people without web development skills.
  • Google Groups is a way to create membership wikis, for project management and online collaboration.

Rad Resource: Go to www.google.com and search for products. Then scroll down the page to check out the business & office options, and the social options.

For a demonstration of how I’ve used google tools in multisite evaluation, join the AEA Coffee Break Webinar on February 27, “Doing it virtually: Online tools for managing multisite evaluation.” You can register here.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

· ·

We’re Amy Schaller and Bryna Koch, Evaluation Specialists at the University of Arizona, Cooperative Extension Services.

How do you support an inaugural cross-site evaluation with a national initiative while incorporating technology and building grantee capacity?  This was the charge of the CYFERnet (Children, Youth, and Families Evaluation & Research Network) Evaluation Team at the University of Arizona.  Our vision was to create a website that would serve as a central platform to house a range of evaluation resources, tools, and training materials for grantees.  And so, CYFERnetSEARCH.org was born.  The evaluation capacity-building website, designed to support the efforts of the Children, Youth, and Families At-Risk (CYFAR) initiative funded by USDA-NIFA, is also available to the public.

The resources we developed for the site included: content for eight learning modules including quizzes and videos, a searchable database of vetted evaluation instruments, a user-account feature, and online logic model and survey builders.  All of these features were developed from the ground-up and much energy was invested in generating original content, design, and features; no small undertaking. The process was, and remains, highly collaborative with the features on the website evolving numerous times since they were first developed.

Lessons Learned – DIY web development is fairly common these days, but to do so successfully requires some strategic vision and input from outside sources.  Here are some “Behind the Scenes” tips we can share from our experience to help those of you whose evaluation work intersects with website development:

  • If you can, work with web-design professionals.  Working across disciplines allows fresh thinking and combined perspectives. Solicit input from professionals with varying backgrounds.
  • Do your design research.  Select websites you like to use and consider what makes those sites appealing to you. Is it the feel or the look? How do they provide information? How do they make that information accessible?
  • Choose the tone for your site.  What feeling do you want to convey to your user?  What colors, images, and features communicate the essence of your project/company?
  • Do your platform research. Drupal, WordPress, Joomla? Should you consider open-source? How much modification is needed so the platform works for you?
  • Be the user. Interact with your site as a user would on a daily basis.
  • Organization and presentation matter. How can the content be intuitively grouped into themes that enable easier navigation for your users?
  • Consider the pace of technology and industry.  Will your design, platform and content be relevant in one or two years?

Rad Resources – 

cyfernetsearch.org – Our website

Trello.com A free online collaboration tool for project management

The American Evaluation Association is celebrating Arizona Evaluation Network (AZENet) Affiliate Week with our colleagues in the AZENet AEA Affiliate. The contributions all this week to aea365 come from our AZE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · · · ·

Archives

To top