AEA365 | A Tip-a-Day by and for Evaluators

TAG | Institutional Review

Hello! We are Dana Gonzales and Lonnie Wederski, institutional review board (IRB) members at Solutions IRB, specialists in the review of evaluation research.

Why talk about IRB review for evaluations of science, technology, engineering, and math (STEM) education projects? Most simply, federally funded projects may require it. You may also ask, “Why aren’t all of these evaluations exempt?” IRB reviewers apply the Code of Federal Regulations (CFR) in their decisions. Many STEM evaluations include children. Under CFR rules, only a narrow range of research is exempt from review when it involves children, like research applying educational tests or observations of public behavior where the investigator does not participate. Interviews and focus groups with minors won’t likely qualify for exempt review, as they are seldom part of the normal educational curriculum. Randomization to a control group would not meet exempt category requirements for the same reason. Both would, however, qualify for expedited review, if there is no more than minimal risk for participants.

So, do you need to use an IRB? Ask these questions:

  • Is IRB required by the grant or foundation funding the project?
  • Does the school district require IRB review?
  • Do you intend to disseminate findings in a publication requiring IRB review?

If the answer to any of those questions is “yes,” you need an IRB—at which point uncertainty strikes! Maybe this is the first time you’ll use an IRB (you are not alone) or you remember unpleasant experiences with an academic IRB. Fear not, evaluators! Many IRB reviewers understand the differences between clinical studies and evaluations. Some specialize in evaluations, employing reviewers with expertise in the methods evaluators use, who recognize that phenomenology, grounded theory, ethnography, and autoethnography are valid study approaches. Who wants to educate an IRB when you are paying them? 

Rad Resources:

Hot Tips:

  • Have questions regarding the ethics of recruitment or consent? Some independent IRBs will brainstorm with you and answer “what if” questions. Ask for a complementary consultation with a reviewer.
  • Ready to submit your evaluation for review? Ask the IRB if free pre-review of study documents is provided, to save time prior to formal review. Ask for a list of the documents required by the IRB.
  • Most important, know the review timeframe in advance! If the IRB requires two weeks for review, you need to plan accordingly. Some IRBs routinely review exempt and expedited studies in 24-48 hours, so timeframes can vary widely.

We hope you found the information provided helpful.

The American Evaluation Association is celebrating Research vs Evaluation week. The contributions all this week to aea365 come from members whose work requires them to reconcile distinctions between research and evaluation, situated in the context of STEM teaching and learning innovations.. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

I’m Jim Hammerman, and I Co-Direct the Evaluation Group at TERC, a non-profit, primarily grant-funded math, science and technology, education research, development, and evaluation company in Cambridge, MA. I’m also a current member and former Chair of TERC’s Institutional Review Board (IRB).

Several of us recently noted on AEA’s LinkedIn group that there are at least three ways to address the issue of Institutional Review for nonprofit organizations:

  1. Partner with a university and use their IRB process;
  2. Contract with a commercial IRB;
  3. Develop the capacity in-house, establishing your own IRB.

Over the course of three days on the aea365 blog, we’ve been sharing a few lessons learned about pursuing these approaches. I’m going to talk about option 3.

Lessons Learned: As a mid-sized non-profit of about 120 employees, several years ago we found it both necessary and advantageous to establish our own IRB. It took a bit of effort to complete the paperwork and establish the policies to obtain and maintain a Federal Wide Assurance (FWA) number through the NIH Office of Human Resource Protection (OHRP).

We started with documents borrowed from collaborators and with advice from some of our own staff who had served on university IRBs. Initially we had paper-based record-keeping, but setting up electronic systems for archiving and tracking IRB review status has been very helpful.

Having our own IRB allows us to review and oversee projects for which we’re prime, or for which our evaluation subcontract is the only real research (e.g., the prime is doing development). But arrangements can be flexible. Sometimes we obtain review for our portion of a project through a university or other institution with their own IRB when they’re the prime; sometimes our IRB oversees research of our partners if they don’t have their own IRB. For liability reasons, we don’t conduct reviews for folks with no affiliation with TERC.

Having our own IRB allows staff to work with peers who know and understand the nature of the work when developing procedures and informed consent documents. Rapid modification of procedures or consents is also easier with an in-house IRB – e.g., when we encounter unanticipated issues in some of our cutting edge work in online learning environments or with out-of-school library or community action groups. Experience as a member of the IRB can also help junior researchers become familiar with a range of study types and issues, building internal capacity. Having your own IRB can provide a number of benefits for your organization.

Resources: IRB guidelines and requirements regarding membership, meetings, initial and continuing review procedures, record-keeping, and the like can be found in the Code of Federal Regulations 45CFR690.101-124 available through the NSF (http://www.nsf.gov/bfa/dias/policy/human.jsp) or NIH (http://www.hhs.gov/ohrp/).

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

·

My name is Stephen Meyer and I am a Senior Research Associate with RMC Research Corporation in Denver, Colorado. RMC Research conducts research and evaluation related to a wide range of education topics.

Colleagues and I recently noted on AEA’s LinkedIn group that there are at least three ways to address the issue of Institutional Review for nonprofit organizations:

  1. Partner with a university and use their IRB process;
  2. Contract with a commercial IRB;
  3. Develop the capacity in-house, establishing your own IRB.

Over the course of three days on the aea365 blog, three of us are going to share a few lessons learned about pursuing one of these approaches.

Most of RMC’s research and evaluation projects require that we seek Institutional Review Board (IRB) approval. We typically work with a university IRB when one is a partner on a particular project, but have found it rare that a university is willing to conduct an external review unless it is in such a partnership role.

Lessons Learned: Recently, for a study involving prison inmates, I approached several IRBs at universities that conduct related research and found that they either were unwilling to conduct reviews for outside organizations or that they lacked the capacity to address the additional requirements related to reviewing studies involving prisoners. While a handful of commercial IRBs also lacked the capacity to review studies involving prisoners, most were able to do so. The costs were substantially higher than what we had been charged by university IRBs–initial review costs ranged from about $1400 to nearly $3000, with additional costs for (annual) continuing reviews, protocol amendments, reviews of Principal Investigator credentials, and reviews of advertisements used for subject recruitment. Although we have had a negative experience with one commercial IRB (related to inconsistent turnaround times and billing practices), our overall experience have been positive. Relative to my experience with university IRBs, I have found that commercial IRBs tend to: have shorter review turnaround times (as little as one week), be more responsive to questions, and have clearer procedures for communication and documentation of project protocols.

Rad Resources: Some options for locating a commercial IRB are below:

  • Citizens for Responsible Care and Research (CIRCARE), a human rights organization dedicated to the protection of human subjects in research and medical treatment, maintains a list of commercial IRBs. http://www.circare.org/info/commercialirb.htm
  • The Office for Human Research Protections (OHRP) maintains a searchable database of registered IRBs. OHRP-registered IRBs have provided assurances that they will abide by federal regulations related to protection of human subjects. http://www.hhs.gov/ohrp
  • The Association for the Accreditation of Human Research Protection Programs (AAHRPP) maintains a list, by state, of the organizations that have met their accreditation standards. http://www.aahrpp.org

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

·

My name is Benjamin Cohen and I am Director of Evaluation at the Center for Schools and Communities, a division of the Central Susquehanna Intermediate Unit. We currently coordinate several interventions and initiatives for school-age youth that are funded by federal, state and private funders. My current focus is the evaluation of a state-wide bullying prevention program operating in over 100 schools and 20+ counties.

My colleagues and I recently noted on AEA’s LinkedIn group that there are at least three ways to address the issue of Institutional Review for nonprofit organizations:

  1. Partner with a university and use their IRB process;
  2. Contract with a commercial IRB;
  3. Develop the capacity in-house, establishing your own IRB.

Over the course of three days on the aea365 blog, three of us are going to share a few lessons learned about pursuing one of these approaches. I’ll be talking about partnering with a university.

Lessons Learned: In some cases we need to utilize an IRB to document the ethical conduct of our work and to be compliant with related regulations. Since our organization is a smaller human services provider and not a research organization, we do not have the capacity to build our own IRB, and thus seek university partners to fulfill that role.

University IRBs have offered us the convenience of a packaged review process for our research and evaluation projects. And because we use university IRBs in concert with a faculty partner, we can often rely on those faculty to navigate our project proposals through the IRB’s waypoints, without having to directly respond to IRB members. Having a faculty member on our evaluation teams permits us to focus on issues directly related to project operations, data collection, logistics and other administrative issues. Furthermore, the training that university IRB members typically have lets us have confidence in their expertise in informed consent issues and procedures, and gives us an assurance that we are conducting our work according to high standards of peer review.

Nonetheless working with a university IRB has some disadvantages. A practical one is covering universities’ overhead costs on a project budget. Universities, in short, can be expensive to include as project partners. Among universities in our region, none appear to offer their IRBs on a fee-for-service basis, therefore university faculty must be included as contracted project staff. Furthermore, university schedules for IRB review do not always correspond to project deadlines, nor do project stakeholders always understand the nature of university IRB protocols or how informed consent issues can manifest in the conduct of a project. For these reasons we are also looking into private IRB services that exist outside universities.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

· ·

Archives

To top