AEA365 | A Tip-a-Day by and for Evaluators

TAG | grants

Greetings! We are Kate LaVelle, Research Associate, and Judith Rhodes, Associate Professor of Research, from the Office of Social Service Research and Development (OSSRD) at Louisiana State University. At OSSRD we write large federal grants to support educational, place-based initiatives for school districts and communities with significant need in southern Louisiana. In this post, we share our lessons learned and tips based on our grant writing experiences.

Hot Tip: Grant applications require a description of the need being addressed; however, applications vary in how much direction they give for presenting information on needs. For example, some applications ask for results from a completed needs assessment or segmentation analysis. Other applications require you to discuss needs within preset categories, such as academic, health, or community needs. To cover these common requirements, we find it helpful to create a Gaps and Solutions table. This concisely presents evidence-based specific gaps that are linked to particular solutions, providing a clear justification for proposed services based on identified needs.

Here is an excerpt from a sample Gaps and Solutions table:

Hot Tip: When writing grant applications that incorporate complex approaches, we find it useful to develop an Intervention Design table that includes the detailed information that funding agencies typically want to know. For example, the table below contains information about who and how many individuals will be served, the cost of services per participant, plans for scaling up services over time, and the funding sources for each planned strategy. We include a list of key partners to show the important collaborations, as well as research-based evidence backing the proposed strategies. This table can also be helpful for communicating the intervention design to colleagues working on other parts of the grant, such as the budget or evaluation sections.

Lessons Learned:

  • Be purposeful in where you place tables in the grant application. For example, we have found that a Gaps and Solutions table works well at the end of the Needs section as a way to summarize key gaps and solutions, as well as provide a transition into the Program Design section, which typically follows. However, a more detailed Intervention Design table might be best placed in the Appendix if page space is limited, assuming that the table is sufficiently referred to in the narrative.
  • If feasible, hire a graphic designer (or graphic design student if cost is an issue) to create a logo specifically for your proposed initiative. We find having a professional logo adds a polished look to the application, as well as provides a visual branding that potential funders may be more likely to remember.

Rad Resource: Grants.gov is a helpful resource for exploring different types of education grants. Federal departmental websites also have previously-awarded proposals available to view, which can provide more ideas of ways to effectively present your next grant proposal. After all, if previously used strategies were successful for another applicant, they might work for you!

The American Evaluation Association is celebrating Ed Eval TIG Week with our colleagues in the PreK-12 Educational Evaluation Topical Interest Group. The contributions all this week to aea365 come from our Ed Eval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

 

My name is Moya Alfonso, and I’m an Assistant Professor at Georgia Southern University and University Sector Representative and Board Member for the Southeast Evaluation Association (SEA), a regional affiliate of the American Evaluation Association (AEA).

So, you need to improve (or develop) your grant writing skills and perform service. A perfect way to address both of these needs is to serve as a grant reviewer!

Lesson Learned: I have honed my grant writing skills by reviewing for local nonprofits, the Centers for Disease Control and Prevention, and the Department of Education, and learning what is expected and seeing the mistakes made by others.  At the same time, I performed an important service to the fields of public health and educational research and evaluation.

Hot Tip: Select the Right Opportunity. When looking for opportunities to be a grant reviewer, consider where your strengths lie. If you’re a program evaluator with a background in education, for example, the Department of Education might be a good place to start. Targeting opportunities will increase your odds of being selected for a review panel – even if you are new to reviews.

Hot Tip: Know What You’re Getting Into. So you’ve found an opportunity that is right up your alley. Now what? It’s time to determine logistics. If detailed information is not provided in the call for reviewers, contact the review administrator about in-person versus remote reviews, estimates of time required, number of applications assigned, grant review dates or time periods, and travel reimbursement or stipends.

Hot Tip: Be Critical Yet Constructive. There’s nothing worse than receiving a “Great Job!” back from a reviewer. No one is perfect. Read (and reread) each application with an eye toward both its strengths and weaknesses. Keep feedback constructive; there is no room for personal insults in grant reviews.

Hot Tip: Know You’re Not Alone. Grant reviewers typically serve on panels comprised of individuals with a variety of perspectives and skill sets. You are not expected to know everything! Feel free to draw upon the wisdom of your grant review administrator and your fellow reviewers.

Hot Tip: Don’t Trust Technology. Technology is amazing – when it works! When completing reviews, you will likely need to learn new technology to complete your reviews. Don’t trust it! Perform your reviews in a word processing program, save your files to your computer, and use the copy and paste functions to complete your reviews.

The American Evaluation Association is celebrating Southeast Evaluation Association (SEA) Affiliate Week with our colleagues in the SEA Affiliate. The contributions all this week to aea365 come from SEA Affiliate members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi! I’m Catherine Jahnes, a Phoenix-based evaluator. In Arizona, there can be the sense that the funding community lacks the resources to support evaluation. Virginia G. Piper Charitable Trust, Valley of the Sun United Way, First Things First, and Arizona Grantmakers Forum (AGF) united to combat this perception by creating an evaluation-focused affinity group comprised of local funders. The group is just up and running, but we already have some lessons to share about bringing funders together to talk about evaluation.

Hot Tip – The group’s name carries weight. While an increasing number of local funders have evaluators on staff, they still amount to only a handful, so when we convened the group it was important not to inadvertently send the message that we were about evaluation with a capital E. We decided to call ourselves the Evaluation and Impact Affinity Group. By focusing our message on impact and effectiveness, we gathered close to thirty people representing funders of all types and sizes for our first meeting.

Lesson Learned – At the end of the day, funders share similar problems related to evaluation. Topics of interest include:

  • Setting philanthropic goals
  • Building evaluation into the grantmaking process from step one
  • Moving beyond evaluating individual grants to understanding overall impact

Hot Tip – Start with a small, dedicated steering committee that can facilitate discussions and maintain the group’s momentum.

Lesson Learned – If the steering committee consists of representatives from a narrow range of funding types or sizes, it is important to get input from the whole group about discussion topics, and look for ways to diversify leadership.

Rad ResourceArizona Grantmakers Forum is a regional networking and professional development organization for funders with ties to Arizona.

Harvard Family Research Project User’s Guide to Advocacy Evaluation Planning helps funders evaluate their advocacy investment.

The American Evaluation Association is celebrating Arizona Evaluation Network (AZENet) Affiliate Week with our colleagues in the AZENet AEA Affiliate. The contributions all this week to aea365 come from our AZE members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· · · ·

Hello! I’m Sheila Robinson Kohn from Greece Central School District and University of Rochester in NY and program chair for the PreK-12 Educational Evaluation TIG. Welcome to our 2nd annual sponsored AEA365 week, in honor of Teacher Appreciation Week!

Here is our TIG’s mission, vision, and values in abbreviated form; please visit our website for the complete version, along with additional information about our TIG.

Mission: Raise the quality of educational evaluation.

Vision: Foster a close community of educational evaluators, become more responsive to context in education, and maintain high standards for educational evaluation practice.

Values: Relevant, responsive, high quality educational evaluation that reflects our beliefs in social justice, equity, and educating the whole child.

We asked our TIG members to identify important topics in educational evaluation and this week’s posts will reflect several of those topics.

Today, I’ll begin our week by sharing a Rad Resource for educational evaluators, the Institute of Education Sciences (IES) website. IES is “the nation’s engine for education research, evaluation, assessment, development and statistics” and offers a virtual treasure trove of resources of interest to education evaluators. IES is comprised of four centers, each featuring many programs:

  • National Center for Education Evaluation (NCEE)
  • National Center for Education Research (NCER)
  • National Center for Education Statistics (NCES)
  • National Center for Special Ed Research (NCSER)

Here’s a sample of what this multi-faceted site offers:

IES also offers a host of funding opportunities for educational researchers and evaluators.

Lesson Learned: Type “evaluation” into the search box on the IES home page, you may get as I did, “about 7260” results! That’s enough to keep any evaluator busy for awhile!

Hot Tip: Subscribe to “NewsFlash” IES’ email-based alert service. Select to receive only the specific information you choose to receive from IES, its four centers, and their programs.

Hot HOT Tip: Join us for our PK12 Educational Evaluation TIG Business Meeting in Minneapolis at AEA 2012 to learn more and get involved!

Hot Tip: Take a minute and thank a teacher this week!

The American Evaluation Association is celebrating Educational Evaluation Week with our colleagues in the PreK-12 Educational Evaluation AEA Topical Interest Group. The contributions all this week to aea365 come from our EdEval TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hi! My name is Laura Sefton. I’m a Research Coordinator in the University of Massachusetts Medical School’s Center for Health Policy and Research. Funding for many of our research and evaluation projects comes from many sources, including state, federal, and foundation grants and contracts. Part of my role is to assist staff in finding the grant funding that supports their research and evaluation interests. I perform a variety of searches and follow several guidelines to ensure successful results.

Hot Tip: Before starting your search, establish the parameters of what you are looking to fund. Identify topics of interest and determine keywords to be used in your search. Determine what the project will entail, who will be involved, and how big the budget might be.

Rad Resources: Identify funders and/or open funding opportunities using three online databases.

1. Grants.gov – The U.S. Department of Health and Human Services manages this free database of funding opportunities for 1,000 federal grant programs. Users can search using keywords and/or by agency and sign up to receive emails of new funding opportunities related to your interests.

2. Foundation Directory Online – FDO is a database developed by the national non-profit Foundation Center. A basic monthly or annual subscription provides access to profiles in their database of over 10,000 foundations. Free or fee-based training about the grant-seeking process is offered by the Foundation Center and comes in many formats, including webinars, interactive self-paced online courses, and part or full-day classroom training.

3. COS.com – COS.com is a database of over 400,000 funding opportunities from foundations, companies, or government agencies. An institutional subscription allows users to create accounts through which searches can be saved and tracked. Their robust search interface allows users to specify parameters such as activity location, funding type, and multiple search terms, and to create email alerts about new opportunities.

Hot Tip: Evaluate potential funders or funding opportunities against your project’s criteria. Look for limitations to their funding, such as requirements that the Principal Investigator have less than 5 years of research experience or that funding cannot be used for evaluation projects.

Hot Tip: If you feel that the funder or the opportunity is a good match for your interests, talk to the contact person at the funding organization. They can advise whether your idea is in fact a good match and provide helpful information for your application. This may allow you to develop a relationship with the organization for the future.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

My name is Alexander Manga, and I am a Ph.D student at Western Michigan University. I served as a scribe at Evaluation 2010 and attended session 356, Differing Perspectives of Quality Throughout an Evaluation. I chose this session because I am very interested in understanding more about the construction and processing of quality evaluations.

Lessons Learned: Quality began at the grant proposal stage

  • The more detailed, accurate, and sensible your grant deliverables were the greater your chances of getting a proposal chosen. The keys are not to over commit, yet hit the deliverables asked for in the RFP. Tailor the proposal to the needs and wants of the agency in a manner that ensures success for both sides.
  • Past work was a predictor of future quality. It was noted that past work performance with program managers or stakeholders will greatly influence future opportunities. A strong reputation for quality work may increase your chances for grant acceptance.

Lessons Learned: Working with stakeholders and managers is a key to success and having a re-occurring relationship

  • Communication throughout the entire evaluation process is key. Perspectives and criteria may change during the evaluation process. There are two different situations that may exist; predictable and unpredictable paths the evaluator can encounter. Obviously, the more predictable the better. Communication between evaluator and stakeholder can mitigate unpredictable situations during the evaluation.
  • Assume change will occur. Nothing will be static, yet dynamic. Remain open minded to continuous change in both planning and practice. Stakeholders may use different criteria to judge the quality by the end of the evaluation.
  • Engage in reflection. After each evaluation, team members should reflect on the process from beginning to end and determine positive and negative points.

Lessons Learned: How is quality judged?

  • Reputation
  • Methodological Rigor
  • Cost-Effectiveness
  • Likelihood to meet expected deliverables
  • Credentialing

Lesson Learned: Increase evaluative inquiry sustainability

  • By incorporating a participatory approach, evaluators can take advantage of current practices and procedures by researching process methods and operational scopes to determine efficiencies and effectiveness. Sustainability can be attenuated by participation of stakeholders or constituents. This process then repeats itself through a cycle that includes: Action, Plan, Observation, and Reflection. This involvement of participants through the entire cycle can enrich the evaluation process by ensuring communication and understanding at the ground level through completion.

The full description of this session and its presenters may be found here. At AEA’s 2010 Annual Conference, session scribes took notes at over 30 sessions and we’ll be sharing their work throughout the winter on aea365. This week’s scribing posts were done by the students in Western Michigan University’s Interdisciplinary PhD program. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice.

·

My name is Susan Kistler, AEA’s Executive Director, and contributor of each Saturday’s aea365 post. Last June, I wrote about Funding Opportunities for Evaluators. Today, I want to expand a bit in that arena. Over the past six months, we’ve been working to improve our listening and identifying processes, to find evaluation Requests for Proposals (RFPs) and grant opportunities to share. Two resources in particular have proven useful:

Rad Resource: Philanthropy News Digest has a customizable RFP Alert system that sends RFP alerts right to your email. This is different from subscribing to their newsletter; instead, you must set up a free account on their website. Then, select profile -> edit profile -> Newsletters/Alert Subscriptions. This great service isn’t an easy find, and takes about two minutes to set up, but it allows you to customize the information you receive by RFP subject and by region.

Rad Resource: Grants.gov allows you to find and apply for United States Federal Grants. Using the Subscription tab or button, you can sign up to receive daily notices of all new grant opportunities or set up advanced criteria to hone in on your area of specialization.

The above two resources have proven the most valuable so far in identifying funding opportunities for evaluators. The AEA staff monitors these on a daily basis and adds opportunities that we find to the AEA Headlines and Resources list. However, our review is a general one, wherein we’re striving to identify opportunities with an evaluation focus. If you are working within a larger firm or agency, subscribing to either or both of the above will help you to identify opportunities that may incorporate evaluation, but have a broader focus.

Hot Tip: As in June, I recommend signing up for AEA’s weekly headlines and resources list that includes the funding opportunity finds for that week. The list is posted each Sunday to AEA’s Listserv, EVALTALK, and appears in the news listing on AEA’s LinkedIn group. But, to receive it in a nice stand-alone compact form, you can subscribe to the headlines and resources list as a single weekly compilation via the Headlines and Resources website.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

· ·

Hello. We are Bill Bickel, Jennifer Iriti, and Julie Meredith. We make up the optimistically named Evaluation for Learning Project (EFL) at the Learning Research and Development Center, University of Pittsburgh. We are writing to share a recent experience we had working with a small, regional foundation interested in learning more from its grant making activities beyond its modest grant reports. Foundation leadership sensed that there was more to be gleaned from grantee experiences, especially about long-term effects of grants, which were not being captured. We were asked to devise a low cost way to help. Our learning protocol tip is what we came up with.

Tip: The following protocol was used with a test set of grantees.

  1. Identify a set of grantees to focus the learning exercise. Some common characteristics enriched the learning opportunity (e.g., grants closed for at least three years; some common goals and/or change methods [e.g., reliance on professional development in grantees’ work]; grantee leadership interested in learning).
  2. Develop an informal, retrospective theory of change (ToC) based upon existing grantee documents (e.g., applications & reports, write-ups, organizational descriptions).
  3. “Test” our ToC with grant leadership and refine. Debrief grantee leadership on what happened since the grant closed to document the enacted ToC.
  4. Poll grantee leadership on what the foundation could do to enhance their own and the foundation’s capacities to support future learning.
  5. Analyze individual and cross-grantee data for implications for foundation processes and write-up results in an accessible brief for all concerned.

EFL’s level of effort was modest. Insights gained about long-term outcomes, capacity building needs, and recommended changes in foundation application and reporting processes by early accounts are potentially useful both in the short and longer term. They are being vetted against the working knowledge of foundation leadership as we write. One can imagine many variations on the protocol; our point here is that past organizational experience has much to offer to a learning agenda if tapped in even an informal way.

Recommended resources that provide additional insights to support learning in “small foundations with big learning agenda” follow.

Resource: Marli Melton, Jan. Kay. Slater & Wendy Constantine have a useful chapter on ways evaluation can support learning in “Strategies for Smaller Foundations”.

Resource: Though not specifically targeted to small foundations Michael Patton, John Bare, & Deborah Bonnet offer insights on “Building Strong Foundation-Grantee Relationships,”– quite relevant to building learning in such contexts.

Both can be found in: Eds. M. Braverman, J. K. Slater, and N. Constantine. (2004). Foundations & Evaluation: Contexts & Practices for Effective Philanthropy. San Francisco: Jossey-Bass.*

*American Evaluation Association members receive 20% off on all Jossey-Bass titles when ordered directly from the publisher. Just sign on to the AEA website and select “Publications Discount Codes” from the members only for details.

The American Evaluation Association is celebrating evaluation in Not For Profits & Foundations (NPF) week with our colleagues in the NPF Topical Interest Group.  The contributions all this week to AEA365 will come from our NPF members and you may wish to consider subscribing to our weekly headlines and resources list where we’ll be highlighting NPF resources.

· · ·

Hi, my name is Rick Axelson. I am a faculty consultant in the Office of Consultation and Research in Medical Education (OCRME) at the University of Iowa. OCRME works with faculty, staff, and students to support innovation and improvement in medical education practices. My colleague, Susan Lenoch, and I conduct workshops and individual consultations with clients – primarily educators – on program evaluation. The workshops attract faculty and staff from across the College of Medicine, all with different needs and interests.

One of the biggest challenges we’ve encountered in designing workshops for audiences of non-evaluators is “where to start.” Participants’ background knowledge seems to vary greatly and, there isn’t sufficient interest, time, or commonality in background on which to base our discussion. So, where do we begin?

Hot Tip: Start at the end. That is, build backwards from the participants’ common goals or interests. Why are they interested in doing an evaluation? Is there a particular product that they need to develop? To provide a common interest for participants, we advertise our evaluation workshops as an aid for completing the evaluation portion of an in-house grant application. Participants are encouraged to bring their grant proposal ideas to the workshop; we open the workshop by discussing their project ideas and refer to their projects throughout the workshop to illustrate evaluation principles. The added advantage of this approach is that it also works as a warm-up activity.

A second major challenge with these workshops is to provide a simple, straightforward process that will enable participants to create their desired evaluation product. Here, we have found the following to be helpful.

Hot Tip: Use examples of completed products to illustrate the evaluation design process. We walk participants through an example of a previously funded project. Breaking the evaluation design process down into a few simple steps, we show how the sample evaluation would have been developed using this method. We also point out areas (e.g., statistical analysis) where participants may need expert assistance to complete their study design.

Rad Resources: We have developed a workbook to lead participants through the development of the evaluation section for an in-house grant application. A previously funded project was used as the central example throughout the workbook to illustrate the steps involved in designing a program evaluation for an educational intervention. You can download it from: http://www.healthcare.uiowa.edu/ocrme/teach_train_sup/online_lean_teach_edu.htm

This aea365 Tip-a-Day contribution comes from the American Evaluation Association. If you want to learn more from Rick on this topic, check out the sessions sponsored by the Teaching of Evaluation TIG on the program for Evaluation 2010, November 10-13 in San Antonio.

·

My name is Susan Kistler and I am AEA’s Executive Director. I contribute each Saturday’s post to the aea365 blog. AEA is striving to help evaluators know about evaluation funding opportunities, working to improve on two fronts:

  1. Listening and Identifying: Improving our processes for knowing about funding opportunities
  2. Disseminating and Communicating: Improving our processes for letting the evaluation community know about funding opportunities

Hot Tip: To learn about funding opportunities for evaluators, subscribe to AEAWEB: Headlines and Resources. AEAWEB includes a range of headlines and announcements of interest to evaluators, and is where we post an announcement of each identified funding opportunity. The easiest ways to receive AEAWEB are via one of the following:

Hot Tip: If you are on Twitter, follow @aeaweb to receive headlines and resources announcements in real time rather than as a weekly compilation.

Hot Tip: Here are four opportunities posted this past week, to give you an example of the announcements included on AEAWEB.

  • Funding: Impact 2.0 – grant opportunity for evaluating web 2.0 in Latin America http://ow.ly/1RA7n
  • Funding: Evaluating Oil Spill Disaster Response? Grants Available Through NSF http://ow.ly/1RBow
  • Funding: Grants available for graduate students evaluating Head Start Programs for their dissertation research http://ow.ly/1RzkFFinally
  • Funding: Establish National Center on Program Management and Fiscal Operations from US DHHS http://ow.ly/1RBFh

If you know of a funding opportunity for evaluators, please share it via an email to headlines@eval.org.

· ·

Older posts >>

Archives

To top