AEA365 | A Tip-a-Day by and for Evaluators

CAT | Military and Veteran’s

Hello, I’m Stephen Axelrad. As the chair of the Military and Veteran Evaluation (MVE) Topical Interest Group, one of the reasons why I wanted to start this TIG was to broaden the military’s understanding of what program evaluation is and broaden program evaluator’s understandings of what the military community is. While there are pockets of the military that conduct evaluations from the health, behavioral, and social science perspective, the majority of program evaluation in the military is conducted from a resource management perspective. In the Department of Defense and the military service branches, program evaluation is embedded in what is considered the program planning, budgeting, and evaluation (PPBE) process. Although different terms are used, the goals of the PPBE process are very similar to what we would recognize in most evaluations.

  1. Describe and understand the key assumptions, inputs, activities, outputs, and outcomes associated with a program, strategy, or initiative.
  2. Identify areas of effectiveness and ineffectiveness, gaps, and innovations in current programs, strategies, initiatives, and systems.
  3. Provide rigorous evidence to support quality decisions from leaders.
  4. Use data to improve program planning and execution.

Hot Tip: Many program evaluators, especially those who exist outside of the military healthcare and community support agencies have financial management, operations research, and systems engineering backgrounds. These fields understand and use some of the same methodological (e.g., surveys, interviews) and analytic techniques (e.g., regression, structural modeling) that evaluators within AEA frequently use. Finding common ground on methodology can help avoid pedagogically based misunderstandings.

Rad Resources: Before engaging with a project or initiative that involves DoD or military service branch officials, review the following web sites to become more familiar with how they may approach evaluation.

Lesson Learned: Evaluators do not have to be in the military in order to understand how to design and execute evaluations affecting the military community. Evaluators who demonstrate an understanding of the military decision-making process (MDMP) and can structure their evaluations as closely as possible to MDMP can maximize buy-in from uniformed and civilian leaders and the use of their evaluation results.

The American Evaluation Association is celebrating MVE TIG Week with our colleagues in the Military and Veteran’s Issues Topical Interest Group. The contributions all this week to aea365 come from our MVE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Stanley Capela. I am the VP for Quality Management for HeartShare Human Services of New York. In addition, I am a military peer reviewer for a nation accreditation organization where I have had the opportunity for participating on reviews for military family readiness programs as well as after school programs on military bases throughout the United States as well as military bases in Germany, Japan and Guam. The purpose of this AEA 365 is to share some tips that I have learned doing these reviews on how best to evaluate military programs.

As a military reviewer I had to undergo training to make sure I have a good understanding on how to approach these reviews. My comments will be based on my experiences as well as what I learned participating in training programs to prepare me for these reviews.

Lesson Learned: First, hierarchy is very important in the military. One key is to be sensitive to their respect for hierarchy and anticipate how you may address this. For example when doing entrance and exit conferences you may see others stand as a higher ranking officer enters the room.

Second, the military has a language that is part of their culture. Therefore, it is important to understand the high usage of acronyms.

Third, their use of various terms such as the use of customers versus clients; staff instead of personnel, service member instead of soldier/sailor or Marine; In-Briefing instead of Entrance Meeting; Cusomer File Review instead of Consumer Record Review.

Fourth, when conducting interviews, it is good to start by letting them know who you are and why you are there. If in my case when doing a review we try to let them know about the accrediting organization and why they are being interview.

Fifth, make sure you stick to a schedule; let people know there may be follow up schedules, avoid casual conversations and if there are people you want to interview outside of the original schedule it is kept to a minimum.

Sixth, when interviewing customers who are veterans, it is important to use the following guidelines: be understanding; be an active listener; listen with empathy, but minimize sympathy; and provide background and context to your interview questions.

These are some tips that I have learned over time.

Rad Resource: If you want to learn more I would suggest looking at the Council on Accreditation website www.coanet.org to learn more about the standards as well as opportunities to possibly become a military reviewer. It has been a very valuable experience and I gained a great deal of knowledge enhancing my evaluation skills.

The American Evaluation Association is celebrating MVE TIG Week with our colleagues in the Military and Veteran’s Issues Topical Interest Group. The contributions all this week to aea365 come from our MVE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Pat Clifford, a consultant working with the Tristate Veterans Community Alliance and Program Chair of the Military and Veteran Evaluation (MVE) TIG. A prominent issue for our collaborative and many other veteran support organizations is the successful transition from military to civilian life. When approaching this task, it’s important for evaluators to recognize some of the primary contextual drivers.

  • Military transition is a life transition: And, like other transitional periods, it comes with unique risk and protective factors.
  • Some stakeholders and factors are common: For example, there are similar national stakeholders like the Department of Veterans Affairs (VA), the Department of Defense (DoD) as well as a strong military culture. National changes like draw downs and restructuring impact everyone across the board.
  • However, transitions play out in local contexts: Each region has different demographics, organizational players, and linkages to bases and installations.
  • And the “Sea of Goodwill” can intensify problem: Many organizations and individuals are quick to start providing services thinking they are the only ones out there. This means that resources are often not targeted effectively or coordinated.

All of these factors contribute to a complex environment for evaluators. Often well-meaning programs and models (especially those that are implemented top-down) tend to run into roadblocks in practice. To counteract this, there are things that evaluators can do to engage with their local context and drive relevance. 

Hot Tips:

  • Prioritize relationships: Often programs are focused on evaluating programs as if they operated in a social vacuum. The reality is that program outcomes hinge on credibility with their target population, and that means relationships. Intentionally explore how programs build trust and display credibility. From my experience, word-of-mouth is key in the veteran space; and cultural missteps can lead to long-term consequences.
  • Help programs recognize and agree on roles: No organization can do everything well. Too often programs want to be the “one stop shops” to help solve all of a veteran or military families’ needs. Evaluators can work with stakeholders to think critically about their scope, identify their roles and set up processes to ensure they work in partnership, not in isolation.
  • Encourage leadership by empowering: Work with programs to explore how they create a space for indigenous leadership. While national models and top-down initiatives can help bring solutions “to scale”, often they work to disempower local leaders and veteran stakeholders. Evaluators can help become a voice for local knowledge and expertise that can help inform larger initiatives.

Rad Resources:

Tristate Veterans Community Alliance

VA Office of Policy and Planning

Ending Veteran Homelessness by Addressing Failed Transition Policies

The American Evaluation Association is celebrating MVE TIG Week with our colleagues in the Military and Veteran’s Issues Topical Interest Group. The contributions all this week to aea365 come from our MVE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

I’m Nathan Graeser with the Center of Innovation and Research on Veterans & Military Families (CIR) at the University of Southern California. Over the last three years, we have lead a collective impact movement dedicated to address reintegration challenges faced by our veterans, service members and their families. As the director/facilitator of the Los Angeles Veterans Collaborative, it has become increasingly clear that despite undergoing massive efforts to collect data on the needs of Veterans (see LA/OC Veterans Study), our strategies were still making little dent in the outcomes we hoped for. Our current efforts, while thoughtful and robust, still failed to address the scale and magnitude of the problem. Much more needed to be done to support creative outreach, better access to care, connecting Veterans back into their community, employment support, and better access to physical and behavioral health care. Despite the tremendous momentum and collective work being done, the list of needs and systemic failures was still daunting.

We needed a process to try out new strategies- based on data- to learn from and adapt our current efforts. Using a developmental evaluation process, we decided to encourage our community to begin to try out new ideas with the hope that we might be able to learn new ways of addressing some of these challenges. It was our hope that by trying out new ideas we might learn as a community and thus change the way we deliver our services and programs which, just maybe, might change the way Veteran experience transition in Los Angeles.

Lesson Learned: In order to encourage these new ideas in our collective impact movement, in October we launched a Community Initiative Grant offering pilots as a way to do this in real time. Members of the Los Angeles Veterans Collaborative now have access to seed money to try out new ideas with the specific purpose of sharing failures, successes and lessons learned to the larger community. Our hope is that this will encourage members to try our ideas- that they will work with the intended outcome to grow our community capacity, not necessarily only their own. The first seven pilots have been submitted and it has already been an amazing process to watch them unfold and the community behind them. We are looking forward to seeing what happens next- hoping to adopt strategies that will allow us to evaluate our be fast enough to meet the growing need for the thousands of Veterans and their families transitioning out of the military.

The American Evaluation Association is celebrating MVE TIG Week with our colleagues in the Military and Veteran’s Issues Topical Interest Group. The contributions all this week to aea365 come from our MVE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

My name is Sarah Baughman and I am the National Project leader for the Military Families Learning Network (MFLN). The MFLN connects the resources of Land Grant Universities and Cooperative Extension with military audiences. Working across institutional cultural contexts has been one of our challenges. Our academic institutions have very different cultures than the military and helping each learn to work with the other has been key to our success. The hierarchical nature of military service branches is distinctly different from the flat, individualistic culture of academia. Understanding the work culture of our partners and clients has improved our communications and our programming.

We’ve learned quite a bit as our program has matured and hope you find the tips and resources below useful in your work with military service members and their families.

Hot Tips: There are five branches in the military, each with a unique culture, language, values and identity. The five branches are: Army, Marine Corps, Air Force, Navy and Coast Guard. Be sure you are using the proper terminology and images with each branch.

Rank matters.

Check your assumptions. The popular narrative around military families is deficit based. While military families certainly have challenges unknown to non-military families they are also resilient.

Include or recruit an advisor or key staff member with military experience to help guide your work with service members and their families.

Operation Iraqi Freedom and Operation Enduring Freedom relied heavily on the National Guard and Reserves. Guard and Reserve families are more likely to be in rural areas or far from common military services available to active duty service members on or near installations. This presents unique opportunities and challenges for these families especially during times of deployment.

Military service members are mission oriented. The mission comes before the individual. Be clear about the mission of your program or project and who is tasked with each component.

Military families often do not identify themselves as military. Consider asking if clients are in the military in your program applications or intake forms to better serve those clients.

Rad Resources:

Dictionary of Military Terms

Understanding rank – What are those stripes and bars?

Guide to Military Uniforms and Insignia

Common military terms and lingo

Evidence based programs and practices for the military can be found at the Penn State Clearinghouse

Research on Military Families can be found at the University of Minnesota REACH program

The Military Family Research Institute at Purdue University

The American Evaluation Association is celebrating MVE TIG Week with our colleagues in the Military and Veteran’s Issues Topical Interest Group. The contributions all this week to aea365 come from our MVE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, I’m Stephen Axelrad. As the chair of the Military and Veteran Evaluation (MVE) Topical Interest Group, I am very excited to introduce this upcoming week for AEA365 dedicate to military and veteran issues in evaluation. Last month, we made our Annual Conference debut. This week is an opportunity of us to introduce our TIG to the broader evaluation community who were unable to attend our conference and are interested in deepening their understanding of the application of evaluation theory, methods, and practice in active duty military, veteran, military family, and national security settings.

Hot Tip: While the military and veteran community goes beyond the Departments of Defense (DoD) and Veterans Affairs (VA), both agencies produce many of the policies and programs that affect millions of Service members, veterans and their families. Getting smart on DoD and VA will give you a good foundation in understanding the issues affecting military and veteran evaluations.

Rad Resource: VA’s Veterans Policy Research Agenda available at http://www.va.gov/op3/docs/StrategicPlanning/VPRA_FY15_Published_Version_9_5_14.pdf describes issues in which the VA needs more evaluation. The Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury’s resource page at http://www.dcoe.mil/About_DCoE/Program_Evaluation.aspx describes how program evaluation is being leveraged to understand the effectiveness of military health and social support programs.

Lesson Learned: Cultural competency is a critical success factor for evaluators who have never served in the military but are working with military and veteran communities. These communities have their own norms, values, and vernacular that vary from the general population.   The Uniformed Services University of Health Sciences’ Center for Deployment Psychology and University of Southern California’s School of Social Work’s Military Social Work have great programs to help evaluators, applied researchers, and providers understand military and veteran communities.

The American Evaluation Association is celebrating MVE TIG Week with our colleagues in the Military and Veteran’s Issues Topical Interest Group. The contributions all this week to aea365 come from our MVE TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Archives

To top