Hello! My name is Jacqueline Singh, MPP, PhD., Executive Advisor Evaluation and Program Design at Qualitative Advantage, LLC. I’m also Co-chair of the AEA Program Design TIG and Board Member at Large, Policy Committee Chair for Indiana Evaluation Association.
Over the years, I’ve learned that program implementors and decision makers are incentivized to advance well designed and impactful programs—or, policies. Funding choices are more decisive when conceptualizations are explicit. As such, I have found that document models are invaluable to program design and evaluation. They facilitate participation early in the evaluation process, by capturing how programs are conceptualized in formalized documents (e.g., RFP’s, grant proposals, literature, websites, etc.). Document models identify goals and objectives; expose problematic areas; clarify assumptions; and, highlight areas needing refinement (Rutman, 1980). In addition, they sustain stakeholders’ curiosity and involvement throughout various data inflection points to identify the types of evidences needed to answer overarching evaluation questions.
Document models are practical tool that add strategic value by mirroring the content of program-related documents in an unbiased way. They take stock of what exists now; work out conceptualization issues in a graphic; and, help sort out issues surrounding program design, evaluation, accountability, transparency, and equity. Stakeholders come to recognize how the evaluator understands their intervention. In addition, they are drawn into conceptualization processes (e.g., conceptual frameworks, logic models, program theory, etc.) that guide evaluation and/or research. Working from the known (i.e., existing documents) to the unknown (e.g., finalized evaluation plan) facilitates buy-in (or, ownership) into a program’s design, its implementation, evaluation questions, and agreed upon evaluation design/plan.
The power of document models is in how they capture what logic models alone can miss. They bring into focus the formalized commitments made to funders (e.g., philanthropy, legislators, etc.). Using document models enhances the relationship evaluation has with policymaking, evidence-based or evidence-informed decision-making, and strategic planning.
For this AEA365, a document model illustration was created using a U.S. Department of Education Federal TRIO program website description—Educational Opportunity Centers (EOC). https://www2.ed.gov/programs/trioeoc/index.html
Hot Tips:
- Analyze documents to create an initial model, or series of models, that help stakeholders understand the inner workings of a program (or, policy) to “see” how the intervention is formally described.
- Stress the importance of engaging in front-end planning activities (e.g., evaluability assessment) for any intervention under examination.
- Creating document models is experiential and helps build evaluation capacity. They facilitate building trust with and among multiple stakeholders—and, establish a greater sense of transparency for a more equitable dynamic between all individuals involved in the evaluation.
Rad Resources:
- Rutman, L. (1980). Planning useful evaluations. Beverly Hills: Sage Publications.
- Shakman, K., & Rodriguez, S. M. (2015). Logic models for program design, implementation, and evaluation: Workshop toolkit (REL 2015-057). Retrieved from: https://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/REL_2015057.pdf
- Whitt, J. E. (2001). Document Analysis. In C. F. Conrad, J. G. Haworth, & L. R. Lattuca (Eds.), Qualitative research in higher education: Expanding perspectives (2nd ed., p. 447). Boston, MA: Pearson.
The American Evaluation Association is celebrating Program Design TIG Week with our colleagues in the Program Design Topical Interest Group. The contributions all this week to aea365 come from our Program Design TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.