AEA365 | A Tip-a-Day by and for Evaluators

Search

We are Caitlin Ruffenach, Researcher, and Kim Leonard, Senior Evaluation Officer, from The Oregon Community Foundation (OCF). Among other things, we are working on an evaluation of the Studio to School Initiative at OCF, which focuses on the development of sustainable arts education programs through partnerships between arts organizations and schools.

This past summer, in collaboration with the Oregon Arts Commission, we conducted a survey of arts organizations in Oregon in an effort to learn about the arts education programming they provide, often in concert with what is available more directly through the school system.

The purpose of this survey was to help the Foundation understand how the grantees of its Studio to School Initiative fit into the broader arts education landscape in Oregon. We hope the survey results will also serve as a resource for grantees, funders, and other stakeholders to understand and identify programs delivering arts education throughout the state.

Lesson Learned: To ensure we would have the most useful information possible, our survey design process included several noteworthy steps:

  1. We started with existing data; by gathering information about organizations who had received funding in arts education in Oregon in the past we were able to target our efforts to recruit respondents.
  2. We consulted with others who have done similar surveys to learn from their successes and challenges;
  3. We paid close attention to survey question wording to ensure that we were focusing as tightly on what was measurable by survey as possible; and
  4. We vetted our early findings with arts education stakeholders.

Hot Tip: A collaborative, inclusive survey design process can result in better survey tools. We used a small, informal advisory group throughout the process that included members who had conducted similar surveys and representatives of our target respondent group. They helped with question wording, as well as with identifying a small survey pilot.

Hot Tip: Vetting preliminary findings with stakeholders is fun and helps support evaluation use. We took advantage of an existing gathering of arts stakeholders in Oregon to share and workshop our initial findings. We used a data placemat, complete with re-useable stickers, to slowly reveal the findings. We then engaged the attendees in discussions about how the findings did or didn’t resonate with their experiences. What we learned during this gathering is reflected in our final report.

Resources: We are not the first to try a more inclusive process both in developing our survey tool and in vetting/interpreting the results! Check out the previous aea365 post about participatory data analysis. And check out the Innovation Network’s slide deck on Data Placemats for more information about that particular tool.

The American Evaluation Association is celebrating Oregon Community Foundation (OCF) week. The contributions all this week to aea365 come from OCF team members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

·

My name is Sophia Guevara, Program Co-Chair for the Social Network Analysis (SNA) TIG.  This week, several evaluation professionals have shared with this blog’s readers their thoughts on social network analysis. With posts discussing logic models to examples of the application of social network analysis on a wide-range of evaluation questions, you’ve hopefully gained a better understanding of it.

Rad Resource: The SNA in Evaluation LinkedIn group. This group provides TIG group members with an opportunity to discuss topics of interest for those utilizing or learning about social network analysis.

Rad Resource: Join the SNA TIG group. As a member, make sure to make use of the eGroup discussion option.

Rad Resource: SNA TIG business meeting. If you are thinking of joining the TIG or have already joined and are looking to connect with other evaluation professionals making use of SNA, the business meeting is an excellent place to do just that. The SNA TIG business meeting is held at the annual American Evaluation Association conference.

Rad Resource: AEA public eLibrary and the Coffee Break Archive. There are a variety of resources that can help you learn more about the topic. For example, if you are looking to learn more about the use of SNA related-programs, check out Dr. Geletta’s coffee break webinar focused on importing spreadsheet data into Gephi.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

No tags

Lopez KemisA

Hello from Andres Lazaro Lopez and Mari Kemis from the Research Institute for Studies in Education at Iowa State University. As STEM education becomes more of a national priority, state governments and education professionals are increasingly collaborating with nonprofits and businesses to implement statewide STEM initiatives. Supported by National Science Foundation funding, we have been tasked to conduct a process evaluation of the Iowa statewide STEM initiative in order to both assess Iowa’s initiative and create a logic model that will help inform other states on model STEM evaluation.

While social network analysis (SNA) has become commonly used to examine STEM challenges and strategies for advancement (particularly for women faculty, racial minorities, young girls, and STEM teacher turnover), to our knowledge we are the first to use SNA specifically to understand a statewide STEM initiative’s collaboration, growth, potential, and bias. Our evaluation focuses specifically on the states’ six regional STEM networks, their growth and density over the initiatives’ years (‘07-’15), and the professional affiliations of its collaborators. How we translated that into actionable decision points for key stakeholders is the focus of this blog.

Lessons Learned: With interest in both the boundaries of the statewide network and ego networks of key STEM players, we decided to use both free and fixed recall approaches. Using data from an extensive document analysis, we identified 391 STEM professionals for our roster approach. We asked respondents to categorize this list by people they knew and worked with. Next, the free recall section allowed respondents to list professionals they rely on most to accomplish their STEM work and their level of weekly communication – generating 483 additional names not identified with the roster approach. Both strategies allowed us to measure the potential and actual collaboration along the lines of the well-known network of STEM professionals (roster) and individual’s local networks (free recall).

Lopez KemisB

Lessons Learned: The data offered compelling information for both regional and statewide use. Centrality measurements helped identify regional players that had important network positions but were underutilized. Network diameter and clique score measurements informed the executive council of overall network health and specific areas that require initiative resources.

Lessons Learned: Most importantly, the SNA data allowed the initiative to see beyond the usual go-to stakeholders. With a variety of SNA measurements and our three variables, we have been successful in identifying a diverse list of stakeholders while offering suggestions of how to trim down the networks’ size without creating single points of fracture. SNA has been an invaluable tool to classify formally and evaluate the logistics of key STEM players. We recommend other STEM initiatives interested in using SNA to begin identifying a roster of collaborators early in the development of their initiative.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

 

Hi, I’m Rebecca Woodland, an Associate Professor of Educational Leadership at UMass Amherst. If there is one thing that I know for certain it’s that relationships matter and how we are connected influences the quality and outcomes of our shared endeavors. Social Network Analysis (SNA) has had a profound influence on my evaluation work. I want to introduce and encourage evaluators (who may not know much about SNA) to consider integrating it into their own practice.

Simply put, SNA is all about telling the story of how “ties” between people or groups form, and how these “links” may influence important program objectives and outcomes. With SNA you can mathematically describe and visually see connections between people. You can use SNA to explain and predict how ties between “actors” influence the attainment of program goals.

Hot Tips: Evaluators can use SNA to address a wide-range of pressing program evaluation questions such as these:

  1. Want to know whether a program has the capacity to spread a new or novel intervention? SNA was used to evaluate school-level capacity to support or constrain instructional innovation.
  2. Want to know how large, inter-agency partnerships develop and how inter-agency collaboration correlates with intended program outcomes? Evaluators used SNA to track the development and impact of a Safe Schools/Healthy Students inter-agency community mental health network.
  3. Want to know who influences the budgeting and disbursement of funds for advocacy programs in fragile environments? SNA was used to map the flow of resources and funding patterns for new-born survival activities in northern Nigeria.

Lesson Learned: Possibly the biggest wow factor is that SNA enables the creation of illustrative visuals that display complex information, such as intra-organizational communication flow and the location of network “brokers,” “hubs,” “isolates” and “cliques”, in user-friendly ways.

WoodlandImage via under Creative Commons Attribution 3.0 License

Rad Resources

  • ®Visualyzer is an easy to use program (with a 30-day free trial) that enables you to create socio-grams on any network of interest to you.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m John Burrett, of Haiku Analytics Inc., Ottawa. One serious problem with logic models is that they usually leave out external influences and feedback effects, even when they may be important, because they make the model “too complex”. It is good to simplify, but ignoring important influences on program success when planning an evaluation may lead evaluators to fail to collect important data and to misinterpret results.

Trying to embrace complexity by drawing a web of boxes and arrows is not helpful: it’s too complex to use and explain and will drive your audience away. This will probably come only from the mind of the evaluator or program manager, thereby easily missing important external influences and other complexities.

Hot Tip: I recently stumbled onto an alternative approach during a mapping of factors of cause and effect related to a complex policy problem. Data was obtained from an expert panel, developing a matrix linking a number of factors with an estimate of strength and direction of relationship between them. Mapping this with network analysis software helped the panel to visualize what they had created.

It followed that this form of data could generate outcomes chains and logic models. Here’s a simple example: a program supporting trades training by providing grants to students and developing state of the art teaching materials in collaboration with trade schools drives the immediate outcomes of…

  • Students gaining the ability to take training and
  • Currency and quality of the training being improved, in order to achieve
  • The ultimate outcome of increased employment.

Exogenous effects influencing these results include cost of living, demand for skills and technical changes affecting the training’s currency. The size of the nodes indicates betweenness centrality, identifying those factors that connect many influences, thus propagating certain effects. The width of the edges indicates the hypothesized strength of influence. Possible unintended effects and a feedback loop are also shown.

Burrett

Lesson Learned: A key advantage of this approach is that that it creates a logic model using expert knowledge, rather than simply an evaluator/manager’s understanding of a program. This could also include other sources of information like findings from literature and program stakeholders’ experiences. Importantly, you could do this without imposing any prior idea of the logic model on those providing the cause-effect data other than including the program/outputs/activities and specifying the immediate/intermediate and ultimate intended outcomes.

A second major advantage is that the logic model utilizes network metrics generated from the data, so how the program and influences are expected to be related can be analyzed. For instance, factors that are thought to have an important role in propagating effects across the system would show high betweenness/eigenvector centralities.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Bethany Laursen, an independent consultant and evaluation specialist for several units at the University of Wisconsin. I fell in love with social network analysis (SNA) as a graduate student because SNA gave me words and pictures to describe how I think. Many evaluators can relate to my story, but one of the challenges to using SNA in evaluation is identifying what counts as a “good” network structure.

Hot Tip: Identify keywords in your evaluation question(s) that tell you what counts as a “good” outcome or goal for your evaluand. An example evaluation question might be: “How can we improve the collaboration capacity of our coalition?” The stated goal is “collaboration capacity.”

Hot Tip: Use published literature to specify which network structure(s) promote that goal. Social networks are complex, requiring rigorous research to understand which functions emerge from different structures. It would be unethical for an evaluator to guess. Fortunately, a lot has been published in journals and in gray and white papers,

Continuing our example, we need to research what kinds of networks foster “collaboration capacity” so we can compare our coalition’s network to this standard and find areas for improvement. You may find a robust definition of “collaboration capacity” in the literature, but if you don’t, you will have to specify what “collaboration capacity” looks like in your coalition. Perhaps you settle on “timely exchange of resources.” Now, what does the literature say about which kinds of networks promote “timely exchange of resources”? Although many social network theories cut across subject domains, it’s best to start with your subject domain (e.g. coalitions) to help ensure assumptions and definitions mesh with your evaluand. Review papers are an excellent resource.

Lesson Learned: Although a lot has been published, many gaps remain. Sometimes the SNA literature may not be clear about which kinds of networks promote the goals you’ve identified for your evaluand. In this case, you can either 1) do some scholarship to synthesize the literature and argue for such a standard, or 2) go back to your evaluation question and redefine the goal in narrower terms that are described in the literature.

In our example, the literature on coalition networks may not have reached consensus about which types of networks promote timely exchange of resources. But perhaps reviews have been published on which types of brokers foster diversity in coalitions. You can either 1) synthesize the coalition literature to create a rigorous standard for “timely exchange of resources,” or 2) reframe the overall evaluation question as, “How can brokers improve diversity in our coalition’s network?”

Rad Resources:

This white paper clearly describes network structures that promote different types of conversations in social media

This short webinar reports a meta-synthesis of which networks promote adaptive co-management capacity at different stages of the adaptive cycle

Different network structures promote different system functions. This is the take home slide from the Rad Resource on ACM capacity. In this case, the evaluand's network goal is timely social learning, collective action, and resilience.

Different network structures promote different system functions. This is the take home slide from the Rad Resource on ACM capacity. In this case, the evaluand’s network goal is timely social learning, collective action, and resilience.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m Maryann Durland, an independent evaluator and Social Network Analysis (SNA) practitioner. In this post I will address the requirements for doing an SNA application, particularly in evaluation, and which we could also call the standard for an application. I will use the early literature that formed and grounded SNA thinking and that continues to be relevant.

Early on in the history of defining SNA, Linton C. Freeman, described four requirements for completing a social network analysis in his book, The Development of Social Network Analysis: A Study in the Sociology of Science:

  1. A structural perspective
  2. Empirical data
  3. Graphics
  4. Mathematical models with analysis

I believe and promote, particularly in evaluation applications, that these are still the requirements for meeting the standard for doing SNA. Evaluations using SNA are distinct from research on SNA theories and measures, which may have different requirements.

In evaluation applications structural perspective means that we can define relationships within the program and these relationships create a structure through which information flows, resources are found, barriers are identified, spaces are found that need connections, and so on. Data is the existence or non-existence of a relationship between two elements. Empirical data refers to verifiable data collected on the relationship between any two elements, also called the members of a set.

Just like traditional data collection, we collect relational data through a variety of methods from observations to surveys about experiences. The data we collect populates matrix cells, indicating the presence or degree of a relationship between two members. Graphics indicate that we can visualize the network, results and/or analysis in graphs such as sociograms. Mathematical models with analysis allow us to calculate SNA measures which are measures of the network, not attributes assigned to individuals. Models called algorithms or a set of procedures, are as much a description of the relationship as they are the algorithm for how to calculate a measure.

Lesson Learned: Clearly, these four requirements delineate a specific methodological basis that is different from traditional quantitative and qualitative analysis. These requirements mean evaluators must think differently, ask questions for a different purpose, and conceptualize an evaluation differently.

Rad Resource: Early literature on SNA sought to develop what we could call standards for applications and one of the most important resources is the work of Linton C. Freeman. Freeman’s work continues to set the standards for SNA applications and the reference for the requirements.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hi, I’m Maryann Durland, Co-Chair of the Social Network Analysis TIG and owner of Durland Consulting, where we specialize in social network analysis. In this post, I show how to enter data into UCINET for a network where you are measuring multiple types of relationships.

An NGO recently wanted to understand how a network of community organizations connected and what each saw as the strengths of the network members. This resulted in 15 different types of collaborative relationships based on the different activities that the network as a whole engaged in. Collaborative activities included collecting data, drafting reports, presenting reports, lobbying, recruiting other members, networking, and planning for the future.

UCINET accepts a variety of data formats. One common approach is to load data from an excel file(.xls, .xlsx, .csv) into a DL format. DL stands for “data language,” and the DL statement at the beginning of the file describes the data, including the number of nodes, labels, and the type of format. UCINET provides a menu-driven process for uploading data and converting to DL. In UCINET, select Data>Import Excel>DL-type formats and copy and paste your data into the DL Editor or open your Excel file using File>Open Excel file. Under Data format, select the appropriate format and other details about your data. For my NGO project, I selected Edgearray1 (ego alter rel1 rel2) to match the layout of my data, which was in the following order: the chooser (ego), the chosen (alter), and the type of collaborative activities (rel1, rel2, …). Though I have column labels, I do not need the labels chooser and chosen,and these will be deleted when I save the file.

Durland 1

Now you will have all relationships saved in one file. When you run measures for the network, you will run them on all relationships at the same time. To visualize the network, you can pull up the data in NetDraw (the network visualization program packaged with UCINET) and code for each relationship and show all of the relationships together in one sociogram or separately:

Durland 2

In the map above, the color of the arrows provides information on the type of collaborative activities reported by community organizations. For example, member 2 reported that they draft reports with member 3 (blue arrow), and member 3 reported they collect data with member 1 (red). The black arrow means a member said that they worked on lobbying efforts together. If members indicated multiple relationships, then the arrows are gray.

Rad Resource: UCINET 6 User’s Guide includes a section on importing data that illustrates how to format various DL files—explore here to learn about the possibilities!

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Greetings! I am Simon Geletta – Program Co-Chair for the SNA-TIG and associate professor of public health atDes Moines University in Iowa. In this post, I illustrate how I used the open-source softwareGephi to conduct a social network analysis of hospital visitation patterns in Iowa.

Hospital utilization patterns are important to healthcare services research because they relate to the concept of access. Using SNA revealed more insight than other more “traditional” analytic approaches such as statistical summary tables and GIS maps, which provide insight on abstract (the statistical approach), or spatial patterns (the GIS approach). SNA allowed me to look at patterns of hospital use in the network, which shed further light on patient access to healthcare services.

Lesson Learned: Getting Data into Gephi. Originally, my data were stored in an Excel spreadsheet. To import the data, I used Gephi’s “Data Laboratory” interface (shown below). In my analysis, I used hospital locations and patient locations as nodes. The edges linked the patient location nodes to the hospital location nodes.

Galetta 1

Hot Tip: You can import data into Gephi from a range of standard data formats, including spreadsheets and delimited files (comma separated, tab delimited, etc).

Lesson Learned: Visualizing Data in Gephi. In Gephi, I used the “Overview” window to visualize and analyze my data. To get an intuitive view of the hospital visitation network, I selected theForce Atlas 2 layout option, which balances speed and precision.

Galetta 2

Hot Tip: Gephi includes 12 layout settings – fit for a wide array of network sizes and complexity. After applying layout settings, you can also tweak your layout to make it less cluttered.

Lesson Learned: Analyzing Data in Gephi. I used the modularity analysis function to identify communities within the network. The analysis revealed five closely knit hospital visitation patterns or communities that reflected geographic characteristics (i.e., most visits were to hospitals closest to the visitors), and organizational characteristics (larger hospitals attracted more patients – not only from their immediate locality but also patients that were further away from their localities.)

Galetta 3

Once the communities were delineated, the “Overview” window also allowed me to filter the network by the community partitions and evaluate each community as a separate network. You can export and use measures from Gephi for modeling and hypothesis testing using other statistical software.

Rad Resource: For more information on the modularity analysis function, check out this article byV. D. Blondale and colleagues.

Lesson Learned: Selecting an appropriate layout for your network is an iterative process. It is not an “exact” science but a combination of science, art, and aesthetics.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello there! This is Jes Koepfler, Principal of UXR Consulting in Philadelphia, PA, and a PhD candidate at the University of Maryland in Information Studies, and Derek Hansen, Assistant Professor at Brigham Young University in the Information Technology department and one of the team members behind NodeXL. We are working with some of our colleagues at the Information Policy and Access Center at the University of Maryland to conduct an outcomes-based summative evaluation of an online civic engagement platform called ACTion Alexandria. As part of our multi-year, mixed-methods study, we’re using NodeXL to analyze social network data to address questions about community engagement through site participation.

Lesson Learned: We created the graph below using NodeXL. It shows the 5 key types of participation that visitors can engage in on the ACTion Alexandria website. In this graph, each dot represents a site visitor. Color represents how many activities site visitors have performed (e.g., orange=all 5 activities, dark blue=just 1 activity). Size also indicates the number of activities site visitors have performed. Each line represents an activity the site visitor engaged in on the website. The thickness of the lines (also referred to as edge thickness) shows the number of times a site visitor has performed an activity.

The graph highlights the following things about site participation:

  1. Most people only perform one activity, although a significant number perform two activities.
  2. Voting is by far the most popular way that people engage with the site.
  3. Blog posters post many times, but don’t tend to comment on each other’s blogs.
  4. People who engage with multiple types of activities are more likely to engage in them multiple times (i.e., have thicker lines).

Koepfler 1

Hot Tips:

  • Bi-modal networks: Social network analysis is commonly used to understand relationships between people, but it can also show interesting patterns of relationships between people and things (like activities on a website). These are called bi-modal networks.
  • Network graph aesthetics: Color, labels, line thickness, and size can all be used simultaneously to represent several types of information from your data in a network graph.
  • Manual layout: Network graphs are easier to read when as few lines overlap each other possible. In NodeXL, use the manual layout feature to ‘untangle’ any criss-crossing edges or to overlapping nodes.

Rad Resources:

  • Chapter 4: Getting Started with NodeXL, Layout, Visual Design, and Labeling of Analyzing Social Media Networks Using NodeXL provides a beginner-level, how-to guide for manipulating network graph aesthetics.
  • Check out the NodeXL graph gallery for ideas on ways to use network graph aesthetics to tell different stories with your data.

The American Evaluation Association is celebrating Social Network Analysis Week with our colleagues in the Social Network Analysis Topical Interest Group. The contributions all this week to aea365 come from our SNA TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Older posts >>

Archives

To top