Outlier Research and Evaluation, CEMSE | University of Chicago conducted one portion of The Source’s overall evaluation effort with a particular focus on STEM, using qualitative data collection methods. The Source team engaged a range of investigators to evaluate other components and address questions of importance to the program leaders. In this role, Outlier evaluators communicate closely with the project team while still maintaining an outsider point of view. Through this process, Outlier evaluators embrace a collaborative approach to program improvement by providing feedback, findings, and recommendations for program improvement.
Guiding Evaluation Principles
Outlier is committed to the following evaluation principles:
Evaluation must be grounded in a commitment to conducting rigorous studies of meaningful questions.
Rigorous studies use valid, reliable, and systematic data collection methods including qualitative and quantitative approaches. Those methods are of little use, however, if the questions they seek to answer are of little or no consequence to the stakeholder(s) or the field. Therefore, meaningful evaluation questions are informed by client and stakeholder needs and interests, and thus contribute to understandings about a program within a local context. Where appropriate to the program and design, evaluation questions also respond to broader concerns in the field.
Evaluation designs must provide useful and practical information to support improved practice.
Although evaluation can contribute to the development of theory, unless it provides information that is applicable and important to the clients, it has failed in its major purpose. Historically, evaluators have lamented the fact that their findings, whether part of formative or summative evaluation efforts, are often not used by the immediate or any extended audience of stakeholders. Outlier is committed to evaluation designs that result in practical, useful findings and to developing and maintaining relationships with stakeholders that increase the likelihood that those findings will be utilized.
Evaluation designs must appropriately match the contexts and conditions of the project and be sensitive to the needs and interests of the stakeholder(s).
The value of an evaluation design comes from the extent to which it is well matched to the circumstances of the project and the needs of project leaders. Only when those needs are defined and circumstances accounted for, can evaluators generate a useful, appropriate evaluation design.
The Evaluation of Ci3’s The Source
The Source is a STEM-based alternate reality game initiated by Ci3 and Game Changer Chicago Design Lab. Outlier’s portion of the larger evaluation effort had two strands. The first strand focused on youth attitudes toward STEM (science, technology, engineering and mathematics) and STEM careers. The second strand focused on the youth’s experience with The Source itself and their perspectives on the parts of The Source that impacted them the most.
Across the two strands, the evaluation focused on three main goals: 1) Describing players’ attitudes about STEM and STEM careers and their perspectives on how participation in The Source affected those attitudes; 2) Describing youth’s perspectives on their development of particular “21st Century Skills” and parts of The Source that had the greatest impacts on them; and 3) Identifying and describing the parts of The Source experience that mentors and participating youth felt were the most and least successful.
With these goals in mind, the evaluation plan targeted three questions:
- To what extent do youth feel that their experience with The Source has affected their abilities to enact twenty-first century skills?
- To what extent do youth feel that their experience with The Source has affected their attitudes toward STEM in general, STEM self-efficacy, and STEM careers?
- What do group leaders and youth suggest for improving The Source?
The Source’s whole evaluation effort includes other questions examined by other parties through questionnaires. The evaluation questions proposed above were measured using only qualitative data collection methods. Data sources include eight youth focus groups, ten youth interviews, eight mentor interviews and examination of youth work product.
The Evaluation Findings and this Report
Outlier provided The Source leadership with formative recommendations regarding the implementation of The Source. Summative findings about The Source and recommendations are presented in this report. This report has been created with password-protected access for different report audiences.