Recommendations
Techie Club Recommendations
Techie Club evaluation recommendations are based on fall and spring 2013-2014 qualitative and quantitative data collection and analysis.
-
Recommendation 1. Examine site selection process.
In both the fall and spring, volunteers and students reported in interviews and focus groups that issues associated with technological barriers, whether that meant access to working computers or dealing with district firewall policies, affected club experiences. TECH CORPS should closely consider site selection and incorporate a process to determine which schools truly have the capacity to host a club. Capacity includes (but is not limited to) having truly sufficient technological resources (number and quality of computers, reliable and open internet access). Additionally, TECH CORPS should consider how to best prepare volunteers for technological issues that may arise during clubs. Even with the best resources and access, it is likely that there will be some tech glitches over the course of the year. In addition to technology sufficiency, volunteers often reported that school-based support was critical to success. Techie Club site selection should consider strong school leader and staff support as a requirement.
Consider an alternative strategy for sites that are interested in Techie Club, but do not have reliable or sufficient resources. How might TECH CORPS provide these sites with resources to expose students to technology?
Implementation has been a concern across a number of sites because of lack of technical resources, curriculum insufficiencies for advanced students, and lack of volunteer readiness (described below). Outlier strongly recommends that TECH CORPS consider reducing the number of club sites to a select, viable sample to focus on high-quality delivery prior to scale-up.
-
Recommendation 2. Evaluate the Techie Club Curriculum and determine the intended audience.
Techie Club was originally meant to be only a one-year program for elementary students. Returning students and middle school students showed no increase in their knowledge and skills from pre to post Club. In addition, while students new to Techie Club showed some increase in knowledge and skills (overall non-significant across all gender, age, race/ethnicity groups), this difference was primarily seen in White students. While a lack of pre-post change in knowledge and skills for minority students may be attributed to implementation issues at particular sites (although dramatic site effects were not noted), there may be other ways to more closely address the needs of these students. Intrinsic motivation (interest) in computer science and technology also decreased for girls at a higher rate than boys. This suggests not only implementation issues but a differential experience for girls. Thus, TECH CORPS should closely consider how to better appeal to and reach students from a variety of demographics. These may include curriculum adjustments and volunteer recruitment and training (see below).
If Techie Club continues to be a multiple year experience for students, the curriculum should better address how volunteers can differentiate instruction for a variety of student skill sets. Quantitative data showed that returning students showed no change in knowledge and skills, which may be attributed to the fact that they are being taught the same curriculum from the prior year. This finding was supported by student feedback in focus groups. If TECH CORPS wants to continue including returning students in clubs, they need to address how to differentiate content based on the skill level of students within Techie Club. In addition, some students felt that there was too much guidance and wanted a more self-directed experience. TECH CORPS may consider how best to work with volunteers to facilitate a more hands-on experience for students.
TECH CORPS staff reported that Techie Club had approximately a 25% to 30% attrition rate across the year. Future data collection may include a closer examination of students who do not continue with the program. Are students leaving the program at key points? Are certain students dropping out at higher rates than others? Do external factors lead to drop-out, and if so, how may these be addressed?
Reconsider the depth vs. breadth curriculum balance. In the questionnaires, students frequently cited parts of the computer, Scratch, and robotics among their favorite parts of Techie Club. During the focus groups, we heard from students that, though they liked robotics, they didn’t like how they didn’t have much time building the robots, and they didn’t like that it was already programmed for them. Rather than cover so much over the course of the year, revisit curriculum to consider a deeper dive on some topics that may resonate most with students (parts of the computer, Scratch, robotics).
-
Recommendation 3. Revisit Techieloka strategy.
Despite observed and reported student excitement across sites about Techieloka, only 4 of the 14 clubs participated in Techieloka this year. Volunteers expressed frustration with trying to figure out how to get kids to Techiloca across town after school. Those sites which were unable to attend Techieloka were on their own when it came to planning an end of year program for the students, often adding additional burden and cost to volunteers. If TECH CORPS would like to continue to make Techieloka a cornerstone of the program, more resources and planning need to be committed to facilitating attendance. Alternatively, TECH CORPS might consider site-specific events for some or all schools.
-
Recommendation 4. Strategize volunteer recruitment, preparation, and resources.
Techie Club has capitalized on the community resources of corporate volunteers for club instruction. While this strategy has merit, TECH CORPS should reconsider its recruitment strategy for greater focus and selectivity. TECH CORPS may put additional resources into recruiting minority and female volunteers. In addition, a more rigorous evaluation process may be necessary to select candidates who have the capacity to be effective instructors. While this may be a challenge, it relates to the above recommendation of scaling back the number of TECH CORPS sites. Focusing on implementing high-quality content with exemplary volunteers should be the first strategy of Techie Club. Once successes are well-articulated, scale-up is more feasible.
Volunteers would also like to know more about the student population of their club and the technology resources that will be available to them prior to starting Techie Club.
Techie Camp Recommendations
Techie Camp evaluation recommendations are based on summer 2014 qualitative and quantitative data collection and analysis.
-
Recommendation 1. Examine how best to meet the needs of female and minority campers.
As with Clubs, closely examine how best to target female and minority campers. Data from the questionnaire suggests some gender and race differences. Girls have significantly lower intrinsic motivation (interest) for computer science. African-American, Mixed, and “Other”-identified students also have lower intrinsic motivation for computer science/technology.
Note however, that in the absence of a pre-questionnaire, we do not know whether gender, race, and age differences existed before Techie Camp. Thus, it’s possible that gaps widened, narrowed, or stayed the same over the camp experience. It is only noted that at post-test, gaps are evident, and thus TECH CORPS may consider additional focus to best reach all campers.
-
Recommendation 2. Consider camp goals and how to best achieve them for students of all ability levels.
When asked about their self-efficacy (confidence) in skills related to Techie Camp programs, most students expressed modestly positive rates. Approximately 40% to 50% of all students across program types “agreed a lot” with items that describe Techie Camp goals, such as “I am good at programming a robot,” and “Even if a computer problem is difficult, I know I’ll be able to solve it if I try.” Approximately 30% to 40% indicated that they “agree a little” with these statements. Approximately 15% to 20% of students indicated that they disagreed (a little or a lot) with these statements. Thus, more differentiation may be needed for thissmaller number of students.
The agreement rates were slightly lower for the App Development program, with about a third of campers agreeing “a lot” with statements such as “I am good at developing apps on a cell phone” and “I know enough to teach myself how to do things in App Inventor.” Approximately a third of students also disagreed with the statement “I am good at developing apps on a cell phone.” Although in general, most students do express some confidence in skills related to App Development, TECH CORPS may reconsider how to best help students achieve goals in this area, and possibly consider the goals themselves.
One major reason for the variation in confidence may be the variety in incoming skill levels for campers. Both students and instructors often expressed frustration with the lack of differentiation within camps. This issue may be solved in one of two ways. First, TECH CORPS may consider having separate camps for students new to the respective technology and experienced with the respective technology. TECH CORPS may also consider focusing the camp to better suit one group or the other. Second, if separating or limiting camp audiences is not feasible, TECH CORPS should invest more time and development into the curricula and instructors to adequately prepare for a variety of student ability levels. Providing instructors with tools to effectively differentiate the curriculum will allow them to provide all campers with a fulfilling experience.
-
Recommendation 3. Consider alternate structures for camp instructor training.
Some camp instructors, particularly those new to Techie Camp, felt that they needed more substantial training. Camp instructors often expressed that they wanted training content to be more relevant to their specific camp program, and more substantial at that.
The training for camp instructors combined both new and experienced instructors. Both groups reported the training sessions to be helpful in the area of classroom management. New instructors reported that they were able to learn from more experienced staff, though explicitly stated that they needed more.
TECH CORPS will need to customize the professional development based on their teacher population. We recommend they provide separate professional development experience for instructors new to the program or have breakout sessions during the workshop that addresse the specific needs of new and experienced instructors. For newer instructors, these sessions may focus on the delivery of the curriculum, the use of materials specific to their curriculum, curriculum review, and an opportunity to practice teaching a lesson.
-
Recommendation 4: Evaluate the size of each camp and respective material/equipment sufficiency.
Although equipment and resource sufficiency was much less of an implementation challenge for camps than for clubs, gaps were still reported by some campers and instructors. When planning the size and scope of each camp, re-evaluate the sufficiency of materials available.