Outlier Research and Evaluation
Evaluation of Code.org Computer Science Education Programs
Elementary - Computer Science Fundamentals – Interviews and Questionnaires
Outlier Research & Evaluation conducted an evaluation of the elementary (K5) Computer Science Fundamentals Courses. The evaluation asked elementary school teachers nationwide how they used the instructional materials during their first year of implementation (2014-15). This report provides detailed information about implementation and how teachers used the courses in different school contexts, in different roles, and with different student populations.
The report also provides program improvement information, including teachers’ reporting of the supports they would need in the coming year to be able to continue teaching the course; whether students had the skills necessary to participate in the courses; and whether they would modify or supplement the instructional materials.
Each teacher that participated in Code.org PD for the CS Fundamentals Courses 1-4 were emailed a link to the survey and were invited to participate. About 7000 teachers were emailed, including teachers that were trained by Code.org affiliates and Code.org staff. A total of 1,464 respondents started the questionnaire, and 744 completed enough items to be used in analysis.
Nearly three-quarters of the questionnaire respondents reported that they worked with more than one section of students. Note that hereafter, “tech facilitator” will refer to teachers that teach multiple sections of students, and “classroom teacher” will refer to teachers that teach only one section of students, unless otherwise noted.
Fifteen teachers were also interviewed, to gather rich information about their experiences with the instructional materials.
HOW DO TEACHERS USE THE MATERIALS?
Most questionnaire respondents used Courses 1 and 2, while about a third used Course 3. Only a small number (8%) used Course 4.
The majority of questionnaire respondents reported using Courses 1 and 2, with 78% and 76% using each course, respectively. Only about a third (31%) of respondents said that they used Course 3, and only 8% reported using Course 4. Only about half or fewer teachers reported using all or more than half of Courses 1 and 2; and only about a third of teachers reporting using all or more than half of Course 3. Most teachers that reported using Course 4 only used “some,” and only a total of 57 teachers reported using Course 4 at all.
All interviewed teachers reported that all of the students in their classroom participated in CS Fundamentals courses. More than half (59%) of questionnaire respondents also agreed or strongly agreed that all of their students participated in the Courses.
The majority of teachers reported that they used the Courses with all of their students. Keep in mind that almost ¾ of the respondents were tech facilitators, meaning that they see more than two sections of students. Questionnaire respondents were also asked specifically about specific groups of students, and whether they used the Courses with those groups. A quarter of all respondents either agreed or strongly agreed that they used the Courses almost exclusively with students that expressed an interest in CS. Another 15% agreed or strongly agreed that they mostly used the Courses with their higher-achieving students, and 10% agreed or strongly agreed that they used the Courses mostly with their lower-achieving students. Thirteen percent (13%) agreed or strongly agreed that they let students decide on their own whether to participate. Finally, 10% of respondents disagreed or strongly disagreed that all of their students participated in the Courses.
The majority of interviewed teachers used Course 1 for K , 1st and 2nd grades; Course 2 for 3rd and 4th; and Course 3 for 4th and 5th. Most questionnaire respondents said that they used Course 1 for grades K, 1st, and 2nd; Course 2 for grades 3rd, 4th, and 5th; and Course 3 for grades 3rd, 4th, and 5th.
Most interviewed teachers decided to start their kindergarten, first, and second grade students on Course 1, and their older students (3rd through 5th grade) on Course 2. As one interviewed teacher said:
I think, honestly, if they start with Course 1 the older kids would get a little bit bored with some of the Course 1 stuff and then move more into Course 2 instead of going through all of Course 1. There were times that we would end up in the Course 2 before they completely finished the Course 1, some of my 3rd and 4th graders.
At least one teacher reported that she assigned students to a course based on ability rather than grade. She said, “Once we did Hour of Code I kind of divided them up. If they were sufficient and they were pretty confident on the Frozen and just the playing around part of it, they started with 2. If they finished with 3, then I could have them go back and help out with younger children.” She added that her students also took the Courses home, and were supported by their parents. See Tables 12–14 for detailed information on the grades that used Courses 1-3.
Fewer than half of questionnaire respondents reported using pair programming as an instructional strategy. Several teachers reported that paired programming was challenging for their students. Others reported that their classroom or school situation didn’t allow for paired programming.
Only 47% of teachers reported using pair programming as an instructional strategy. In interviews, several teachers said that their younger students in particular were unable to effectively participate in paired programming. For example, one said, “I don't think I want to try that yet with kindergarten, and some classes are incapable of it just because of behavior. I tried it and it went out of control, and so we couldn't.” Another said, “Oh, we did that once and that was a big struggle for second graders, because they can't break down the steps as well as you would think.” A third teacher said, “we have a lot of only children in our classroom. The idea that the navigator is just as important or more important than the pilot is hard for them to understand because I don't think they are truly listening to each other.”
Other teachers said that their school context was not conducive to pair programming. As one teacher said, “I didn’t because we are a one to one school, and so every child has their own iPad…they would log into their own account ant they liked to see their own progress.” However, this teacher said that students would work together at times, especially when they hit a roadblock. Working with a partner to solve a specific challenge, however, is not the same as one student acting as a ‘driver’ and the other acting as a ‘navigator.’
The teachers that did report using paired programming said that it was best used when students that ‘get it’ are paired with students that don’t. One teacher explained, “I’ll have the one that’s really getting it navigate, and have the other kid do the driving part. Eventually you see them start picking it up and moving a little quicker on their own.” Another teacher, a tech facilitator, said that effectively pairing students was “the biggest challenge I had last year,” in part because, “I only see them once a week for 50 minutes at a time, so I want to make sure I partner them up correctly. You don’t want two friends together, you don’t want two high flyers together, you want a mix of both…before I even do partners I talk to the [classroom] teachers to get a good sense.”
Teachers reported that adjusting their instruction of the same lessons for different grade and ability levels was challenging.
Teachers were challenged by the fact that multiple grades used each of the Courses. One of the interviewed teachers said, “that was one thing that I, as the teacher, really had to think about, because especially the unplugged were written basically at one level. And I had to go down a level for the third graders, and then I had to up it a bit for the fifth graders, depending on the lesson.”
About half of questionnaire respondents said that they used the K5 Courses as stand-alone lessons. Less than one fifth said that they used the K5 courses to supplement other subject areas.
Just more than half (55%) of respondents reported using the CS Fundamentals Courses as stand-alone lessons, while more than one quarter (28%) used them to supplement computer science time. Only 17% reported that they used the Courses to supplement other subject areas. Of the respondents that reported using the CS Fundamentals Courses to supplement other subject areas, most (65%) said they used them to supplement math.
Interviewed teachers also reported that they mostly used the CS Fundamentals Courses for computer science only, though some said that the unplugged lessons in particular were suited for reinforcing other subjects. As one said, “I was able to make the connections with what they're doing in math, just because the unplugged for me was more direct instruction than the lessons on the computer.” She went on to say that when she touched on vocabulary words, such as ‘algorithm,’ she made the connection to their math time. She also reminded students of math time while talking about debugging, saying, “When I talked about finding their mistakes... we talked about checking over your work and ‘Huh, I wonder if you've ever heard that before?’”
Teachers did report in the questionnaire that it was easy for them to see how the CS Fundamentals Courses connected to other subject areas, with 65% agreeing or strongly agreeing. More than half (53%) also agreed or strongly agreed that it was easy for them to find the time to use the CS Fundamentals Courses. However, keep in mind that the survey respondents are already using the Courses, and that they were motivated enough to respond to this questionnaire. It is likely that less motivated teachers will need additional supports to make it as easy as possible to use the Courses.
Student Interest
All interviewed teachers reported that nearly all students were highly interested in the lessons, with a few exceptions.
Respondents said that there were typically only a few students that were not interested. Some of these students were said to be uninterested most of the time they were in school. As one teacher said, “I have a very small group that’s not engaged. In any of the classes you might have one or two students if that, that aren’t engaged [with Code.org Courses]. The majority of them enjoy doing Code.org.” Another teacher summed up why she thought that students liked the Courses, saying:
I can usually judge a project by the amount of kids who start saying, ‘Do we have to do this today? Aren't we going to move on to something else?’ I have not heard that in all of the lessons that I’ve had that 4th and 5th grade is doing. I haven't had a single person say that. Sometimes kids just get so bored with something that they want to move on. I don't know. I think their rewards are pretty good for them. They can get immediate results. They get that feedback of is it right or is it wrong.
Teachers reported that the Courses were interesting for students that were not high achieving, and that they were also engaging for girls.
Some teachers said that some students that were not typically interested in school were nonetheless very interested and successful with the K5 Courses. One teacher explained:
We had one third-grader who has extreme reading problems and he was one of the best coders in third grade. We were just like, how does he do this, but he must be very visual and it was amazing what he could figure out.
This teacher, a tech facilitator, said once she saw how the student worked through the puzzles, she was able to work with his reading teacher to adjust the strategies they used to teach him reading, and that:
…it made a huge difference in [the reading teacher’s] conferences with the parents, because she had positive things to say…and any child that can do things hands-on like that and be able to see the problem and fix it, even though he can’t read very well, there’s definitely a possibility of a future for him.
Interviewed teachers also said that the Courses were reaching girls, even those that don’t typically like math and science. One teacher said, “I know one girl in particular hated it [Codeables, the CS materials used in their school the previous year] because she didn’t understand it. [She thought] math and science were dumb. Now, I think with some more direct instruction of it and the breakdown of it…she’s much more open to it and wasn’t negatively talking about it, which is exciting…I think in a lot of ways it shows that it could be fun and applied to other things, and that it wasn’t just for the boys.”
Some students that were not interested in CS Fundamentals Courses also did not have much exposure to computers and technology outside of school.
Teachers noticed that some of the students that were not interested in the Courses also did not have much experience with computers. One teacher spoke of a student in particular that refused to participate in the Courses. She said, “We ended up having a parent conference…because his mom got mad that he was singled out. We asked, ‘don’t you play computer games at home or anything?’ And they said no.” Another teacher echoed this, saying, “Not everybody's going to be comfortable on the computers, especially if they don't have the opportunity at home. Those are usually the students I have a harder time getting them comfortable at school using computers.”
Teacher self-efficacy
Teachers that are new to computer science wanted more content support.
More than half of questionnaire respondents (55%) reported that this was their first year teaching computer science. While many of the interviewed teachers with little CS background felt that they were fully prepared to teach the Courses after attending the CS Fundamentals PD, many would also like opportunities for in-depth explorations of CS content. One teacher explained that she was “very lost on the binary,” but had students that were fascinated by it. This teacher said, “I was just scrambling to find things, and I actually went to our tech department because I knew that they had taken computer science…they gave me some pointers, like for beginners, you stick with this, because there’s different binary languages. And so that was totally new to me.” Another teacher reported that she felt as though she needed to go back and teach herself programming, and that she hit a wall with the Courses, and that around that time, so did her students. She explained that “after I did the workshop, I felt so inspired and I thought I couldn’t understand this stuff, and yet I am about to do this curriculum…even though I don’t have the background…but [now] I feel like I need to take a class or learn more to go forward.”
More than half of questionnaire respondents (55%) somewhat disagreed or disagreed that they had all of the skills they needed to teach computer science well.
Questionnaire respondents were lukewarm when rating their understanding of CS concepts, vocabulary, and teaching skills and strategies. This is not surprising, considering that 55% of respondents were first-time teachers of computer science. Interviewed teachers expressed a desire to learn more, though some were vague, saying that they would like to “go in more depth” with the Courses. In short, they didn’t know what they didn’t know. One teacher summed her situation up well, by both acknowledging that she “loved how they made it [CS] so accessible,” even though, “before the workshop, I thought it was like an impenetrable foreign language.” She said, “Like me, I think other teachers would probably love more [time spent] as a student learning the computer science…I think if I would want to go further, over the summer I’m probably going to be doing a lot of research and talking to people to find out more about content-wise.”
Teacher feedback on professional development
Many teachers began teaching CS Fundamentals Courses before they attended affiliate training. Those teachers especially requested further PD that delved deeper into the courses beyond getting set up.
Several interviewed teachers said that they began teaching the CS Fundamental Courses at the beginning of the school year, but didn’t attend a PD until later. Those teachers were disappointed that the PD focused mainly on getting started with the Courses, which they had already done. They hoped for more in-depth discussions about best practices and content. As one teacher said, “I didn't really have anyone to compare notes with, so I think PD level 2 would be good, you know? Level 1 is how to get set up. Level 2 is, "Okay, you've been doing this, let's share ideas, let's figure out where you can go next." I would've loved that. I think that would've been great.”
Few teachers would prefer online PD to in-person PD. However, some reported that online supplements to the PD would be useful.
Half of questionnaire respondents disagreed or strongly disagreed that they would prefer to participate in the PD online rather than in person. However, about half (46%) agreed or strongly agreed that an online supplement to the PD would be useful.
Teacher feedback on instructional materials
Teachers wanted the unplugged lessons to be more firmly linked to the plugged activities.
Interviewed teachers pointed out that the unplugged lessons felt disconnected from the plugged lessons. One said that her kids struggled with why they were playing a game in an unplugged lesson, and didn’t understand how it connected to the plugged activities. The teacher also struggled with making the connection for students, given her lack of computer science background. She finally was able to give students some examples of a vocabulary word from the plugged lessons, but said, “it was not laid out in the unplugged activity that that was the connection.”
Most teachers agreed or strongly agreed that the Courses are developmentally appropriate for their students (81%). The majority (58%) also agreed or strongly agreed that their students had all of the prerequisite skills needed to participate in the CS Fundamentals Courses.
Most teachers reported that the courses were developmentally appropriate for their students, and that their students had all of the prerequisite skills needed to participate in the CS Fundamentals Courses. Nearly all interviewed teachers also said that the Courses were a good fit for their students. However, teachers still identified some students that needed additional supports to fully participate in the Courses.
Teachers reported in interviews that while the Courses were generally a good fit for their students, some had challenges completing certain parts of courses.
Some interviewed teachers said that their kindergarten students needed skill building with the mouse and keyboard before they could participate in the Courses. “They're used to their iPads…my school is a one-to-one iPad school. They were trying to swipe the screen.” This teacher said that they had to do mouse practice for three weeks, and then drag and drop practice after that. Once those mechanics were worked out, students still struggled with problem-solving. The teacher said, “it was really hard for them to figure out ‘what do I need to do first to get to the next step?’” Other teachers said that their students were not socially or emotionally ready to fully participate in the communication and collaboration aspects of the Courses.
Several teachers reported in interviews that their students, especially those in the younger grades, were very impatient when participating in both plugged and unplugged activities. While this created the opportunity to work on persistence and perseverance, teachers reported that they were unable to implement practices such as pair programming effectively.
Several teachers said that they didn’t use pair programming, for example, because students “don’t come to the classroom ready to share.” She pointed out that, “discussing your ideas and listening to others is a lifelong process…there are some adults that could work on it, too.” She said that she needed to “take a step back with that and maybe do some more spelled out lessons of how you talk to each other.” Other teachers reported that some classes, especially the younger grades, became too frustrated with the plugged activities, and that they needed to stop using them after 8 or so weeks.
School support and teacher attitudes about computer science
All interviewed elementary teachers said that their school administration was completely supportive of their CS teaching. The majority of teachers said that their administration was also aware of the K5 Courses and had seen some of their students engage with the courses.
Teachers reported in interviews that both their school and district administration was very supportive of their CS teaching. Many also said that they supported Code.org’s goals of teaching CS to all students. One teacher said that her superintendent and school leaders:
don't want to get that above average student. This is something that we're hoping the average or even the below average students grasp and take a liking to it and just run with it. Give them a different opportunity for a career.
More than half (59%) of questionnaire respondents disagreed or strongly disagreed that their school or district didn’t know they were teaching the CS Fundamentals Courses.
However, fewer than half of questionnaire respondents (44%) agreed or strongly agreed that their school leaders expected all students to regularly participate in computer science learning. But, most questionnaire respondents (82%) reported that they disagreed or strongly disagreed that their school leaders wouldn’t want them to teach the CS Fundamentals Courses next school year.
Teachers most frequently reported that student enjoyment of computer science, and data showing student learning, were the factors that would most influence their decision to teach elementary computer science.
Nearly a quarter of respondents (23%) reported that student enjoyment of CS would most influence their decision to teach computer science, while 19% reported that data showing that students do better in other subjects when they also participate in computer science would be most influential.
The majority of questionnaire respondents and interviewed teachers agreed that CS is valuable for students to learn.
Nearly all of interviewed teachers agreed that CS was valuable for elementary students to learn. Though most said they felt this way because they thought that CS content is important for students to learn when they are younger, some said they appreciated the other benefits they could see students getting from participating in CS, including their perception that the Courses are “teaching the social skills that especially in the younger grades they definitely need. It's teaching logic, a way of thinking.” A large majority (84%) of questionnaire respondents also agreed or strongly agreed that they think computer science is just as important for students to learn as math or English/Language Arts.