Outlier Research & Evaluation conducted a study of the status of implementation and the factors affecting implementation of the Exploring Computer Science (ECS) curriculum in Spring 2015. This report contains findings from the Teacher Implementation Questionnaire, which was developed to measure various dimensions of ECS implementation. These dimensions, developed through our research-based methods for measuring instructional materials implementation, include structural elements (e.g. number, order, and duration of units) as well as interactional elements (e.g. teacher facilitation of student engagement, student participation in small group work). The questionnaire also asks about factors that contribute to or inhibit implementation as well as teachers’ attitudes toward, and understanding of, ECS specifically, and CS more generally.
This questionnaire was administered to participants in all of Code.org’s 2014-2015 regions participating in the Code.org/ECS PD at the end of the school year. In total, 186 teachers responded to the questionnaire, with an overall response rate of 78%. . The Code.org/ECS workshops are made up of four phases: Phase 1 is online only; Phase 2 is a week-long, in-person workshop during the summer; Phase 3 contains four one-day, in-person workshops distributed throughout the school year; and Phase 4 is another week-long, in-person summer workshop. Note that the sample is 78% of all teachers who completed Phase 2 and at least one Phase 3 PD. Teachers who attended Phase 2 but no additional PD were excluded from analysis.
Teachers reported varying levels of teaching experience. The largest percentage of teachers reported experience teaching for 6-10 years (25%), while the smallest percentage reported being in their first year of teaching (2%). More than one-fifth of teachers (21%) reported experience teaching for twenty-one years or more.
Teachers also reported varying years of experience teaching CS-related courses. Most teachers reported they were in their first year of teaching ECS (70%), with far fewer teachers reporting having taught the course for two (12%) and three (12%) years. The majority of teachers (84%) reported having never taught AP Computer Science, while large percentages reported having many years of experience teaching applications courses, like Keyboarding, Microsoft Office Suite, or Other Software Applications (44%), and “Other Computing Courses” such as Networking, Databases, Web Design, etc. (37%). The table below presents the six or more years of teaching these courses in a collapsed category labeled “6+ Years”, however 30% of the overall total reported for Keyboarding, Microsoft Office Suite, or Other Software Applications and 20% of the overall total reported for “Other Computing Courses” have taught the course for ten years or more.
Teachers were asked to report their feelings about the Exploring Computer Science (ECS) curriculum including their commitment to the ECS philosophy, as well as their attitude toward use of the instructional materials. In the questionnaire, on average, participants reported having positive feelings about the curriculum (4.047 out of 5). When asked whether they felt “committed to what ECS is trying to do,” respondents agreed “quite a bit” on average, giving that item a rating of 4.392 out of 5. Teachers reported that they “liked teaching with ECS materials” a “fair amount,” with an average rating of 3.782 out of 5. They also reported that “teaching with ECS materials is personally rewarding,” on average, agreeing “quite a bit” with the statement (3.965 out of 5).
Teachers were also asked more generally how they felt about teaching with a complete introductory compute science curriculum, like ECS. These items were asked with a six point scale. On average teachers agreed or strongly agreed that they liked having “a more complete introductory computer science curriculum, like ECS” (mean of 5.093 out of 6). They also reported wishing there were “more complete curricula for teaching computer science, like ECS,” giving that item an average rating of 4.5 out of 6.
Teachers were asked to report on the units they had completed at the time of the spring questionnaire. The timeframe in which a teacher could have responded to the questionnaire varied from late-March to early-June, so reports of completed units depends both on individual teachers’ progress, as well as when they took the questionnaire. Only a few teachers (2%) reported that they had not completed any units. Nearly all of the teachers (91%) reported completing Unit 1: Human Computer Interaction with nearly the same amount (90%) reporting that they had completed Unit 2: Problem-Solving. A majority (79%) completed Unit 3: Web Design, and 63% completed Unit 4: Introduction to Programming.
In all, nearly half of the teachers reported completing four units total (47%), with 23% reporting that they had completed three units. Few teachers (2%) had completed all six units at the time of questionnaire administration.
The amount of time teachers spent on each of the first 4 units of ECS varied widely; between 19 – 24% reported completing Units 1-4 in the amount of time specified in the ECS curriculum. For the rest of the teachers in the sample, most (23-65%) reported spending one or more weeks of additional time on Units 1-3 than specified in the materials. However, only 31% of teachers reported spending more than the time specified on Unit 4.
Teachers were asked to report on the ways they assess student learning, selecting all specified in the questionnaire that applied to their classroom. The most popular approach across all ECS units was unit/final projects and presentations (86%). The second most utilized assessment method was evaluation of student artifacts and portfolios (70%), followed closely by simple observation of student interactions with the computer and other course tools (67%). A sizeable percentage of teachers also reported using “other types of assessments” (39%) to assess student learning. These are described below.
Most teachers (92%) reported using at least two types of assessments to grade students in their introductory computer science class with over a third (34%) reporting that they used four different types of assessments, and 17% reporting that they used all of the assessment approaches, including an “other” type.
Nearly all of the teachers who reported using “other” assessments (i.e., those not specified in ECS, and not included in the list of questionnaire options) reported that they had used “something [they] created [themselves]” (97%; n = 68), while about one-third (32%) reported using “something another teacher created. Nearly half of these teachers (46%) reported using an “assessment from another source” (46%) including “online lessons or quizzes” and “Internet resources,” (e.g. Khan Academy, W3schools, Scratch.mit.edu forums, sam.course.com, typing.com, Google First, Code.org).
In addition to asking all teachers what they were using to assess their students’ learning, we also asked them to rate their confidence in their ability to provide a grade to students in their introductory computer science class. The majority of teachers in the sample (80%) reported high levels of confidence by responding they were “reasonably confident” (53%) or “totally confident (27%) on a 4-point response scale ranging from “Not at all confident” to “Totally confident.”
Teachers were also asked to comment on their confidence rating. Some teachers who reported low levels of confidence (“somewhat confident” or “not confident at all”) in their ability to grade their introductory CS students disclosed their challenges associated with assessment; the following quotes are a sample of responses specific to assessment challenges:
I question whether or not the assessments are good. I struggle with evaluating student work. Many times it seems subjective.
[I] haven’t been sure of what is important and should be graded and what is not.
In hindsight, I probably graded more heavily on the final product that I did on the process. I am still struggling with rubrics that adequately reflect a grading process.
There is not right or wrong answer for most of the projects. I feel there should be some solid projects that can be graded with a definite answer.
Unit tests are difficult to create.
This is my first time teaching computer science. I feel like I am learning with the students.
When there are mistakes, for instance, [when] code that doesn't work, I feel that I ought to figure out why it doesn't work. That is very time consuming. It is difficult for me, as a science teacher, to grade projects where there is no real ‘right’ answer.
It is hard to not get an A in this course if you are a member of the class and contribute to discussions, partner, and group activities. It is one of the problems I have with the work being done, I don't feel the students all deserve A's but the way the curriculum is written it is hard to separate out the A's, B's, and C's.
I don't think I have come up with good consistent methods of evaluating student work. Especially as primarily being a math teacher, I feel the grades are very subjective and arbitrary which I am not used to.
I would like more concrete means of assessment that wouldn't depend on my perception of their participation.