The BASICS Study: Teacher and Student Implementation Questionnaire for the Exploring Computer Science (ECS) Curriculum (Fall 2015)

The BASICS (Barriers and Supports to Implementing Computer Science) study is a three-year research project funded by the National Science Foundation. Throughout the study, the BASICS team at Outlier Research & Evaluation at the University of Chicago has been committed to generating useful, applicable, and accessible resources and findings for others in the computer science education community. All of Outlier’s work seeks to support communication and collaboration in research and evaluation.

One BASICS study goal is to develop and share valid and reliable instruments that rigorously measure implementation of the Exploring Computer Science (ECS) curriculum. More specifically, these instruments will measure how the ECS materials themselves are used in practice as well as supports for and barriers to their use.

This goal stems from our belief that the growth and endurance of K-12 computer science education is reliant on systematic study of instructional materials implementation and the alignment of efforts to collect and describe implementation data. We are taking steps toward that goal by sharing conceptual language and measures developed in this study for others to use.

In this spirit of collaboration, we are sharing our most recent versions (May/June 2015) of the student and teacher questionnaires used in the BASICS study. Our approach to implementation measurement was guided by two conceptual frameworks – the Innovation Implementation Framework and the Factor Framework (Century & Cassata, 2014) — developed from earlier National Science Foundation-supported studies of implementation, spread and sustainability of educational innovations. For more information on the creation of these frameworks, go here.

These questionnaires were administered to high school teachers (N=205) and students (N=957) using the ECS curriculum. However because they were developed using conceptual frameworks that can be applied to many curricular innovations, the items can be adapted for use with other computer science curricula.

The measures in this document are not final. Psychometric analyses and revision are ongoing and items and/or subscales may be modified further to achieve optimal reliability and validity. Scale revision and refinement is informed by results of psychometric analysis on each year of student and teacher data collected. The measures will undergo final revision once 2016 data analysis is complete.

This work is supported by the National Science Foundation, Study # 1339256

Suggested Citation: Outlier Research & Evaluation (Fall 2015). BASICS Study ECS Teacher and Student Implementation Questionnaire Measures. [Measurement scales]. Retrieved from Chicago, IL; Outlier Research & Evaluation | CEMSE | The University of Chicago. outlier.uchicago.edu/basics/

Basics Study Teacher Questionnaire1 – Implementation of the ECS Curriculum

Table 1. Teacher instruction with ECS.
Subscales, reliability (internal consistency), and itemsResponse scale
Teacher Facilitation of Cognitively Demanding Work
(Cronbach’s α 0.81)
Think about the first class you teach each week and (your most recently completed unit). How often did you explicitly ask students to do the following?2

1=Never; 2=One class session3; 3=A few class sessions; 4=About half the class sessions; 5=Many class sessions; 6=Once or more per class session; 7=N/A4
Consider alternative approaches to their work.
Analyze (organize, process, manipulate, evaluate) data.
Explain the logic and reasoning supporting their solutions.
Explain why they agree or disagree with the work of other students.
Communicate their thought process to others.
Subscales, reliability (internal consistency), and itemsResponse scale
Teacher Facilitation of Student Autonomy
(Cronbach’s α 0.82)
Think about the first class you teach each week and (your most recently completed unit). How often did you do the following?

1=Never; 2=One class session; 3=A few class sessions; 4=About half the class sessions; 5=Many class sessions; 6=Once or more per class session
Intentionally step back so students can determine how to figure out answers/solutions on their own.
Give students choices that significantly shape their learning experiences.
Give students the opportunities to work without my participation or input during instruction time.
Give students activities that require them to manage their own time.
Subscales, reliability (internal consistency), and itemsResponse scale
Teacher Facilitation of Student Interest
(Cronbach’s α 0.88)
Think about the first class you teach each week and (your most recently completed unit). How often did you explicitly do the following?

1=Never; 2=One class session; 3=A few class sessions; 4=About half the class sessions; 5=Many class sessions; 6=Once or more per class session
Connect lessons or activities to students’ lives.
Ask students to consider relationships between lesson content and real world problems.
Engage student interest by connecting lesson content with current events.
Subscales, reliability (internal consistency), and itemsResponse scale
Teacher Facilitation of Intellectual Risk Taking
(Cronbach’s α 0.87)
Think about the first class you teach each week and (your most recently completed unit). How often did you explicitly do the following?

1=Never; 2=One class session; 3=A few class sessions; 4=About half the class sessions; 5=Many class sessions; 6=Once or more per class session
Encourage students to answer questions even if they are not sure they are correct.
Encourage students to take risks in trying new things.
Encourage students to ask questions if they don’t understand something.
Encourage students to use problem-solving strategies that they have not tried before.
Encourage students to create original solutions to computational problems.
Urge students to ask peers they don’t know well for help.
Subscales, reliability (internal consistency), and itemsResponse scale
Teacher Facilitation of Group Work
(Cronbach’s α 0.89)
Think about the first class you teach each week and (your most recently completed unit). How often did you explicitly do the following?

1=Never; 2=One class session; 3=A few class sessions; 4=About half the class sessions; 5=Many class sessions; 6=Once or more per class session
Encourage all group members to contribute verbally or nonverbally.
Ensure all group members understand the task at hand.
Encourage cooperative work among students.

1 Based on total sample of 205 high school teachers using the ECS curriculum.

2 Teachers were asked to report on their use of instructional strategies in the most recent ECS unit they had completed at the time of the questionnaire.

3 Class sessions were defined as the designated time in the school schedule when the ECS curriculum was taught.

4 N/A responses were converted to missing and not included in reliability analyses.

Basics Study Teacher Questionnaire – Teacher Beliefs and Dispositions

Table 2. Teachers’ perspectives on ECS (Exploring Computer Science) curriculum.
Subscales, reliability (internal consistency), and itemsResponse scale
Understanding of ECS
(Cronbach’s α 0.85)
How much do you agree or disagree with the following?

1= Strongly disagree; 2= Disagree; 3= Somewhat disagree; 4=Somewhat agree; 5= Agree; 6= Strongly agree
Overall, I completely understand the ECS [Exploring Computer Science] learning objectives.
Overall, I completely understand the teaching strategies I am supposed to use in the ECS [Exploring Computer Science] curriculum.
Subscales, reliability (internal consistency), and itemsResponse scale
ECS Fit with Students’ Needs
(Cronbach’s α 0.90)
How much do you agree or disagree?

The ECS [Exploring Computer Science] curriculum is a perfect fit for my students'....

1= Strongly disagree; 2= Disagree; 3= Somewhat disagree; 4=Somewhat agree; 5= Agree; 6= Strongly agree
Academic needs.
Cultural identity and background.
College and career pathway needs.
Learning styles or differences.
Table 3. Teachers’ beliefs and dispositions.
Subscales, reliability (internal consistency), and itemsResponse scale
Sense of Efficacy in Teaching Computer Science
(Cronbach’s α 0.81)
How much do you agree or disagree with the following statements about teaching introductory computer science?

1= Strongly disagree; 2= Disagree; 3= Somewhat disagree; 4=Somewhat agree; 5= Agree; 6= Strongly agree
I understand computer science concepts well enough to be a very effective teacher of introductory computer science.
I have nearly every skill I need to teach computer science well.
I am about the best high school computer science teacher I know.
I have almost nothing else to learn to be a better computer science teacher.
Subscales, reliability (internal consistency), and itemsResponse scale
Sense of Efficacy in Teaching in General
(Cronbach’s α 0.77)
How much do you agree or disagree with the following statements about teaching?

1= Strongly disagree; 2= Disagree; 3= Somewhat disagree; 4=Somewhat agree; 5= Agree; 6= Strongly agree
I have nearly every skill I need to teach well.
I am a very effective teacher.
I am about the best high school teacher I know.
I have almost nothing else to learn to be a better teacher.
Subscales, reliability (internal consistency), and itemsResponse scale
Teacher Resourcefulness and Coping
(Cronbach’s α 0.86)
How much do you agree or disagree with the following?

1= Strongly disagree; 2= Disagree; 3= Somewhat disagree; 4=Somewhat agree; 5= Agree; 6= Strongly agree
I am able to manage the pressure and stress at my school well.
I see difficult tasks through to the end.
I find ways to accomplish my goals.
When planning for my work, I prepare for potential challenges.
I am able to manage my work even when there are unexpected changes and constraints.
Subscales, reliability (internal consistency), and itemsResponse scale
Innovativeness in Teaching
(Cronbach’s α 0.82)
How much do you agree or disagree?

1= Strongly disagree; 2= Disagree; 3= Somewhat disagree; 4=Somewhat agree; 5= Agree; 6= Strongly agree
I experiment with new practices all the time.
I am always looking for new ways of doing things in my teaching.
I am constantly the first to try new things in my school.

Basics Study Student Questionnaire5 – Implementation of the ECS Curriculum

Table 4. Student perception of teacher instruction with ECS.
Subscales, reliability (internal consistency), and itemsResponse scale
Teacher Facilitation of Cognitively Demanding Work
(Cronbach’s α 0.92)
Think about what happened in your computer science class in the past month. During that time, how often did your teacher ask you to do the following?

1=Never; 2=A few class sessions; 3=About half the class sessions; 4=Many class sessions; 5=Once or more per class session
Consider alternative approaches to my work.
Analyze data (organize, process, manipulate, or evaluate data).
Explain the logic and reasoning supporting my solutions to problems.
Explain why I agreed or disagreed with the work of other students in the class.
Communicate my thought processes to others.
Problem solve when something didn’t work the way I wanted it to work.
Subscales, reliability (internal consistency), and itemsResponse scale
Teacher Facilitation of Student Autonomy
(Cronbach’s α 0.73)
How often in the past month did your teacher ask you to do the following?

1=Never; 2=A few class sessions; 3=About half the class sessions; 4=Many class sessions; 5=Once or more per class session
Make my own goals for learning computer science.
Make my own choices about assignments related to my computer science class.
Work in my computer science class without the teacher telling me what to do or managing my work time.
Subscales, reliability (internal consistency), and itemsResponse scale
Teacher Facilitation of Student Interest
(Cronbach’s α 0.88)
How often in the past month did your computer science teacher do the following?

1=Never; 2=A few class sessions; 3=About half the class sessions; 4=Many class sessions; 5=Once or more per class session
Asked me to consider relationships between what I was learning in the lesson and real world problems.
Connected a lesson or classroom activities to my own life.
Made the class fun.
Made activities and projects interesting and relevant to me.
Table 5. Student participation in ECS.
Subscales, reliability (internal consistency), and itemsResponse scale
Students Contribute to Small Group Work
(Cronbach’s α 0.93)
This next set of questions asks about how you worked with other students in your computer science class in the past month.

During that time, how often did you do the following?

1=Never; 2=A few class sessions; 3=About half the class sessions; 4=Many class sessions; 5=Once or more per class session
Contributed to small group.
Worked collaboratively with other students.
Shared responsibility for activity and project work with group members.
Subscales, reliability (internal consistency), and itemsResponse scale
Students Engage in Discussion
(Cronbach’s α 0.89)
How often did you do the following in your computer science class in the past month?

1=Never; 2=A few class sessions; 3=About half the class sessions; 4=Many class sessions; 5=Once or more per class session
Talked to other students about my computer science work.
Responded to questions other students had about their computer science work.
Discussed what I was learning with other students in the class.
Subscales, reliability (internal consistency), and itemsResponse scale
Students Engage in Cognitively Demanding Work
(Cronbach’s α 0.93)
Think very carefully about your work in computer science class over the past month.

During that time, how often did you do the following?

1=Never; 2=A few class sessions; 3=About half the class sessions; 4=Many class sessions; 5=Once or more per class session
Identified real world applications for the work I did in class.
Communicated my thought process to others.
Explained why I agreed or disagreed with the work of other students.
Problem solved when something didn’t work the way I wanted it to work.
Considered alternative approaches to my work.
Analyzed data (organized processed, manipulated, evaluated data).
Subscales, reliability (internal consistency), and itemsResponse scale
Students Demonstrate Autonomy
(Cronbach’s α 0.90)
How often did you do the following in your computer science class in the past month?

1=Never; 2=A few class sessions; 3=About half the class sessions; 4=Many class sessions; 5=Once or more per class session; 6=N/A
Set my own goals for things I did or learned in class.
Made my own choices for assignments in computer science class.
Worked on my own to find information I needed to solve computing problems in class.
Identified problems I wanted to solve in class.
Subscales, reliability (internal consistency), and itemsResponse scale
Students Take Intellectual Risks
(Cronbach’s α 0.86)
How often did you do the following in your computer science class in the past month?

1=Never; 2=A few class sessions; 3=About half the class sessions; 4=Many class sessions; 5=Once or more per class session; 6=N/A
Asked questions when I was confused about activities or assignments.
Tried new things in class even when I was not sure how to do them.
Created something original that no one had done before.
Tried something I thought I might fail.

5 Based on total sample of 957 high school students in classes using the ECS curriculum.

Basics Study Student Questionnaire – Student Beliefs and Dispositions About Computer Science

Table 6. Students’ beliefs and dispositions about computer science.
Subscales, reliability (internal consistency), and itemsResponse scale
Beliefs and Values of Friends and Family
(Cronbach’s α 0.89)
How much do you agree or disagree with the following statements?

1=Completely disagree; 2=Mostly disagree; 3=Slightly disagree; 4=Slightly agree; 5=Mostly agree; 6=Completely agree; 7=I don’t know what they think
My friends think it’s cool to choose a job/career in computer science.
My friends think it is important for students to take computer science.
My family members think it would be good for me to choose a job/career in computer science.
My family thinks it is very useful for me to take this course.
My family thinks I should take more computer science courses.
Subscales, reliability (internal consistency), and itemsResponse scale
Perceived Relevance to Student’s Future
(Cronbach’s α 0.92)
How much do you agree or disagree with the following statements about your computer science class?

1=Completely disagree; 2=Mostly disagree; 3=Slightly disagree; 4=Slightly agree; 5=Mostly agree; 6=Completely agree
Taking computer science is necessary for me to accomplish what I want in school.
Taking computer science will help me reach my goals for college/career.
What I learn in computer science will benefit my future.
I think it is useful for me to take computer science.
Subscales, reliability (internal consistency), and itemsResponse scale
Intrinsic Motivation: Interest/Enjoyment in Computer Science Class
(Cronbach’s α 0.94)
How much do you agree or disagree with the following statements about how you feel about your computer science class?

1=Completely disagree; 2=Mostly disagree; 3=Slightly disagree; 4=Slightly agree; 5=Mostly agree; 6=Completely agree
I like my computer science class.
I think my computer science class is interesting.
I enjoy my time in computer science class.
I like doing the activities we do in my computer science class.
I think it would be cool to choose a job/career in computer science.6
Subscales, reliability (internal consistency), and itemsResponse scale
Sense of Efficacy in Computer Science
(Cronbach’s α 0.88)
How much do you agree or disagree with the following statements?

1=Completely disagree; 2=Mostly disagree; 3=Slightly disagree; 4=Slightly agree; 5=Mostly agree; 6=Completely agree
I have the skills and ability to learn computer science.
I am better at computer science than most of the other kids at my school.
I am very good at computer science.

6 Not specific to computer science class.