In E. Dunican & T.R.G. Green (Eds). Proc. PPIG 16

Pages 97-110

Factors Affecting Course Outcomes in Introductory Programming
Susan Wiedenbeck Drexel University susan.wiedenbeck@cis.drexel.edu Deborah LaBelle Pennsylvania State University-Delaware County dm19@psu.edu Vennila N.R. Kain Media Laboratory Masssachusetts Institute of Technology vennila@vennila.net
Keywords: POP-I.B, programmer education, POP-II.A, novices, POP-5.A, mental models, POP-5.A, selfefficacy

Abstract Learning to program is difficult for many students. Although several factors that affect learning to program have been identified over the years, we are still far from a full understanding of why some students learn to program easily and quickly while others flounder. Two constructs that may affect learning to program are self-efficacy and mental models. Self-efficacy is the individual’s judgment of his or her ability to perform a task in a specific domain (Bandura 1986). A mental model is a person’s internal (mental) representation of real world objects and systems (Norman 1983). Separate research on self-efficacy and mental models has shown that both are important to knowledge acquisition and transfer. Using a path-analytic approach, this research investigates the joint effects of self-efficacy, mental model, and previous experience on learning to program in an introductory course. The results show that self-efficacy for programming is influenced by previous programming experience, and student self-efficacy increases substantially during an introductory programming course. Furthermore, students’ mental models of programming influence their self-efficacy, and both the mental model and self-efficacy have a direct effect on overall success in an introductory course. Introduction The dropout and failure rates in introductory programming courses at the university level are evidence to the fact that learning to program is a difficult task. One source suggests that the dropout and failure rate is as high as 30 percent (Guzdial & Soloway, 2002). Decisions about majoring in computer science and related fields are often determined by a student’s success or failure in the introductory course. If a student drops out, fails, or passes with a struggle, that student is unlikely to enroll for a follow-on course. In spite of research on factors that influence the enrolment and success of novices in introductory programming, it is still not fully understood what makes an introductory programming course positive and successful for some, but difficult and frustrating for others. Several factors that may influence novices’ success in an introductory university-level programming course have been discussed in the literature. The most frequently mentioned factor is previous computer programming experience, usually in secondary school (Bunderson & Christensen, 1995; Byrne & Lyons 2001; Hagan & Markham 2000; Sacrowitz & Parelius 1996; Taylor & Mounfield 1989; Wilson & Shrock 2001). These studies provide converging evidence that prior programming experience has a positive effect on success in an introductory university course. Other factors that may affect course success have been less well investigated. Two recent studies have shown a positive relationship of mathematics or science background to computer programming success (Byrne &
16th Workshop of the Psychology of Programming Interest Group. Carlow, Ireland, April 2004 www.ppig.org

ii

Lyons 2001; Wilson & Shrock 2001). A relationship between student learning styles and learning to program has been found by both Byrne and Lyons (2001), and Thomas, Woodbury, and Jarman (2002). Other intriguing factors that have been addressed in recent studies include student attributions of success to oneself or to outside forces (Wilson & Shrock 2001), students’ course outcome expectations (Rountree, Rountree & Robins 2002), and self-efficacy (Wilson & Shrock 2001). A factor of potential interest that has been studied in basic computer training, but not to our knowledge in computer programming, is computer playfulness during training (Martocchio & Webster 1992; Potosky 2002). Finally, there is a body of research on the student’s mental model of programming in relation to success in specific programming tasks (Cañas, Bajo & Gonzalvo 1994; Corritore & Wiedenbeck 1991; Soloway & Ehrlich 1984; Wiedenbeck, Ramalingam, Sarasamma & Corritore 1999). In summary, there is a substantial literature on factors affecting success in the initial programming course. However, more research is needed to determine which are the key factors, how they interact with each other, and how they combine to affect course outcomes. We are interested in creating and testing models that incorporate factors that appear to be important on theoretical or empirical grounds. Two important constructs in cognitive and social cognitive theory of the past 20 years are mental models and self-efficacy. Our goal in this research is to study self-efficacy and mental models of beginning programmers, explore the relationship between these concepts, and investigate their combined influence on course performance. Background on Self-Efficacy and Mental Models Self-Efficacy and Its Role in Learning Bandura (1986, p. 391) defines self-efficacy as “people’s judgments of their capabilities to organize and execute courses of action required to attain designated types of performance.” Self-efficacy beliefs are a key element in human performance over a very broad range of situations, for example, efficacy for work tasks, for physical activities, for personal relationships (Bandura 1977, 1986; Gist & Mitchell 1992). Self-efficacy is important in learning activities because learning involves more than just acquiring skills. As Bandura says, “competent functioning requires both skills and self-beliefs of efficacy to use them effectively” (1986, p. 391). In learning situations, self-efficacy influences the use of cognitive strategies while solving problems, the amount of effort expended, the type of coping strategies adopted, the level of persistence in the face of failure, and the ultimate performance outcomes (Bandura 1986; Gist & Mitchell 1992; Zimmerman 1995). According to self-efficacy theory (Bandura 1977, 1986), judgments of self-efficacy are based on four sources of information: the individual’s performance attainments, experiences of observing the performance of others, verbal persuasion, and physiological reactions that people use partly to judge their capableness and vulnerabilities. Of these four sources, the most important is performance attainments, that is, the individual’s evaluation of the outcomes of his or her direct attempts to perform an activity. Educational researchers recognize that, because skills and self-beliefs are so intertwined, one way of improving student performance is to improve student self-efficacy. Interventions to improve student self-efficacy focus on specific skills or knowledge and target the four sources of information that students use to evaluate their self-efficacy, as defined above. Providing students with direct hands-on experiences in an activity is critical, since the strongest source of information is performance outcomes (Bandura 1977, 1986; Pajares 1996; Zimmerman 1995). Making hands-on experiences positive is also important, especially in the early stage of learning, when the task may seem overwhelming. Attempts have also been made, with some success, to increase self-efficacy in learning by peer modeling of tasks, verbal persuasion, or other types of social influences, such as cooperative learning environments (Bandura 1986; Compeau & Higgins 1995; Gist, Schwoerer & Rosen 1989). Several studies of self-efficacy in learning to use personal computers or computer applications have been carried out (e.g., Compeau & Higgins 1995). However, to our knowledge the only study that has directly targeted introductory computer programming students is Wilson and Shrock (2001). This

16th Workshop of the Psychology of Programming Interest Group. Carlow, Ireland, April 2004

www.ppig.org

iii

study, contrary to theoretical expectations, did not find a significant effect of student self-efficacy on course outcomes. Mental Models and Programming Norman (1983) defines mental models as predictive representations of real world systems. People create internal representations of objects and processes in the world, and they use these mental representations to reason about, explain, and predict the behavior of external systems. Mental models are critical in debugging a process when things go wrong because the mental model supports the person in reasoning about and localizing possible faults (Allen 1997). Mental models have been studied in many domains and situations (e.g., Stevens & Gentner 1983). In recent years, the mental models concept has been popularized by practitioner magazines and web sites in areas such as humancomputer interaction (e.g., McDaniel, 2003). Programming is a cognitive activity that requires the programmer to develop abstract representations of a process and express them in the form of logic structures. In the case of creating, modifying, reusing, or debugging a program, the programmer must also translate these abstract representations into completely correct code using a formal language. Having a well-developed and accurate mental model is likely to affect the success of a novice programmer in an introductory programming course. A programmer’s mental model could encompass useful knowledge about how programs work in general, stereotypical ways of solving common programming problems, and how a particular program is structured and functions, as well as knowledge about the syntax and semantics of a specific language (Cañas et al. 1994). Mental models (also referred to as schemas or plans) have been shown to play an important role in program comprehension (Soloway & Ehrlich 1984; Littman, Pinto, Letovsky & Soloway 1986; Nanja & Cook 1987; Pennington 1987; Corritore & Wiedenbeck 1991; Wiedenbeck et al. 1999) and also in comprehension-related tasks, such as modification and debugging. For example, Littman and his colleagues (1986) found strong effects of mental model formation in a program modification task. Participants were asked to modify a program but were not given any explicit instructions about how to approach the task. The results showed that programmers who first attempted to systematically read and comprehend the program were much more successful in doing the modifications than programmers who jumped immediately into making modifications. The difference in performance between programmers who built a mental model of the program and those who did not was especially great in modifications that involved interactions with code in other parts of the program. Similar results were reported by Nanja and Cook (1987) in a comparison of novices and experts debugging a program. A conclusion that can be made from these studies is that novices’ success in programming tasks may be increased by greater attention to building a good mental model of the program. These studies of mental models in programming do not deal directly with the issue of success in introductory programming courses. However, the relationship between a good mental model and success in programming tasks suggests that having a good mental model may be an important contributor to course outcomes. Model of Factors Affecting Performance in Introductory Programming This study proposes a model of performance of novice programmers based on their previous programming experience, self-efficacy for programming, and mental model. Our model is represented in Figure 1. The ovals represent factors, or variables, that may affect success in introductory programming. The arrows represent predicted relationships between the variables. These relationships are directional, as shown by the arrow heads. Multiple links into an oval indicate that an oval is affected by more than one factor. Multiple links out of an oval indicate that the factor represented by the oval affects more than one other factor. With respect to terminology, if variable A points to variable B, then in that relationship A is referred to either as the independent or the predictor variable, and B is referred to as the dependent or response variable.

16th Workshop of the Psychology of Programming Interest Group. Carlow, Ireland, April 2004

www.ppig.org

iv

Pre Self-efficacy Previous Experience Mental Model

Post Self-efficacy Performance

Figure 1 – Proposed model of factors affecting student performance in an introductory programming course In line with the considerable existing research (Byrne & Lyons 2001; Bunderson & Christensen, 1995; Hagan & Markham 2000; Sacrowitz & Parelius 1996; Taylor & Mounfield 1989; Wilson & Shrock 2001), we expect previous experience to be important to success in an introductory programming course. Our hypothesis is that previous experience acts as a significant predictor of both students’ self-efficacy and mental models of programming, which in turn predict course performance. Based on self-efficacy theory (Bandura 1977, 1986), we expect that students’ self-efficacy will increase as a result of instruction and continued hands-on exposure to programming, so post-selfefficacy should be higher than pre-self-efficacy. We also hypothesize that students’ mental models of programming will have a significant effect on their self-efficacy beliefs. That is, having a clear mental model of what programs do and how they do it will increases students’ feelings of efficacy about programming. Finally, it is expected that both mental model and self-efficacy will explain a significant amount of course performance. In keeping with Bandura, students’ knowledge content and organization, as represented in the mental model, and their self-beliefs will be intertwined in successful course outcomes. Our model includes the concept of mediation of the effect of variables. A variable may not affect performance directly, but instead may affect it indirectly through another variable. This is seen in the pre-self-efficacy variable, which is hypothesized to affect performance indirectly through its effect on post-pre-self-efficacy. Therefore, pre-self-efficacy does not have a direct effect on performance, but its effect is mediated by the intervening variable, post-self-efficacy. Likewise, the previous experience variable’s effect on performance is indirect and mediated through several other variables. The model also suggests that one variable may affect another variable both directly and indirectly. For example, the mental model variable is shown as having a direct effect on performance, and also an indirect effect on performance via its effect in strengthening post-self-efficacy. Methodology Participants Seventy-five students took part in the study. The students were enrolled in four sections of an introductory C++ programming course at a large public university in the United States. The four sections were taught by four different instructors who coordinated with each other to cover the same material. Twenty-five of the participants were female and 75 were male. The average age of the participants was 20 years. The majority of the participants were second or third year undergraduates. The participants came from a wide variety of majors, ranging from computer science to agricultural studies. The majority were not computer science major, but were taking the course as a requirement of their major or for personal interest.

16th Workshop of the Psychology of Programming Interest Group. Carlow, Ireland, April 2004

www.ppig.org

v

The participants had low experience in computer programming. Based on their self-reports, it was found that on average they had taken 1.27 programming courses in secondary school and had been exposed to 1.37 programming languages. Aooroximately half of the participants had not previously studied programming. Materials The materials included a background questionnaire, a self-efficacy scale, and two instruments to measure mental models. The background questionnaire used five questions measuring the breadth of participants’ prior computer and programming background: number of courses taken that used computer applications as a mandatory part of course work (e.g., spreadsheets, databases), number of programming courses taken, number of programming languages used, number of programs written, and length of the programs written. Self-efficacy was measured using the Computer Programming Self-Efficacy Scale (Ramalingam & Wiedenbeck 1998). This validated instrument was used previously by Wilson and Shrock (2001) in their research on success factors in introductory computer science courses. The scale consists of thirtythree items that ask students to judge their capabilities in a wide range of programming tasks and situations. Based on self-efficacy theory, four categories of questions are included in the scale: Simple programming tasks (9 questions, e.g., “I would be able to write a program that computes the average of 3 numbers”) Complex programming tasks (11 questions, e.g., “I would be able to comprehend a long, complex multi-file program”) Independence and persistence (8 questions, e.g., “I would be able to find ways of overcoming the problem if I got stuck at a point while working on a programming project”) Self-regulation (4 questions, “I would be able to find a way to concentrate on my program, even when there were many distractions”). Responses were marked on a 7-point Likert scale ranging from “not confident at all” to “absolutely confident.” The students’ mental models were evaluated by using two measurements, program comprehension and program recall. The program comprehension booklet consisted of six short C++ programs (each 15-20 lines long). The programs consisted of a class definition, a constructor, a member function of the class, and a main function. (See Appendix A for an example.) Each of the programs was followed by a list of five true/false questions covering each of the information categories developed by Pennington (1987) to measure the mental model of programmers: elementary operations, control flow, data flow, program function, and program state. These categories have been used in more recent research on mental models (Good & Brna in press), including object-oriented programming (Wiedenbeck et al. 1999). Program recall has been used as a measure of mental organization or mental models in past programming research, e.g., Shneiderman (1976). Recall is usually measured as the number of lines recalled correctly. Our program recall booklet, modeled on Shneiderman’s, contained a C++ program that dealt with temperature conversions. The program consisted of 27 lines of code. Procedure The study was carried out over the course of a fifteen week semester in two parts. The first part took place in the second week of the semester, and the second part in the thirteenth week of the semester. The first phase of the study involved collecting the student programming background information and having students complete the self-efficacy scale, which yielded the pre-self-efficacy score.

16th Workshop of the Psychology of Programming Interest Group. Carlow, Ireland, April 2004

www.ppig.org

vi

The second part of the study involved completion of the two programming tasks designed to assess the student’s mental model and a repetition of the same self-efficacy scale used in the first part. The participants were given a program comprehension booklet containing the six programs and associated questions. For each of the six programs, they had 1.5 minutes to study the program and two minutes to answer the questions. The participants were not allowed to look back at the programs while answering the questions. After completing the program comprehension booklet, participants were given the recall booklet. They had five minutes to study the program, then closed the booklet and had five more minutes to recall and reproduce the program from memory, as best they could. The participants also completed the self-efficacy scale, yielding the post-self-efficacy score. The performance measure was the student’s final course grade and was obtained from the instructor at the end of the semester. For the purposes of the experiment a participant’s course grade (given as a letter grade: A, B, C, etc.) was translated to a numerical scale of 0-9, where 0 represented failure in the course and 9 represented the highest level of achievement. Results Self-Efficacy of Novice Programmers The alpha-reliability of the self-efficacy scale was .98, indicating a highly reliable scale. The mean pre-self-efficacy score was 94.63 and the mean post-efficacy score was 163.37 out of a maximum possible score of 231 (see Table 1). Participants’ self-efficacy increased significantly over the course of a semester of instruction (t = 12.78, p