Journal of University Teaching & Learning Practice
Volume 9 | Issue 1

Article 4

2012

Evaluation of the Effectiveness of Online Resources in Developing Student Critical Thinking: Review of
Literature and Case Study of a Critical Thinking
Online Site
Erst Carmichael
University of Western Sydney, e.carmichael@uws.edu.au

Helen Farrell
University of New South Wales, h.farrell@unsw.edu.au

Follow this and additional works at: http://ro.uow.edu.au/jutlp
Recommended Citation
Carmichael, Erst and Farrell, Helen, Evaluation of the Effectiveness of Online Resources in
Developing Student Critical Thinking: Review of Literature and Case Study of a Critical Thinking
Online Site, Journal of University Teaching & Learning Practice, 9(1), 2012.
Available at:http://ro.uow.edu.au/jutlp/vol9/iss1/4
Research Online is the open access institutional repository for the
University of Wollongong. For further information contact the UOW
Library: research-pubs@uow.edu.au

Evaluation of the Effectiveness of Online Resources in Developing
Student Critical Thinking: Review of Literature and Case Study of a
Critical Thinking Online Site
Abstract

A graduate's ability to be a critical thinker is expected by many employers; therefore development of students’ critical-thinking skills in higher education is important. There is also a perception that today’s students are technologically "savvy", and appreciate the inclusion of a technological approach to learning. However, the complexity of the concept of critical thinking and the assumptions about students’ technological skills are debatable issues that require clarification and evidence-based research in terms of teaching and learning. This paper reports on a case study of an online Blackboard site at the University of Western Sydney, where analysis of patterns of usage of the online site and qualitative analysis of student feedback provide evidence to support its effectiveness for encouraging students' critical thinking. There is potential to expand this into a more widely usable teaching and learning resource in the future, and for further research to explore the benefits for student learning. Keywords

critical thinking; effective student learning; higher education; technology as a teaching tool
Cover Page Footnote

Acknowledgements: We would like to thank our partners for their support during a hectic writing period, for their feedback and technical assistance.

This journal article is available in Journal of University Teaching & Learning Practice: http://ro.uow.edu.au/jutlp/vol9/iss1/4

Carmichael and Farrell: Can online resources develop student critical thinking?

Introduction
Background
Critical thinking is expected of students from the first year in higher education (HE) onwards. One of the most difficult challenges for students in the transition phase to academic writing in HE is critical thinking. Lecturers frequently comment that students are not "critical enough"; however, students do not always understand what this means (Price & O’Donovan 2007). A website dedicated to developing critical thinking is one way for students to begin the process of exploring the nature of critical thinking in relation to their discipline area. While the literature on critical thinking has been historically somewhat focussed on the debate about whether critical thinking skills are generic (Norris & Ennis 1990; Swartz & Perkins 1990) or subject-specific (McPeck
1990; Moore 2004), in more recent years there has been a definite move towards a focus on learning and teaching. This is emphasised by those proposing a constructivist approach to critical thinking (Laurillard 2002; Moon 2008). This paper is more concerned with contributing to information regarding students’ development of critical-thinking skills, particularly in relation to an online learning context, than to the debate about the essence of critical thinking.
The use of online learning resources can be of assistance in the process of continually developing students’ critical skills – for “Gen Y” students as well as for other cohorts within the university community. Moreover, it has become an internationally accepted practice for universities to use electronic programs to enhance student learning, including many "new technologies" or tools that may have had an interactive component, such as CD-ROMs, the Internet, video and online databases (Barron & Orwig 1997). Current technological learning tools include digital video and audio equipment (notably, iPods); learning-management systems such as Blackboard; RSS feeds and blogs; and even social communication sites such as Wikis, Facebook, Twitter and YouTube.
Interestingly, debate is also occurring regarding the technological competence of students. Prensky
(2001; 2005; 2007) argues that technologically adept students need to learn in a technologically enhanced environment. However, research in Australia (Kennedy, Dalgarno, Bennett, Judd, Gray
& Chang 2008) warns that presumptions of students’ technological competence may seriously hamper student learning.
In 2006 a Blackboard site was set up at the University of Western Sydney (UWS) to address the expectation that students need to acquire critical thinking as part of their professional and lifelong learning skills. While many commencing students at UWS are high-school leavers, and are commonly assumed to be quite comfortable with the use of information, a number of these and other new students are challenged by the academic culture they encounter. This is to be expected, as over 50% of students are the first in their families to attend university; approximately 30% are mature-age students; and approximately 30% are from Culturally and Linguistically Diverse
(CALD) backgrounds, both local and international (Office of Planning & Quality 2011; Reid
2003). This paper will give a brief overview of the literature about critical thinking and online learning before examining a case study of the critical-thinking site at UWS.
The nature of critical thinking
Because "critical thinking" is an umbrella term used for a complex set of thinking skills, it is important that the term be clarified for HE students. There are many definitions of critical thinking; this can be confusing for students unless each subject lecturer is explicit about their own meaning (Lloyd & Bahr 2010; Moon 2008), particularly if marking criteria include critical

1

Journal of University Teaching & Learning Practice, Vol. 9 [2012], Iss. 1, Art. 4

analysis. A working definition of critical thinking that seems transportable to most HE contexts is
Moon’s (2008, p7): "Critical thinking is a capacity to work with complex ideas whereby a person can make effective provision of evidence to justify a reasonable judgement. The evidence, and therefore the judgement, will pay appropriate attention to context". Furthermore, Duron, Limbach and Waugh (2006) emphasise the higher levels of thinking involved, such as evaluation and reflection. For students, these descriptions highlight that critical thinking is a sophisticated skill set rather than a superficial requirement easily acquired by rote learning.
The process of clarification of the term "critical thinking" may serve, in fact, to highlight the complexity of the term and the various skills it encompasses, such as analysis, making judgements, problem-solving, evaluating, questioning, reflecting and the like. Such expected skills may even vary between different units of study within a discipline (Lloyd & Bahr 2010). At the same time, it has to be taken into consideration that students are at different developmental levels of thinking
(Perry 1970). More recently, an extensive study of undergraduate and post-college students
(Baxter Magolda 1992; 2001) identified four levels of thinking. The basic level, absolute knowing, describes students who hold a black-and-white view of the world. At level two, transitional knowing, students accept that uncertainty is a reality in some areas of knowledge; and at level three, independent knowing describes students who can come to a personal conclusion. The highest level, contextual knowing, is achieved when students appreciate that a particular perspective may be affected by its context.
Critical thinking and technology
Bringing together critical thinking and digital technologies may be beneficial in that it provides an additional opportunity for interested students to achieve Baxter Magolda’s (1992; 2001) higher levels of knowing and/or to practise the seven processes of critical thinking (Carmichael, Craigie,
Driscoll, Farrell, James & Scoufis 1998). Since the Internet is an increasing source of information for learning (Astleitner 2002), it is also important for students to develop critical-thinking skills to apply to web-based materials.
Moreover, there is global recognition of the importance of information literacy for a variety of everyday tasks, for lifelong learning and for successful engagement with the local and international community (Lorenzo & Dziuban 2006; UNSW 2011). The term "information literacy" can be defined as: "the ability to locate, evaluate and use effectively…needed information" (Webber, Boon & Johnston 2005). Prensky (2001) and Oblinger and Oblinger
(2005), writing from the North American perspective, support the notion of current university students' competence with a range of personal and social technologies such as iPods, the advanced features on mobile phones and social networking sites. Prensky (2001) in particular claims that current HE students are greatly disadvantaged by teaching that does not take into consideration their lived experience.
Yet, while acknowledging the social technology skills of Australian HE students, Bennett, Maton
& Kervin (2008) question their overall technological skills and information-literacy levels in terms of approaching academic tasks. In Australia, recent research has reported on survey findings from three universities where 108 university staff and 2,588 students were surveyed regarding their use of technology (Kennedy et al. 2008). Surprisingly, one finding was that many first-year students were "relatively unfamiliar with a range of emerging technologies and tools" (p. 490). Also,
Kennedy et al. (2008) found that cultural and socio-economic factors seem to affect the usage patterns of technologies among some students (Bennett et al. 2008). The implication of these findings is that HE providers cannot assume sophisticated technological skills among their

http://ro.uow.edu.au/jutlp/vol9/iss1/4

2

Carmichael and Farrell: Can online resources develop student critical thinking?

students, and need to take a variety of factors into account when designing web-based materials.
Moreover, while graduates typically display high levels of proficiency in using the required technology once they are in the workplace, they still lack critical-thinking skills (Lorenzo &
Dziuban 2006). Thus care needs to be taken when designing learning materials that the technology is appropriate for the students who will use it, so that they can understand and, in this case, learn about critical thinking.
Learning online in HE – considerations for designers
As with face-to-face situations, successful online learning depends on high-quality planning and delivery, appropriate learning tasks and timely feedback (Clark & Mayer 2008). In the last decade or more, researchers have been attempting to develop a composite picture of the effectiveness of online learning (Astleitner 2002; Perez 2002; Bravo, Van Joolingen & De Jong 2006; Coates
2007; UNSW 2011). Learning-management systems (LMSes), such as Blackboard, are commonly used in Australia and elsewhere as a part of the student learning experience (Coates 2007).
However, little is known about the impact of LMSes on student learning and learning outcomes
(Coates 2007). For example, Coates surveyed over 1,000 students across four disciplinary areas at four institutions to gauge the level of academic and social engagement of commencing students who used online LMSes. From the data obtained from the study, Coates developed a typology suggesting four likely types of online engagement: "intense, collaborative, independent or passive"
(p132). Briefly, intense engagement correlates with high levels of involvement with study, students and staff; collaboratively engaged students prefer the social aspects of learning over independent learning; an independent style of engagement is characterised as highly motivated academically but less socially; while passive engagement is used to describe those students whose participation in all university activities is low. This research goes some way toward an understanding of how students engage with online learning, and how to identify student competence with the technology in an Australian university context.
Taking a pragmatic approach that is localised to the institutional context, Clark and Mayer (2008) suggest that the beginning point for the development of effective web-based resources is consideration of how the e-Learning architecture will reflect different learning purposes. These can be described as: information acquisition; a directive approach or "guided discovery" (p27) through the use of interactive activities; and student-created educational resources that are shared beyond their own institution. Guided discovery supports users’ construction of knowledge and their ability to apply this new knowledge to other contexts, and is therefore the architecture most suited to the development of critical thinking. Ideally, Keats and Schmidt (2007) argue, the social interactivity of technology – especially in relation to sharing and discussion of ideas – has the potential to connect HE students to the international community to create a wider socio-political learning environment. This wider environment should provide additional opportunities to develop critical thinking. This categorisation is considered to be appropriate for the case study to be examined in this paper.
Beneath the e-learning architecture sit four key pedagogical considerations: content, instructional methods, practice examples and feedback (Clark & Mayer 2008). These considerations are important, since together they can support quality student learning, provided students are technologically proficient and have easy access to learning tools and resources. Good educational design also entails a supportive physical and social environment (Ellsworth 1994). The physical environment includes IT resources that are available to students off campus and the university’s IT support structures, as well the design of the materials themselves (Goodyear 2005). These need to be compatible to avoid student dissatisfaction. The social environment will usually involve social

3

Journal of University Teaching & Learning Practice, Vol. 9 [2012], Iss. 1, Art. 4

networking with peers and facilitators on open-forum discussion boards, and may include private feedback from the facilitator (Goodyear 2005).
Critical thinking skills needed for professional and lifelong learning are also important attributes for technology-based learning environments, so that learners can synthesise and evaluate sources of knowledge and integrate learning from social-networking sites. To maximise learning, care needs to be taken to engage HE students who may vary in their learning development, interests and experience with technological skills. However, the blending of technological tools with the development of critical-thinking resources offers a potential for HE students to develop lifelong learning practices and vital social networking; hence the current case study.
The case study
The critical-thinking Blackboard site was established to provide an interactive resource for independent and group student development of the concept and practice of critical thinking. One approach to address the variation in students’ developmental levels, cultural backgrounds and academic experience is to draw on terms reported by HE students themselves to describe critical thinking. These terms can then be used as a starting point to explore more commonly used abstract terms when discussing critical thinking. This approach was first used in a book of sample essays that successfully demonstrated critical thinking in six undergraduate disciplines (Carmichael et al.
1998). The researchers interviewed students who had been awarded a credit grade or higher for an assignment that the marker identified as exhibiting critical thinking. The interviews explored the writers’ and markers’ perceptions of critical thinking and the textual features in these assignments that expressed it. These observations were used to annotate the sample assignments. The collated data from these student interviews suggested seven processes that the student writers frequently used to achieve a critical approach in their writing. These processes are iterative and interrelated, and the learning modules on the Blackboard site at UWS represent each process (Figure 1). This provides an opportunity for students to learn about different aspects of critical thinking. While this is not explicitly teaching critical thinking (Choy & Cheah 2009), it is a resource for students to explore the processes that other students have used in developing their critical thinking (Farrell,
James, Carmichael & Scoufis 1997).

http://ro.uow.edu.au/jutlp/vol9/iss1/4

4

Carmichael and Farrell: Can online resources develop student critical thinking?

Figure 1: Critical thinking site: learning modules

Table 1 names each process, defines its meaning and provides examples of exercises to be found in that particular module on the site. Figure 2 illustrates the introductory module: What is critical thinking? As well as some text and short video clips, there are links to the seven processes that were identified by students in the previous study.

Table 1: Explanation of critical thinking processes in the learning modules
Process
1. Question interpretation Descriptor
Analysing and interpreting the question, before researching and reading the literature

2. Immersion

Understanding the topic through reading, discussion and thought

3. Questioning

Asking questions that help to explore the topic

Example of an exercise
Exercise on breaking down the question into topic, instruction words and focus. Connection between question and point of view expressed in example.
Quiz about how students immerse themselves in a topic; e.g., reading widely. Developing a concept map of questions; e.g., Why is this issue important? 5

Journal of University Teaching & Learning Practice, Vol. 9 [2012], Iss. 1, Art. 4

4. Making links/ connections Tying together a concept with similar/opposing concepts in another field or related to workplace practice or life experience Quiz relating theories to "real life" practical situations, for example work/life balance.

5. Views

Understanding the topic from a variety of perspectives, evaluation of different views

Work-related problem scenario asking for a solution according to the perspective of an employee/manager/HR professional.
Finding concepts, theoretical perspectives or paradigms in a short text. 6. Frameworks

Understanding the relevant concepts within a wider context of a paradigm and/or theory within the subject or course
7. Position
Developing a position and the arguments to support it
Adapted from Scoufis et al. (1999).

Use of a concept pyramid to compare theories and evidence.

Figure 2: Learning module: What is critical thinking?
Design of the online critical thinking site
This Blackboard project is an extension of previously funded projects at UWS encouraging the development of students' critical thinking. The project encompassed Goodyear’s (2005, p2) principles, such as “designing good learning tasks, …good access to robust and appropriate technology and helping create a convivial learning culture”. Blackboard was chosen as the LMS

http://ro.uow.edu.au/jutlp/vol9/iss1/4

6

Carmichael and Farrell: Can online resources develop student critical thinking?

because students are familiar with the system, as it is a common learning tool across the university; there is a high level of system compatibility both on and off campus; and Blackboard is supported by the university’s IT unit. The Blackboard critical-thinking site encompasses interactive learning modules, multi-media segments, the latest edition of the e-book containing sample assignments (Carmichael et al. 2009) and an opportunity for students to communicate via email and discussion board with each other and the site instructors. A place for a reflective journal entry is supplied at the conclusion of each learning module. The design of the site is based on a clear understanding of its purpose (Perez 2002; UNSW 2011). The site is open to all students at
UWS, and is also available for lecturers to use in class to stimulate discussion about critical thinking. The pilot online project began in 2006, when materials were trialled for (i) a site that was available to all students at UWS and accessed through the Student Learning Unit (SLU) portal; and (ii) a module added to the Blackboard site for the subject Engineering and Industrial Design Practice
(EIDP). In 2009 the SLU online resource was further developed, the e-book was updated with additional model assignments from the fields of arts, social sciences and environmental studies, and a specific cohort of students (those enrolled in the subject Business Academic Skills) made use of the site through a more embedded approach. This initiative had instructional input as well as incorporating the independent, social and collaborative aspects of online learning (Astleitner
2002), and obtained evaluative feedback from students to further develop the resource.
While there is a strong element of a student-centred constructivist approach (Laurillard 2002) in this site, it also embodies a "free access" approach (Perez 2002, p225) that allows students to explore materials according to their needs. The learning modules were developed from the interviews with students and lecturers, with explanations of critical thinking by lecturers and students, in video and audio formats. Other modules are based on the seven processes identified above (Table 1). Interactive exercises and quizzes are included in most of the learning modules. A preliminary quiz directs students to the appropriate module/s to enhance their understanding of critical thinking. Another quiz asks them to identify strategies that would help develop critical thinking, or identify issues from various scenarios. Students are also provided with a description of the processes and an example, often taken from student writing. However, what evidence of student learning can be gleaned from their use of this critical-thinking site?
Within this context a research study was undertaken to evaluate this online critical-thinking resource at UWS. An important purpose of the research was to ascertain whether students gain an understanding of critical thinking and are able to transfer this to their discipline and new contexts.
The project not only aimed to stimulate students' critical thinking and independent learning, but also to encourage discussion within the academic community, which had occurred at conference presentations in 2009.

Methodology
Recent research has looked at trends of student usage of online resources (Coates 2007; Kennedy et al. 2008). This evaluative study will add a more in-depth view to the composite picture to date in attempting to examine both the student interactions with the materials and their perceptions of the benefits of using the site. An integrated methodology based on pragmatism and using numerical and qualitative analysis (Denzin & Lincoln 2003) provided information about usage patterns and students’ learning experiences. As Creswell (2003, p12) indicates, a pragmatic approach can use mixed methods that are relevant to finding out “knowledge about the problem”;

7

Journal of University Teaching & Learning Practice, Vol. 9 [2012], Iss. 1, Art. 4

this was the approach used in this case study. First, a numerical analysis of student usage patterns of the two pilot sites established in 2006, SLU and EIDP, was undertaken. In 2009 a similar analysis was carried out. Analysis of student usage of the site in the first semester of 2011 was also determined. In both 2006 and 2009 a questionnaire was distributed, followed by semi-structured
.
semi telephone interviews, and each was analysed thematically as qualitative data.

Findings and Implications
Patterns of usage – what evidence do they provide?
A total of 45 students logged on to the generic SLU website in May 2006, the first month of its operation. Various patterns of usage emerged for example, 33 students had fewer than 10 visits to emerged; , the site, six students had between 11 and 20 visits and six accessed the site more than 21 times
(Figure 1). The highest users (32 105 visits) were from a variety of faculties (Law, Nursing, Arts,
(32-105
Adult Education). It could be expected that they comprehensively explored the site; for instance, instance one student accessed the site 93 times to view 90% of the site. This student's last visit was to explore the pages that describe critical thinking. A second student accessed the site 87 times to view 84% of the site. This student completed an investigation of the site by looking at Handy
Hints, a page that provides some critical-thinking strategies. In contrast, a mid-range user (14 range visits) viewed five pages of the site repeatedly and ended usage of the site by exploring descriptions of critical thinking. Unfortunately due to the under-development of university-wide
Unfortunately,
university
Blackboard reporting practices for the first semester of 2006, it is not possible to ascertain how much time each student spent on the website.

Figure 3: Range of student visits to the SLU site in May 2006 (N=45)
:
However, by the end of 2006 reporting practices had been expanded and it could be determined expanded, that a total number of 66 students had made 1 448 voluntary visits to the site. Length of time for
1,448
t each visit varied from 18 seconds to more than three hours and the number of visits ranged from hours, one to 27. In general, the fewer the visits to the site the more time was spent, but there were some site, notable exceptions. For instance, one student had seven visits for a total of 12 minutes; another accessed the site five times for a total of three hours and explored sample essays and modules on the aspects of critical thinking; one student’s only visit lasted 2 hours 51 minutes. Again a range
Again,
of disciplines was represented. plines Comparison of the usage patterns of the SLU critical-thinking website with the discipline-specific discipline EIDP Critical Thinking module was undertaken. The latter site used the same learning modules modules, but only supplied the engineering model assignments. Approximately 300 students were enrolled model in this first-semester core unit which takes an enabling approach to inducting students into their semester unit, discipline. The EIDP site was used to instruct, support student learning and serve as a

http://ro.uow.edu.au/jutlp/vol9/iss1/4

8

Carmichael and Farrell: Can online resources develop student critical thinking?

communication tool. A random sample of the use of the critical-thinking modules by the 300 students was based on every 20th user (N=14). Usage of the entire EIDP site was spread over the semester, and ranged from 206 to 2,890 visits in total. Visits to the critical-thinking folder within that site were clustered in early May (13 of 14 students), with this sample of students mostly visiting the critical-thinking modules over one to two days, although several had visits spread over one to two months. Students had been advised to use the site over a two-week period as part of their course. It was not possible to determine exactly how many students accessed the criticalthinking materials on the EIDP site. Table 2 shows how this small sample of students made use of the materials.
Table 2: Variation in patterns of usage for random sample of students EIDP (N=14)
Modules in the folder
Descriptions of critical thinking CT slide show
Handy hints
Seven processes of CT
Self-diagnosis of understanding of CT
Sample texts demonstrating
CT
Practice exercises
Answers to exercises
Introduction to model assignments Model Assignment 1
Model Assignment 2
Model Assignment 3

Number of student visits
13
4
11
1
1

Number of repeat visits
4 (1 returned 6 times; another
5 times)
1 (returned twice)
1 (student returned 3 times)
0
0

8

3 (1 returned 4 times)

8
3
7

4 (1 student returned 3 times; another 2)
0
3

4
6
5

1 (returned 4 times)
0
2

This snapshot of usage patterns indicates that students’ exploration of the site varied considerably.
For instance, students 4 and 9 seemed to take a somewhat logical but "short cut" approach, beginning with a brief overview of critical thinking, handy hints, a practice example and answer and finally another practice text and answer. Student 4 then accessed the model assignments in order on three different days, suggesting a need to gain more in-depth knowledge. Students 8, 11 and 12 all took a methodical approach by accessing the conceptual items (definitions, descriptions and slide show) followed by examples and practice exercises. However, only Student 12 of this sub-group looked at the suggested answers. Two looked at the introduction to model assignments but did not explore the models. In terms of the range of approaches, the students using either site demonstrated a "free access" approach (Perez 2002) or independent learning, following their interests or taking a "need to know" approach. Returning to particular items seems to suggest engagement (Coates 2007). Closer analysis of the EIDP site demonstrates in more detail the variation in student usage found through the SLU site.
Usage patterns in 2009 show some differences to those in 2006 (Table 3). Student use of the site was voluntary, though one group of students in a first-year core subject, Business Academic Skills

9

Journal of University Teaching & Learning Practice, Vol. 9 [2012], Iss. 1, Art. 4

(BAS), was encouraged to use the site as part of a learning portfolio. This group presumably contributed to a higher usage rate in 2009, when 904 users were recorded. These users made 2,821 visits to the site, ranging from one to 66 return visits for any one individual. While the number of users of the SLU site has greatly increased, the time spent on the site for many students has been minimal. For instance, in 2009 44% of the users spent less than 10 minutes on the site. The average user time was 41 minutes and the most time spent by any user was 12 hours. However, time spent on a site may be complicated by students leaving the computer on while they do something else, or they may print out information to read later. The usefulness of the site for some students can be measured by those who revisited the site over a nine-month period. An examination of several students’ use of the learning modules in 2009 indicated that, as noted above, they tended to "dip" into modules randomly, as it suited them (Perez 2002), and spent differing amounts of time exploring any one module. Students were interested in the discussion messages, with 30% of them reading the messages regularly. However, only 7% actually contributed to the discussion by posting a message. Twenty-seven percent attempted the quizzes, while most students viewed the learning module and book files between one and 127 times. Late at night was the most common time students used the site. This pattern of usage continued to demonstrate a "need to know" approach. Those who could be seen to "overuse" the site may have been genuinely learning from revisiting modules or model assignments, or were perhaps seeking more interactive communication with peers (Coates 2007).
Finally, in the first half of 2011 there have been 1,170 visits to the site from 520 users (Table 3).
The maximum time spent on the site by any one user was 9 hours and 35 minutes, with a maximum of 20 return visits by any one student. Sixty students spent over half an hour using the site, with 39 of those using the site between one and 9.5 hours. Again, it is uncertain whether extended time spent on a site may be complicated by students leaving the computer unattended, or whether short logon times are indicative of printing out information to read in hard copy.

Table 3: Variation in patterns of usage for students using the site, 2006-2011
Site

Number of users

2006 SLU site

66

2006 EIDP site

300

2009 SLU site

904

2011 SLU site
(semester 1)

520

Number of visits
1,448
206 - 2,890
(entire EIDP site, numbers unknown for critical thinking site)
2,821
1,170

Time accessed
18 seconds - 3 hours 2 days - over 2 months 22 seconds - 12 hours 4 seconds - 9.5 hours Number of return visits
1 - 27
Different
patterns (see samples in Table
2)
1 - 66 return visits 1 - 20 return visits Student feedback on the usefulness of the site
To gain student feedback, a questionnaire was distributed to 448 students who had used the site during 2009, with follow-up telephone interviews. Responses from 113 students (25%), the

http://ro.uow.edu.au/jutlp/vol9/iss1/4

10

Carmichael and Farrell: Can online resources develop student critical thinking?

majority of whom (64%) were studying business, were obtained. This may relate to the learning portfolio for BAS students, or to the fact that the largest cohort of students at UWS is business students. The remainder were from various degree courses including nursing and health-related disciplines, teaching/education, sciences and arts. Year of study varied: approximately 68% were first-year students, 27% were second- or third-year students and 5% were fourth-year or more, including postgraduate and part-time students.
Numerical analysis of the questionnaire responses was indicative of student engagement with the critical-thinking process as a result of using the site (Table 4). Importantly, 95% of the respondents would recommend this site to other students, of whom 73% indicated that they found the site to be useful by rating it 3.5 or above on a Likert scale of 1 to 5, where 1 = not useful and 5 = very useful. Although only 38% of the students responding to the questionnaire had read sample assignments in the e-book, all who did read them indicated that they were useful. This deeper reading of information is more suggestive of Coates's (2007) notion of intensive and independent engagement. On the other hand, 67% of the students found the exercises and activities in the interactive learning modules to be useful, indicating that more students prefer the instant feedback provided in those modules. Ninety-four percent of respondents did not find the modules difficult to understand. Table 4: Responses to feedback survey (N=113)
Question
Would you recommend this site to other students?
Was the site useful in terms of understanding critical thinking?
Did you read the sample student assignments in the e-book? Did you find these sample assignments useful? (N=
33)
Did you find the interactive exercises in the modules useful?
Did you find the modules too difficult to understand? Yes
95%

N/A or No
5%

Total
100%

73%

27%

100%

38%

62%

100%

100%

0%

100%

67%

33%

100%

6%

94%

100%

Qualitative evaluation of student experience of the site
The numerical analysis of student usage and engagement shown in Table 4 indicates that students took different approaches to exploring the site and found it useful. However, this data does not delve into the students' experiences of their learning. Qualitative information was gained from the general SLU site in 2009 via questionnaires, emailed comments on the site and semi-structured interviews (Freebody 2003). This provided a more in-depth understanding of learning for students, especially undergraduate students.

11

Journal of University Teaching & Learning Practice, Vol. 9 [2012], Iss. 1, Art. 4

Students who interacted with the instructor via the discussion board on the general SLU site from
2006 initially discussed practical questions, such as access problems, but also expressed positive feedback that suggested they were motivated and engaged, and which corresponded to Coates’ intense engagement (Coates 2007). Statements included: "I’ve found the site helpful and hope I can now put it into practice" and "this site is really great, just wish I looked at it sooner". Others were enthusiastic, with statements such as: "I wanted to comment on how important and helpful this site has been…[I] have an exam where critical thinking will be questioned and the information found here has really helped". More recently, students have interacted with their peers in comments such as: "I agree with XXX, our life experiences assist us with analysing the given information…." Information relating to their perceptions of the usefulness of the site was obtained through student comments in response to the questionnaire. One theme that emerged was that the information and exercises were helpful for writing assignments, and with various subjects. One student said ‘[I] liked doing it at home in my own time. It helped for questioning information when researching to make up a table with both sides of the argument’. Other students appeared to believe that the modules had scaffolded their learning about academic writing, such as the following comment:
‘Helped the transition between high school and uni.’ An undergraduate business student made the following observation about writing an essay for Macroeconomics: ‘The part about reading different texts and contrasting those ideas with the assigned text book was particularly helpful…
The site provided a good outline to start off an essay’. This indicates a broader context in which students could synthesise information and transfer it to other situations (Lorenzo & Dziuban
2006). This reported transference in three contexts by three separate students, is indicative of respondents’ confidence and ability to operate at Baxter Magolda’s higher levels of thinking:
Independent Knowing or even Contextual Knowing (1992; 2001).
Some students made comments specifically about the accessibility of the content on the site and how it helped their understanding of critical thinking. For example, one student stated that
‘examples were easy to understand, helped define what critical thinking is. Quizzes were useful’.
Another student explained that the site ‘helped me to understand critical thinking and pinpoint areas to analyse text [to] use in my assignment’. One international Nursing student made the following comments on her questionnaire: ‘it was very useful as I am a Mature Age student and from Brazil. It gave me ideas about the way to study in Australia. I read through it from beginning to end and I answered all the questions. Exercises were very useful as I got knowledge and feedback’. Another student said: ‘The examples and descriptions are not too short, but to the point. You don’t have to read too much’. This is reinforcing of the approach taken in developing the interactive online materials as short, bite sized sections for ease of reading (Perez 2002), and in keeping with the principles of educational technology espoused by Goodyear (2005) and Clark and
Mayer (2008). One student found the site ‘unhelpful’ as she wanted to gain high distinctions for her essays. After several email communications with the site instructor, she began sharing information and resources regarding academic argument with the other students via the Discussion
Board. This demonstrated a change in attitude and a more mature interaction with the site, particularly when she began to achieve HDs for her assignments.
Of the seven learning modules, the students rated the most useful modules to be Questioning and
Question Interpretation, which are possibly the most "concrete" of the modules, and a practical starting point for critical thinking. These preferences were supported by a closer examination of the learning reflections written by 50 students, which indicated that the former module had raised awareness of breaking the question into key parts: topic, instruction words and focus. One student

http://ro.uow.edu.au/jutlp/vol9/iss1/4

12

Carmichael and Farrell: Can online resources develop student critical thinking?

wrote: "before I have never concentrated too much on analysis [of the question], too time consuming, especially in exams [now] I realise it is not a waste of time". Another wrote that "this activity is really very useful, it helped me to understand the first aspect of thinking critically, that is to understand the question". Many of the students’ reflections indicated that interpreting the question was important for guiding a clearer and better essay answer. This is evidence of stimulation of student thinking, albeit at a rudimentary level.
Furthermore, the reflections suggest that students learned useful strategies from the modules, which assisted their approach to critical thinking in assignment preparation. For example, one student wrote: "this module helped me to understand that ...I have to take into account the different views that are within the text"; another student commented about developing a questioning approach: "now I am better equipped to put into practice my understanding…" One student explained, "I did not realise how important questioning is until I read through this topic…the concept map was also very helpful as it addressed all the issues we should explore before we answer a question".
Another theme in the questionnaire comments was that the assignments in the e-book were good
"example assignments/papers" that gave them more confidence in understanding what was expected. This was acknowledged by a number of students, with one stating that it gave "insight into how to answer the question". Another stated that the most useful aspect of the model assignments was "teacher and student comments". Overall, these reflective comments suggest that the respondents gained insight into the complex term "critical thinking" in relation to their academic writing.
One BAS tutor, who recommended the resources to her class during 2009, also indicated that student learning was enhanced by using them. She taught approximately 100 students altogether in four separate tutorial groups. Over 95% of the students were first-years. The site, she suggested, provided transitional support for new students: "for a lot of students seeing what actually qualifies as a good piece of writing is really very important because coming in from high school they haven’t seen the level of thinking that is required". After marking assignments, she reviewed student usage patterns of the site and identified that those students who had used the site achieved better marks in their reflective assessment than those who had not used it.

Conclusion
The literature indicates that the success of using online resources for the development of students' critical thinking in the HE context at least partially depends on students’ developmental levels, their experience with the technology used in academic settings and their levels of engagement.
Successful learning about critical thinking in an online environment is also contingent on the quality of online resources and tools. This case study demonstrates that many students find online learning about critical thinking to be helpful, stimulating and engaging. It verifies that some students enjoy learning in their own space and time and that this site contained suitable content, sample texts, practice examples and timely feedback (Clark & Mayer 2008).
Research suggests that a site embedded within a unit of study with instructional and peer interaction could achieve better critical-thinking outcomes (Ramsden 1992; Skillen 2006) in relation to teaching skills, whether they be thinking or writing skills. However, findings in this case study indicate that stand-alone resources can achieve perceived benefits for students. Further research comparing student understanding of critical thinking via a stand-alone site with one

13

Journal of University Teaching & Learning Practice, Vol. 9 [2012], Iss. 1, Art. 4

embedded in a face-to-face learning environment would offer an opportunity to more closely evaluate student outcomes. This could include indicators of learning in terms of assignment marks or grades.
There is also potential to expand the current site into a freely accessible website, with opportunity for greater national and international interaction together with interesting research opportunities
(Keats & Schmidt 2007). This could provide students with the experience of technology to broaden their perspectives regarding critical thinking through networking in a wider crossinstitutional and cross-country environment.

References
Astleitner, H. (2002). Teaching critical thinking online. Journal of Instructional Psychology,
29(2), 53-76.
Barron, A. & Orwig, G. (1997). New technologies for education: A beginner’s guide, 3rd edn.
Librarians Unlimited Inc., Englewood, CO.
Baxter Magolda, M. (1992). Knowing and reasoning in college: Gender-related patterns in students' intellectual development. Jossey-Bass, San Francisco.
Baxter Magolda, M. (2001). Making their own way: narratives for transforming higher education to promote self-development. Stylus, Sterling, VA .
Bennett, S., Maton, K. & Kervin, L. (2008). The ‘digital natives’ debate: A critical review of the evidence. British Journal of Educational Technology, 39(5), 775-786.
Bravo, C., van Joolingen,W. & De Jong, T. (2006). Modelling and simulation in inquiry learning:
Checking solutions and giving intelligent advice. Simulation, 82(11), 769-784.
Carmichael, E., Craigie, D., Driscoll, K., Farrell, H., James, B. & Scoufis, M. (1998). Critical
Analysis – What is it? 2nd edn, University of Western Sydney Nepean, Kingswood, Sydney.
Carmichael, E., Craigie, D., Driscoll, K., Farrell, H., James, B. & Scoufis, M. (2009). Critical
Analysis – What is it? 5th edn, University of Western Sydney, Kingswood, Sydney.
Choy, S. C. & Cheah, P. K. (2009). Teacher perceptions of critical thinking among students and its influence on higher education. International Journal of Teaching and Learning in Higher
Education, 20(2), 198-206.
Clark, R. & Mayer, R. (2008). E-Learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning, 2nd edn. Pfeiffer, San Francisco.
Coates, H. 2007, A model of online and general campus based student engagement. Assessment and Evaluation in Higher Education, 32(2), 121-141.
Creswell, J. (2003). Research design: qualitative, quantitative, and mixed methods approaches, 2nd edn. Sage Publications, Thousand Oaks, CA.

http://ro.uow.edu.au/jutlp/vol9/iss1/4

14

Carmichael and Farrell: Can online resources develop student critical thinking?

Denzin, N. & Lincoln, Y. (Eds) (2003). Collecting and interpreting qualitative materials, 2nd edn.
Sage Publications, London.
Duron, R., Limbach, B. & Waugh, W. (2006). Critical thinking framework for any discipline.
International Journal of Teaching and Learning in Higher Education, 17(2), 160-166.
Ellsworth, J., (1994). Education on the Internet: A hands-on book of ideas, resources, projects and advice, Sams Publishing, Indianapolis, IN.
Farrell, H., James, B., Carmichael, E. & Scoufis, M. (1997, Critical practice and undergraduate students. In Golebiowski, Z. (Ed.), Policy and Practice of tertiary literacy: Selected proceedings of The First National Conference on Tertiary Literacy: Research and Practice,
Victoria University, Melbourne, 1, 150-159.
Freebody, P. (2003). Qualitative research in education. Sage Publications, London.
Goodyear, P., (2005). Educational design and networked learning: Patterns, pattern languages and design practice. Australasian Journal of Educational Technology, 21(1), 82-101.
Keats, D. & Schmidt, J. (2007). The genesis and emergence of Education 3.0 in higher education and its potential for Africa. First Monday, 12(3), 1-14. Accessed 2 September 2011 from http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/1625/15. Kennedy, G., Dalgarno, B., Bennett, S., Judd, T., Gray, K. & Chang, R. (2008). Immigrants and natives: Investigating differences between staff and students’ use of technology. In Hello!
Where are you in the landscape of educational technology? Proceedings ASCILITE
Melbourne 2008. Accessed 23 August 2009 from http://www.ascilite.org.au/conferences/melbourne08/procs/kennedy.pdf. Laurillard, D. (2002). Rethinking teaching for the knowledge society. Educause Review, 37(1), 1625.
Lorenzo, G. & Dziuban, C. (2006). Ensuring the Net Generation is Net Savvy. Educause Learning
Initiative. Accessed 2 September 2011 from www.educause.edu/ELI/Ensuring theNetGenerationIsNetS/156766. Lloyd, M. & Bahr, N. (2010). Thinking critically about critical thinking in higher education.
International Journal for the Scholarship of Teaching and Learning, 4(2), vol.4, 1-16.
Accessed 11 July 2011 from http://www.georgiasouthern.edu/ijsotl.
McPeck, J. (1990). Teaching critical thinking. Routledge, New York.
Moon, J. (2008). Critical thinking: An exploration of theory and practice. Routledge, London.
Moore, T. (2004). ‘The critical thinking debate: how general are the general thinking skills?’.
Higher Education Research & Development, 23(1), 3-18.
Norris, S. P. & Ennis, R. H. (1990). Evaluating Critical Thinking: Practitioners’ Guide to
Teaching Thinking Series. Hawker Brownlow Education, Cheltenham, Vic.

15

Journal of University Teaching & Learning Practice, Vol. 9 [2012], Iss. 1, Art. 4

Oblinger, D. & Oblinger, J. (2005). Is it age or IT: First steps towards understanding the net generation. In Oblinger, D. & Oblinger, J. (Eds), Educating the net generation, 2.1-2.20.
Accessed 30 August 2011 from http://www.educause.edu/educatingthenetgen.
Office of Planning and Quality, University of Western Sydney. Commencing Student Survey, TILT
System. Accessed 11 July 2011 from http://www.uws.edu.au/opq/planning_and_quality/tracking_and_improving_performance#1. Perez, A. (2002). Evaluating web-based learning skills support: Effective learning tool or distraction? In Webb, J. & McLean, P., Academic Skills Advising: Evaluating for program improvement and accountability, Victoria Language and Learning Network. 222-233.
Perry, W. (1970). Intellectual and ethical development in college years. Holt, Rinehart and
Winston, New York.
Prensky, M. (2001). Digital natives, digital immigrants. On the Horizon, 9(5). Accessed 2
September 2011 from http://www.marcprensky.com/writing/Prensky%20%20Digital%20Natives,%20Digital%20Immigrants%20-%20Part1.pdf.
Prensky, M. (2005). Listen to the Natives. Accessed 2 November 2009 from http://www.ascd.org/cms/objectlib/ascdframeset/index.cfm?publication=http://www.ascd.org /authors/ed_lead/el200512_prensky.html.
Prensky, M. (2007). Changing Paradigms. Accessed 2 September 2011 from http://www.marcprensky.com/writing/Prensky-ChangingParadigms-01-EdTech.pdf. Price, M. & O’Donovan, B. (2007). Making meaning out of assessment feedback – getting more than the message. In Enhancing Higher Education, Theory and Scholarship, Proceedings of the 30th HERDSA Annual conference [CD-ROM], Adelaide, 8-11 July.
Reid, J. (2003). Why the Government’s figures don’t add up. University of Western Sydney Media release, September 2003. Accessed 19 September 2005 from http://www.gws.org.au/cgibin/mediareleases.pl?form_type=mainmenu. Ramsden, P. (1992). Learning to teach in higher education. Routledge, London.
Scoufis, M., James, B., Carmichael, E. & Farrell, H. (1999). On the track: Critical thinking in assignment writing [CD Rom]. Video Education Australasia, Bendigo.
Skillen, J. (2006). Teaching academic writing from the “centre” in Australian universities. In
Ganobcsik-Williams, L. (Ed.), Teaching academic writing in UK higher education: Theories, practices and models. Palgrave MacMillan, New York.
Swartz, J. & Perkins, D.N. (1990). Teaching thinking: Issues & approaches. Midwest
Publications, Pacific Grove, CA.
UNSW, COFA online Gateway 2011, Learning to teach online episodes. Accessed 1 September
2011 from http://online.cofa.unsw.edu.au/learning-to-teach-online/ltto-episodes.

http://ro.uow.edu.au/jutlp/vol9/iss1/4

16

Carmichael and Farrell: Can online resources develop student critical thinking?

Webber, S., Boon, S. & Johnston, B. (2005). A comparison of UK academics’ conceptions of information literacy in two disciplines: English and Marketing. Library and Information
Research, 29(93), 4-15. Accessed 3 September 2011 fromhttp://www.cilip.org.uk/specialinterestgroups/bysubject/research/publications/jjournal/ar chive.

17