Evaluation

Many members of the working group have been and are involved in studies that demonstrate the efficacy of the guidance given in the Sections 2 and 3 of the 2003 ITiCSE Working Group report (to be published here). These are collected online and provide concrete realizations of the principles that are described in the report.

Presently, this list includes the following studies:

  • Stephen Cooper and Wanda Dann -- This study examines the use of program visualization for introducing objects and their behaviors using Alice (DCP:2000), a 3D program visualization environment. Statistical data is collected to show evidence of student performance and retention in CS1 for test and control groups. Early summaries may be found in CDP:2003.

    Classification:
    • completed but not yet published
    • oriented toward measuring student outcomes with respect to everyday teaching situations
    • oriented toward more formal education research
  • Rudolf Fleischer -- This study will measure the effectiveness of visualizations in the context of a second-year course on the theory of computation (finite automata, context-free grammars, Turing machines) (Flei:2003) The study will be done in Spring 2004 at the Hong Kong University of Science and Technology.

    Classification:
    • in progress
    • oriented toward measuring student outcomes with respect to everyday teaching situations
  • Boris Koldehofe -- By using the framework presented in this working group report and considering the results of a previous study (KPT:2003), a new study of instructor's satisfaction with LYDIAN (KPT:2000), a simulation-visualization environment for learning and teaching distributed algorithms, is planned. The purpose is to evaluate which features of LYDIAN contribute to instructor satisfaction and the teachers' performance when integrating LYDIAN in their course. The study also tries to determine the factors which may prevent instructors from actually using LYDIAN after they download the tool.

    Classification:
    • in progress
    • oriented toward measuring instructor satisfaction
  • Ari Korhonen and Lauri Malmi -- Intervention study with automatic assessment (KMMS:2002)-- the paper presents the results of the large scale (N=550 students) intervention study carried out in a virtual learning environment based on the TRAKLA system (KoMa:2000). The system provides individually tailored exercises that the learner solves on the Web using an algorithm simulation (KMNS:2002). The learner receives immediate feedback around the clock on his or her performance. The system has features that traditional instruction cannot provide. Thus, the study was pursued to research the quality, advantages and limitations of this novel approach. One of the conclusions was that the learning environment was as good as if they were solving the same exercises in class room with human tutors giving the feedback.

    Classification:
    • completed and published
    • oriented toward more formal education research
  • Ari Korhonen and Lauri Malmi -- This study presents some experiences and observations (MKS:2002) made during 10 years of using the TRAKLA system (KoMa:2000). It is inevitable that learners perform better when they are allowed to resubmit their work. However, this is not the whole story; it is also important to organize the course so that the skills and challenges of the learner coincide. Moreover, the grading policy seems to have a major impact on performance. The study summarizes the results and points out changes in learner performance under the several changes the course has gone through during the time period.

    Classification:
    • completed and published
    • oriented toward measuring student outcomes with respect to everyday teaching situations
  • Ari Korhonen and Lauri Malmi -- This study focuses on the effect of different learner engagement levels on learning. The ITiCSE 2002 Working Group report on visualization prepared a taxonomy of engagement in order to identify and differentiate various types of actions the learners may perform while using visualizations. The plan for this study will appear at the Web site of the Computer Science Education Research Group at Helsinki University of Technology.

    Classification:
    • in progress
    • oriented toward measuring student outcomes with respect to everyday teaching situations
  • Marja Kuittinen and Jorma Sajaniemi -- This study evaluates the use of variable roles and role-based animation in teaching introductory programming. The role concept captures tacit expert knowledge in a form that can be taught to novices. Details about the project and evaluation studies can be found at http://cs.joensuu.fi/~saja/var_roles/.

    Classification:
    • completed and published
    • oriented toward more formal education research
  • Charles Leska -- This study examines the impact on student performance of using visualization on a unit in a computer literacy course. The unit focuses on the binary representation of data and the interplay of the processor components during the fetch-execution cycle. Details about the study can be found at http://faculty.rmc.edu/cleska/public_html/algvizstudy.htm.

    Classification:
    • in progress
    • oriented toward measuring student outcomes with respect to everyday teaching situations
  • Scott Grissom, Myles McNally, and Tom Naps -- This study compares the effect of three different learner engagement levels with visualization in a course module on introductory sorting algorithms. Results of the study have been published in a paper.

    Classification:
    • completed and published
    • oriented toward measuring student outcomes with respect to everyday teaching situations
  • Jarmo Rantakokko -- This study evaluates the effects of using algorithm visualization in parallel computing. The study focuses on student learning outcomes. Details about the project and this particular study can be found at http://user.it.uu.se/~jarmo/hgur.html.

    Classification:
    • in progress
    • oriented toward more formal education research

Publications

  • CDP:20003
    S. Cooper, W. Dann, and R. Pausch, Introduction to OO: Teaching Objects-First in Introductory Computer Science. In Proceedings of the 34th ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE 2003), Reno, Nevada (2003). ACM Press, New York, pp. 191-195.
    Abstract and PDF are available at the ACM Digital Library.
  • DCP:2000
    W. Dann, S. Cooper, and R. Pausch, Making the Connection: Programming With Animated Small World. In Proceedings of the 5th Annual ACM SIGCSE/SIGCUE Conference on Innovation and Technology in Computer Science Eduation (ITiCSE 2000), Helsinki, Finland (2000). ACM Press, New York, pp. 41-44.
    Abstract and PDF are available at the ACM Digital Library.
  • Flei:2003
    R. Fleischer, COMP 272: Theory of Computation - A proposed study on the learning effectiveness of visualizations, manuscript (2003).
  • GMN:2003
    S. Grissom, M. McNally, and T. Naps, Algorithm Visualization in Computer Science Education: Comparing Levels of Student Engagement. In Proceedings of the First ACM Symposium on Software Visualization, San Diego, California (2003). ACM Press, New York, pp. 87-94.
    Abstract and PDF are available at the ACM Digital Library.
  • KPT:2000
    B. Koldehofe, M. Papatriantafilou, and P. Tsigas, LYDIAN, An Extensible Educational Animation Environment for Distributed Algorithms. In Proceedings of the 4th Annual ACM SIGCSE/SIGCUE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2000), Helsinki, Finland (2000). ACM Press, New York, p. 189.
    Abstract and PDF are available at the ACM Digital Library.
  • KPT:2003
    B. Koldehofe, M. Papatriantafilou, and P. Tsigas, Integrating a Simulation-Visualization Environment in a Basic Distributed Systems Course: A Case Study Using LYDIAN. In Proceedings of the 8th Annual ACM SIGCSE/SIGCUE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2003), Thessaloniki, Greece (2003). ACM Press, New York, p. 226.
    PDF available at Chalmers University.
  • KMMS:2002
    A. Korhonen, L. Malmi, P. Myllyselkä, and P. Scheinin, Does it Make a Difference if Students Exercise on the Web or in the Classroom?. In Proceedings of the 7th Annual ACM SIGCSE/SIGCUE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2002), Arhus, Denmark (2002). ACM Press, New York, pp. 121-124.
    Abstract and PDF are available at the ACM Digital Library.
  • KMMSS:2002
    A. Korhonen, L. Malmi, P. Mard, H. Salonen, and P. Silvasti, Electronic course material on Data structures and algorithms. In Proceedings of the Second Annual Finnish / Baltic Sea Conference on Computer Science Education (2002), pp. 16-20.
  • KMNS:2000
    A. Korhonen, L. Malmi, J. Nikander, and P. Silvasti, Algorithm Simulation - A Novel Way to Specify Animations. In Proceedings of the the Second International Program Visualization Workshop, Hornstrup Centret, Denmark (2002).
    The PDF file is available at University of Aarhus, Denmark press.
  • KoMa:2000
    A. Korhonen and L. Malmi, Algorithm Simulation with Automatic Assessment. In Proceedings of the 5th Annual ACM SIGCSE/SIGCUE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2000), Helsinki, Finland. ACM Press, New York, pp. 160-163.
    Abstract and PDF are available at the ACM Digital Library.
  • KuSa:2003
    M. Kuittinen and J. Sajaniemi, First Results of An Experiment on Using Roles of Variables in Teaching. In EASE & PPIG 2003, Papers of the Joint Conference at Keele University, England (2003), pp. 347-357.
    For more information about this and related publications, please see the literature on the role of variables.
  • MKS:2002
    L. Malmi, A. Korhonen, and R. Saikkonen, Experiences in Automatic Assessment on Mass Courses and Issues for Designing Virtual Course. In Proceedings of the 7thAnnual ACM SIGCSE/SIGCUE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2002), Arhus, Denmark. ACM Press, New York, pp. 55-59.
    Abstract and PDF are available at the ACM Digital Library.
  • NR+:2003
    T. Naps, G. Rößling, V. Almstrum, W. Dann, R. Fleischer, C. Hundhausen, A. Korhonen, L. Malmi, M. McNally, S. Rodger and J. Á. Valázquez-Iturbide, Exploring the Role of Visualization and Engagement in Computer Science Education. ACM SIGCSE Bulletin 35:2, June 2003, pp. 131-152.
    Abstract and PDF are available at the ACM Digital Library.
  • SaKu:2003
    J. Sajaniemi and M. Kuittinen, An Experiment on Using Roles of Variables in Teaching Introductory Programming. Submitted to Empirical Software Engineering, 2003.
    For more information about this and related publications, please see the literature on the role of variables.


© Dr. Guido Roessling 2015