Chapter 10: User Studies


Karoulis, Athanasis, Ioannis Stamelos, Lefteris Angelis. “Experimental Evaluation of an Instructional Supporting Tool in Distance Learning.” Educational Technology & Society 11, no.3 (2008) 67-81.
This article provides an overview of a research project conducted at a Greek university, which sought to analyze the effects of an electronic learning aid on student performance in a distance education course.  Using an already established learning aid, the researchers aimed to assess the effectiveness of the tool and measure usability in a distance education setting.  After detailed testing, which is thoroughly described and documented in the article, the researchers discovered that the learning aid had significant effects on student performance in the distance education course.  M. Houlihan

McMartin, Flora, Ellen Iveson, Alan Wolk, Joshua Morrill, Glenda Morgan, and Cathryn Manduca. “The Use of Online Digital Resources and Educational Digital Libraries in Higher Education.” International Journal on Digital Libraries 9, no. 1 (2008): pp.65-79.  
This paper presents the results of a survey of U.S. faculty aimed at understanding their use of educational digital libraries.  Approximately a fifth of the respondents taught online courses.  Results revealed that faculty are more alike than different in their use of digital materials – variables such as institution type and teaching experience did not highly influence use. Lack of time was a common barrier to use.  Valuing digital resources did not always result in higher use, and there was a heavy reliance on search engines to find digital material, thus faculty were not often aware that they were using a digital library.  These findings reveal the challenges of educating faculty about the existence of digital libraries and encouraging instructors to utilize these resources in their teaching.  T. Clayton

Miller, Rush, Hong Xu, Xiuying Zou. “Global Document Delivery, User Studies, and Service Evaluation: The Gateway Experience.” Journal of Academic Librarianship 34, no. 4 (2008): 314-321.
The East Asian Gateway Service (EAGS) at the University of Pittsburgh is a document delivery service that provides U.S. scholars with Chinese language articles and East Asian scholars with English language articles. This article examines user characteristics and document request patterns of the EAGS service from 2002-2006.  Important trends revealed by the study include an increase of requests by U.S. users for Chinese articles in the sciences and an overwhelming demand for Western medical journal articles by Chinese scholars.  The analysis reveals that the EAGS is an important model for global document delivery and offers an excellent case study for cross country user studies and service evaluations.  T. Clayton


Bolman, Catherine; C. Tattersall, W. Waterink, J. Janssen, B. van den Berg, R. van Es and Koper, R. “Learners' Evaluation of a Navigation Support Tool in Distance Education.” Journal of Computer Assisted Learning 23, no. 5 (2007): 384-392.
This article describes an experimental study that examines the usefulness of a navigation support tool used in an online course to support distance learning.  In a self-directed online internet skills course, the online navigation tool provided advice to the experimental group about which course module would be best to complete next.  System logs and questionnaires provided data about how learners evaluated the tool and whether or not subjects followed the advice.  The study showed that more than half of the learners in the experimental group followed the advice and felt that the tool helped them continue the course.  However, a majority of subjects expressed a desire to know the source of the tool’s advice.  The results suggest that online learners will put more trust, and hence find more usefulness, in navigation tools if the source of the advice is revealed.  T. Clayton

Connaway, Lynn Silipigni, Lawrence Olszewski, Lillie R. Jenkins. “What is enough?:  Satisficing information needs.” Journal of documentation 64, no. 1 (2007): pp 74-89.
Using focus group interviews with faculty, undergraduate, and graduate students, the authors investigated what leads researchers to stop searching for more information when there is no confidence that pertinent information has been identified. The subjects report using both qualitative and quantitative criteria that help them make rational choices that determine when enough information has been discovered.  Graduate and undergraduate students' criteria include what is required to complete a class assignment or achieve a good final grade.  The faculty take into consideration the time required in light of their other duties such as giving lectures or speaking at conferences or workshops.  When researching articles for publication, faculty normally seek feedback from their colleagues and reviewers.  The authors include implications for libraries and information science practice.  R. Leech

de Villiers, M. Ruth. “An Action Research Approach to the Design, Development and Evaluation of an Interactive E-Learning Tutorial in a Cognitive Domain.” In Journal of Information Technology Education 6 (2007): 455-479.
This article provides an overview of a research project conducted at the University of South Africa that evaluated an interactive e-learning application called Relations.  Relations is used to teach advanced mathematical skills to students in a cognitive domain.  The author describes the content of the application as well as the evolution of and assessment process of the application since 1994.  Four different evaluation methods were used to evaluate the application to ensure accurate and superior results—questionnaire surveys, interviews, heuristic evaluation and a post-test.  It was discovered that Relations did indeed have a positive effect on student learning.  M. Houlihan

Gilbert, Jennifer, Susan Morton, Jennifer Rowley. “e-Learning: The Student Experience.” British Journal of Educational Technology 38, no. 4 (2007): 560-573.
This article seeks to provide critical feedback of students’ perceptions and experiences with e-learning and also shed light on the student learning processes and habits in an e-learning environment.  In order to gain qualitative and quantitative information, the authors distributed a student evaluation form to Masters level students enrolled in an Information Technologies and Management course.  The survey results offer great insights into the satisfaction and dissatisfaction students have with e-learning environments, and prove that all students interact with the materials differently.  Also, often students are hesitant about the role of the instructor in online course.  M. Houlihan

Haya, Glenn, Else Nygren, Wilhelm Widmark.“Metalib and Google Scholar: a user study.” Online Information Review 31, no. 3 (2007): 365-375.
This article documents a user study of Google Scholar and Metalib conducted with 32 Uppsala University undergraduates in 2005-06.  Participants were asked to spend twenty minutes with each search tool to locate relevant articles for their thesis research and to fill out questionnaires after each session.  Prior to their sessions, half of the subjects received basic training on using the two search tools.  The authors draw several conclusions relating to Google Scholar’s relatively higher usability and neither tool’s suitability as a “first choice” resource.  In addition, a crucial finding shows that instruction has impact.  Students with prior instruction found more articles using both Metalib and Google Scholar.  Thus, librarians must think critically about how best to present and deliver instruction for these resources, particularly for distance education students.  T. Clayton

Johnson, Megan, Louise Ochoa, Geraldine Purpur. “Virtually Usable: A Test of the Information Gardens.” Journal of Academic Librarianship 33, no. 5 (2007): 593-601.
This article presents the results of a usability study conducted by the Appalachian State University (ASU) Distance Learning Library Services team. They tested the functionality of a 3-dimensional desktop virtual reality library they call the Information Gardens.  This virtual library supports three graduate programs in the Reich College of Education Leadership and Educational Studies Program at ASU.  The authors´ goal was to identify areas for improvement while still in the design stage.  They describe a series of three informative evaluations, using questionnaires, post hoc interviews and recordings using Morae software.  R. Leech

Lang, Daniela, Christoph Mengelkamp, Reinhold Jaeger, Didier Geoffroy, Michael Billaud, Thomas Zimmer. “Pedagogical Evaluation of Remote Laboratories in eMerge Project.” European Journal of Engineering Education 32, no. 1 (2007): 57-72.
This research compares opportunities for conducting electrical engineering experiments on the internet as opposed to the physical laboratory.  The test comprised 84 students at Bordeaux University 1, where half performed experiments in a laboratory, and the other half via the internet.  Students' opinions about acceptance, usability, learning effects, usefulness in studying and vocational terms were measured through questionnaires.  Results show that experiments conducted on the internet are just as successful as those conducted in an actual laboratory.  It was suggested that webcams be used to allow for more direct contact with experiments.  R. Leech


Pisciotta, Henry, Michael Dooris, James Frost, and Michael Halm. “Penn State's Visual Image User Study.” Portal: Libraries and the Academy 5, no. 1 (2005) 33-58.
The Visual Image User Study (VIUS) used multiple modes of assessment to examine the needs of digital image delivery at Penn State.  This article details components of the study related to academic users of pictures.  The authors document the market for digital image systems, critical factors that ensure users’ willingness to use an image delivery system, and user expectations for systems that are effective tools for teaching, learning, and managing collections. The development of two prototype image systems is also described.  The project team observed that success in image delivery requires a user-centered focus and the cooperation of multiple university units, including libraries. Thus, digital library development must operate in the wider context of other university initiatives.  T. Clayton