Evaluating TELT


A note for the reader

The TELT Evaluation Framework, developed in 2009 through to 2011, is designed to undergo iterative cyclical refinement and ongoing development, based on the results of the sub-layer evaluations themselves and an ever-changing staff, student and application landscape in which it is applied.

Therefore, readers are reminded that the nature, validity and applicability of the reviewed literature, the proposed processes, the suggested composition of the survey instruments, and the construction of the sub-layers are all likely to change in the future and undergo refinement and improvement in order to adapt to the evolving social, technological and institutional milieu.

Readers are certainly encouraged to use these document as the basis for understanding the aims, goals and structure of the Framework, but are invited to refer to the refinement documents for the most up-to-date information on Framework refinement, latest research, and the latest survey instruments and evaluation results.

We welcome questions and suggestions – please contact us for any clarification of the document or subsequent refinements.

TELT Evaluation Framework Vision

To devise a "living" evaluation framework that affords an iterative refinement process to underpin:
  1. All TELT technology selections and developments
  2. Efficient tracking of the trajectory of success (and failure) of educational technologies used across faculties and over time
  3. Enhancement of the divergent L & T approaches and practices in online learning at UNSW

evaluation-framework-img1.png


TELT Evaluation Framework Frequently Asked Questions

Read about, or ask, the most Frequently Asked Questions (FAQ) for the TELT Evaluation Framework.

TELT Evaluation Framework Downloads

Please access the comprehensive documents below to learn more about the TELT Evaluation Framework:

CORE DOCUMENTS:

pdficon_large.gifEvaluation of the TELT Platform - Essential Elements and Methodologies
Quinton, S., Pachman, M., & He, R. (2010). Evaluation of the TELT Platform - Essential Elements and Methodologies. Sydney, Australia: University of New South Wales, Learning and Teaching @ UNSW

pdficon_large.gifTELT Evaluation - Stage 3 - A Multi-Definitional Approach to Educational Technology Evaluation at UNSW / Stage 3 Evaluation Criteria
Quinton, S., Logunov, A. (2010). TELT Evaluation - A Multi-Definitional Approach to Educational Technology Evaluation at UNSW; Stage 3 Evaluation Criteria. Sydney, Australia: University of New South Wales, Learning and Teaching @ UNSW

pdficon_large.gifTELT Faculty Policy Plans and Strategies Construction Kit
Quinton, S. (2010). TELT Faculty Policy Plans and Strategies Construction Kit. Sydney, Australia: University of New South Wales, Learning and Teaching @ UNSW.

PAPERS AND REPORTS:

pdficon_large.gifTELT Evaluation Framework - Refinement of TELT Survey Instrument (1st iteration)
Pachman, M., Logunov, A., Quinton, S. (2010). Refinement of the TELT Survey Instrument. Sydney, Australia: University of New South Wales, Learning and Teaching @ UNSW.

pdficon_large.gifTELT Evaluation Framework - Refinement of TELT Survey Instrument (2nd iteration)
Pachman, M., Logunov, A., Quinton, S. (2011). Refinement of the TELT Survey Instrument (Stage 3, iteration 2). Sydney, Australia: University of New South Wales, Learning and Teaching @ UNSW.

Pilot Evaluation Results:


pdficon_large.gifBrief summary of evaluation pilots (Elluminate), S2, 2010
Quinton, S., Pachman, M., & Logunov, A. (2011). Brief summary of evaluation pilots (Elluminate), S2, 2010. Sydney, Australia: University of New South Wales, Learning and Teaching @ UNSW.

pdficon_large.gifBrief summary of evaluation pilots (Moodle), S2, 2010
Quinton, S., Pachman, M., & Logunov, A. (2011). Brief summary of evaluation pilots (Moodle), S2, 2010. Sydney, Australia: University of New South Wales, Learning and Teaching @ UNSW.

pdficon_large.gifBrief summary of evaluation pilots (Moodle + Wikispaces), S1, 2010
Quinton, S., Pachman, M., & Logunov, A. (2010). Brief summary of evaluation pilots (Moodle + Wikispaces), S1, 2010. Sydney, Australia: University of New South Wales, Learning and Teaching @ UNSW.



TELT Evaluation Framework


Why the need for a TELT Evaluation Framework?


Rationale and argument for devising a generic evaluation framework:
  1. Lack of availability of a widely recognised framework
  2. The need to avoid the many bias inherent in most frameworks
  3. The need to position L&T at the top of a hierarchy of evaluation criteria consistent with the TELT vision
  4. Identify the principles and models on which the framework is to be established

An extensive literature search conducted over several months has confirmed the difficulty in identifying a comprehensive, generic approach and methodologies for facilitating the selection and evaluation of the effectiveness of educational technology applications and solutions that focus exclusively on serving the needs of learning and teaching.
Pedagogical concerns over the actual learning effectiveness of various eLearning applications have increased alongside the meteoric rise in eLearning popularity and implementation. The chief criticism is that the increasingly feature-rich technological platforms are not properly developed for an appropriate learning context, often improperly integrated and under-utilized, and in many situations simply become an expensive means of indexing ordinary content, adding little educational value.
The TELT Evaluation Framework is thus being developed as a comprehensive, flexible and continuous approach to assess and evaluate the suitability of different educational technologies for use within the TELT Platform, as guided by informed Learning and Teaching practice and research.

What is the TELT Evaluation Framework?


For UNSW, the priority for its TELT evaluation strategy is to devise a theoretical framework from which to assess the suitability of selected educational technologies for inclusion in the TELT platform. The Evaluation Framework is an extensive, multi-faceted tool for providing an evidence-based assessment of learning and teaching technologies that may be used by the University.
Once a technology has passed through a three stage evaluation process and been approved for general use, a perpetual, iterative process of evaluation is initiated to assess the educational effectiveness of that technology in terms of the depth of student comprehension and the measurable improvements to the teaching and learning processes.

What the TELT Evaluation Framework is not


The Framework is designed to be underpinned by educational and eLearning theory and to learn from and to improve on the results and experiences of other eLearning evaluations, with the aim of improving the TELT environment within the University.
We understand that some staff may misunderstand the Evaluation, perceiving it as an assessment of their skills, teaching practices, or ability. The Evaluation is focused on understanding and improving eLearning technologies now and for the future, and is thus very critical of the usability, true learning effectiveness, and integration with student learning environments.
It is NOT in any way intended to be an assessment of the performance or expertise of individual teaching staff.

Framework Structure


Framework Structure Overview


The Framework itself incorporates continual effectiveness re-assessment of the evaluation processes: how well they identify suitable technology applications and tools for inclusion in the platform, and to what extent they improve technology supported learning and teaching. Thus, the Evaluation is conducted over the entire life-cycle of the technologies, delineated across three stages:
  1. Stage 1: An initial vendor assessment: to determine the suitability, stability and viability of the prospective vendors/open-source solutions.
  2. Stage 2: A pre-pilot evaluation score-sheet: to determine the suitability of technology for full pilot evaluation testing in Stage 3. Stage 2 assesses the features and functionality of the application from educational and technical perspectives.
  3. Stage 3: An on-going full-pilot evaluation: to determine suitability for the TELT Platform, as measured by the educational value of the technology application.

evaluation-framework-img2.png

Categories of TELT Applications


To permit relevance to different types of TELT applications (categorised for example as large-scale LMS, medium-scale virtual classroom or collaborative suites, and small-scale special purpose software), evaluation methodologies are to be identified to address the specific requirements of each category.

Categories of TELT Applications that have been identified, include:
  1. LMS - Learning Management System
  2. CMC - Computer-Mediated Communication Systems
  3. PDS - Plagiarism Detection Systems
  4. CAA - Computer Assisted Assessment Tools
  5. EMS - Educational Multimedia Software
  6. LC - Lecture Capture Systems
  7. Smaller-scale Special Purpose Software

Evaluation Framework - Stage 3


Stage 3 begins an ongoing L&T focused evaluative sub-cycle that aims to facilitate a culture of evidence-based excellence at the faculty and institutional levels to improve UNSW technology supported L&T practices.
Stage 3 determines the suitability of the TELT technology for inclusion in the TELT Platform as a whole. Stage 3 consists of three layers:
  1. Pedagogy Layer
  2. Technical Layer
  3. Business Layer

evaluation-framework-img3.png

  1. The Pedagogy Layer is emphasized first and foremost as it is directly informed by the UNSW Graduate Attributes, Learning and Teaching Principles and the 16 Guidelines for Learning and Teaching Practices.
    The evaluation criteria specified for this layer are also guided by the teaching approaches and models applied in faculties and schools.


    evaluation-framework-img4.png

    The pedagogical criteria for evaluating a TELT technology are further divided into three sub-layers:
    1. The educational value and usefulness of the selected application
    2. The learning effectiveness of the TELT technologies
    3. The learning environments that use TELT technologies

evaluation-framework-img5-1.png
  1. The Pedagogy Layer
  2. The Business Layer reflects the formalised processes of organisational change at UNSW that are informed first by pedagogical and then by technological factors.
  3. The Technical Layer reflects the need to account for the technical requirements, standards and protocols established at UNSW that are informed first by the pedagogical and then the business process criteria.

Generic Usability Evaluation


In conjunction with the overarching concerns over evaluating the actual pedagogical (learning) effectiveness of integrated eLearning environments as assessed through various educational theories and methodologies, it is also important to begin considering usability and usefulness as two fundamental factors that create the foundation for eLearning applications to be both conducive to effective learning and supportive of positive pedagogical strategies, thus having education value. Prior to undertaking a comprehensive assessment of the learning effectiveness of an eLearning application, a preliminary examination should consider the application’s usability, usefulness, and educational value as a function of usability and usefulness within a learning and teaching context.
If the usability of a particular technology application is a hindrance to the users’ learning and teaching processes, and if it is not evaluated for separately, the entire evaluation could present a biased impression of the overall educational value and subsequent learning effectiveness of the application.
On a practical level, the six-stage heuristic life-cycle forms an evidence-based starting point for the creation of tailored and thematically grouped questions revolving around the entire application usability experience, for surveys and other instruments that would physically empower the Evaluation. The questions would assess the most common usability, usefulness and educational value attributes, and typical usability problems, as identified through our extensive evaluation literature review, which builds on the review undertaken in the supporting document titled : “Evaluation of the TELT platform: Essential elements and methodologies”.
It is the nexus of various ad-hoc usability elements and assessable attributes identified through research that creates the ephemeral, intuitive "look and feel" of user experience and satisfaction that has been so inherently difficult to isolate and evaluate.
The Generic Usability Evaluation relies heavily on educational and technology literature -- please find literature references and supporting documentation for each heuristic usability element, in the comprehensive reports available for download.

evaluation-framework-usability-lifecycle.png

Other links