Evaluation of Essential Skills Online (ESO): A Utilization-Focused Approach to a Developmental Evaluation of a Workplace Literacy and Numeracy Resource

It appears your Web browser is not configured to display PDF files. Download adobe Acrobat or click here to download the PDF file.

Click here to download the PDF file.

Creator: 

Reynolds, Claire

Date: 

2022

Abstract: 

Essential Skills Online (ESO) was designed to address the current national shortage of online resources that provide adults seeking employment with practice in the skills of the Essential Skills Framework (~ 1996-2021). During ESO's development, an evaluation of its design and pedagogical content was commissioned by its developer. In response, pragmatic, collaborative, sociocultural research was undertaken using Patton's (2012) utilization-focused evaluation (U-FE) framework, within a developmental evaluation (DE) approach which, unlike traditional evaluation, focused on program development in process. This dissertation reports on the DE in a qualitative, multiphase case study which: 1) provided recommendations for the ongoing development of ESO; and 2) evaluated the contributions of a U-FE framework to ESO's DE. Two organizations participated in the iteration of this DE: an Indigenous-led educational organization and a non-Indigenous-led organization that supported recently incarcerated adults. In Phase 1, pre-implementation, a situation analysis was conducted, as the evaluator and stakeholders collaborated to design the evaluation, prepare ESO for use, and organize the administration of the pre-and post-test (Test of Workplace Essential Skills). Phase 2, the field evaluation, consisted of fieldwork and correspondence with the field participants. Due to the COVID-19 pandemic and other unforeseen events, a full U-FE could not be completed. Therefore, in Phase 3, interviews with experts in program evaluation, adult literacy, assessment, and online learning were conducted to assess the feasibility of Phase 2's recommendations and future directions. Data (e.g., from field notes, semi-structured interviews, journal entries, focus groups, emails, documents) were analysed through coding, narrative analysis, and quantitization. The findings concerning ESO, which were corroborated by the industry experts, included: 1) pedagogical and design improvements, and 2) issues concerning the pre-and post-test. The industry experts also emphasized the need for increased collaboration with ESO's users to enhance its usefulness. Although some evaluators would be hesitant to use U-FE within a DE given the time constraints placed on this evaluation, the results reported here suggest its value. The U-FE framework provided an effective, holistic, structure for DE. Further, adaptability and strong communication were essential evaluator characteristics given the emergent nature of DE. Implications and future direction are discussed.

Subject: 

Education - Adult and Continuing
Reading
Education - Technology

Language: 

English

Publisher: 

Carleton University

Contributor: 

resource developer: 
Cummins Patrick

Thesis Degree Name: 

Doctor of Philosophy: 
Ph.D.

Thesis Degree Level: 

Doctoral

Thesis Degree Discipline: 

Applied Linguistics and Discourse Studies

Parent Collection: 

Theses and Dissertations

Items in CURVE are protected by copyright, with all rights reserved, unless otherwise indicated. They are made available with permission from the author(s).