Evaluation of Essential Skills Online (ESO): A Utilization-Focused Approach to a Developmental Evaluation of a Workplace Literacy and Numeracy Resource

Essential Skills Online (ESO) was designed to address the current national shortage of online resources that provide adults seeking employment with practice in the skills of the Essential Skills Framework (~ 1996-2021). During ESO's development, an evaluation of its design and pedagogical content was commissioned by its developer. In response, pragmatic, collaborative, sociocultural research was undertaken using Patton's (2012) utilization-focused evaluation (U-FE) framework, within a developmental evaluation (DE) approach which, unlike traditional evaluation, focused on program development in process. This dissertation reports on the DE in a qualitative, multiphase case study which: 1) provided recommendations for the ongoing development of ESO; and 2) evaluated the contributions of a U-FE framework to ESO's DE. Two organizations participated in the iteration of this DE: an Indigenous-led educational organization and a non-Indigenous-led organization that supported recently incarcerated adults. In Phase 1, pre-implementation, a situation analysis was conducted, as the evaluator and stakeholders collaborated to design the evaluation, prepare ESO for use, and organize the administration of the pre-and post-test (Test of Workplace Essential Skills). Phase 2, the field evaluation, consisted of fieldwork and correspondence with the field participants. Due to the COVID-19 pandemic and other unforeseen events, a full U-FE could not be completed. Therefore, in Phase 3, interviews with experts in program evaluation, adult literacy, assessment, and online learning were conducted to assess the feasibility of Phase 2's recommendations and future directions. Data (e.g., from field notes, semi-structured interviews, journal entries, focus groups, emails, documents) were analysed through coding, narrative analysis, and quantitization. The findings concerning ESO, which were corroborated by the industry experts, included: 1) pedagogical and design improvements, and 2) issues concerning the pre-and post-test. The industry experts also emphasized the need for increased collaboration with ESO's users to enhance its usefulness. Although some evaluators would be hesitant to use U-FE within a DE given the time constraints placed on this evaluation, the results reported here suggest its value. The U-FE framework provided an effective, holistic, structure for DE. Further, adaptability and strong communication were essential evaluator characteristics given the emergent nature of DE. Implications and future direction are discussed.