Assessment of Higher Order Thinking Skills in Virtual Learning Environments

This research addresses the assessment of Higher-Order Thinking Skills (HOTS), such as metacognition, reflection, and problem-solving, in Virtual Learning Environments (VLEs). We particularly focus on the use of process metrics, their combinations, and various analysis methods that allow VLE platforms to perform automated HOTS assessments. Traditional learning assessments rely mostly on outputs and are not suitable for HOTS assessment that requires process observation. As a result, it is a challenge for learners and educators to identify the areas of weakness for customized help when it comes to HOTS. Our objective is to understand the requirements of a VLE-based HOTS assessment framework and explore what process metrics can be used and how they can be analyzed to offer insight into learners' HOTS development. To achieve the above objective, a series of four studies were performed within this research. Study 1 was an initial exploratory investigation that suggested 3D VLEs as a possible HOTS fostering platform though associating their unique affordances to the requirements of common learning theories. This study was a motivational activity and initiated our research. Study 2 was performed on a text-based VLE and provided new insight into how aggregated process metrics can be used to represent student attention and participation, which are linked to HOTS. Study 3 focused on identifying 3D VLE process metrics and their alignment with HOTS components. Study 3 results suggested that the rich data coming from a 3D VLE, and the combination of process metrics as small groups (motifs) and time series, can offer more insights about HOTS. Finally, Study 4 employed motifs and time series-based similarity analysis on process metrics for performing HOTS assessment during learning tasks in a 3D VLE. Study 4 investigated task compatibility with four different similarity indexes, and findings suggested employing different similarity indexes depending on the learning tasks. Overall, the studies conducted within the scope of this research provided supporting evidence of the possibility of automated HOTS assessment on VLEs using process metrics. They showed the value of motifs (small yet meaningful series of process metrics) as a measure for HOTS. However, they suggested that there is no single method, and different learning tasks might use different data analysis strategies.