Virtual Reality for Education

The second phase involved prototype development.  Informed by general CVLE design principles and the literature on engagement, a prototype incorporating design elements hypothesized to influence social presence was created. This was an iterative process of design, testing, and refinement, prioritizing user needs.

The third phase focused on prototype evaluation.  Given the limited research on room layout and social presence, participant numbers were determined based on a combination of methodological best practices and practical constraints.  Data collection methods in this phase included a background questionnaire and interviews.  While most participants were postgraduates due to the timing of the research coinciding with an undergraduate break, the study also included other participants.  The timing of the project presented some challenges regarding participant availability, as many postgraduate students were focused on their final projects.  This, coupled with the general demand for research participants within the academic environment, required careful time management.  A more detailed discussion of these limitations will follow.

The collected data provided insights into the prototype's strengths and weaknesses, specifically how design choices influenced the perception of social presence.  This feedback then informed further iterations of the prototype, creating a cyclical process of design, evaluation, and refinement central to the case study methodology. This iterative approach facilitated a deeper understanding of the complex interplay between design, technology, and user experience.

Prototype and Formative Testing:

In this phase, I moved from ideation to tangible prototypes, ensuring they were ready for user testing. My approach involved a blend of efficient design and rigorous testing, adapting as challenges arose.

Prototyping:

My prototyping process took a slightly different path than the traditional wireframe-to-high-fidelity progression. Given the project's focus on virtual environments, I prioritized creating functional prototypes that users could experience.

For the FrameVR prototype, I focused on meeting the requirements derived from the primary research. Specifically, P3 had mentioned the importance of a whiteboard, which I knew was also present in the meeting room that inspired the design. To expedite the process, I explored existing templates within FrameVR, carefully evaluating them against my needs. The "Neo Office" template proved to be the best fit, offering the necessary elements and resembling the design school environment I aimed to replicate. This unconventional approach allowed me to leverage available resources and streamline development.

Formative Testing:

Before launching full-scale user testing, I conducted a pilot test with a participant. This step was crucial for identifying potential issues that might arise during the actual testing sessions. It wasn't about collecting data at this stage; it was about refining my process.

The pilot test was incredibly valuable. It brought to light several key areas for improvement. For instance, I realized that the original discussion topic (Artificial Intelligence) might be challenging for participants without a strong technical background. To ensure a smoother experience, I pivoted to a more relatable topic: movies and TV shows. I also put a plan in place to have prompts to make the conversation smooth.

A significant technical challenge emerged during the pilot test: echoes caused by having multiple laptops with microphones in close proximity. Unfortunately, I couldn't secure appropriately spaced rooms or the VR headsets in time, so I made the tough call to shift the user testing to a remote format. While this wasn't the original plan, it opened up new possibilities. Remote testing made participation more appealing, addressing recruitment challenges.

I also anticipated that some participants might need help navigating the FrameVR interface or engaging in conversation. To address this, I decided to allocate buffer time to each session to ensure participants were comfortable with the technology before diving into the discussion and I would provide short prompts to guide the conversation. FrameVR's intuitive design also helped minimize potential technical hurdles.

These decisions were also supported by the literature review and primary research.

Throughout the prototyping and formative testing process, I was guided by user-centered design and accessibility principles. My focus remained on creating a positive and inclusive experience for the participants, ensuring their needs and comfort were prioritized.

Research Approach:

This study employed a mixed-methods approach, combining both qualitative and quantitative data collection and analysis, to gain a comprehensive understanding of the research questions. The research was conducted in two main phases: primary research using interviews and user testing with the prototypes.

Primary Research:

The primary research phase involved conducting interviews with nine participants. These interviews aimed to explore the influence of classroom environments, specifically layouts, on social presence and engagement within virtual reality settings. This phase was crucial for gathering in-depth qualitative data to inform the design of the prototypes. The insights gained from the interviews helped to identify key elements for the FrameVR prototype, such as the inclusion of a whiteboard and the preference for a cluster layout for collaborative activities.

User Testing:

The user testing phase focused on evaluating the two prototypes: Microsoft Teams and FrameVR. Participants engaged with each prototype, and their experiences were measured through a combination of quantitative and qualitative methods.

• Quantitative Measures: Participants completed a questionnaire after interacting with each prototype, which included questions using a 1-to-5 scale. This questionnaire, adapted from the "Measure of Social Presence" (Harms & Biocca, 2004), assessed various aspects of social presence, including co-presence, attentional allocation, perceived message understanding, perceived affective understanding, perceived emotional interdependence, and perceived behavioral interdependence.

• Qualitative Measures: The questionnaire also included two open-ended questions designed to gather qualitative feedback and capture participants' experiences in their own words. These questions aimed to gather quotes from the users for the results.

• Background Questionnaire: To provide context for the user testing data, a background questionnaire was used to collect demographic information from participants, including gender and education level.

Prototype Testing Design:

The study employed a within-subjects design for the user testing. This design was chosen to minimize the impact of individual differences between participants and to maximize statistical power with a smaller sample size. To mitigate potential knowledge transfer effects, the order in which participants interacted with the prototypes was alternated.

Participants:

The number of participants for the user testing was initially calculated using the GPower 3.1 tool, which indicated a need for thirty-four participants to achieve a 95% confidence level. However, due to practical constraints, such as the availability of postgraduate students, only twenty-one participants were recruited for the user testing.

Independent Variable:

The independent variable in this study was the type of platform used for interaction: FrameVR and Microsoft Teams.  

• FrameVR: This prototype offered a virtual environment designed to simulate a real-life classroom, with features like a whiteboard and the ability to move around.

• Microsoft Teams: This prototype represented a typical online meeting platform, providing a contrast to the immersive nature of VR.

Why other questionnaires were not used:

Initially, the Presence Questionnaire by Witmer and Singer and the Slater-Usoh-Steed questionnaire were considered to measure social presence and engagement. However, these questionnaires were deemed less suitable for the study's specific requirements compared to the "Measure of Social Presence."

Results:

This section presents the findings from the user testing phase, combining demographic information, observational data, and quantitative results from the social presence questionnaire.

Demographics:

A brief background questionnaire was used to gather demographic information from the 21 participants. The participant group consisted of 8 males and 13 females. The majority (95%) of participants were postgraduate students currently pursuing their studies, with the remaining 5% having completed their undergraduate studies. 71% of the participants had used VR before, while 29% had not. Interestingly, the same percentages (71% and 29%) applied to the participants' previous use of VR for educational purposes.

Observations:

Throughout the user testing sessions, several key observations were noted:

• Group Dynamics: In the first session with four participants, two individuals appeared to lead the conversation, while the others were quieter. This led to assumptions about extroverted and introverted tendencies, though other factors like gender dynamics were also considered but ultimately remained speculative.

• Technical Issues: The first session also encountered technical difficulties with echoes when participants used both Teams and FrameVR simultaneously.

• Whiteboard Interaction: The whiteboard feature in FrameVR saw varied usage. In one group, a participant interacted with the whiteboard and erased its content, while in other groups, it was actively used as part of the discussion.

(Insert Whiteboard Image from pg 20, Fig 5)

• Introverted Behavior: The second group displayed more introverted behavior, with participants initially hesitant to speak. However, they became more comfortable as the session progressed.

• Technical Troubleshooting: One participant in the second group experienced microphone issues in FrameVR, which was quickly resolved by adjusting browser permissions.

• VR Exploration: In the third group, a participant briefly got lost while exploring the FrameVR environment, highlighting the need for intuitive navigation.

• Enthusiasm for VR: There was a general excitement in some groups towards interacting with the FrameVR prototype, particularly the whiteboard feature.

• Comfort Level: The fourth group's quick comfort level was attributed to potential prior familiarity among participants.

• Conversation Struggles: The fifth group, characterized as slightly introverted, initially struggled to maintain conversation flow in Teams but improved over time.

Social Presence:

After interacting with each prototype, participants completed the "Measure of Social Presence" questionnaire. The results were analyzed using SPSS to determine any significant differences in social presence between the two platforms.

• Co-Presence: A significant difference was found in co-presence, with the Microsoft Teams prototype scoring higher than the FrameVR prototype (Teams Mean = 4.5476, FrameVR Mean = 3.7698, p = 0.003).

• Attentional Allocation: No significant difference was observed between the two prototypes in attentional allocation.

• Perceived Message Understanding: Similarly, no significant difference was found in perceived message understanding between the prototypes.

• Perceived Affective Understanding: A significant difference was observed, with the Microsoft Teams prototype again scoring slightly higher (Teams Mean = 3.2857, FrameVR Mean = 3.0238, p = 0.015).

• Perceived Emotional Interdependence: The Microsoft Teams prototype showed a significantly higher score in perceived emotional interdependence compared to the FrameVR prototype (Teams Mean = 3.8250, FrameVR Mean = 3.0667, p = 0.002).

• Perceived Behavioral Interdependence: A significant difference was found, with the Microsoft Teams prototype scoring higher (Teams Mean = 3.8730, FrameVR Mean = 3.3254, p = 0.018).

Overall Rating:

In the exit interview, participants provided an overall rating of their social presence and engagement in each prototype.

Engagement Measurement:

Due to time constraints and the literature review's finding that social presence influences engagement, a separate questionnaire for engagement was not used. Instead, a direct rating question was included in the exit interview to assess participants' engagement with each prototype.

Insights and Future Directions:

This study explored the influence of classroom layout on social presence and engagement in virtual reality learning environments. While the initial aim was to determine if room layout influenced social presence and engagement, the focus shifted slightly to the cluster layout due to its preference among participants for collaborative tasks.  

The literature review confirmed the potential of VR to enhance social presence but found a gap in research on the specific impact of room layout within VR. User interviews revealed a preference for the cluster layout in group work scenarios, though the interviews did not conclusively establish how this layout affects social presence and engagement.  

Unexpectedly, user testing showed that participants favored the Microsoft Teams prototype over the FrameVR prototype, contradicting both the literature review and primary research findings. This discrepancy, along with limitations in the study, suggests a need for further investigation. The use of the whiteboard in the FrameVR prototype was observed to positively influence engagement in three out of five groups.  

The study faced limitations, primarily the remote testing environment and the use of laptops instead of VR headsets, which may have affected the results. The sample size was also smaller than the initial calculation, potentially impacting the study's statistical power.  

Despite these limitations, the research highlights the importance of collaborative and interactive elements in both online meeting platforms and VR environments.  

Recommendations for VR Classroom Design:

• The design of a VR classroom should align with its purpose; an auditorium layout suits lectures, while a cluster layout is better for group work.

• Interactive elements, such as whiteboards, can enhance engagement but should be balanced to avoid distractions.

• VR offers advantages as an alternative to in-person learning.

• Future research should explore methods for displaying facial expressions in VR without specialized equipment, as this may improve social presence.

Conclusion:

The study partially answered the research questions, but further research is needed to provide more conclusive findings.

Theoretical Implications:

• Using VR headsets might yield different results than using FrameVR on a laptop.

• Investigating other visual cues, like facial expressions, could expand the study's foundation.

• Advances in VR technology may alter the current findings.

• Research on improving the comfort and accessibility of VR headsets is warranted.

UX Design Practitioner Implications:

Further studies could explore alternative methods to enhance social presence in XR technologies.

Limitations and Future Research:

• The study's participant pool was primarily postgraduate students, which may limit generalizability.

• The absence of VR headsets in the user testing is a significant limitation.

• Future research should aim for a larger sample size (34 participants) when comparing two prototypes.

References:

• Barreda-Ángeles, M., Horneber, S., & Hartmann, T. (2023). Easily applicable social virtual reality and social presence in online higher education during the covid-19 pandemic: A qualitative study. Computers & Education: X Reality, 2, 100024. https://doi.org/10.1016/J.CEXR.2023.100024

• Dzardanova, E. (n.d.). Virtual Reality in Education: The Impact of High-Fidelity Avatars on the Learning Experience. https://www.researchgate.net/publication/365604762

• Hanaysha, J. R., Shriedeh, F. B., & In'airat, M. (2023). Impact of classroom environment, teacher competency, information and communication technology resources, and university facilities on student engagement and academic performance. International Journal of Information Management Data Insights, 3(2), 100188. https://doi.org/10.1016/J.IJIMEI.2023.100188

• Harms, C., & Biocca, & A. (2004). Internal consistency and reliability of the networked minds social presence measure.

• Harrison, V., Kemp, R., Brace, N., & Snelgar, R. (2020). SPSS for psychologists (7th ed.). MACMILLAN.

• Lege, R. (2024). A social presence benchmark framework for extended reality (XR) technologies. Computers & Education: X Reality, 4, 100062. https://doi.org/10.1016/J.CEXR.2024.100062