You're a middle school student working in a small group in your class. Your group members are arguing over who has the best idea for completing the group's task. You try to bring up your idea, but others talk over you. You start to feel left out, and the direction of the discussion feels aimless and confusing. You can feel yourself withdrawing from the task.

This scenario represents a common and major challenge in implementing effective collaborative learning techniques in kindergarten through 12th-grade (K-12) classrooms. On one side is the research on how people learn that has converged toward a perspective of learning as fundamentally collaborative and social [1,2,3]. On the other are the classroom challenges teachers face when executing these collaborative learning techniques. They must monitor group progress on time sensitive tasks, provide guidance on learning activities and goals, support students in knowledge-building conversations, and differentiate resources across six to eight small groups simultaneously, depending upon their class size [4, 5]. Teachers also face the impossible expectation of being omnipresent in all groups to catch those precious "teachable moments" occurring in individual groups (for example, eureka moments that generate ideas).

  Reframing the Role of AI in Education

How can teachers and students overcome these challenges to create effective socio-collaborative learning environments? Solving this question is the driving motivation behind the National Science Foundation (NSF) AI Institute for Student-AI Teaming (iSAT). iSAT's researchers believe artificial intelligence (AI) is part of the solution. Currently, AI functions more as an intelligent tool supporting personalized learning—think learning management systems, student metric analyses, digital textbooks, and so on. iSAT researchers are working to reframe this role into one in which AI is viewed as a social, collaborative partner.

Let's revisit the earlier classroom scenario, this time adding in AI as a collaborative partner. This "AI Partner" knows the lesson, can hear and understand the students' conversations, identify who is speaking, and report useful learning analytics and needs of the various groups to the teacher. The Partner is also trained to encourage students to practice collaborative learning skills and to adhere to inclusive community agreements set by the students and teacher.

When the Partner hears the students in our hypothetical group arguing, for example, it can remind students of their community agreements or suggest a helpful collaborative learning technique. When the AI Partner detects a student hasn't had a chance to contribute, it can amplify their voice by encouraging the student to share an idea they expressed earlier that might push the task forward.

  It Takes a Village to Raise an AI Partner

Several components need to work together to create this future AI Partner. The Partner will need to:

• have multimodal processing, natural language understanding, and knowledge representation to autonomously monitor and support the unfolding collaborative learning processes at multiple levels;
• apply theoretical frameworks, innovative interaction paradigms, and novel architectures for orchestrating effective student and teacher interactions; and
• understand equitable and ethical approaches to monitoring student interactions to engage with and amplify diverse voices.

Bringing these components together to build something with the potential to change the American K-12 classroom requires the very thing iSAT set out to create—an effective socio-collaborative learning environment. To create this environment within iSAT, the Institute brings together experts from diverse research backgrounds across nine institutions with K-12 community partners. iSAT also provides its student members with a platform to express and explore their ideas in impactful ways, and iSAT members meet frequently to collaboratively solve our biggest challenges. These values-centered approaches to foundational research—practicing collaborative problem solving, coordinating diverse perspectives, and encouraging student-driven design—are reflected throughout iSAT's work.

Our researchers and community partners work across three interconnected research strands (see Figure 1). Strand 3 researchers investigate new approaches to broadening participation in who envisions, creates, critiques, and applies AI learning technologies to ensure these technologies reflect the needs, interests, and values of diverse community stakeholders. Strand 2 makes foundational theoretical and technological advances to develop the science of student-AI teaming. Strand 1 researchers work toward the AI advances needed to understand and facilitate collaborative learning conversations. Finally, a team of institute-wide researchers oversees iSAT's data collection and analysis, technical architecture, and the AI Partner designs, implementation, and testing.

  Student and Teacher-Driven Design

Bringing an AI Partner who can listen to and understand student speech into K-12 classrooms raises important questions around ethics and responsible innovation. What data does the AI Partner collect, and what does it do with it? Who can see this data? How much control will students have over this data? How can we avoid doing harm with the data? What should the AI Partner be able to do? What shouldn't it do?

To address these questions, Strand 3's team of learning sciences, equity, and ethics experts created three subgroups, called "themes," dedicated to different aspects of co-designing with students and teachers. The Sensor Immersion Unit theme members work with CU Boulder's SchoolWide Labs team to co-design AI Partner embedded curriculum units with teachers from two school districts in Colorado and implement these units in classrooms across the two school districts. This curriculum uses paired programming and group work to drive student collaboration to better understand how sensors perceive the world. The Games Unit theme members also employed co-design to create a curriculum unit designed to push students to think about how video games can become biased and the potential harm this can create.

The researchers within both themes work with teachers and students to inform the Institute's data collection and analysis work along with the functions of the AI Partner. Researchers and teachers co-design and implement the curricula used to gather consented student multimodal data across Denver Public Schools and St. Vrain Valley Schools District in Colorado, which is protected under institutional review board (IRB) approval.

Strand 3's third theme, Learning Futures Workshops (LFW), is dedicated to co-designing with high school aged young people from California and Colorado through a series of workshops with the goal of incorporating the visions these young people have for ideal classroom collaboration and an intelligent agent into the AI Partner's design. The students constructed community agreements—norms for the group on how they would work together and hold each other accountable—shifting the discourse from "disciplining" youth to the youth thinking about accountability for their agreed upon norms. From this work, they designed a prototype for displaying when these norms are violated, which iSAT's researchers are incorporating into their own design prototype.

  Facilitating Collaborative Problem Solving with Human-Centered Design

Strand 3's work with teachers and students informs the work of Strand 2's team science experts throughout their three themes. The Framework & Measures theme works to identify and measure collaborative problem solving (CPS) skills. The underlying emphasis of this work focuses on the theoretical fundamentals of CPS including how communication is coded, students' perspectives on collaborative problem solving, and the dynamics of collaboration. Theme team members also work on ways to tie dynamic speech flow measures based on recurrence analysis and mutual information to human-and automatic-coded CPS skills.

Strand 2 created its Collaboration Processes & Orchestration theme to identify non-verbal behaviors including eye gaze (joint attention, looking at tools/computer, among others), gestures and body language (pointing, leaning, manipulating objects, or nodding), and emotion (a smile or frown) that can be applied to video data of students collaborating on the Sensor Immersion unit. Their work found the coding scheme works across different sets of data and contexts, indicating that it is robust in identifying non-verbal communication during collaborative learning in groups [7].

Strand 2's third theme, UX Design & Multimodal Modeling, investigates multimodal measurements to inform the design iterations of different AI partners. Theme members developed the iSAT Lab at the University of Colorado Boulder and support the sister lab run by Strand 1 researchers at Colorado State University. The team collects data on collaboration within these labs to model collaborative behaviors and correspondingly implement these models as part of the future AI Partner. The team implemented an initial study in Fall 2022 consisting of 30 groups total (15 groups of two and 15 groups of three participants) engaging in five collaborative tasks. The team collected approximately 2.5 hours of data for each group in multiple forms including Kinect video, regular video and audio, screen recording, eye tracking with Tobii glasses, and state and trait surveys.

iSAT and the Center for Human, Artificial Intelligence, and Robot Teaming (CHART) at Arizona State University held a Data Jam in late February of this year. Working with the iSAT Lab data, attendees analyzed and presented their perspectives on the collaboration analytics that can be used to drive the development of the AI Partner.

  Creating an Agent to Understand and Facilitate Collaborations

While Strands 2 and 3 work on identifying what the AI Partner should hear, see, understand, and do when working with teams of students, Strand 1 is busy developing new advances in foundational AI technologies to create a Partner capable of processing and executing the correct actions for these inputs. Strand 1 members divided their team into three themes that tackle the different components outlined by the Strands 2 and 3 teams.

Strand 1's Content Analysis & Dialogue Management theme is working on the component that will enable the AI Partner to understand what students are working on and how students work together to understand and complete tasks. To this end, the team has annotated students' worksheets from the Sensor Immersion unit with Abstract Meaning Representation (AMR), OnTask, and Academic Productive Talk (APT) and trained models for each layer while also analyzing how automatic speech recognition (ASR) errors impact each of these downstream tasks. The team developed the annotation scheme design of the Dependency Dialog Acts (DDA), which capture the speaker intention and the threading structure in student conversations. The team is working on annotating and utilizing this framework to provide linguistic insight into high-level content analysis tasks, such as CPS, APT, and equitable conversation.

The Speech Processing and Diarization theme members are focused on helping our AI Partner hear and understand students when they talk (speech processing). They are also ensuring the AI Partner will be able to identify who is speaking and when (diarization). They established a corpus of transcribed data for benchmarking, training, and fine-tuning speaker verification models on child speech. This includes transcripts from the Sensor Immersion unit and close-talking microphone recordings from existing corpora. The team is also training an interruption detection model. To help the Partner identify who is speaking and when, the team is working on person re-identification applied to school classroom environments, including the application of a state-of-the-art person re-ID system to a challenging real-world classroom dataset.

---

Researchers and teachers co-design and implement the curricula used to gather consented student multimodal data.

---

Students and teachers establish common ground when interacting with one another through both behavioral and verbal cues, as well as prior goals, expectations, and beliefs. The Situated Grounding theme team is tasked with identifying this common ground through discourse and gesture. To accomplish this, the team finalized the initial guidelines and is working on annotating targeted datasets, including EggNOG, a biological information database; the Fibonacci weights experiments out of our CU Boulder and Colorado State University (CSU) labs; and Sensor Immersion both in the lab and classrooms. Brandeis University iSAT members are adjudicating the original Gesture Abstract Meaning Representation (GAMR) annotations on data from EggNOG. To help our AI Partner understand when speakers are referring back to previously expressed thoughts or ideas later on in a conversation, the team is working with student annotators on the co-reference multi-sentence AMR annotation across the adjudicated EggNOG speech and gesture AMR. Lab data collection is underway at CSU and Brandeis, supplemented by Strand 2 data from CU Boulder. This data is being annotated for gesture semantics and object grounding, which is being used to train novel gesture recognition and object grounding models for use on lab, and eventually, classroom data. The group is also working with Strand 2 on non-verbal behavior annotation in the classroom.

  Converging on the AI Partner Design

In the beginning of its third year, iSAT members converged on the design of the AI Partner—work that wouldn't have been possible without iSAT's students and early career scholars from all three strands coming together in iSAT's second year to design a set of metaphors to represent the roles of the AI Partner. This group came up with three metaphors for the AI Partner—a community builder, a co-pilot, and an augmenter—and reshaped how the greater Institute initially imagined the intelligent agent. From these roles emerged two new prototypes: CoBi, based on the community builder metaphor, and the Jigsaw Interactive Agent (JIA), based on the co-pilot metaphor.

Building effective collaborative communities. CoBi's raison d'être is to help students build strong collaborative relationships that will enable them to dive deep into their classwork and create rich learning experiences together. Its design is based on a visualization that emerged from the students who participated in last spring's Learning Futures Workshop. The student-imagined display consist of light rods arranged in a row. Each rod is a different color, and each color represents a violation of the community agreements created by students and a teacher before embarking on group work. Whenever the device would detect language that would harm collaboration, it would shine the violation's representative color.

The CoBi team used this concept to create designs for how an AI-powered visualization could display how well a class did with upholding its community agreements. To avoid CoBi acting as a spy for the teacher—a role the Learning Futures Workshop students did not want for the AI Partner—the team created designs to identify and display indicators of these agreements at the classroom level.

CoBi team members are currently working on a roadmap to get the community builder into classrooms, which includes collaborating with the Institute's teacher advisory board to get their feedback and then to start testing version 1 in classrooms in Spring 2023.

---

iSAT researchers are working to reframe this role into one in which AI is viewed as a social, collaborative partner.

---

Facilitating brainstorming. The second design prototype, known as JIA, focuses on how the AI Partner will support small group task progress and collaboration by providing guidance and facilitation (from the co-pilot metaphor), while also distilling information from small group collaborations to teachers to support classroom orchestration (from the augmenter metaphor) [8]. The JIA team consists of cross-strand and institute-wide members who work together to analyze the limitations of breakout group conversations around the Jigsaw worksheet in lesson four of the Sensor Immersion curriculum unit, brainstorm potential improvements to the flow of lesson four, and conceptualize a set of lab experiments to further define the role of the interactive agent in the Jigsaw worksheet application.

  Conclusion: The Collaborative Future of AI in Education

Collaboration is the key to a successful future for AI in education. This is shown in not only the research on how people learn converging on learning as fundamentally collaborative and social, but also in the educational reform efforts that emphasize the need to engage students in collaborative knowledge-building activities and productive discourse to develop the problem-solving skills needed to tackle the challenges of the future [6].

For AI to thrive in this collaboration-centered educational landscape, researchers will need to reframe AI from a personal interactive study guide into an AI Partner focused on building communities within classrooms and driving collaborative problem solving. AI will also need to become a tool for inclusion, providing an equitable platform for all students to share their ideas to unlock the innovative solutions that die in silos.

iSAT's development of its AI Partner strives to address the current and future needs of education by providing students with an agent that will help them develop healthy relationships with their peers while empowering them with the collaborative problem-solving skills they'll need to succeed in the classroom and beyond.

  References

[1] Bransford, D., Brown A., and Cocking A. How people learn: Brain, mind, experience, and school committee on developments in the science of learning. The National Academies Press, Washington, D.C., 2000.

[2] National Academies of Sciences, Engineering, and Medicine. How People Learn II: Learners, contexts, and cultures. The National Academies Press, Washington, D.C., 2018.

[3] Vygotsky, L. Mind in Society: The development of higher psychological processes. Harvard University Press, Cambridge, 1978.

[4] Tissenbaum, M. and Slotta, J.D. Scripting and orchestration of learning across contexts: A role for intelligent agents and data mining. In Seamless Learning in the Age of Mobile Connectivity. Springer, 2015, 223–257.

[5] Roschelle, J. and Dimitriadis and Y. and Hoppe, U. Classroom orchestration: Synthesis. Computers & Education 69 (2013), 523–526.

[6] Ananiadou, K. and Claro, M. 21st century skills and competences for new millennium learners in OECD countries. In OECD Education Working Papers No. 41. OECD Publishing, 2009.

[7] Reddy, G., Eloy, L., Dickler, R., et al. Beyond joint visual attention: Identifying gaze dynamics that underlie successful collaborations. In Symposium on Eye Tracking Research and Applications (ETRA '23). 2023. (Under Review).

[8] Cao, J., Dickler, R., Grace, M., et al. Designing an AI Partner for Jigsaw classrooms. Workshop on Language-Based AI Agent Interaction with Children (AIAIC'2023), Los Angeles, California, 2023.

  Author

Alayne Benson is the AI Communications & Outreach Coordinator for the National Science Foundation AI Institute for Student-AI Teaming (iSAT), housed within the Institute of Cognitive Science at the University of Colorado Boulder. She obtained her master's degree in history at the University of Central Florida. A former university history instructor and learning architect for K-12 learning products, she pivoted to becoming a science communicator—turning her lifelong admiration of science into a career dedicated to sharing this admiration with others.

  Figures

Figure 1. iSAT's three interconnected research strands support each other to advance the development of iSAT's AI Partner.

Figure. Denver Public School (DPS) teachers attend a Sensor Immersion professional development workshop with iSAT researchers.

Figure. Middle school students work on a Sensor Immersion lesson together with the help of iSAT Strand 3 researcher, Quinten Biddy.

Figure. iSAT Lab participants examine collaborative problem-solving using an eye-tracker.

 

This work is licensed under Creative Commons Attribution International 4.0.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2023 ACM, Inc.