Researcher | Developer | Designer
A research project to learn how embodied and Multimodal Interaction to Support Preschool Children's Musical Understanding
Keywords: Multimodal, Embodied interaction, Music, Children
Plantform
Touchdesigner | Python
Kinect
Unity | C#
GIMI3 Projector
Researchers
Baihui Chen｜Interaction designer & Development
Kun Qian｜Animator
1. Background
Music education benefits children’s mental and intellectual development in early childhood (Hallam, 2010). Guided by child-centered design principles, we propose the Multimodal Interaction of embodied cognition for 4-6-year-old children (Chen, Druga, Tang & Xie, 2019). In order to lower the barrier of entry for younger kids, we want to tap into natural and accessible ways that they understand and experience musical knowledge. This study will improve preschoolers' perception of music through multimodal interaction (Giannopulu, 2013).
2. Research Problem
How to explore a multimodal and embodied interaction to help children understand complex musical knowledge？
-
Embodied learning environments have a substantial share in teaching interventions and research for enhancing learning in science, technology, engineering, and mathematics (STEM) education (Duijzer, Heuvel-Panhuizen, Veldhuis, Doorman & Leseman, 2019).
-
Through learning environments based on embodied cognition theory students are provided with opportunities to ground abstract formal concepts in concrete bodily experiences (Glenberg 2010).
3. Breaking Down The Research Progress
3-1Fig. The Research Progress
4. Interaction Design Principles
4.1 Interactive systems
-
The overall duration of the work should be appropriate to the attention of the child and limited to 10 minutes.
-
The projection should match the height of the preschooler to the size of the child's interaction.
4.2 Children's interactive movement design
-
The purpose of the interaction should be clear in concrete scenarios, and the design of the action should be intuitive and effective.
-
Interaction mapping should be established between the user's body and the agents in the digital world, such as movement mapping, and limb position.
4.3 Interaction scene construction
-
The work should be guided by the intentionality of the scenario and should suggest situational guidance and metaphors. This can be achieved through the design of guiding characters and animated story narratives that provide clear guidelines for interaction
-
The work should have a sense of spatial orientation, as there are z-coordinates in addition to x and y coordinates in 3D space. Orientation metaphors are needed in interactive projection design to guide and control audience behavior.
-
For interactive projection there should be real-time feedback in multiple media forms. Therefore, visual and auditory complementary feedback results should be taken into account in the design process.
5. Interaction Design Sketches & Interaction Design Storyboard
The work is mainly divided into four parts: "TAN," "XUN," "XING," and "YOU." In terms of content structure, it mainly adopts a general division format, with the opening part "Seek," guiding users to gradually establish a cognitive relationship with the scenario, allowing them to take the initiative and participate in the development of the whole work. The opening section, "Seek," guides the user to gradually establish a cognitive relationship with the scenario, allowing the user to actively participate in developing the whole work. The musical elements are experienced mainly in natural ambient sounds, loudness (left hand), speed (right hand) when conducting, and the rhythm of the music.
5-1Fig. Interaction Design Sketches
5-2Fig. Interaction Design Storyboard
6. Interaction System
Based on the above design principles, the characteristics of the children's age and behavior, and the international standard height data, the projection size of the interactive system was set at 170cm*40cm (Figure 6-1) and the projection length was set at 3 minutes and 15 seconds without interaction. The interaction was mainly in the form of physical interaction, with the use of embodied and familiar interaction actions. In addition, the environment is also an important part of the experience, so the work builds immersive scenarios to give children a better experience, hence the choice of space-based physical interaction projection.
6-1Fig. Interaction system design diagram
7. Research Methodology
This study focuses on a mixed research methodology using research through design to explore the impact of Multimodal interaction through the Kinect and unity with animation to achieve the form of projection. The children's natural movements and gestures are mainly used to interact naturally with the music and images.
The target group of this research is preschool children. As an adult is not easy to stand in their shoes to think about problems, so using participatory design, we could easily interact with them to observe and gather feedback. I think it is a beautiful tool to avoid biases in the design process.
8. Design Practice
8. 1 The main interaction settings
Interaction mapping relationships are also used to enhance the embodiment of children in the scene, and research has shown that immediate feedback is an essential element of the embodiment experience. When interaction mapping relationships are on screen, it is quicker and more intuitive for the user to receive feedback on their behavior and thus stimulate engagement.
8. 2 Visual guidance
Environmental features in the context of embodied interaction are an important factor in the experience of interaction, and the influence of the environment on cognition includes the specific situation and context of the interaction. One of the earliest theories of environmental relevance comes from Gibson's ecological psychology, which argues that visuals can be used as environmental illustrations to guide people's behavior and interactions. In interactive work, children can be helped to quickly receive behavioral feedback through intuitive, multi-dimensional perceptual feedback
8. 3 Scene orientation perception
With the rapid development of technology, user interaction has gradually expanded from a two-dimensional plane to a three-dimensional space, gradually moving from two-dimensional plane interaction to three-dimensional somatosensory interaction. In addition to x and y coordinates, there are also z coordinates, i.e. the sensor can sense the distance from the user to the sensor. According to children's cognitive stage development, when the user operates in three-dimensional space is the picture should also conform to the metaphor of operation in three-dimensional space, therefore, in the scene design process two-dimensional picture is not simply layer superposition, but the picture through three-dimensional collision detection, simulating the orientation of the real space to reduce the difficulty of cognition and understanding when children interact.
9. Development
TouchDesigner receives and processes the Kinect data in the TouchDesigner integrated environment, first providing feedback on the screen, and then transferring the data to Ableton Live. The music elements that need to be called are configured in advance in Ableton Live through the Python language, and the data is transferred between the serial time and Touchdesigner to enable the music dynamics to change.
（ Part of the development display ）