Results

weDRAW contributed to progress beyond the state of the art along with several different directions:

  • Novel concepts and approaches

    Research in weDRAW studied the best sensory modalities for understanding specific arithmetic and geometrical concepts. Results in this direction were obtained in terms of a series of psychophysical studies. These results, integrated with the pedagogical inputs, informed design and development of technologies.

  • A novel ICT-supported paradigm to learn arithmetic and geometry

    The serious games and the activities weDRAW developed link body movement with visual and auditory feedback, taking art (namely music and drawing) as a fundamental basis to design feedback.

  • A novel unique language

    A relevant aspect in weDRAW is that its approach and technology enable the application of the same learning paradigm to typically developed and impaired children. weDRAW organized workshops to identify the best approaches to teach geometry to children with visual disability, and the prototypical activities weDRAW conceived for teaching some geometric concepts were tested with visually impaired children.

  • A novel embodied and enactive pedagogical approach

    weDRAW technologies contribute to innovative pedagogical approaches, enabling teachers to design activities meeting specific needs of their students. weDRAW focused a lot on teachers and their role. Teachers were involved in the design process of technology.

  • Novel algorithms for analysis of nonverbal affective motoric and social interaction

    The project worked on algorithms for analysis of nonverbal affective motoric interaction in the learning process. Techniques were integrated into the weDRAW libraries and platform.

  • Novel ICT tools for early dyslexic diagnosis

    research suggests that dyslexic children have a lower precision on time perception. To enable fast screening of dyslexia, an app was developed in weDRAW for assessing precision on time perception from different sensory modalities.

  • Novel ICT products and services for technology-enhanced learning

    weDRAW released a platform, three serious games, and other prototypes that represent novel ICT products and services for technology-enhanced learning (see below for details). 

Exploited results

Cartesian Garden Game

Ignition Factory

Info avaialble soon

weDRAW web App for Schools' interactive whiteboard

IIT - DeAgostini - Università di Genova

Info avaialble soon

Librieries

Università di Genova

Info avaialble soon

weDRAW results are expected to have a direct impact on society, especially for enhancing the teaching and learning processes, for increasing educational possibilities for pupils with special (dis)abilities, and for increasing the capacity to retain pupils in the areas threatened by lack of specialized schools.

weDRAW is also expected to induce a chain of events encouraging both institutions and citizens to adopt more widely specialized ICT methods and tools into sustainable educational development models of the local and regional education systems.

Read more: Results

  • Hits: 2540

Scientific Publications

Scientific Publications

  • What cognitive and affective states should technology monitor to support learning?

    Olugbade, Temitayo; Cuturi, Luigi; Cappagli, Giulia; Volta, Erica; Alborno, Paolo; Newbold, Joseph; Bianchi-Berthouze, Nadia; Baud-Bovy, Gabriel; Volpe, Gualtiero; Gori, Monica

    This paper discusses self-efficacy, curiosity, and reflectivity as cognitive and affective states that are critical to learning but are overlooked in the context of affect-aware technology for learning. This discussion sits within the opportunities offered by the weDRAW project aiming at an embodied approach to the design of technology to support exploration and learning of mathematical concepts. We first review existing literature to clarify how the three states facilitate learning and how, if not supported, they may instead hinder learning. We then review the literature to understand how bodily expressions communicate these states and how technology could be used to monitor them. We conclude by presenting initial movement cues currently explored in the context of weDRAW.

    Read more

  • The Influence of Auditory Information on Visual Size Adaptation

    Alessia Tonelli, Luigi F. Cuturi and Monica Gori

    Size perception can be influenced by several visual cues, such as spatial (e.g., depth or vergence) and temporal contextual cues (e.g., adaptation to steady visual stimulation). Nevertheless, perception is generally multisensory and other sensory modalities, such as auditory, can contribute to the functional estimation of the size of objects. In this study, we investigate whether auditory stimuli at different sound pitches can influence visual size perception after visual adaptation. To this aim, we used an adaptation paradigm (Pooresmaeili et al., 2013) in three experimental conditions: visual-only, visual-sound at 100 Hz and visual-sound at 9,000 Hz. We asked participants to judge the size of a test stimulus in a size discrimination task. 

    Read more

  • Developing a pedagogical framework for designing a multisensory serious gaming environment

    Sara Price, Sam Duffy, Monica Gori

    The importance of multisensory interaction for learning has increased with improved understanding of children’s sensory development, and a flourishing interest in embodied cognition. The potential to foster new forms of multisensory interaction through various sensor, mobile and haptic technologies is promising in providing new ways for young children to engage with key mathematical concepts. However, designing effective learning environments for real world classrooms is challenging, and requires a pedagogically, rather than technologically, driven approach to design. This paper describes initial work underpinning the development of a pedagogical framework, intended to inform the design of a multisensory serious gaming environment. It identifies the theoretical basis of the framework, illustrates how this informs teaching strategies, and outlines key technology research driven perspectives and considerations important for informing design. An initial table mapping mathematical concepts to design, a framework of considerations for design, and a process model of how the framework will continue to be developed across the design process are provided.

    Read more

  • WeDRAW: USING MULTISENSORY SERIOUS GAMES TO EXPLORE CONCEPTS IN PRIMARY MATHEMATICS

    Sam Duffy, Sara Price, Gualtiero Volpe, Paul Marshall, Nadia Berthouze, Giulia Cappagli, Luigi Cuturi, Nicolo Balzarotti, David Trainor, Monica Gori

    WeDRAW aims to mediate learning of primary school mathematical concepts, such as geometry and arithmetic, through the design, development and evaluation of multisensory serious games, using a combination of sensory interactive technologies. Working closely with schools, using participatory design techniques, the WeDRAW system will be embedded into the school curricula, and configurable by teachers. Besides application to typically developing children, a major goal is to examine this multisensory approach with visually impaired and dyslexic children. 

     

    Read more

  • A multimodal serious-game to teach fractions in primary school

    Simone Ghisio, Paolo Alborno, Erica Volta, Gualtiero Volpe

    Multisensory learning is considered a relevant pedagogical framework for education since a very long time and several authors support the use of a multisensory and kinesthetic approach in children learning. Moreover, results from psychophysics and developmental psychology show that children have a preferential sensory channel to learn specific concepts (spatial and/or temporal), hence a further evidence for the need of a multisensory approach. In this work, we present an example of serious game for learning a particularly complicated mathematical concept: fractions. The main novelty of our proposal comes from the role covered by the communication between sensory modalities in particular, movement, vision, and sound. The game has been developed in the context of the EU-ICT-H2020 weDRAW Project aiming at developing new multimodal technologies for multisensory serious-games on mathematical concepts for primary school children.

    Read more

  • The Effect of Visual Experience on Perceived Haptic Verticality When Tilted in the Roll Plane

    Luigi F. Cuturi, Monica Gori

    The orientation of the body in space can influence perception of verticality leading sometimes to biases consistent with priors peaked at the most common head and body orientation, that is upright. In this study, we investigate haptic perception of verticality in sighted individuals and early and late blind adults when tilted counterclockwise in the roll plane. Participants were asked to perform a stimulus orientation discrimination task with their body tilted to their left ear side 90° relative to gravity. Stimuli were presented by using a motorized haptic bar. In order to test whether different reference frames relative to the head influenced perception of verticality, we varied the position of the stimulus on the body longitudinal axis. Depending on the stimulus position sighted participants tended to have biases away or toward their body tilt. Visually impaired individuals instead show a different pattern of verticality estimations. 

    Read more

  • An open platform for full-body multisensory serious-game to teach geometry in primary school

    Simone Ghisio, Erica Volta, Paolo Alborno, Gualtiero Volpe

    Recent results from psychophysics and developmental psychology show that children have a preferential sensory channel to learn specific concepts. In this work, we explore the possibility of developing and evaluating novel multisensory technologies for deeper learning of arithmetic and geometry. The main novelty of such new technologies comes from the renewed understanding of the role of communication between sensory modalities during development that is that specific sensory systems have specific roles for learning specific concepts. 

    Read more

  • Angle Discrimination by Walking in Children

    Luigi Cuturi, Giulia Cappagli, Monica Gori

    In primary school, children tend to have difficulties in discriminating angles of different degrees and categorizing them either as acute or obtuse, especially at the first stages of development (6-7 y.o.). In the context of a novel approach that intends to use other sensory modalities than visual to teach geometrical concepts, we ran a psychophysical study investigating angle perception by spatially navigating in space. Our results show that the youngest group of children tend to be more imprecise when asked to discriminate the walking angle of 90°, pivotal to learn how to differentiate between acute and obtuse angles. These results are then discussed in terms of the development of novel technological solutions aimed to integrate locomotion in the teaching of geometrical concepts.

    Read more

  • Using force-feedback devices in educational settings: a short review

    Gabriel Baud-Bovy, Nicolò Balzarotti

    In this short review, we aim at providing an update about recent research on force-feedback devices in educational settings, with a particular focus on primary school teaching. This review describes haptic devices and education virtual environments before entering into the details of domain-specific applications of this technology in schools. Currently, the number of studies that investigated the potential of haptic devices in educational settings is limited, in particular for primary schools. 

    Read more

  • Opportunities and Challenges of Bodily Interaction for Geometry Learning to Inform Technology Design

    Sara Price, Sam Duffy

    An increasing body of work provides evidence of the importance of bodily experience for cognition and the learning of mathematics. Sensor-based technologies have potential for guiding sensori-motor engagement with challenging mathematical ideas in new ways. Yet, designing environments that promote an appropriate sensori-motoric interaction that effectively supports salient foundations of mathematical concepts is challenging and requires understanding of opportunities and challenges that bodily interaction offers. This study aimed to better understand how young children can, and do, use their bodies to explore geometrical concepts of angle and shape, and what contribution the different sensori-motor experiences make to the comprehension of mathematical ideas.

    Read more

  • Exploring how children interact with 3D shapes using haptic technologies

    Nikoleta Yiannoutsou, Rose Johnson, Sara Price

    Haptic devices have the potential to enhance the learning experience by foregrounding embodied, sensory and multi-modal elements of learning topics. In this paper, we report on-going work investigating a game prototype with haptic feedback for seven year old children’s engagement with geometrical concepts as part of an iterative design study. Our findings include a new game play mode adopted by the children, that empowers the use of haptic feedback in game play and has the potential to enable the enactment of shape properties in the game play process.

    Read more

  • Effects of Chai3D Texture Rendering Parameters on Texture Perception.

    Balzarotti N, Baud-Bovy G (2018)

    Touching objects in a virtual environment is a challenge that has yet to be addressed convincingly, in part because haptic technology and, in particular, low-cost haptic technology have strong limitations. This work aimed at assessing the impact of Chai3D texture rendering parameters on texture perception. We used Multidimensional Scaling techniques to build psychological scales for the texture level, stiffness, dynamic friction and several texture patterns. Two perceptual dimensions were in general necessary to fully account for the one-dimensional parameter change. The scales for the texture level, dynamic friction and texture pattern parameters were markedly larger than the stiffness scale, indicating the potential of these parameters to generate well differentiated textures.

Read more: Scientific Publications

  • Hits: 3388

Videos

This game allows understanding how the brain can adapt when the sensory channels are altered. In this case, the visual rays are deviated by the prismatic glasses and the sensory-motor coordination undergoes its effects. The brain needs time and evidence to adapt to this change, falling into several errors before completely recalibrating the movement of the arms and hands from the distorted view.

Questo gioco permette di comprendere come il cervello riesce a riadattarsi quando i canali sensoriali sono distorti. In questo caso, i raggi visivi vengono deviati dagli occhiali prismatici e la coordinazione sensori-motoria ne subisce gli effetti. Il cervello ha bisogno di tempo e prove per adattarsi a questo cambiamento, cadendo in diversi errori prima di ricalibrare completamente il movimento di braccia e mani rispetto alla vista ormai distorta.

How can we guess which action a person is miming in darkness? With this activity, we will discover how simple it is to understand both gestures and emotions in the dark if you attach few phosphorescent balls on the body of our mime. Thanks to this capacity of our brain, we are able to distinguish an inanimate object from an animate one since the first months of life.

Come facciamo ad indovinare cosa sta mimando una persona al buio? Con questa attività scopriremo che non solo capire le azioni ma anche intuire le emozioni al buio è davvero facile se si applicano poche palline fosforescenti sul corpo del nostro mimo. Grazie a questa capacità del nostro cervello, siamo in grado di distinguere un oggetto inanimato da uno animato a partire dai primi mesi di vita!

The communication of a message seems slightly simple when using principally one sense, but what happens when each of us uses a different sense? This activity aims to show that more senses we use, more likely it is that the final addressee will get a diverse message from the original! This phenomenon happens because every sense is devoted to a specific task: for example, vision is specialized in perceiving objects position in space whereas hearing is specialized in distinguishing fast music from slow music. Therefore, when we want to use our hearing to understand where objects are in a room, the task will become more difficult and communication gets worse.

Trasmettere un messaggio ad altre persone sembra piuttosto semplice quando si utilizza un solo senso, ma cosa succede quando ognuno di noi utilizza un senso diverso dagli altri? Questa attività ha lo scopo di mostrare che più sensi vogliamo usare, più è probabile che al destinatario arrivi un messaggio diverso dall’originale! Questo accade perché ogni senso è specializzato in un preciso compito, ad esempio la visione è specializzata nel percepire dove gli oggetti sono nello spazio mentre l’udito è specializzato nel distinguere musiche veloci da musiche lente. Quindi quando vogliamo utilizzare l’udito per capire dove sono gli oggetti in una stanza, il compito diventerà sempre più difficile!

In this activity, the child is asked to recognize different objects by touching them blindfolded or with the use of blurry glasses (see dedicated video). This task, apparently trivial, can be very difficult when the sensory channel of touch is compromised. To do this, you can use gloves of different thickness. The increase of thickness reduces the tactile sensory capacity, which makes perception less precise and therefore more prone to errors.

In questa attività, al bambino viene chiesto di riconoscere degli oggetti toccandoli da bendati o con l’utilizzo degli occhiali sfocati (si veda video dedicato). Un compito all’apparenza banale ma che può essere reso molto difficile quando il canale sensoriale del tatto viene compromesso! Per farlo, si possono utilizzare guanti di diverso spessore. L’aumentare dello spessore riduce la capacità sensoriale tattile, ovvero rende la percezione meno precisa e quindi più incline ad errori.

This activity allows us to understand that sometimes our brain deceives us so that we feel what we are not actually experiencing. For example, it is possible that a rubber hand seems to belong to our body just because the brain sees it moving like our real hand. The authors who have discovered this illusion said "the hands feel what the eyes see".

Questa attività ci consente di capire che alcune volte il nostro cervello ci inganna e ci fa illudere di provare sensazioni che in realtà non stiamo provando. Ad esempio, è possibile che una qualunque mano di gomma ci sembri appartenere al nostro corpo solo perché il cervello la vede muoversi come la nostra vera mano. Gli autori che hanno scoperto questa illusione dissero: le mani sentono quello che gli occhi vedono.

This video shows how to build a pair of blurry glasses. The blurry vision recreates an effect known in the scientific literature as the Ganzfeld effect: vision can perceive the reality thanks to the different distribution of light in the space, if the light is uniformly distributed, then the visual perception gets worse. This simple object makes this concept clear in a practical and funny way. The blurry glasses can be used in different activities to understand how the sight, even if present, is not enough to carry out daily activities. Try them also with the basketball game!

Questo video illustra come costruire dei semplici occhiali sfocati. La sfocatura ricrea un effetto conosciuto in letteratura come l’effetto Ganzfeld: la vista permette di percepire la realtà solo grazie alla diversa distribuzione della luce nello spazio, se la luce viene distribuita in modo uniforme ecco che la percezione visiva peggiora. Questo semplice oggetto rende chiaro questo concetto in modo pratico e divertente. Può essere utilizzato in varie attività per far capire quanto la vista, anche se leggermente presente, certe volte non basti per svolgere delle normali attività quotidiane. Provateli anche con l’attività del tiro a canestro!

In this video, we created an object that allows us to study the perception of verticality. The concept of verticality corresponds to the direction of gravity. Although verticality can be correctly perceived when we are standing, we can face perceptual errors when lying on one side. These errors depend on how our brain codes the verticality in the most frequent situations, which is when we are standing or sitting. In fact, the brain assumes that the body is more often in a standing position and therefore parallel to gravity. On the other hand, when we are lying on one side, we are in an unusual position and the perceptual system tends to recognize verticality as more tilted towards the body thus making a mistake.

In questo video, abbiamo costruito un oggetto che permette di studiare la percezione di verticalità. Il concetto di verticalità corrisponde alla direzione di gravità e, nonostante si riesca a percepirla correttamente quando siamo in piedi, si possono presentare degli errori percettivi quando si è sdraiati su un fianco. Questi errori dipendono da come il nostro cervello codifica la verticalità nelle situazioni più frequenti, ovvero quando siamo in piedi o seduti. Il cervello infatti assume che il corpo sia più spesso in posizione eretta e quindi parallelo alla gravità. Quando invece siamo sdraiati su un fianco siamo in una posizione inusuale e il sistema percettivo tende a indicare la verticalità come più inclinata verso il corpo commettendo quindi un errore.

In this video, we present three activities where movements sonification technologies are used to support mathematics understanding. In the first, we show how to create angles by moving the arms. Different angles are associated to different musical notes: sounds with high pitch are used to sonify acute angles and sound with low pitch for obtuse angles. In the second, we used the mathematical relationship between rhythmic patterns performed by hands clapping and fractions. In the third, the fraction becomes the user’s body: the openings of arms and legs controls numerator and denominator of the fraction, respectively  

In questo video presentiamo tre attività dove le tecnologie di sonificazione dei movimenti sono usati per supportare la comprensione della matematica. Nella prima demo, mostriamo come creare angoli usando il movimento delle braccia. Diversi angoli sono associati a diverse note musicali: suoni acuti sono usati per sonificare angoli acuti e suoni gravi per angoli ottusi. Nella seconda, usiamo la relazione matematica fra pattern ritmici creati attraverso il battito delle mani e le frazioni. Nella terza demo, la frazione diventa il corpo dell’utente: l’apertura delle braccia e delle gambe controlla il numeratore e denominatore della frazione, rispettivamente

The video shows the activities developed by the weDRAW partners of Infomus Lab-Casa Paganini (University of Genoa). These activites take advantange of motion capture systems, auditory and visual systems to support teaching fractions, proportions and angles.  

Il video mostra le attività sviluppate dai partner weDRAW di Infomus Lab-Casa Paganini (Università di Genova). Queste attività si avvalgono di sistemi di motion capture, sistemi uditivi e visivi per l'insegnamento delle frazioni, delle proporzioni e degli angoli.

Read more: Videos

  • Hits: 4781

Objectives

Objectives

Scientific objectives

To gain a deeper understanding of the rhythmic and motoric capabilities of typically developing, visual impaired, and dyslexic children at different stages of their development (6-7yo and 8-10yo) through carrying out psychophysical experiments.

Identifying the primary and most suitable sensory modality to teach typically developing children different arithmetical and geometrical concepts.

Identifying the sensory modalities that can effectively supplement the primary sensory modality when this is compromised in visually impaired and dyslexic.

Pedagogical objectives

Defining a multisensory embodied and enactive pedagogical framework for teaching and learning arithmetical and geometrical concepts with primary school children, exploiting art – namely music and drawing – as mediator of the learning experience.

Understanding how and the extent to which the developed pedagogical framework can be applied to both typically developing and dyslexic and visual impaired children, in order to overcome barriers and promote social inclusion of impaired children.

Evaluating the pedagogical effectiveness of this new paradigm and of the developed technology for both typically developing and impaired children.

Technological objectives

Developing a library of software modules for analysis of nonverbal motoric expressive and social behaviour of both individuals and small groups of children and for real-time control of sound, music, haptic, and visual feedback.

Developing a library of software modules for multimodal (tactile, motor) analysis of nonverbal behaviour to capture the child’s learning-related affective states.

Developing an integration hardware and software platform, supporting multiple different input and output devices, and scalable to different learning environments (e.g. school and home), integrating the above-mentioned libraries and supporting design and development of serious games.

Developing three serious games, exploiting both the envisaged platform and libraries, and applying the art-inspired multisensory embodied and enactive pedagogical framework developed in the project. The first serious game concerns learning arithmetic through music, the second one addresses learning geometry through drawing, and the third one is a proof-of-concept of shared and social learning, for groups including typically developed and impaired children.

Developing tools for early diagnosis of dyslexia, grounding on recent evidence that problems in rhythm perception can be an indicator for dyslexia.

Read more: Objectives

  • Hits: 4495

© 2018 Wedraw Project
WeDraw has received funding from the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No. 732391
This website reflects only the author's view and that the Commission is not responsible for any use that may be made of the information it contains.