This project will build and develop JOKER, a generic intelligent user interface providing a multimodal dialogue system with social communication skills including humor, empathy, compassion, charm, and other informal socially-oriented behavior. Talk during social interactions naturally involves the exchange of propositional content but also and perhaps more importantly the expression of interpersonal relationships, as well as displays of emotion, affect, interest, etc. This project will facilitate advanced dialogues employing complex social behaviors in order to provide a companion-machine (robot or ECA) with the skills to create and maintain a long term social relationship through verbal and non verbal language interaction. Such social interaction requires that the robot has the ability to represent and understand some complex human social behavior. It is not straightforward to design a robot with such abilities. Social interactions require social intelligence and ‘understanding’ (for planning ahead and dealing with new circumstances) and employ theory of mind for inferring the cognitive states of another person. JOKER will emphasize the fusion of verbal and non-verbal channels for emotional and social behavior perception, interaction and generation capabilities. Our paradigm invokes two types of decision: intuitive (mainly based upon non-verbal multimodal cues) and cognitive (based upon fusion of semantic and contextual information with non-verbal multimodal cues.) The intuitive type will be used dynamically in the interaction at the non-verbal level (empathic behavior: synchrony of mimics such as smile, nods) but also at verbal levels for reflex small- talk (politeness behavior: verbal synchrony with hello, how are you, thanks, etc). Cognitive decisions will be used for reasoning on the strategy of the dialog and deciding more complex social behaviors (humor, compassion, white lies, etc.) taking into account the user profile and contextual information. JOKER will react in real-time with a robust perception module (sensing user's facial expressions, gaze, voice, audio and speech style and content), a social interaction module modelling user and context, with long-term memories, and a generation and synthesis module for maintaining social engagement with the user. The research will provide a generic intelligent user interface for use with various platforms such as robots or ECAs, a collection of multimodal data with different socially-oriented behavior scenarios in two languages (French and English) and an evaluation protocol for such systems. Using the database collected in a human-machine context, cultural aspects of emotions and natural social interaction including chat, jokes, and other informal socially-oriented behavior will be incorporated.

TRIAGE (Traceability, Acknowledgement, Identification and Management of Disasters Victims) is a research project aiming to an identity management solution for a large number of victims (any person, living or lifeless) during and after a disaster (natural, accidental or malevolent). The proposed solution is mobile, communicating, flexible, efficient, ergonomic and secured. The solution is designed to meet expectations of first responders, when encountering numerous victims. When facing a mass population situation after a natural disaster (such as Katrina hurricane, tsunami in Thailand, earthquake in Italy or Haiti, triple disaster in Japan, earthquake in Turkey, etc.), after a terrorist attack (Madrid, London, Moscow, Oslo/Utøya, etc.) or after an accident (Concordia, Port Saïd, Furiani, etc.), rescues services are required to identify all victims (conscious, unconscious, or lifeless) to build up and manage identity, medical and administrative files. TRIAGE is especially designed to bring an efficient solution in a situation of a disaster generating movements of large numbers of victims (hundreds up to thousands of them). Victims could be injured, unconscious, or lifeless. The living victims are usually not carrying any identity documents while they are fleeing the area. TRIAGE thus aims to provide a complete, compact and ergonomic set of tools required for the rescuing services to help those victims. When a disaster occurs, rescuers have to face an important flow of disoriented and frightened people that they need to census immediately. TRIAGE provides an efficient mean of identity management in order to allow rescuers to concentrate their efforts on the victims themselves. TRIAGE will design a scalable infrastructure based on a mobile platform and a set of inter connected mobile devices. The device will provide the first responders with powerful tools to carry their activity (sorting of injuries and related level of urgency, identification and position). It will support contactless biometric data capture of the victims (fingerprints, face) on low cost sensors, allowing in the same time a speech transcription of the rescuer describing the victim medical state, including a translation capability for missions abroad, a geo-location capability and time-stamping. All those functionalities will be integrated in a simple and intuitive ergonomic design, and a step by step guidance for the using of the device. TRIAGE is very cautious to protect the confidentiality of data and give ethical attention to the respect of privacy. The collected data will be recorded in a temporary local database, gathered during the rescue operation and stored at the mobile platform level, solely maintained during the medical follow up of the victims and securely destroyed afterward. Crucial information on the victim will be stored on a RFID chip (face and fingerprint templates, medical file), carried by the victims. The mobile platform will assist first responders’ actions, will detect position of the handheld devices, and will gather the flow of collected data from the victims. The platform will also be able to consolidate the victims' identity by securely transferring information to a backbone infrastructure (e.g. by transferring ID, medical and administrative data to the hospital). The rescuers will conduct the operations by following defined procedures. If the secured wireless network is available, the mobile devices will remain connected with the mobile platform. If the network is not available, all the information will be processed locally and stored into the device and on the RFID chip. Geo-location will always be activated to guide the first responders and to provide information on the deployment. TRIAGE will set up demonstrations in a large scale situation to prove its efficiency in a real deployment scenario. Several public events are scheduled as important standardisation participation in order to disseminate the results for a broader impact.

The urbanization of society has gradually separated humans from the plant world. Most people have forgotten the names of plants and their potential uses. Yet there is a growing awareness that biodiversity is a treasure we must preserve and transmit to future generations. The identification of plant species is a necessary step to understand our environment. However, for most people, botany remains difficult to understand and to learn. It is not easy to decrypt botanical literature because it requires a solid theoretical background. In the ReVeRIES project (French acronym that means “dreams” and that stands for Interactive, Fun and Educational Plant Recognition on Smartphones), we propose to use mobile technologies in order to help humans recognize plants that surround them. We believe that a promising way to recreate the relationship between modern human beings and their natural environment is to provide smartphone applications that help them recognize and learn about plants. The ReVeRIES project relies on a mobile application called Folia and developed during the ANR ReVeS project. This application is capable of recognizing species of trees and shrubs (taller than 1m20 and originating from France) by analyzing photos of their leaves. This prototype simulates the behavior of a botanist when trying to determine the plant species, which makes it different from all the other tools available on the market. In the context of ReVeRIES, we propose to go much further by developing the following aspects: game-based mobile learning, multimodal images recognition and citizen sciences. First of all, we intend to design mobile learning games that will help users learn about plant characteristics and especially learn the methods, used by expert botanists, to recognize plant families, genera and species. In order to motivate children and botanical neophytes to learn about plants and explore their natural environment, we also intend to use game mechanics for creating fun activities based on plant recognition. The users will be able to improve their skills by comparing their results to those found by the recognition algorithm. Concerning the image recognition process, we intend to extend the previous prototype to the main exotic woody trees and shrubs. Moreover, we aim to take into account various organs of the plant. This multimodality is essential if we want users to learn and practice the correct recognition method, for which botanists use a variety of organs (i.e. leaf, bark, size of plant, flower, fruit, etc.). In addition, the use of organs should greatly improve the algorithm’s accuracy. In terms of image processing, the work done on the leaves cannot be extended directly to flowers, fruits and barks. This will greatly increase the complexity of the data fusion process. Finally, we intend to explore ways in order to enhance social awareness of our natural resources and to support citizen science. The geolocated photos and information taken with the application and validated by experts, could be transferred to specialized networks, such as Tela Botanica, integrated into the OpenStreetMap geographic information system and mobilized by local institutions to support actions and projects involving citizens. This addresses problems related to the field of Volunteered Geographical Information. The project raises many scientific challenges in TEL (Technology Enhanced Learning), Serious Game, image analysis, data fusion, HCI, and also in the field of collaborative environmental inventories. The possible impacts are numerous: teaching of botany at different levels and with various learning audiences, collective intelligence, citizen sciences, nature preservation and environmental collaborative games. In addition to citizens interested in nature, this system could be very useful for teachers and their students, botanists and also nature parks.

The proposal aims at developing tools for diagnostic, localization, and measurements of automatic transcription errors. This proposal is based on a consortium of academic actors of very first plan in this field. The objective is to study in detail (at the perceptive, acoustico-phonetics, lexical, and syntactic levels) the errors in order to bring a precise diagnosis of possible lacks of the current classical models on certain classes of linguistic phenomena. At the application level, the proposal is justified by an observation: a high number of applications in the field of content access from multimedia data are made possible by the use of automatic transcriptions of speech: subtitling of video emissions, search for precise extracts in audio-visual archives, automated reports of meetings, extraction of information and structuring of information (Speech Analytics) in contents multimedia (Web, call centers, ...). However their deployment on a large scale is often slowed down by the fact that transcription from automatic speech recognition systems contains too many errors. Research and development in speech recognition related, successfully until now, to the improvement of methods and models implemented in the process of transcription, measured thanks to the word error rate; however, last a certain performance level, the marginal cost induced to reduce the residual errors increases then exponentially. Transcription errors thus persist, which are more or less awkward according to the applications. Information retrieval is tolerant with errors (up to 30%), but systematic errors on certain named entities can be prohibitive. On the contrary, subtitling or meeting transcription have a very weak tolerance with the errors, and even very low word error rates compared to the state of the art (lower than 5%) are too high for the end-users. Error processing is not limited to increase the acceptability of the applications based on the automatic transcription of the word. Error classification, impact measurement by perceptive tests, error diagnosis state-of-the-art systems, are the first crucial stage in order to identify the lacks of the current models and to prepare the future Automatic Speech Recognition system generations. The proposal aims, by a close cooperation between complementary partners who excel in their field, to set up an infrastructure of detection, diagnosis, and qualitative measurement which makes it possible to create a virtuous circle of improvement of large and very large vocabulary continuous speech recognition systems.

Along with the democratization of increasingly high-performance digital and communication technologies, higher education and training for adults are constantly challenged by both the renewal and the adaptation of teaching practices. While the frontiers between guided learning and self-learning are becoming less obvious, which tends to redefine the role of the teacher and the learner, the great accessibility of technologies, on the other hand, enables a diversity of interaction modes between teachers and learners, as well as between learners and learners. We believe that the widespread use of digital technologies, especially online courses starts with the development of SPOC (Small Private Online Courses) at a reduced cost while capable of largely covering numerous educational areas. For that matter, the engineering process needs to better involve teachers in charge of the lectures, and to allow them to personalize their content and teaching methods in order to develop blended learning, thus the entanglement of the use of digital content and classroom teaching. PASTEL is a research project that aims to explore the potential of real time and automatic transcriptions for the instrumentation of mixed educational situations where the modalities of the interactions can be face-to-face or online, synchronous or asynchronous. The speech recognition technology approaches a maturity level that allows new opportunities of instrumentation in pedagogical practices and their new uses. More specifically, we develop (1) a real-time transcription application, and (2) educational outreach applications based on the transcription system outputs. We will use these results to automatically generate the materials of a basic SPOC. A set of editing features will be implemented for the mentioned applications that will allow the teacher to adapt and customize these contents according to their needs. Then, the developed applications will be made available to public institutions for higher education and research, and will also be transferred to the industry through Orange or start-ups associated to the research laboratories involved in the project. The major innovations of PASTEL cover the discourse structure from automatic transcriptions that are linked to its educational objectives. The innovation also features the challenging flow processing in real time, which is required when the discourse structure is being used in a face-to-face situation. The project also brings innovative solutions in terms of instrumentation, and diversification of pedagogical practices, as well as a new approach to design and structure online educational contents, based on the use of speech recognition technology.