The EMIR Laboratory (Exploration of Media Immersion for Rehabilitation) is now well underway to becoming a reality. We have a space, albeit still temporary as we shall eventually be moving to a completely refurbished space a few doors down the corridor, several computers and are in the process of acquiring our first major piece, a floor projection system. Combined with our efforts in collaboration with Bloorview Kids Rehab, we will be working with the full range of human sensory perception - visual and audio of course, but also tactile, movement, physiological (heart rate, skin conductance, breathing, etc.), olfactive and even taste as well as using a brain-computer interface. The goal is to generate immersive experiences - creative, game-like, artistic, etc. - that challenge rehab patients, clinicians and/or researchers to view themselves in new ways.
However, few people have any understanding of what can be achieved or how to go about doing this. In addition, even our team, which has been exploring multisensory immersive environments for some time, needs good intermediate tools to support our ongoing research, and we are not always aware of what is possible either. With a view to both helping ourselves, but also encouraging collaboration and participation in the new laboratory, we have embarked upon the process of developing a "toolkit" for delivering multisensory immersive experiences with a minimum of technical expertise.
Called an Affordance Toolkit (because each tool affords different sets of activities - we are drawing on Gibson's affordance theory for this), the framework consists of matching a set of controller interfaces to a set of viewer modules as a function of particular tasks. Controllers include cameras that are able to read and interpret gestures, tactile screens and pressure carpets able to register different forms of body contact, microphones for recording and interpreting sounds, and sensors for recording physiological or neurological signals. Viewers include 1-, 2- or 4-wall projection, ceiling and floor projection, surround spatialized sound, motor-driven devices - both large and small, scent diffusers, and so on.
Tools under development that bridge these two sets of functionalities include the following :
1) Mirror Space - using webcams and full wall prujections where the real-time video images are horizontally flipped to generate a pseudo-mirror image (occupying 1, 2 or all for walls), combined with the addition of digital enhancements, virtual objects and annotations added to the projected image, we are able to deliver an environment that supports a variety of tasks, including various physical games (tug of war, zone avoidance, tag, etc.), cognitive games or tasks (draw in the outlines of objects, paint by numbers, etc.) or controlled exercise and/or balance task (raise your feet until they hit a gong, move along a virtual line, etc.);
2) Master at Work - using data gloves or alternate controllers for those unable to use their hands, use gestures and manipulation to create and modify sounds, visual objects, odors, etc. to make a "multisensory composition" akin to a musical composition. This might be done in a darkened room and avoid the use of vision;
3) Room of Presence - Similarly to the previous tool, this will allow for the materialization of virtual characters that then interact with the user. The user will be able to draw on a bank of virtual characters with a range of pre-deteermined bheaviors, or be able to create very simple "characters" with new behaviors;
4) Multisensory Logbook - In order to record, annotate, archive and playback the expriences created in the EMIR laboratory, we are working on the development of a multisensory logbook system involving video cameras and microphones as well as a computerized logbook of programmed functions;
5) Social Atlas - Using GPS for outdoor environments and RFID tracers combined with other location technologies for interiors, we will provide the ability to both track volonteers or friends and to represent these movements within the EMIR laboratory;
6) Experiensorium - Using geographical database structures, we shall be able to provide the possibility of navigating large and complex virtual environments filled with a multitude of sensory experiences. This will be particularly effective in the presence of non-realistic visuals or no visuals at all. For example, walking through a sketched farmyard, but hearing and smelling the animals, feeling thir presence through air currents and the occasional sense of touch. Within the experiensorium, it will be possible to play out games or narrative experiences.
In addition to these macro-tools, we will also be developing and using a range of microtools such as the ability to call up a pop-up menu on the wall-screens using gestures, to partition the visual, audio or tactile spaces, to inject text into these different spaces (e.g. written, audio or braille), and so on.
Each of the proposed tools represents significant research and development challenges, but working on them is both satisfying and engaging. We look forward to reporting on progress on the development of the toolkit over the coming months.
Tuesday, July 28, 2009
Thursday, April 30, 2009
Transformative Installations - Global perspective
Since the early development work on the Bloorview initiative (called at the time the "Hidden Magician" project), our efforts to develop a whole range of "transformative" or "resonant" installations has moved forward by leaps and bounds into several major initiatives. We are currently active in the development of a major "new generation environment" at Bloorview Kids Rehab that we are calling the "Living Walls Initiative". Within this project, we are developing a highly interactive, one might say "reactive" wall mural that responds to the presence of children with disability in many different ways. Our goal is to change the way the children understand their relationship to their surrounding space.
Children with disability struggle within environments which are highly disabling. Indeed, we call the children "disabled" but we might more usefully call the environments within which they (and we) function disabling environments. As a result, these children often feel like they are a burden on others, that they have to struggle with the environment, that they are what's "wrong". By developing new environments that are much more responsive to a variety of forms and levels of disability, we aim to challenge this understanding, to offer these children an insight into other possible relations they might have to the spaces that surround them and with which they engage.
An early conceptualization of the Living Walls Initiative
The Living Walls initiative is the first major attempt to do this. The overall concept is to develop a large wall mural (we're thinking 8 feet high by 20 feet long) that is made up of motorized elements that will respond, via appropriately designed interfaces, to children with various forms of disability. The mural will depict a scene of relevance to the hospital - a depiction of the ravine that drops away behind the hospital and which has already been incorporated in a number of ways into the design of the hospital building. This allows the children to be attuned to the presence of natural elements in the local environment of the hospital. We are designing into the mural elements which may change color and shape and hence depict the changing seasons. However, the main focus of the mural is to allow the children to interact with the scene and to make interesting changes to it. For example, we are building in animal figures that may hide or emerge at different moments, when the mural senses a child in its proximity. By making some sort of movement, whether using a wheelchair or a gesture, children will be able to change several aspects of the mural - the intensity of water flow in the built-in waterfall, the shape and color of leaves in the trees, the overflight of planes, and so on.
The project is moving from its conceptual design phase into the development of early prototypes that will be used to test the implementation before this is fully fleshed out. At the same time, funds are being sought, both from private donors and funding agencies, in support of the project. Many of the partnerships needed for its success are already in place.
A second "next generation environment" projet also aimed at helping children with disability has been named the "Ado-Matrix Project". This project focuses particularly on the plight of adolescents with disability, who face a situation where they tend to become isolated from their peers and are in a difficult position to build new friendships. To serve their needs, we are developing a tele-gaming environment that "equalizes" player access across different levels of ability, so that a severely handicapped adolescent may play on an equal footing as an able-bodied friend. Our project seeks to create remotely controlled robots that must work together in a common, physically real environment to achieve group goals. Each adolescent will control his or her own robot, an semi-independent webcam and will have access to group chat either through text or voice or a combination of these. Different robots will have different functionality, however. For this project we are still building partnerships and doing conceptual design.
A third installation project on which we are working addresses the issue of climate change and environmental responsibility. Here our aim is to develop an installation that can be taken to the urban public and which will sensitize participants not only to the issues of the environment but do so in a manner that is informed by an awareness of the inequities in urban life and how different elements of the community may learn to find common ground in addressing these issues. The project bears the title "Voices of Transition".
Children with disability struggle within environments which are highly disabling. Indeed, we call the children "disabled" but we might more usefully call the environments within which they (and we) function disabling environments. As a result, these children often feel like they are a burden on others, that they have to struggle with the environment, that they are what's "wrong". By developing new environments that are much more responsive to a variety of forms and levels of disability, we aim to challenge this understanding, to offer these children an insight into other possible relations they might have to the spaces that surround them and with which they engage.
The Living Walls initiative is the first major attempt to do this. The overall concept is to develop a large wall mural (we're thinking 8 feet high by 20 feet long) that is made up of motorized elements that will respond, via appropriately designed interfaces, to children with various forms of disability. The mural will depict a scene of relevance to the hospital - a depiction of the ravine that drops away behind the hospital and which has already been incorporated in a number of ways into the design of the hospital building. This allows the children to be attuned to the presence of natural elements in the local environment of the hospital. We are designing into the mural elements which may change color and shape and hence depict the changing seasons. However, the main focus of the mural is to allow the children to interact with the scene and to make interesting changes to it. For example, we are building in animal figures that may hide or emerge at different moments, when the mural senses a child in its proximity. By making some sort of movement, whether using a wheelchair or a gesture, children will be able to change several aspects of the mural - the intensity of water flow in the built-in waterfall, the shape and color of leaves in the trees, the overflight of planes, and so on.
The project is moving from its conceptual design phase into the development of early prototypes that will be used to test the implementation before this is fully fleshed out. At the same time, funds are being sought, both from private donors and funding agencies, in support of the project. Many of the partnerships needed for its success are already in place.
A second "next generation environment" projet also aimed at helping children with disability has been named the "Ado-Matrix Project". This project focuses particularly on the plight of adolescents with disability, who face a situation where they tend to become isolated from their peers and are in a difficult position to build new friendships. To serve their needs, we are developing a tele-gaming environment that "equalizes" player access across different levels of ability, so that a severely handicapped adolescent may play on an equal footing as an able-bodied friend. Our project seeks to create remotely controlled robots that must work together in a common, physically real environment to achieve group goals. Each adolescent will control his or her own robot, an semi-independent webcam and will have access to group chat either through text or voice or a combination of these. Different robots will have different functionality, however. For this project we are still building partnerships and doing conceptual design.
A third installation project on which we are working addresses the issue of climate change and environmental responsibility. Here our aim is to develop an installation that can be taken to the urban public and which will sensitize participants not only to the issues of the environment but do so in a manner that is informed by an awareness of the inequities in urban life and how different elements of the community may learn to find common ground in addressing these issues. The project bears the title "Voices of Transition".