A Day in the Life...

This has been an amazing autumn. I am doing some of the most creative work I have ever done in my career, and at the same time it has been a highly productive period, generating six full peer-reviewed papers over the past year, with three-plus more in the works. So, a typical day :

1. Designing erotic accessories for people with disabilities : This is a two-year project funded by Quebec's Fonds de recherche québécois en société et culture (FQRSC), led by my collaborator Dr Ernesto Morales and involving, in addition to myself, our sexologist collaborator Dr Frédérique Courtois. We organized a series with people with a range of disabilities, both men and women, to discuss their masturbation practices and the difficulties and challenges they face. Based on these interviews, we have elaborated some preliminary designs. Most of these are of the "handle" type, that is, extensions that connect to existing sex toys that allow people with a range of disabilities to use these more effectively. In addition, some actual sex toys are also under development where there don't appear to be any commercial ones available. The work is inspiring in the sense that we are clearly serving a need that exists and for which little work has been done in the past. Our work in this area will probably provide a kind of basis for encouraging other projects in this area over the coming years. Furthermore, the designs we are producing can be either 3d printed directly, or one can 3d print molds that are used to construct the object from non 3d-printed materials such as silicone gels.

2. Understanding how children with and without disability move together : This is a three-year project funded by the Canadian Institutes of Health Research under the leadership of my collaborator Dr Coralee McLaren and involving, in addition to myself, Dr Tom Chau and Dr Barbara Gibson from Holland-Blooview Kids Rehab and Dr Cheryl Missiuna from the University of Waterloo. The project involves bringing a group of children, both with and without disabilities, together within a dance exercise that encompasses both structured and unstructured movement exercises. We will be monitoring the children's neurological responses via their own movements, but especially as they observe and respond to the movements of other children. Although we have not yet begin the full data analysis phase, we have used the EEG cap to study one child's responses to the movements of others, and the results, even for this one individual, are fascinating. Certain observed movements trigger "motor imagery" related to those movements, but not all movements have this effect. It seems that children "feel" certain movements of their peers more than others, and these "felt movements" trigger them to move in similar ways (a kind of mimicking). These observations are changing how we understand the way children play together. There will be much more to come, but even these preliminary results are intriguing.

3. Designing interactive and immersive installations for people with disabilities : Artists across the world are designing and implementing immersive and interactive installations that provide unusual experiences to participants. These installations are often not accessible to people with disabilities. In addition, the experience of disability can itself generate new possibilities for unusual experiences for those without disability, and unusual experiences of the body can lead to new insights for people with disabilities about their own interactions with the environment. A Ph.D. student, Ms. Afnen Arfaoui, has taken on the challenge of developing a design methodology for creating interactive and immersive installations that address issues of disability. She is drawing on Alfred North Whitehead's "process philosophy" in order to structure this design process. This is leading to a novel approach to design of these unusual installations, an approach which integrates both scientific preoccupations and artistic sensitivities.

4. Designing a participative, interactive opera : Given today's technology and the possibilities offered by social networking, it seems obvious that the traditional presentation formats of performance art - theatre, music, opera, dance - should evolve. Modifying the formats of public presentations of these art forms is not easy, however - these have settled into "ways of doing" that are normative and generally accepted by the public. We propose to open up the opera form to new forms of interaction with the audience. To do this, we are working with an unpublished manuscript (see the discussion of the opera project below), a science fiction story still under development. The manuscript is part of a "vast narrative" type project, that is to say, the story is told across 15 books in over 2500 pages, and parts of the story are finding their way into a variety of formats, including game environments and visual art (paintings). Furthermore, the story draws inspiration from the Greek myths and legends of gods and heroes, in particular the stories of Agamemnon Atreus and his family (the Oresteia), of Jason and Medea, of Odysseus and his compatriots and family, and of Orpheus and his family. The Greeks "invented" dramatic presentations, but in their day the public played an active role in the presentation of theatre. We propose to restore this participation. Right now, the issue we are struggling with is how to "segment" the story (in essence, the libretto) into short pieces which can be recombined in different ways through audience choices. The segments have to work not just as text, but also musically and choreographically. You can't just take an arbitrary piece of music and move it around, the result will be incoherent. Choreography also imposes constraints on the process. So as a team we are working on finding a kind of "basic unit" that incorporates the constraints on music, dance and text recombination.

5. Ongoing projects in other areas : Beyond these main challenges, which tend to be present every day in one form or another, several other projects are also ongoing. As part of the effort of developing smart garments and intelligent environments, I am drawing on ecoscience theory to develop a new understanding of complex sensor systems to assist the design process, and to recognize the need to address issues of personal vulnerability as these systems encroach on the highly personal spaces of the body. Another project under development focusses on designing techniques to test and enhance the combined effect of viewing and feeling the body, as a means to improve movement retraining following stroke, for example. In a third area we are looking at how to recruit adolescents with disabilities for a study of virtual reality as a tool for skill development for eventual employment. And this work excludes my ongoing writing initiatives and my work on fashion design...

On Designing Immersive Spaces for Rehabilitation Research

Over the course of the past ten years, my team and I have successfully designed and implemented (produced) more than a dozen immersive installations within a research focus. It is with the emergence of technologies for creating and working with immersive, interactive environments (virtual reality, augmented reality, sensor networks, wearable computing, smart environments, etc.), that the design of immersive spaces has become a viable process. Hence working with such spaces and installations is still very new. Artists have been creating interactive experiences using these technologies for a little more than a decade, while the use of these environments in research has been mostly confined to virtual reality and related technologies. The convergence of immersive technologies from disparate fields into a coherent and powerful set of tools for immersive design is no older than a few years, and, of course, just as the technologies are still evolving rapidly, research based on their use is only beginning to develop.

Research that embraces immersive design requires new methodologies and these, too, are still very experimental. Immersive spaces are different from conventional experimental designs. Immersive environments are increasingly multimodal - that is, visual and auditory and tactile or haptic. They are "ecological" in the sense of being global, holistic and systemic - they engage all of a person, not just targeted aspects. Furthermore, they often generate surprises - this is a consequence of their ecological nature.

Our experience has demonstrated that the creation of these environments demands a three phase process :

CONCEPTION/DESIGN

I

I

DEVELOPMENT/PRODUCTION

I

I

EXPERIMENTATION/EVALUATION

This might look like the classic life cycle of any project, but each of these phases is substantial - these are complex projects to organize. Let us examine just one of our projects to get a better sense of what is involved. I like to call this project "Virtuoso" - all of our installation projects have a distinct title. This project draws upon another key principle we have identified over the course of our development work - immersive installations that have a research (scientific) focus should serve a specific and well-identified need. This is distinct from artistic installations which may or may not serve such a need. "Virtuoso" is being developed to provide an engaging experience to adolescents with motor impairments, who experience both growing isolation and a tendency towards depression as they leave behind their childhood friends. By providing these adolescents with a virtual, online, and shared experience of architectural design within a virtual world environment, we hope not only to break the cycle of isolation and depression but potentially offer also an opportunity to develop an interest that might lead to employment.

The Design Phase of this project lasted over four years. Although this was longer than many, it is by no means atypical of the collection of installations we developed. The Design work actually consisted of two more intensive periods, each lasting from 4 to 6 months, separated by an interval during which the project was shelved. The reasons for the break were multiple, but experience shows that such breaks may be a useful part of the project development cycle (although they need not be as long as in this particular case!). Lack of resources, funding and difficulties developing a mature concept were among the reasons, and possibly also our lack of experience developing and financing complex projects. During the first intensive planning period, we organized a series of meetings with both scientists and clinicians, and agreed to develop a kind of video game experience that would include a social networking dimension as well as some form of cooperative manipulation. However, the exact nature of the video game itself never became clear.

Over the fallow period, the team pursued other projects, including several projects that harnessed the virtual world environment called "Second Life". During this work, the existence of several simulators inspired by the "Second Life" environment were identified and explored. One of these, a simulator called "OpenSim", seemed particularly interesting because it shares with "Second Life" the same simplicity of operation. The arrival of a new researcher led to renewed interest in the "Virtuoso" project, and this time the possibility of using OpenSim as a framing environment for the project gave a clear direction for development. The second period of intensive work on design was therefore able to finalize a design concept - we would lead the adolescents through the process of designing and constructing architectural projects using the virtual world environment. Furthermore, the design concept included not just the design of the immersive environment itself, but also details about the experimental protocols that would need to be followed in the third phase of the project. Indeed, we would evaluate the effect on users of the environment of the design process, compared to a control group that would play a video game.

The Production Phase involved the hiring of a summer student to do the development work. The OpenSim simulation was acquired and the necessary procedures for creating, storing and running a variety of virtual landscapes were studied and adopted. A thematic context for the project was created, pedagogical tools for guiding the young people through the process of virtual construction, and different ways for encouraging collaboration and sharing results were discussed and adopted. The summer student hired to develop the environment is an architectural student with an interest in virtual environments. The development phase is expected to last 3 to 4 months.

The Evaluation Phase is expected to take place over the course of another 4 to 5 months. This phase is, of course, completely different than the development phase - it requires very different kinds of expertise. Indeed, each of the three phases requires different mixes of expertise - this is one of the challenges posed by projects of this kind. In Phase Three, what is required are experts on experimental protocols and data analysis. Technical support for the environment itself must also be in place, and it may be necessary to make adjustments to the design of the immersive environment as a result of challenges in addressing the evaluation phase.

A brief analysis of the dozen or so projects that have been developed using this approach shows that the average duration of the Design Phase was 24 months, of the Production Phase was 10 months, and of the Testing Phase was 9 months. At least 5 of the projects were characterized by a two-step design phase with a fallow period between the two more intense design periods. From initial steps to completion took, on average, 4 years. Some of this duration is almost certainly due to working with limited monetary and human resources, but other factors included the fact that we had to invent research methodology as we progressed, that the projects require different mixes of expertise at different phases, and that creation, implementation and testing of these environments is necessarily complex and demanding work. Perhaps with the benefit of experience, it will be possible to limit the time span to less than 3 years. This is important, since most grants are awarded for three year periods - projects which extend significantly beyond three years are therefore extremely difficult to bring to term in an academic environment. Indeed the early projects in our slate of initiatives were actually taken through only to the end of Phase Two, a result of the need to provide some constraints before procedures and research methodologies became clearer. (Nine of the dozen projects completed or underway are shown in the table below. The projects which are excluded from this list are those which have not yet progressed beyond the Design Phase.)

#	Project	Phase	Duration (mo.)	Expertise
1	Ariadne Emerging	Design	24	Performance designer, choreographer, researcher
		Production	08	Singer, dancer/choreographer, video production team
		Documentation	04	Video production team, researcher, designer
-----
2	Incarnatus	Design	08	Performance designer, singer, theatre technician, programmer, researcher
		Production	08	Singer, theatre technician, programmer, researcher
-----
3	Virtualities in Dusseldorf	Design	24	Designer, researcher
		Production	06	Researcher, designers, programmer/technician, virtual content creators, museum staff, translator
-----
4	Augmented Reality for Bloorview Kids Rehab	Design	08	Researchers, designers, programmer/technician, clinicians
		Production	04	Researcher, designer, programmer/technician, hospital patients
-----
5	Ulysses : A Sound Geography	Design	24	Researchers, designer, composer
		Production	04	Researcher, designer, composer, programmer/technician
-----
6	EcoOracle	Design	24	Researchers, designer, programmer
		Production	06	Researcher, designer, programmer/technician
-----
7	EMIR Demos	Design	06	Researcher, designers/programmers
		Production	24	Researcher, programmers/technicians
-----
8	Pro(x)thèse	Design	36	Researchers, designer, composer
		Production	24	Researchers, sexologist, composer, programmer/technician, fashion engineer, photographers, artists, clinicians
		Experimentation	tbd	Researchers, programmer/technician, clinicians, sexologist
-----
9	Virtuoso	Design	48	Researchers, designers
		Production	04	Researchers, designers, architect, programmer/technician
		Experimentation	06	Researchers, designers, programmer/technician, data analyst
-----

It is worth noting that work of this nature may have been possible only within the context of extended funding such as within a Research Chair. The availability of significant funding levels every year over a seven year grant, obviating the need to reapply for funding every two to three years, made this effort viable. Now that procedures and methods are more well defined, such projects can be managed within shorter term funding arrangements, but the initial development work would have been very difficult to undertake within a standard academic grant environment.

EMIR Laboratory Now Functional

It has taken longer to get the lab completed and operational than expected, but this has finally been achieved. In conjunction with the efforts to complete the installation of the laboratory, we have developed five demos that showcase the potential of the laboratory to support diverse projects in rehabilitation and disability studies. These are as follows :

1. Fragment de vie (Fragmented Life) : Demo that provides different viewpoints of disability by offering different soundtracks for the same film delivered to wireless headphones (collaboration with CinéScène Inc.)
2. Vertiges (Vertigo) : Demo that provides an experience of vertigo by simulating walking across a narrow pathway suspended above Quebec City (collaboration with CinéScène Inc.)
3. Deuxième peau (Second Skin) : Demo that provides an observer a vicarious experience of disability. The movements of a person in a motion capture suit are transfered in real time to an avatar in a virtual kitchen, but attempts to interact with the kitchen and manipulate objects are restricted because the avatar experiences various types of motor impairment (collaboration with CinéScène Inc.)
4. Viscères (Visceres) : Demonstration of a body-based interface for virtual navigation of spaces (Google Street Map)
5. Meta-laboratoire (Meta-laboratory) : Demonstration of the use of the lab to support immersive experiments, in this case, a study of the effects of heminegligence on spatial orientation and judgement in the far field (collaboration with CinéScène Inc. and Dr. Julien Voisin)

In addition to the demos, a growing number of research projects are taking place that harness the possibilities that the lab offers. I have subdivided these projects into six categories : Bimodal Environments, Trimodal Environments, Movement-based Environments, Visceral Environments, Virtual Environments and Geographic Environments.

1. Bimodal Environments : Immersive environments that engage two major sense modalities

   Hemispheres (Hémisphères)

   Experimental study in planning stages of far-field effects of heminegligence (collaboration with Dr. Julien Voisin)

   Co-breather (Co-respirateur)

   Design and validation study underway to build and test four co-breathers for possible clinical applications. Co-breathers provide auditory-tactile immersion (collaboration with Ms. M.L. Bourbeau and C. Légaré)

2. Trimodal Environments : Immersive environments that engage three major sense modalities

   Living Wall (Mur vivant)

   Project in preparation, aimed at developing a playful immersive installation for waiting areas in clinics for children and adolescents that offers an engaging and absorbing environment that provides motor training opportunities (dexterity exercises) for individuals with fine motor impairments (collaboration with Dr. Ernesto Morales; projected Ph.D. thesis of Walid Baccari)

   Pro(x)thèse (Pro(x)thesis)

   Project underway in design phase, aimed at developing a clinical tool for allowing people with disability to explore sexual/sensual imagery and providing the means to track image choices over time. The tool involves the use of a touch-sensitive smart garment and an immersive visual environment, and we are commissioning photos by a professional photographer (collaboration with Dr. Ernesto Morales and Dr Frédérique Courtois).

   Auric Space (Espace aurique)

   Current in the planning stage, this project seeks to provide a training environment for people who have difficulties locating sounds in their immediate environment. We will use a bimodal environment (visual and auditory and haptic) to provide cues to and test for sound location (projected Ph.D. thesis of Afnen Arfaoui)

3. Movement-based Environments : Immersive environments that explore movement modalities

   Third Skin (Troisième peau)

   Project in planning stages that seeks to extend the work initiated in the project "Second Skin" to provide a variety of vicarious experiences of disability and ability

   Choreographic maps (Cartes chorégraphiques)

   Planned project that seeks to study how dance may contribute to emerging ideas about how children play (collaboration with Dr. Cora McLaren)

4. Visceral Environments : Immersive environments that explore actions rooted in the engagement of the body's visceral organs

   Visceres II (Viscères)

   Project in planning stages that aims to test the hypothesis that viscerally learned spaces are more fully understood and remembered than spaces learned by more traditional means

   OrienT (OrienT)

   Planned project that seeks to use a smart garment to help people who get easily disoriented to resituate themselves in their environments (collaboration with Dr. Claude Vincent)

5. Virtual Environments : Immersive virtual environments

   Virtuarch (Virtuarch)

   Project in design development stage that seeks to provide adolescents with disabilities who feel isolated and have a tendency towards depression, access to an environment and situation that engages them in the creative design of architectural spaces (collaboration with Dr. Ernesto Morales)

6. Geographic Environments : Immersive environments that encourage an appropriation of geographic space

   Multimodal Online Mapping Interface (Interface multimodal pour la cartographie en ligne)

   Project in planning stage that seeks to design and implement an on-line multimodal interface for a mapping application that draws on cognitive design principles (collaboration with Dr. Mir Mostafavi; projected Ph.D. thesis of Bilel Saadani)

These different projects seek to serve a variety of populations of people with impairments, including the deaf, the deafblind, the blind, those with low vision, people with either gross or fine motor impairments, people with attention deficits and people with intellectual impairments. Furthermore, different projects have different scientific goals and involve distinct methodologies. These include projects that are experimental or involve evaluation and assessment (Hemispheres, Visceres, etc.), projects that have more educational or pedagogical objectives (Second/Third Skin, etc.), projects that seek the development of assistive technologies or that are focused on design issues (OrienT, etc.), projects aimed at developing training environments (Auric Space, etc.), projects aimed at enhancing personal development among those struggling with issues of disability and impairment (Virtuarch, Pro(x)thesis, etc.), and projects that are far more exploratory in nature (Choreographic Maps, etc.).

Transformative Installations - Global perspective

Since the early development work on the Bloorview initiative (called at the time the "Hidden Magician" project), our efforts to develop a whole range of "transformative" or "resonant" installations has moved forward by leaps and bounds into several major initiatives. We are currently active in the development of a major "new generation environment" at Bloorview Kids Rehab that we are calling the "Living Walls Initiative". Within this project, we are developing a highly interactive, one might say "reactive" wall mural that responds to the presence of children with disability in many different ways. Our goal is to change the way the children understand their relationship to their surrounding space.

Children with disability struggle within environments which are highly disabling. Indeed, we call the children "disabled" but we might more usefully call the environments within which they (and we) function disabling environments. As a result, these children often feel like they are a burden on others, that they have to struggle with the environment, that they are what's "wrong". By developing new environments that are much more responsive to a variety of forms and levels of disability, we aim to challenge this understanding, to offer these children an insight into other possible relations they might have to the spaces that surround them and with which they engage.


An early conceptualization of the Living Walls Initiative
The Living Walls initiative is the first major attempt to do this. The overall concept is to develop a large wall mural (we're thinking 8 feet high by 20 feet long) that is made up of motorized elements that will respond, via appropriately designed interfaces, to children with various forms of disability. The mural will depict a scene of relevance to the hospital - a depiction of the ravine that drops away behind the hospital and which has already been incorporated in a number of ways into the design of the hospital building. This allows the children to be attuned to the presence of natural elements in the local environment of the hospital. We are designing into the mural elements which may change color and shape and hence depict the changing seasons. However, the main focus of the mural is to allow the children to interact with the scene and to make interesting changes to it. For example, we are building in animal figures that may hide or emerge at different moments, when the mural senses a child in its proximity. By making some sort of movement, whether using a wheelchair or a gesture, children will be able to change several aspects of the mural - the intensity of water flow in the built-in waterfall, the shape and color of leaves in the trees, the overflight of planes, and so on.

The project is moving from its conceptual design phase into the development of early prototypes that will be used to test the implementation before this is fully fleshed out. At the same time, funds are being sought, both from private donors and funding agencies, in support of the project. Many of the partnerships needed for its success are already in place.

A second "next generation environment" projet also aimed at helping children with disability has been named the "Ado-Matrix Project". This project focuses particularly on the plight of adolescents with disability, who face a situation where they tend to become isolated from their peers and are in a difficult position to build new friendships. To serve their needs, we are developing a tele-gaming environment that "equalizes" player access across different levels of ability, so that a severely handicapped adolescent may play on an equal footing as an able-bodied friend. Our project seeks to create remotely controlled robots that must work together in a common, physically real environment to achieve group goals. Each adolescent will control his or her own robot, an semi-independent webcam and will have access to group chat either through text or voice or a combination of these. Different robots will have different functionality, however. For this project we are still building partnerships and doing conceptual design.

A third installation project on which we are working addresses the issue of climate change and environmental responsibility. Here our aim is to develop an installation that can be taken to the urban public and which will sensitize participants not only to the issues of the environment but do so in a manner that is informed by an awareness of the inequities in urban life and how different elements of the community may learn to find common ground in addressing these issues. The project bears the title "Voices of Transition".

The Hidden Magician - A Resonant Installation for Children with Cerebral Palsy

Early in 2008, the Canada Research Chair in Cognitive Geomatics, in partnership with Bloorview Kids Rehab (BKR), the Institut de réadaptation en déficience physique du Québec (IRDPQ) and Studio BourbeauVoiceDynamics, began to work on the creation of a "resonant installation" addressing the needs of children with cerebral palsy and other motor deficits. Under the title "The Hidden Magician", this broad collaborative effort seeks to develop a participative, immersive installation in which children with cerebral and motor deficits can establish a different relationship with their immediate environment and feel more empowered and recognized for who they are.

Like our other installation initiatives, the approach adopted is to develop an installation design through a broad consultative process that includes researchers, artists, engineers, hospital administration staff, clinicians, students, parents, and the children themselves. Installations must address the needs of the children in ways that are conducive to enhancing their physical and emotional states of being, and yet also generate powerful experiences that are aesthetically interesting and are challenging, even transforming. We use new media technologies including surround projections, gesture recognition interfaces, spatialized sound and tactile environments, combined with engineering skills to develop specialized interfaces that provide enhanced environmental responsiveness for these children.

The project embraces a variety of research areas, from issues about design methodologies, questions concerning the impacts of immersive and participative experiences on children struggling with issues of growth and identity, and efforts to develop measurement and evaluation tools that can better characterize the effectiveness of these installations.

The design concept is still in its early stages. The overall concept has been presented to a broad cross-section of individuals - researchers, artists, clinicians and administrators where it has elicited a great deal of interest and support - both at Bloorview Kids Rehab in Toronto and the IRDPQ in Quebec City. The work is now moving forward into a second stage, focussed on the development of a series of workshops with this diverse clientèle that will feed the design process. Workshops involve a combination of physical activities that aim to allow participants to "think with their bodies" rather than "staying in their heads", and brainstorming and sharing exercises that explore design values and principles. We use dancers, clowns and other specialists in movement to facilitate these exercises.

It is expected that the installation, when completed, will be able to "go on tour" to other interested locations (hospitals, clinics, schools, etc.), and that it will serve as much to sensitize a broader public to the unique qualities of these children as it will enable both the children themselves and their caregivers to rethink their perceptions of who they are.