Research Nirvana?

Have I died and gone to researcher heaven? The set of projects on which I and my collaborators are currently working feels very much like that could be the case. Here is a brief overview :

1. The Winter Project : This project is the brain child of my colleague, Dr. Ernesto Morales. Essentially, we are working in collaboration with the city council for Quebec City and a range of organisations for people with disability, to redesign city policies, including snow removal practices, to make the city more "disability friendly" during winter conditions. Early results from the project include a redesign of pavement ramps to include a drainage system allowing snow to be melted and removed - pavement ramps are of key importance for people in wheelchairs and are often the site of an accumulation of snow and ice ; the development of a wheelchair-adapted snow-mobile device ; and discussions concerning changes to snow removal practices.

2. Pro(x)thèse : This project, under my oversight, seeks to create an interactive smart garment that is embedded within an active, immersive environment, to facilitate the exploration of self image in relation to sensuality and sexuality and disability. A prototype poncho has been developed which incorporates 12 pressure sensors, interface software has been completed, and photography of people with disability has been acquired. The project includes collaborators Dr. Frédérique Courtois, Dr. Ernesto Morales and Ms. Nancy Dubé.

3. Sex toys : Another collaborative project with Dr. Ernesto Morales, in this project we are designing aids that can be combined with existing sex toys, as well as developing new ones, that provide masturbation ability to people with disability who are often unable to use existing sex aids. The designs can be printed on a 3d printer, ensuring privacy and customizability.

4. ProBE : ProBE (Proximal Body Environments) seeks to develop a diagnostic tool for clinics to assess the presence of brain trauma, both mild and more severe, in patients. The project is predicated on the observation, tested across decades of research on obstacle avoidance strategies, that people who have even mild brain trauma move differently in space and in proximity to other objects (or people) than do healthy individuals. ProBE involves researchers from Quebec City (Drs Brad McFadyen, the leader of the project and Denis Laurendeau) and Toronto (Dr Karl Zabjek) as well as clinicians from both cities.

5. Participatory Opera : The crown of my ongoing research program, developing a massively participative one-act opera based on my own science fiction opus. The concept is to develop the opera itself in parallel with a range of tools that allow active participation by diverse publics in all stages of development as well as the final real-time performance of the opera - input concerning the music, the choreography and the story line itself. These tools include virtual and social environments and specially designed smart garments. The opera project, led by myself, involves musicians Jocelyn Kiss, Serge Lacasse and Sophie Stévance from the Department of Music at Laval University, Marie Louise Bourbeau, a lyrical singer in private practice, Dr Adel Elmaghraby, a computer scientist and dancer-researchers Erin Manning and Cora McLaren. Both myself and Dr. Manning are also fashion designers - we will also be developing the costumes for the opera.

On Designing Immersive Spaces for Rehabilitation Research

Over the course of the past ten years, my team and I have successfully designed and implemented (produced) more than a dozen immersive installations within a research focus. It is with the emergence of technologies for creating and working with immersive, interactive environments (virtual reality, augmented reality, sensor networks, wearable computing, smart environments, etc.), that the design of immersive spaces has become a viable process. Hence working with such spaces and installations is still very new. Artists have been creating interactive experiences using these technologies for a little more than a decade, while the use of these environments in research has been mostly confined to virtual reality and related technologies. The convergence of immersive technologies from disparate fields into a coherent and powerful set of tools for immersive design is no older than a few years, and, of course, just as the technologies are still evolving rapidly, research based on their use is only beginning to develop.

Research that embraces immersive design requires new methodologies and these, too, are still very experimental. Immersive spaces are different from conventional experimental designs. Immersive environments are increasingly multimodal - that is, visual and auditory and tactile or haptic. They are "ecological" in the sense of being global, holistic and systemic - they engage all of a person, not just targeted aspects. Furthermore, they often generate surprises - this is a consequence of their ecological nature.

Our experience has demonstrated that the creation of these environments demands a three phase process :

CONCEPTION/DESIGN

I

I

DEVELOPMENT/PRODUCTION

I

I

EXPERIMENTATION/EVALUATION

This might look like the classic life cycle of any project, but each of these phases is substantial - these are complex projects to organize. Let us examine just one of our projects to get a better sense of what is involved. I like to call this project "Virtuoso" - all of our installation projects have a distinct title. This project draws upon another key principle we have identified over the course of our development work - immersive installations that have a research (scientific) focus should serve a specific and well-identified need. This is distinct from artistic installations which may or may not serve such a need. "Virtuoso" is being developed to provide an engaging experience to adolescents with motor impairments, who experience both growing isolation and a tendency towards depression as they leave behind their childhood friends. By providing these adolescents with a virtual, online, and shared experience of architectural design within a virtual world environment, we hope not only to break the cycle of isolation and depression but potentially offer also an opportunity to develop an interest that might lead to employment.

The Design Phase of this project lasted over four years. Although this was longer than many, it is by no means atypical of the collection of installations we developed. The Design work actually consisted of two more intensive periods, each lasting from 4 to 6 months, separated by an interval during which the project was shelved. The reasons for the break were multiple, but experience shows that such breaks may be a useful part of the project development cycle (although they need not be as long as in this particular case!). Lack of resources, funding and difficulties developing a mature concept were among the reasons, and possibly also our lack of experience developing and financing complex projects. During the first intensive planning period, we organized a series of meetings with both scientists and clinicians, and agreed to develop a kind of video game experience that would include a social networking dimension as well as some form of cooperative manipulation. However, the exact nature of the video game itself never became clear.

Over the fallow period, the team pursued other projects, including several projects that harnessed the virtual world environment called "Second Life". During this work, the existence of several simulators inspired by the "Second Life" environment were identified and explored. One of these, a simulator called "OpenSim", seemed particularly interesting because it shares with "Second Life" the same simplicity of operation. The arrival of a new researcher led to renewed interest in the "Virtuoso" project, and this time the possibility of using OpenSim as a framing environment for the project gave a clear direction for development. The second period of intensive work on design was therefore able to finalize a design concept - we would lead the adolescents through the process of designing and constructing architectural projects using the virtual world environment. Furthermore, the design concept included not just the design of the immersive environment itself, but also details about the experimental protocols that would need to be followed in the third phase of the project. Indeed, we would evaluate the effect on users of the environment of the design process, compared to a control group that would play a video game.

The Production Phase involved the hiring of a summer student to do the development work. The OpenSim simulation was acquired and the necessary procedures for creating, storing and running a variety of virtual landscapes were studied and adopted. A thematic context for the project was created, pedagogical tools for guiding the young people through the process of virtual construction, and different ways for encouraging collaboration and sharing results were discussed and adopted. The summer student hired to develop the environment is an architectural student with an interest in virtual environments. The development phase is expected to last 3 to 4 months.

The Evaluation Phase is expected to take place over the course of another 4 to 5 months. This phase is, of course, completely different than the development phase - it requires very different kinds of expertise. Indeed, each of the three phases requires different mixes of expertise - this is one of the challenges posed by projects of this kind. In Phase Three, what is required are experts on experimental protocols and data analysis. Technical support for the environment itself must also be in place, and it may be necessary to make adjustments to the design of the immersive environment as a result of challenges in addressing the evaluation phase.

A brief analysis of the dozen or so projects that have been developed using this approach shows that the average duration of the Design Phase was 24 months, of the Production Phase was 10 months, and of the Testing Phase was 9 months. At least 5 of the projects were characterized by a two-step design phase with a fallow period between the two more intense design periods. From initial steps to completion took, on average, 4 years. Some of this duration is almost certainly due to working with limited monetary and human resources, but other factors included the fact that we had to invent research methodology as we progressed, that the projects require different mixes of expertise at different phases, and that creation, implementation and testing of these environments is necessarily complex and demanding work. Perhaps with the benefit of experience, it will be possible to limit the time span to less than 3 years. This is important, since most grants are awarded for three year periods - projects which extend significantly beyond three years are therefore extremely difficult to bring to term in an academic environment. Indeed the early projects in our slate of initiatives were actually taken through only to the end of Phase Two, a result of the need to provide some constraints before procedures and research methodologies became clearer. (Nine of the dozen projects completed or underway are shown in the table below. The projects which are excluded from this list are those which have not yet progressed beyond the Design Phase.)

#	Project	Phase	Duration (mo.)	Expertise
1	Ariadne Emerging	Design	24	Performance designer, choreographer, researcher
		Production	08	Singer, dancer/choreographer, video production team
		Documentation	04	Video production team, researcher, designer
-----
2	Incarnatus	Design	08	Performance designer, singer, theatre technician, programmer, researcher
		Production	08	Singer, theatre technician, programmer, researcher
-----
3	Virtualities in Dusseldorf	Design	24	Designer, researcher
		Production	06	Researcher, designers, programmer/technician, virtual content creators, museum staff, translator
-----
4	Augmented Reality for Bloorview Kids Rehab	Design	08	Researchers, designers, programmer/technician, clinicians
		Production	04	Researcher, designer, programmer/technician, hospital patients
-----
5	Ulysses : A Sound Geography	Design	24	Researchers, designer, composer
		Production	04	Researcher, designer, composer, programmer/technician
-----
6	EcoOracle	Design	24	Researchers, designer, programmer
		Production	06	Researcher, designer, programmer/technician
-----
7	EMIR Demos	Design	06	Researcher, designers/programmers
		Production	24	Researcher, programmers/technicians
-----
8	Pro(x)thèse	Design	36	Researchers, designer, composer
		Production	24	Researchers, sexologist, composer, programmer/technician, fashion engineer, photographers, artists, clinicians
		Experimentation	tbd	Researchers, programmer/technician, clinicians, sexologist
-----
9	Virtuoso	Design	48	Researchers, designers
		Production	04	Researchers, designers, architect, programmer/technician
		Experimentation	06	Researchers, designers, programmer/technician, data analyst
-----

It is worth noting that work of this nature may have been possible only within the context of extended funding such as within a Research Chair. The availability of significant funding levels every year over a seven year grant, obviating the need to reapply for funding every two to three years, made this effort viable. Now that procedures and methods are more well defined, such projects can be managed within shorter term funding arrangements, but the initial development work would have been very difficult to undertake within a standard academic grant environment.

EMIR Laboratory Now Functional

It has taken longer to get the lab completed and operational than expected, but this has finally been achieved. In conjunction with the efforts to complete the installation of the laboratory, we have developed five demos that showcase the potential of the laboratory to support diverse projects in rehabilitation and disability studies. These are as follows :

1. Fragment de vie (Fragmented Life) : Demo that provides different viewpoints of disability by offering different soundtracks for the same film delivered to wireless headphones (collaboration with CinéScène Inc.)
2. Vertiges (Vertigo) : Demo that provides an experience of vertigo by simulating walking across a narrow pathway suspended above Quebec City (collaboration with CinéScène Inc.)
3. Deuxième peau (Second Skin) : Demo that provides an observer a vicarious experience of disability. The movements of a person in a motion capture suit are transfered in real time to an avatar in a virtual kitchen, but attempts to interact with the kitchen and manipulate objects are restricted because the avatar experiences various types of motor impairment (collaboration with CinéScène Inc.)
4. Viscères (Visceres) : Demonstration of a body-based interface for virtual navigation of spaces (Google Street Map)
5. Meta-laboratoire (Meta-laboratory) : Demonstration of the use of the lab to support immersive experiments, in this case, a study of the effects of heminegligence on spatial orientation and judgement in the far field (collaboration with CinéScène Inc. and Dr. Julien Voisin)

In addition to the demos, a growing number of research projects are taking place that harness the possibilities that the lab offers. I have subdivided these projects into six categories : Bimodal Environments, Trimodal Environments, Movement-based Environments, Visceral Environments, Virtual Environments and Geographic Environments.

1. Bimodal Environments : Immersive environments that engage two major sense modalities

   Hemispheres (Hémisphères)

   Experimental study in planning stages of far-field effects of heminegligence (collaboration with Dr. Julien Voisin)

   Co-breather (Co-respirateur)

   Design and validation study underway to build and test four co-breathers for possible clinical applications. Co-breathers provide auditory-tactile immersion (collaboration with Ms. M.L. Bourbeau and C. Légaré)

2. Trimodal Environments : Immersive environments that engage three major sense modalities

   Living Wall (Mur vivant)

   Project in preparation, aimed at developing a playful immersive installation for waiting areas in clinics for children and adolescents that offers an engaging and absorbing environment that provides motor training opportunities (dexterity exercises) for individuals with fine motor impairments (collaboration with Dr. Ernesto Morales; projected Ph.D. thesis of Walid Baccari)

   Pro(x)thèse (Pro(x)thesis)

   Project underway in design phase, aimed at developing a clinical tool for allowing people with disability to explore sexual/sensual imagery and providing the means to track image choices over time. The tool involves the use of a touch-sensitive smart garment and an immersive visual environment, and we are commissioning photos by a professional photographer (collaboration with Dr. Ernesto Morales and Dr Frédérique Courtois).

   Auric Space (Espace aurique)

   Current in the planning stage, this project seeks to provide a training environment for people who have difficulties locating sounds in their immediate environment. We will use a bimodal environment (visual and auditory and haptic) to provide cues to and test for sound location (projected Ph.D. thesis of Afnen Arfaoui)

3. Movement-based Environments : Immersive environments that explore movement modalities

   Third Skin (Troisième peau)

   Project in planning stages that seeks to extend the work initiated in the project "Second Skin" to provide a variety of vicarious experiences of disability and ability

   Choreographic maps (Cartes chorégraphiques)

   Planned project that seeks to study how dance may contribute to emerging ideas about how children play (collaboration with Dr. Cora McLaren)

4. Visceral Environments : Immersive environments that explore actions rooted in the engagement of the body's visceral organs

   Visceres II (Viscères)

   Project in planning stages that aims to test the hypothesis that viscerally learned spaces are more fully understood and remembered than spaces learned by more traditional means

   OrienT (OrienT)

   Planned project that seeks to use a smart garment to help people who get easily disoriented to resituate themselves in their environments (collaboration with Dr. Claude Vincent)

5. Virtual Environments : Immersive virtual environments

   Virtuarch (Virtuarch)

   Project in design development stage that seeks to provide adolescents with disabilities who feel isolated and have a tendency towards depression, access to an environment and situation that engages them in the creative design of architectural spaces (collaboration with Dr. Ernesto Morales)

6. Geographic Environments : Immersive environments that encourage an appropriation of geographic space

   Multimodal Online Mapping Interface (Interface multimodal pour la cartographie en ligne)

   Project in planning stage that seeks to design and implement an on-line multimodal interface for a mapping application that draws on cognitive design principles (collaboration with Dr. Mir Mostafavi; projected Ph.D. thesis of Bilel Saadani)

These different projects seek to serve a variety of populations of people with impairments, including the deaf, the deafblind, the blind, those with low vision, people with either gross or fine motor impairments, people with attention deficits and people with intellectual impairments. Furthermore, different projects have different scientific goals and involve distinct methodologies. These include projects that are experimental or involve evaluation and assessment (Hemispheres, Visceres, etc.), projects that have more educational or pedagogical objectives (Second/Third Skin, etc.), projects that seek the development of assistive technologies or that are focused on design issues (OrienT, etc.), projects aimed at developing training environments (Auric Space, etc.), projects aimed at enhancing personal development among those struggling with issues of disability and impairment (Virtuarch, Pro(x)thesis, etc.), and projects that are far more exploratory in nature (Choreographic Maps, etc.).

Transformative Installations - Global perspective

Since the early development work on the Bloorview initiative (called at the time the "Hidden Magician" project), our efforts to develop a whole range of "transformative" or "resonant" installations has moved forward by leaps and bounds into several major initiatives. We are currently active in the development of a major "new generation environment" at Bloorview Kids Rehab that we are calling the "Living Walls Initiative". Within this project, we are developing a highly interactive, one might say "reactive" wall mural that responds to the presence of children with disability in many different ways. Our goal is to change the way the children understand their relationship to their surrounding space.

Children with disability struggle within environments which are highly disabling. Indeed, we call the children "disabled" but we might more usefully call the environments within which they (and we) function disabling environments. As a result, these children often feel like they are a burden on others, that they have to struggle with the environment, that they are what's "wrong". By developing new environments that are much more responsive to a variety of forms and levels of disability, we aim to challenge this understanding, to offer these children an insight into other possible relations they might have to the spaces that surround them and with which they engage.


An early conceptualization of the Living Walls Initiative
The Living Walls initiative is the first major attempt to do this. The overall concept is to develop a large wall mural (we're thinking 8 feet high by 20 feet long) that is made up of motorized elements that will respond, via appropriately designed interfaces, to children with various forms of disability. The mural will depict a scene of relevance to the hospital - a depiction of the ravine that drops away behind the hospital and which has already been incorporated in a number of ways into the design of the hospital building. This allows the children to be attuned to the presence of natural elements in the local environment of the hospital. We are designing into the mural elements which may change color and shape and hence depict the changing seasons. However, the main focus of the mural is to allow the children to interact with the scene and to make interesting changes to it. For example, we are building in animal figures that may hide or emerge at different moments, when the mural senses a child in its proximity. By making some sort of movement, whether using a wheelchair or a gesture, children will be able to change several aspects of the mural - the intensity of water flow in the built-in waterfall, the shape and color of leaves in the trees, the overflight of planes, and so on.

The project is moving from its conceptual design phase into the development of early prototypes that will be used to test the implementation before this is fully fleshed out. At the same time, funds are being sought, both from private donors and funding agencies, in support of the project. Many of the partnerships needed for its success are already in place.

A second "next generation environment" projet also aimed at helping children with disability has been named the "Ado-Matrix Project". This project focuses particularly on the plight of adolescents with disability, who face a situation where they tend to become isolated from their peers and are in a difficult position to build new friendships. To serve their needs, we are developing a tele-gaming environment that "equalizes" player access across different levels of ability, so that a severely handicapped adolescent may play on an equal footing as an able-bodied friend. Our project seeks to create remotely controlled robots that must work together in a common, physically real environment to achieve group goals. Each adolescent will control his or her own robot, an semi-independent webcam and will have access to group chat either through text or voice or a combination of these. Different robots will have different functionality, however. For this project we are still building partnerships and doing conceptual design.

A third installation project on which we are working addresses the issue of climate change and environmental responsibility. Here our aim is to develop an installation that can be taken to the urban public and which will sensitize participants not only to the issues of the environment but do so in a manner that is informed by an awareness of the inequities in urban life and how different elements of the community may learn to find common ground in addressing these issues. The project bears the title "Voices of Transition".

Perception and Spatial Representation - Deliverables from the First Mandate of the Chair

While the second seven year mandate of the Canada Research Chair on Cognitive Geomatics is focussed on the relationships between identity, body and space, the first mandate was concerned with understanding our mental representations of space, as derived from perception and mental imagery, and with the development of tools and software that put this knowledge to use. Application areas for this work included rehabilitation, the performing arts, navigation, landscape design, and database design.

Nested perceptions of the world

A variety of researchers have studied how our perceptions of the world are organized as a function of scale. Several schemas that describe scaled perceptions exist. One of the most interesting is that put forward by Dan Montello in 1993 (Scale and multiple psychologies of space). Montello's is interesting in part because it synthesizes the work of several researchers, but also because it critically examines the different approaches. Within Montello's framework, roughly four spaces exist at different scales :
(a) Figural space
(b) Vista space
(c) Environmental space
(d) Geographic space
Montello describes Figural space as being the space of drawings and maps, representations of the world. Vista space is conceived of as the region that can be viewed from a single location. Environmental space is defined as the region accessible via displacement or navigation. Geographic space is the space that is too large to be visited. A fifth space,
(e) Cosmic space,
covers spaces that are not accessible on the Earth.

Other categorisations of spaces also exist, some of them very useful. Hence several researchers focus on what are called "table top spaces" to describe the spaces in which objects can be picked up and manipulated (for example, Andrew Frank). In Montello's scheme, Table top space may be viewed as intermediate between Figural and Vista space. Another approach distinguishes between "within body" space, "body space" and "near body space" (Three spaces of Spatial Cognition by B. Tversky et al.). These are also spaces that are smaller than Vista space. Tversky also points out the cognitive importance of barriers within a space.


Figure 1 : Local displacement space for a household in Sillery, Quebec City

Along with my collaborator-postdoc Dr. Isabelle Reginster, we found that to apply these theoretical ideas to a real application, it was necessary to subdivide Montello's Environmental Space into two different scale spaces, what we called the Local Displacement Space (see Figure 1) and the Extended Displacement Space (see Figure 2). Local Displacement Space dealt with the part of Environmental Space that can be accessed by foot, while the Extended Displacement Space accommodated the region accessed by car. We applied a time limit for displacement as a means to characterize the size of these spaces, and used the three scales (Vista, Local Displacement and Extended Displacement) as spatial units within which information was aggregated to infer perceptions for different households - perceptions of how many municipal services were accessible, of access to schools, and perceptions of how much green areas were to be found within the local environment. We showed how to determine the Vista, LDS and EDS spaces from satellite imagery, and we used the aggregated statistics to explore the relationships between scale, perception and house prices (see the publication Reginster and Edwards, 2001, for details). We tracked, in particular, the location of barriers in the space, both perceptual and navigational barriers.

At the time this work was undertaken, the displacements were tracked using phone interviews carried out with a variety of households in an Origin-Destination survey that had been conducted by colleagues in the Département d'Aménagament of Laval University. These days, the survey could be carried out much more cheaply using a portable GPS unit with a data logger.


Figure 2 : Extended displacement space for the household in Sillery, Quebec City. Note that the extended displacement space consists of corridors around each road used, and that frequency of travel along the road reinforces the intensity of that part of the displacement space

Hence we were able to implement these theoretical constructions of embedded spaces in a study on scaled perceptions of the local environment from the point of view of members of a household.

Rooms and gateways

A second study, undertaken with Dr. Gerard Ligozat and later with his daughter, Anne Laure Ligozat, focused on the development of formal representations of perceived space, especially outdoor, natural spaces. In this project, we were interested in developing a formal (i.e. mathematical) representation of perceived space and in implementing this representation on a computer. We were particularly interested in the fact that, within exterior environments, one may move a certain distance and yet still conclude that one was within the same place as before the move. What determines when we conclude that a change has occurred in our location?

We determined that either the neighborhood had changed, or the order of landmarks on the horizon had change (the latter is called the "panorama" in technical terms). Therefore, we set about to characterise a space in terms of its neighborhoods and panoramas. This led to the idea of "perceptually stable zones" and "zones of transition", which one may metaphorically associate with "rooms" and "gateways". We found that all outdoor spaces could be reconfigured as a set of "rooms" and "gateways", making them analagous to interior spaces. Visual barriers act, within such a viewpoint, as metaphorical "walls". The set of rooms and gateways forms a kind of dual or alternate representation to the set of neighborhoods and panoramas, and we found that one could infer the one from the other and vice versa, to some extent. This work was published in two papers (Ligozat and Edwards, 1999; Edwards and Ligozat, 2004).


Figure 3 : A fictional landscape created within the software prototype PERSEUS

A software prototype called PERSEUS was developped to showcase the model. The prototype divides space up in terms of what are often called "viewsheds", that is, areas of intervisibility, and then subdivides these areas in terms of panoramas, defined as regions in which the order of landmarks on the horizon is stable. To some extent, therefore, the maps produced depend on what objects are labelled as landmarks by the user. We have been able to generate maps of stable perceptual zones for both fictitious landscapes, but also for a study of the Plains of Abraham, the large park within Quebec City.


Figure 4 : Map of the perceptually stable zones for the three landmarks within the fictional landscape used by the PERSEUS prototype

Understanding near-body spaces as a function of disability

The work by Reginster and Edwards, and that by Edwards, Ligozat and Ligozat, constitute new material representations of space (i.e. maps) that incorporate understanding of our mental representations of space as derived from modern cognitive psychology. However, they handle vista spaces and larger regions.

In work aimed at supporting the movement of disabled users in the landscape, another postdoc, Pierre-Emmanual Michon, and a full time Research Professional, David Duguay, and I, developed a new kind of representation, this time a special kind of 3D map, aimed at representing near-body spaces.


Figure 5 : An part of the research centre in rehab in Quebec City, as portrayed within the CADMUS prototype software

For this work we drew on the concept of affordances as proposed by James J. Gibson in the 1950s. The idea is that objects permit certain kinds of functional use but not others - they are said to "afford" such uses. Hence a chair affords that one can sit on it, but not that one can eat it (unless it were a chocolate chair!). We implemented the concept of affordances in a 3D database (see Edwards, 2006, for a description of this process). Hence in our database, doors may afford opening via a "door handle" or a "push button" as in some hospitals. In addition, we matched the affordances of such objects to the physical capabilities of the user. Hence a "door handle" requires the ability to twist as well as a certain level of physical strength, whereas the push button requires a much lower level of strength. Using the combination of affordances and user profiling, we were able to generate maps that showed areas of different accessibility levels as a function of a users physical profile. In a second version of the prototype, which is called CADMUS, we also incorporated mental competencies as well as physical competencies.


Figure 6 : The same region as shown in Figure 5, but color coded in terms of accessibility for a given class of disabled user. Red means access is difficult, green that access is easy

Image schemas and performance design

The work on affordances and user profiling, although it led to the creation of a new kind of 3D map, could also be used to evaluate the effectiveness of particular environmental or building designs for different handicapped profiles. Likewise, the rooms-and-gateways representation of outdoor spaces could be used not only to understand an outdoor space, but also as a support for redesigning such a space.

Our interest in designing spaces extended into another arena, that of performance design (i.e. for the performing arts). Here, the understanding of space requires a connection to their emotional impact and not just their perceptual impact. A useful tool for capturing the relationship between space and emotion is found in image schemata.

Image schemata were unearthed by philosopher Mark Johnson in the early 1980s (see his book The Body in The Mind for a clear exposition of the concept), and their study and use matured under Johnson's collaborator with the linguist, George Lakoff (see Women, Fire and Dangerous Things : What Categories Reveal About the Mind) for this later work. They are basic images that are found to be common across most languages, and that are used to talk about abstract ideas. Common examples of image schemata include CONTAINER, PATH, CYCLE, LINK, ENABLEMENT, FORCE, BLOCKAGE, SPLIT, and COLLECTION. Later studies have found that image schemata also turn up in most forms of expression, including the visual arts, music, gesture and dance, sculpture, and cinema - as such, they constitute a powerful means of coordinating design that must serve our many different senses.

Lakoff and Johnson developed a theory that image schemata are formed during early childhood by a process of binding embodied actions to word concepts. Within this framework, therefore, image schemata are linked to emotional responses, albeit in a manner that is itself rather complex and likely to vary from one individual to another. Nevertheless, artistic design uses image schemata, often unconsciously and intuitively.

In a study callaborative study carried out in 2005, Marie Louise Bourbeau, a mezzo-soprano soloist, and I used image schemata to design a performance of Claudio Monteverdi's opera fragment, Arianna. We showed that image schemata, when used consciously and explicitly, constitute a powerful tool for performance design, for delivering an experience to an audience. In a sense, image schemata allow us to design experiences directly rather than just their progenitors, the objects or events that produce experiences. More details of this work can be found in our paper on the subject (Edwards and Bourbeau, 2005), and the results of the design can be viewed on youTube.


Ariadne Emerging Video Clip

Cognitive Design of Assistive Technologies

In addition to the work on designing maps and spaces, we have been interesting in designing tools that facilitate the navigation and movement within spaces, not just mapping tools. The first significant effort in this direction has been undertaken by a Ph.D. student, Mr. Reda Yaagoubi. The idea is to use what we know about how people represent spaces mentally to assist in the navigation of the blind.

Modern geomatics technology that is useful in this context is, of course, the GPS receiver. However, all current GPS devices rely on the visualization of a map to provide what one might call situational awareness. Instructions on where to go might be provided by a computerized voice, but the devices rely on the visual availability of a map to let people know where they are and the location of objects and landmarks in their immediate environment. Without such landmarks, directional instructions are useless. For the blind, this is a problem - situation awareness is lost many times over the course of a day, and direction that are given without situation awareness may be less than useless.

We are therefore using information about how people, in particular people without recourse to sight, store and maintain mental representations of their immediate surroundings. The tool we are developing seeks to use natural strategies to help individuals update their local mental representation in such a way that a GPS directional instructions may become meaningful. The design process is quite challenging, because it requires that one understands both the cognitive processes and representations in operation and that these inform the engineering and technical principles that must be used to develop a particular form of technology. A paper has been submitted to a journal describing this work (Yaagoubi and Edwards, 2007).

We also undertook behavioral experiments that tested the ability of blind subjects to understand and manipulate mental representations of space. A paper has been submitted describing this work as well (Eardley, A., G. Edwards, F. Malouin, P.-E. Michon and J. Kennedy, 2007). We found that a certain group of people without sight (those born with sight but who lost it a year or so after birth) actually perform better than the sighted at certain tasks involving spatial reasoning on their mental representations. Those born blind from birth with no neurological complications had similar competency as the sighted. Only those born blind with neurological complications performed significantly worse than the sighted on these manipulation tasks.

Resonant Installations - Designing the Immersive Experience for Maximum Impact

The work on image schemata in support of performance design was aimed at connecting performance spaces to their emotional impact. Although image schemata were found to be a powerful tool for design, their connection to emotional response was weaker than we would have liked. In an attempt to develop a stronger connection to emotional response, we (Marie Louise Bourbeau and myself) investigated the use of devices that enhance our awareness of our own bodies during performance.

We began this work by focusing on the act of breathing, perhaps the most important aspect of body-awareness because it is the source of ongoing life. So many studies and body-training disciplines are all based on the act of taking a breath, including all of modern athletics, but also all the performance arts. Marie Louise Bourbeau is a specialist in breath training for singers and dancers, so this choice made double sense as a first target.

We developed an installation, called Incarnatus, that sought to create a new relationship between the participating audience and classical lyrical music. Using one of Mahler's lieder (the Schildwache Nachtlied, or Soldier's Nightsong), sung in German and based on a traditional folk verse, we developed an instatallation that culminated in the use of a device we call the "co-breather". This is a cushion that breathes at the same time as the singer, while the participant is listening to the music sung by the same singer. Far from being experienced as an imposition, participants adapted their breathing to that provided by the co-breather within seconds, and many participants reported a near-ecstatic connection with the music, completely unexpected.


Incarnatus Video clip

Following the public presentation of the Incarnatus installation, we determined that the co-breather creates a rather paradoxical state that includes a heightened state of body awareness combined with a "loss of self" (as well as a stronger sense of identification with the music), a dropping of the barriers that define and protect the self.

We are now in the process of developing new installations that put such "body opening experiences" first, and follow it up with other body exploration processes as well as an integration and a closure phase. The installation that generates such a sequencing of experiences we call a "Resonant Installation", and several examples are presently under development (for more details about Resonant Installations, see the blogsite ResonantInstallations). These form the heart of the second mandate of the Canada Research Chair in Cognitive Geomatics.

Virtual and Mixed Reality Environments - Embodiement and Identity

Finally, during the final year of the first mandate of the Canada Research Chair in Cognitive Geomatics, we have begun a systematic investigation of the relationship between virtual worlds and embodied experience. This is another paradoxical study. At first site, virtual worlds would appear to be a perfect example of a "disembodied experience". This intuitive evaluation that many of us form at a distance is almost completely false, as it turns out. Virtual worlds generate highly embodied experiences. But our understanding of what embodiment is has changed, as a result of this work.

This work is supported by the ongoing discussions undertaken by the Embodied Research Group (ERG), an active group of researchers that meets every week online (on Second Life) to discuss our understanding of embodiment. For more information about the work of the group and the results of the discussions, see their blogsite, EmbodiedResearch. Within this context, it has become apparent that embodiment is "performative" rather than simply "physical". As a result, it is possible to develop a sense of embodiment within virtual worlds, even though our physical bodies are not directly engaged.

This results in a profound rethinking of what it means to be embodied, and even what constitutes "body awareness". In a recent discussion by the Embodied Research Group, it was noted that within Second Life we may actually develop a kind of "reflexive muscle" for an embodied functionality available within Second Life but not in our physical environments. An example is the use of a virtual camera to look at objects that would normally be "out of sight" of one's body. When the reflex develops, we find ourselves "trying" to use it in our physical bodies, and frustrated that we cannot. We have developed our virtual camera as a kind of phantom limb - the neurons still activate it, but there is no follow through to a muscular action.

Within virtual worlds, our identity is also multiplied, resulting in another rethinking of what it means to be a person. The multiplication of identity and the spatial redistribution of our sense of embodiment are two startling mutations in our sense of self that derive from an engaged presence within virtual worlds of such complexity.

This is of special interest for so-called "mixed reality" environments, that is, environments that combine parts of virtual worlds within our physical, material experience. Gaining understanding of the impacts of such mixed reality environments on our sense of self and our ability to act in the world has become a major source of study for the Canada Research Chair in Cogntive Geomatics, and this work will also form part of the second term mandate. A paper on this was recently presented (in French) at the Geocongrès International in Quebec City in October 2007.


Conclusions



Table I : Representations (R) and Tools (T) developed during the first seven year term of the Canada Research Chair in Cognitive Geomatics

Table I shows a summary of the innovations of the first seven year term of the Canada Research Chair in Cognitive Geomatics, in the area of Perceptions and Representations. At the scale of the body, we developed tools we call Resonant Installations. At the near body scale, we developed tools that use image schemata and representations based on the theory of affordances. At the scale of vista space, we developed a map representation that views all spaces, indoors and outdoors, in terms of conceptual rooms, barriers and gateways. At the scale of environmental spaces, we developed map representations in terms of local and extended displacement spaces that can be derived from satellite imagery (e.g. Google Earth). Together, these constitute an "end-to-end" collection of tools for representating, handling and manipulating the full range of perceptual spaces, and informing design processes focussed on the spaces themselves, their map representations, and tools and methodologies that facilitate their understanding. Applications presented include aids for the disabled, for the performing arts and museology, and aids for architecture and landscape design.