Understanding the Physics of the Mind: a proposal for a Perceptual Science Initiative (Fall 2010)

Abstract. Today, perceptual science is standing at the threshold of producing paradigm-shifting discoveries and transformative knowledge, with implications in domains of critical national interest, such as education, medicine, health sciences, bio-engineering, computer science, robotics, economics, among others. Perceptual science has the promise to be both significant and influential, in part because advances in its understanding has often been the gateway for new technologies (e.g., brain machine interfaces) but also because new ideas in perception often resonate with other fields like computer vision, neuroscience and robotics. The field has reached a level of maturity where fundamental research and extraordinary resources will revolutionize how we understand our interface with the outside world. Accelerated progress in the field will require fuller integration between all related areas of investigation, including the creation of laboratories and funding mechanisms specifically designed to foster integrated research.

Fundamental questions

The role of perception is to decode external states that the organism finds itself in so that this information can inform other cognitive processes such as decision-making. Being at the front end of the sciences of the mind (social neuroscience, decision making, science of learning), perceptual science is discovering how sensory systems process vast quantities of data with remarkable spatial and temporal precision. Current directions in perceptual science are towards integrating mathematical formalism and resources from rapidly advancing core disciplines (engineering, computer science, robotics, network applications) to engineer and model the data processing of the mind. Perceptual science has reached a level of maturity where fundamental research and extraordinary resources will revolutionize how we understand our interface with the outside world and how we rehabilitate the interface when it goes awry. The field possesses the talents and the technology to produce transformative knowledge in grand challenges of the study of the mind:

  • An Atlas of the Human Brain. A grand initiative on building an atlas of the human brain will foster a mapping at the level of the circuits, systems and functions that will have an unprecedented impact on the development of sensory-substitution devices, brain-machine interfaces and motor rehabilitation.
  • A Comparative Perception Approach. The comparative study of perceptual systems and their neural correlates across closely related species and across phylogenetic branches will yield unparalleled discoveries into the neural mechanisms of cognitive processes and the functional structure of the brain. Understanding the development and evolutionary consequence of perceptual apparatuses across species may begin to help us understand the underpinnings of cognition in complex environments. One implication of this comparative approach is that we can attain a better understanding of how the human mind changes if we modify our perceptual bandwidth. Certain developmental disorders, such as Autism Spectrum Disorder, may stand to benefit from this knowledge.
  • An Integrated Perceptual Intelligence program. Artificial vision systems have reached an unprecedented degree of maturity, with computer vision applications appearing in consumer products (e.g., digital cameras, online search engines, etc.). As both natural and artificial perceptual systems try to accomplish a common set of goals, parallel progress and interactions between these two fields will open the door to new artificial systems and a better understanding of human perception.

Perceptual science is standing at the threshold of producing paradigm-shifting discoveries and transformative knowledge, with implications in domains of critical national interest, such as education, medicine, health sciences, bio-engineering, computer science, robotics, economics, among others. Today, perceptual science is poised at an ideal moment to transform how we understand the perceptual and conceptual abilities of the human brain.

Context

The 20th century witnessed a dramatic understanding of the underlying principles of perception. Most of the progress made during the past century relied on stimuli that could be separated into simple and quantifiable components, with measurable perceptual effects. Despite the great advances in our understanding of visual perception, the use of simple stimuli do not scale up to explain the complexity of the real-world situations with which perceptual systems are faced. At present, technological advances, access to large quantities of visual data that came with the explosion of the Internet, and a better understanding of real world stimuli, open the door to new fundamental advances.

As a result of new non-invasive brain-imaging technologies, our understanding of the functional aspects of the brain has dramatically improved is the last decade. Two of the most important findings of modern human visual neuroscience is the existence of specific brain regions dedicated to low, medium and high level visual processing, dedicated to specific objects (such as faces) and the discovery of the spatial organization of functional areas in the brain. For instance, the ability to synthesize topographic maps has been found in multiple regions of the human brain [1]. Unraveling the spatial code present in visual, tactile, auditory and cognitive maps will provide a structural basis for descriptions of functional organization of brain tissues. Moreover, a mapping of the brain at the level of the circuits, systems and functions will have an unprecedented impact on the development of sensory-substitution devices, brain- machine interfaces, and motor rehabilitation.

Differences in cognition/mind across species might be due to the way they perceive and decode the world. If we were to understand what it is to “see through sounds” like a bat, or feel through the whiskers of mices, this could open the door to understand how would the human mind change if it was to modify its perceptual bandwidth via neural prosthesis that try to enhance human perception or to compensate for lost of perceptual capabilities in people with disabilities. Progress in this field would have unprecedented impact on many technologies daily used by people.

Despite the recent progress on brain imaging, current research on the neural basis of human perception is largely restricted to methods that still leave us a long way from the high temporal and spatial resolution needed to understand the actual code the brain uses for perception. Research in technologies for non- invasive functional brain analysis, together with a strong integration between human research and research in non-human primates will be necessary to speed up the understanding of how information is coded at the neuronal level by the brain. Current animals’ models (bird, rodents) of perception and action provide data at a level of cortical analysis not accessible using functional MRI in human. Bootstrapping the development of neuro-technologies will be a fundamental component of the functional mapping of the brain. Towards this goal, we need to provide the structure and funds to develop comparative perceptual science research.

Capability to be created

In order to reach its potential perceptual science needs to bring together teams with deep expertise in multiple areas (psychophysics, signal processing, cognitive science, neuroscience, computer science, robotics, among others). For instance, still today, many physiologists work in relative isolation from theorists who have developed insights into information coding in the brain, and methods for exploring information coding that go beyond dreaming up possible features a neuron might be looking for and presenting such features to see what causes a neuron to fire. Currently, academic and publication outlets have created walls between different areas of science and engineering that can be problematic for making progress in interdisciplinary fields such as perceptual science. It is imperative that new scientists can safely cross over the disciplines and methods, without fear. We need to offer mechanisms to encourage and protect interdisciplinary researchers.

Further progress in perceptual science will require fuller integration between all related areas of investigation. This will benefit from a recognized status given to interdisciplinary areas of research: some successful examples are the new emerging fields of “computational perception,” “computational photography,” and “neuro-economics.” More are needed. This helps the community to understand and appreciate the benefits that come from the interaction among two or more fields, and gives a valid title to new and potentially transformative research program. Institutions should call for the creation of laboratories and funding mechanisms specifically designed to foster “grand challenges” in perceptual science. For instance, encouraging the grand initiative of building the Atlas of the Brain, in human and across species.

It would also be natural to call for the creation of laboratories and funding mechanisms specifically designed to foster integrated research, linking theoretical investigation at the level of information coding principles, psychophysical studies, neurophysiological investigation in primates, and functional imaging in humans.

Faster progress could be made if there was an effort among different fields to create and share databases. The access to standardized, and shared databases has already made a big impact in order fields such as language processing and computer vision. The benefits from having open access to large corpuses of data gives the basis for quantifiable progress and enables the wider researcher community to make new findings without expending valuable time and resources.

Many breakthroughs can emerge from a deeper integration of areas of research and resources. For instance, breakthroughs in human perception could lead to machines that correctly re-construct the three-dimensional structure of the external world, that integrate multiple sensory modalities (tactile, visual, and auditory), and that are able to recognize objects in the field of view and the relationships among them. Better understanding of the neural basis of perception could lead to new prosthetic devices that could be interfaced with the brains of the blind and/or deaf and that would provide them with the information that would ordinarily be conveyed by the missing modality.

Implications

Perceptual science has the promise to be both significant and influential, in part because advances in its understanding has often been the gateway for new technologies (e.g., brain machine interfaces) but also because new ideas in perception often resonate with other fields like computer vision, neuroscience and robotics.

Furthermore, advances in the understanding of the spatio-temporal processing of early visual and auditory areas has led to the development of new products which have dramatically transformed modern-day society, such as visual acuity test, prostheses for the visually impaired, cochlear implants, color and 3D displays, new visualization techniques, advances in computer graphics, intelligent digital cameras, etc. Better understanding of higher-level cognitive functions has also been a driving force in the development of voice recognition systems, education, computer vision for safety and surveillance, computational visual design, understanding of three-dimensional scenes for surveillance, assistive systems and the prediction of social behaviors, as well as medical image interpretation, and the study of new approaches to treat autism and dyslexia.

As an example of integration between perceptual science, computer vision and material engineering, Adelson’s group at MIT [2] developed recently a new tactile sensing technology able to sense the shape of the surface it touches with extremely high spatial resolution and with compliance similar to that of a human fingertip. Such a technology has direct applications for medical, robotic and industrial domains where the mechanical properties of material or tissues touched needs to be recovered with fine detail and with minimal invasion (e.g., robotic applications with sensitive gripping surfaces, surgery, brain machine tactile interface, wearable computing). As tactile, visual, and auditory sensors become increasingly ubiquitous, they will enable us to transcend current human limitations. Another example of integrative research between natural language, computer vision and human perception is the visual dictionary of 80.000.000 images from Torralba’s group at MIT [3]. This project produced a large database of visual information organized according to 50.000 concepts extracted from the Wordnet dictionary developed by linguists at Princeton University. This database allows the study of the relationships between visual and linguistic concepts and provides the grounds to study new computer vision algorithms dealing with large quantities of visual concepts as humans do. Another example of interdisciplinary research is the field of optogenetics, [4], which combines optical and genetic techniques to develop new methods to study the neural mechanisms of the brain. This technique has potential for many areas in neuroscience: for example, this method can explore how neural activity relates to blood flow, providing a bridge between systems neuroscience in animal models and functional imaging of humans.

The principles of perception are deeply integrated into the empirical and theoretical foundations of the sciences of the mind. With more synergetic interactions between perceptual science and other sciences of the mind, we should see new advances in both fundamental and technical areas of psychological and neuroscience investigation. An integration of resources and researchers towards the grand questions of perceptual science, within the larger initiative of the Science of the Mind (Understanding the Mechanisms of the Mind through an Integrated Science of the Mind Initiative, An SBE 2020 white paper by Jay McClelland and collaborators), will produce paradigm-shifting progress, with concrete implications in many domains of national interests.

References

  • [1] B. A. Wandell, S.O. Dumoulin and A. A. Brewer (2007). Visual Field Maps in Human Cortex. Neuron, V. 56 , p. 366-383.
  • [2] M.K. Johnson and E.H. Adelson. (2009). Retrographic sensing for the measurement of surface texture and shape. Computer Vision and Pattern Recognition.
  • [3] A. Torralba, R. Fergus, and W.T. Freeman. (2008). 80 million tiny images: a large dataset for non- parametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 30, pp. 1958-1971, 2008.
  • [4]. J. H. Lee, R. Durand, V. Gradinaru, F. Zhang, I. Goshen, D. Kim, L. E. Fenno, C. Ramakrishnan and K. Deisseroth. (2010). Global and local fMRI signals driven by neurons defined optogenetically by type and wiring. Nature.