Nettime mailing list archives

<nettime> Artificial Perception as Reality Check
twsherma on Fri, 10 Jan 2003 12:49:13 +0100 (CET)

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

<nettime> Artificial Perception as Reality Check

Artificial Perception as Reality Check
Thinking About MIT's Tangible Bits

By Tom Sherman

[this text was commissioned for and previously published in Horizon Zero,
the webzine of the Banff Centre:  http://www.horizonzero.ca]

"Tangible Bits is an attempt to bridge the gap between cyberspace and the
physical environment by making digital information (bits) tangible."

    --Hiroshi Ishii, from his Website:

Tangible Media

Hiroshi Ishii started the Tangible Media research group and their ongoing
Tangible Bits project in 1995, when he joined MIT's Media Laboratory
[http://www.media.mit.edu/] as a professor of Media Arts and Sciences.
Ishii relocated from Japan's NTT Human Interface Laboratories
[http://www.ntt.co.jp/index_e.html] in Kyoto, where he had made his mark
in Human Computer Interaction (HCI) and Computer-Supported Cooperative
Work (CSCW) in the early 1990s.

I met Ishii when I visited his lab in 1997 while conducting research for
Ars Electronica's FleshFactor [www.aec.at/fleshfactor/index.html] symposia
and exhibitions. At the time, there was (as there still is) a lot of new
renaissance hype coming out of MIT. But despite what anyone may have heard
to the contrary, engineering still rules at MIT. To Ishii's credit, he
doesn't pretend to be an artist. His research focus has always been on the
design of seamless interfaces between humans, digital information, and the
physical environment. Ishii is an engineer interested in perception. That
said, his use of written language to over-state the creative dimensions of
Tangible Bits research sometimes verges on poetic hyperbole.

Seamless Surfaces

By 1997 there was a steady stream of rhetorically sophisticated
"literature" pouring out of the Tangible Media lab. The story begins with
the shortcomings of computer interfaces to date. Graphic user interfaces
(GUIs) - screens and keyboards and mice - prohibit people from using their
higher, natural skills for sensing and interacting with their physical
environments. Computers are currently anti-body. You can't touch the data
you are working with, or use your body to move around it. But some day
computing will be more accommodating to multiple intelligences
[http://www.accelerated-learning.net/multiple.htm] - including
bodily/kinesthetic and musical/rhythmic intelligences. Tangible Bits seeks
"to build upon these skills by giving physical form to digital
information, seamlessly coupling the dual worlds of bits and atoms." The
idea of a "seamless" integration of digital language and devices into the
physical domain is a central theme.

One strategy for eliminating the "frame" separating computing from the
rest of world is to spread digital information into the background.
Ideally, hands-on, foreground interactions with computers will be informed
by information lingering at the periphery of the user's senses. The
Tangible Bits philosophy is anchored on the gestalt theory of Max
Wertheimer. [http://www.enabling.org/ia/gestalt/gerhards/wert1.html] In
all learning environments, context is important. According to the
pervasive-computing scenario of Tangible Bits, in the near future we will
live surrounded by things such as "interactive surfaces, whereby walls,
desktops, ceilings, doors, windows, etc. become an active interface
between the physical and virtual worlds." The rooms we live and work in,
the cars we drive, the terrain, vegetation, and water we encounter, will
all eventually yield digital information. Ishii's group "is seeking ways
to turn each state of physical matter - not only solid matter, but also
liquids and gases within everyday architectural spaces - into interfaces
between people and digital information."

Pervasive Prototypes

These lofty goals have been substantiated in the somewhat primitive
technical achievements of the Tangible Media Group to date. Throughout the
1990s and into the present, Ishii and his research associates (mostly MIT
graduate students) have typically demonstrated half-a-dozen "tangible
interface" prototypes a year. Their projects have resulted in curiosities
like Audiopad, a real-time musical instrument comprised of movable pucks
on a flat display surface.
[http://tangible.media.mit.edu/projects/Audiopad/Audiopad.htm] Also, see
Illuminating Clay,
[www.tangible.media.mit.edu/projects/IlluminatingClay/Illuminating Clay] a
computational landscape-modeling system featuring digital graphics
projected over malleable putty. Without diminishing the difficulties of
trying to close the great divide between atoms and bits, these devices are
clearly master's thesis-sized projects in terms of achievement, and appear
to be baby-steps in a rather gimmicky research field. My personal
favourites include musicBottles,
wherein different parts in a musical arrangement are "played" by removing
the caps from three transparent containers - this project is said to
exploit "the emotional aspects of glass bottles." Also, LumiTouch,
[http://tangible.media.mit.edu/projects/LumiTouch/lumitouch.htm] a pair of
picture frames networked so that they light up when long-distance lovers
hold photographs of each other. The shallowness of these touchy-feely
attempts to communicate emotional content only serve to undermine the
Tangible rhetoric.

The vulnerability of this research is its extreme literalness, its nuts
and bolts lack of poetics. It is ironic that these hardware-based
prototypes serve to deconstruct and demystify, rather than to strengthen,
some of the group's best claims. Tangible Bits research is conducted by
computer scientists and students in interdisciplinary teams (different
species of engineers, cognitive psychologists, and so on). The profiles of
these researchers generally reveal parallel, hobby-like interests in music
and the visual arts, plus lots of hiking, camping, wind-surfing, and yoga.
It's clear that being creative and pragmatic, killing two birds with one
stone, is an art form in Ishii's lab.

With the above criticism levied, it is hard to argue against the wisdom of
developing dual, or multiple-purpose systems. And all these modest,
thesis-level projects will eventually accrue into a significant
engineering domain. MIT attracts brilliant scientists and students, and I
have no doubt that there is more behind the Tangible Bits initiative than
meets the eye. Just look at the wonderful promises.

Perceptive Engineering

Max Wertheimer said that we should seek to discover the underlying nature
of things (the relationships between elements, both figure and ground).
Ishii is a gestaltist, and he learned a great deal from the late Mark
Weiser, [http://www-sul.stanford.edu/weiser] the brilliant former chief
technologist at Xerox PARC. Weiser launched the idea of ubiquitous, or
pervasive, computing. Mainframes gave way to personal computing, and
computing will now move out into the physical environment, in what Weiser
said would be an era of "calm technology"
[http://www-sul.stanford.edu/weiser/Ubiqforum.html] In other words,
technology will recede into the background.

Hiroshi Ishii has distilled Wertheimer and Weiser's thinking into the key
research goals of Tangible Bits: develop interactive surfaces, couple
atoms and bits (so that the surface of physical objects will reveal
digital information), and move digital information into ambient media.
(Background interfaces at the periphery of sense perception are absolutely
key: again, context matters.)

The extreme literalness which typifies the way engineers apply perceptual
theory leads me to predict the next twist in Ishii's rhetorical narrative:
If human perception depends entirely upon information in the environment
(the Tangible Bits vision is a literal projection of the act of perception
onto the environment), then the way we colour or distort the world in our
internal cognitive processes can be over-written and ignored. Advertising
agencies will love the idea of living rooms where every single surface
reinforces a pitch!

Perception in Ishii's model will end up being a direct consequence of the
properties of the environment. The imagination, and our "memory" of prior
learning, will actually be composed by the environment. We will slip into
a sub-symbolic reality: a childlike state of sensual reverie. Rhetorical
substantiation for such a vision may be obtained from two texts by J. J.
Gibson: The Senses Considered as Perceptual Systems (1966), and The
Ecological Approach to Visual Perception (1979). Gibson, in his theories
of ecological psychology, stressed the importance of interaction: give and
take between the organism and the environment. Active, physical learning,
where major information is picked up by moving around and finding out what
happens, is the guiding principle of Gibson's thinking.

There's no question that today's GUIs pin us down, immobilizing our
bodies, restricting our computing environment to a symbolic, physically
inactive space. But what will happen to our internal, cognitive processes
when we start slipping in and out of cyberspace by physically moving
around: walking, running, jumping, bumping, and caressing? This is where
the passion of engineers who love to hike and bike, windsurf and practice
yoga, comes into play.


Tom Sherman [http://ams.syr.edu/video/sher.html] is an artist and theorist
who splits his time between Port Mouton, Nova Scotia, and Syracuse, New
York, where he teaches media art history, theory, and practice at Syracuse
University's Department of Art Media Studies. [http://ams.syr.edu/] His
latest book, Before and After the I-Bomb: An Artist in the Information
Environment, [http://www.banffcentre.ca/press/publications/ibomb.asp] was
published by the Banff Centre Press in 2002.


1. James Jerome Gibson's The Senses Considered as Perceptual Systems
(1966) and The Ecological Approach to Visual Perception (1979) were both
published by Boston's Houghton Mifflin. An extensive bibliography
[http://www.kyb.tuebingen.mpg.de/bu/people/astros/gibson.htm] of Gibson's
work, as well as explanations of his Information Pickup Theory,
[http://tip.psychology.org/gibson.html] are available online.

2. The artifacts and goals of the Tangible Media Group have been made even
more graspable in the 1997 paper
Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms
by Hiroshi Ishii and Brygg Ullmer. A complete list of Tangible Bits
interfaces is viewable on Tangible Media's projects page.

3. Unless otherwise noted, quotes from Hiroshi Ishii concerning Tangible
Media and Tangible Bits have been taken directly from Ishii's Web site.

#  distributed via <nettime>: no commercial use without permission
#  <nettime> is a moderated mailing list for net criticism,
#  collaborative text filtering and cultural politics of the nets
#  more info: majordomo {AT} bbs.thing.net and "info nettime-l" in the msg body
#  archive: http://www.nettime.org contact: nettime {AT} bbs.thing.net