George Legrady
«Pockets Full of Memories: An interactive museum installation»

George Legrady
«Pockets Full of Memories: An interactive museum installation»

Abstract
Conceived as an installation on the topic of the archive and memory, "Pockets Full of Memories" was exhibited on the main floor of the Centre Pompidou National Museum of Modern Art, Paris, France from April 10 to September 3, 2001. During this time, approximately 20000 visitors came to view the installation and contributed over 3300 objects in their possession, digitally scanning and describing them. This information was stored in a database and organized by the Kohonen self-organizing map algorithm that positioned objects of similar descriptions near each other in a two-dimensional map. The map of objects was projected in the gallery space and also accessible online where individuals in the gallery and at home could review the objects and add comments and stories to any of them.


KeyWords
Archive - cultural artifact - interactive installation - Kohonen self-organizing map algorithm - datastructures - memory


Production History
Planning of "Pockets Full of Memories" (PFOM) began in the spring of 1999 as a result of a meeting with Boris Tissot, exhibition coordinator from the Centre Pompidou who was interested in an exhibition that would integrate issues related to the intersections of memory, the archive, digital technology and the general public. A collaboration was developed with Dr. Timo Honkela from the Media Lab, University of Art and Design in Helsinki through CIRCUS (Content Integrated Research in Creative User Systems) , a research and development project funded by the European Union under the Esprit program. Timo Koskenniemi from UIAH Media Lab contributed the implementation of the Self-Organizing Map algorithm. Kohonen's algorithm applied in a manner outlined earlier in Honkela's research (Honkela 1997) became a key conceptual framework for the PFOM project.

Production began in the summer of 2000, following funding by a Daniel Langlois Foundation for Art, Science & Technology grant and continued until the opening of the exhibition in April 2001. In addition to the contribution by the UIAH Media Lab team (Prof. Timo Honkela, Timo Koskenniemi and Petri Saarikko), an international team of specialists realized various components of the project. Marton Fernezelyi and Zoltan Szegedy-Maszák, from the C3 Center for Culture and Communication, Budapest designed and constructed the scanning station unit consisting of a touchscreen interactive software, an image digitizing system with image processing software, the database and networks software and the interactive station structure itself. The exhibition's visual identity, which included the exhibition design, scanning station designs, interface design for the questionnaire and website, was produced in Stuttgart by the Projekttriangle design team consisting of Danijela Djokic, Martin Grothmaak, and Juergen Spaeth. The internet site providing access to the database and SOM algorithm was developed by Andreas Engberg at the CREATE lab University of California, Santa Barbara. Dr. Brigitte Steinheider from the Fraunhofer Institute for Industrial Engineering, Stuttgart contributed with the development of the data description questionnaire and is currently analyzing the data of the archive from a social science perspective. In addition to this work that examines the cultural and social nature of the collected information , she is also studying the complexity and impact of multinational and multidisciplinary teamwork practice i n the production of the PFOM project.


Project Description
PFOM integrates the real space environment of a museum installation with virtual access to the database through the internet. Visitors to the exhibition contribute visual and descriptive information to the digital archive about an object in their possession at the time of their visit. The data contribution takes place in the entrance area where the public interacts with a kiosk-like scanning station in a two-step process that consists of the scanning of the object followed by filling out an questionnaire through a touchscreen interface to describe its attributes. The image of the object and descriptive data are then stored in a database that grows throughout the duration of the exhibition. Presentation and access to the database occurs both online and in a large-scale projection in the gallery space of the museum.


The Database Archive
According to the museum's records, approximately 20000 visitors came to view the installation during the four-month exhibition of the installation, resulting in a contribution of over 3300 objects in the database archive. A large chronological map from the beginning to the end of the exhibition can be viewed online (requires adequate RAM memory) and detailed information about each contribution allows comparisons of descriptions.

The archive of objects consists of objects that museum visitors carried with them, for instance, such common items as phones, keys, toys, fragments of clothing, personal documents, currency, reading material, and others. The size of the scanning box was the only limiting factor that determined what could be added to the archive. The expectation in the early stages of planning was that the majority of contributions would consist of everyday common objects and the final result would provide an overview of the range of things people carry with them. Nonetheless there was the hope that some members of the public would be creative in their contribution choices, methods and descriptions and in fact there are a few objects that go beyond the everyday, for instance a marriage proposal note. We were also aware that such public interactive systems are testing situations to a segment of the audience who want to see how robust such systems are by trying to break through both the technical and conceptual limits of the project. One of the unexpected contributions to the archive included the numerous scans of body extensions such as objects: heads, hands and feet which increased exponentially once the initial examples entered the archive. The overall quality of the archive is a consequence of the dialogue that occurs between the audience's perception of the archive's holdings followed by a contribution that functions as a statement of participation and an engagement to leave a trace behind once the public has exited the museum. Contributing an image and descriptive information of one's personal object into an institutional environment such as a museum or database or archive certainly provides a sense of fulfillment but the best trace left behind seems to be a direct image of one's body parts. The many scanned heads, hands and feet have augmented the archive from simply being a collection of objects to encoding it with the corporeal presence of the contributors transforming the digital data environment into a metaphoric extension of the human body.


Ordering Data and the SOM Algorithm
The database archive of objects is ordered by the Kohonen Self-Organizing Map prior to being projected in the gallery space and accessed online. Due to current bandwidth limitations and scale of the gallery projection, the map of objects has been limited to 280 objects, so a selection is first made out of the total database each time the SOM is activated (once per minute). The selection is based on a percentage sampling of the database's life with priority given to the ten most recent entries. Honkela describes the function of the SOM in this project as follows:

"The Self-Organizing Map (SOM, also called Kohonen map) algorithm is the basic method that is used to create the "wall of objects". The SOM organizes the input items (contributed objects by the public) into an ordered display, a 2 dimensional map. On the map two items tend to appear close to each other if they have similar input features. In this exhibition, the input features consist of attributes and keywords. The attribute values and keywords are given by the exhibition visitor. They are transformed into numerical form that can serve as inputs. The algorithm may start from a random state of the map. Through the process of iteratively (repeated recalculation) handling the inputs it reaches an ordered state.

The map consists of a collection of map nodes that can be thought as places on the map landscape. On the map, nearby nodes tend to have similar items. Close to each there may be items that have been given similar attribute values, or items that have been named similarly. Thus, all the items with a particular keyword are not necessarily next to each other if the other features vary. Moreover, even if the visual qualities of an image are very similar, it may very well happen that two persons evaluate the item very differently based on their subjective point of view. There are also cases in which even the neighbors are rather far from each as there are occasionally dividing valleys and mountains ranges also in the natural landscapes.

The order of the final map is a consequence of all the inputs. The phenomenon is called emergence: the order is not determined beforehand. The order emerges through the audience contributions. The classification system is not specified by hand but it is created through the large number of local interactions on the map. This is why the system can be called 'self-organizing'. Metaphorically, similar items look for each other without any centralized command.

The principle of the SOM was developed originally by academician Teuvo Kohonen. The inspiration for this innovation has stemmed from the numerous neurophysiological studies in which it was shown that in the cortex of the brain similar kind of maps can be found. Perhaps this is a reason why the method seems to be appealing in several application areas." (Timo Honkela letter to the curator Boris Tissot)


Semantic Meaning and Linguistic Description
Because the positioning of an object in the SOM map is completely dependent on the way the contributor describes them, the emphasis in this work shifts to the function of linguistic description and semantic interpretation as opposed to the object's immediately recognizable literal or physical properties. Contributors engage in a creative process when they prescribe keywords and evaluate their object according to the questionnaire's attributes. Even though the map may contain numerous cell phones, watches and hands, their positioning across the map reflects the contributor's perception of them and the choice of words used to describe them. One of the key experiences of the exhibition is to watch the positioning of the object's image on the large-scale projection. After a short wait following data entry, the image appears on the screen highlighted with an orange frame to make it easier to identify. As the SOM processes the data every minute, it scans each line of the map and replaces or moves objects based on the new order. This process goes on continuously and provides the opportunity for comparison and reflection about the descriptive choices the contributor has made to define the object. Accessibility on the internet further enhances study of the map as each object can be clicked on to see its properties and attached stories, resulting in comparisons to surrounding neighbors. Internet interaction has provided another means by which to extend the dialogue for visitors, as the museum and internet audience have the opportunity to add comments and stories to any object, and from anywhere in the world. Many visitors who have traveled from other geographical areas have used this as a means to make contact with friends and family back home who then have added their own responses.


Visual Identity, Graphic and Interface Design
All graphic design aspects of the PFOM project have been designed by the Projektriangle team as a result of extensive dialogues about the meaning and function of all of the elements: the signage, the interactive questionnaire, the installation and website. The information environments within which data is organized and accessed impose a meta-level of meaning that redefine the content passing through it, much of which is implicit and necessitates extensive study of its functioning. I was very much interested in handing over the design solutions to Projektriangle as their work places an emphasis on coherent systems and the development of a specific visual vocabulary determined by the needs of the project. The thematic repetition throughout all the design elements focused on the process of the production of meaning beginning with the public's personal contributions and descriptions which become synthesized into an institutional archive . The signage in the exhibition space floor exemplifies the three-phase transformative process from object to descriptive parameters to data. From the entrance to the scanning station the floor is covered by graphic icons of objects physically guiding the public's movements to the various stages of the installation. Between the triangular space from the scanning station to the web viewing stations, the floor markings have changed to keywords which are also listed on the large wall to the right of the projected map. The markings in front of the final stage of viewing: the projected map, consist of id numbers.


Future Exhibitions & Next Step Developments
PFOM is an installation that has been planned to be exhibited in a number of different cultural environments. We currently have a large database contribution that reflects the particular population that have visited the Centre Pompidou during summer vacation time. The questionnaire has provided descriptions of the contributors' backgrounds. The data reveals a diversity of ages, nationalities, professionals, and interests. The current research work on the semantic specificities of the contributed data of 3300 objects in the archive is led by Dr. Steinheider. Once the data is analyzed resulting in an evaluation of the overall archive, we will be able to present a culture overview of how the members of this particular audience have described themselves through their object and description choices. These results will be meaningful in themselves but will be even more interesting when contrasted to the results of future collections assembled through travelling the exhibition to other cultural regions.

The realization of the exhibition at the Centre Pompidou has made it possible for us to study its functioning and consequently we have identified a number of areas that could benefit from further development. These include data collecting and data mining processes, dynamic feedback systems and visualization methods. Future developments will therefore focus on a number of key components:

A. REAL-TIME, PERSONALLY DEFINED ADAPTIVE MULTIPLE DATA VIEWS: Datasets are organized according to criteria and structures which can be dynamically activated to suit the viewer's personalized interests. The viewer is given the opportunity to move between various levels of simple to complex representations thereby providing clarity of meaning through contextualization.

B. SELF-ORGANIZING MAP ALGORITHMS & OTHER DATA MINING APPROACHES: Neural-net based data organizing algorithms provide highly interesting models by which data structures can be enhanced in their function and results. It is an approach that has great potential for revealing data relationships through unexpected juxtapositions.

C. MICRO to MACRO READINGS THROUGH TIMEFRAME SELECTION: Self-organizing map imply data relationship changes over a particular timeframe. Time segments can be reviewed, to track particular data, or particular attribute groups as they move in the map. The results can then be excerpted and made available for analysis.


Previous Work
"Pockets Full of Memories" is the latest installation work in a series begun in the early 1990's dealing with the topics of archive, cultural identities, audience contribution and technological processing of information. Two works that closely relate to this current project are "An Anecdoted Archive from the Cold War', first exhibited in 1993 at the Yerba Buena Center for the Arts, San Francisco and "Equivalents II" , first exhibited at the International Center of Photography, NY in 1994.


References
Honkela, Timo, Self-Organizing Maps in Symbol Processing. Hybrid Neural Systems, Stefan Wermter, Ron Sun (eds.), Springer, Heidelberg, 2000, pp. 348-362.

Honkela, Timo, Koskinen Ilpo, Koskenniemi Timo and Karvonen Sakari: Kohonen's Self-Organizing Map in Contextual Analysis of Data. K. Tanaka, and S. Ghandeharizadeh (eds.), Information Organization and Databases, Helsinki, Boston, Kluwer, 2000.

Honkela, Timo, Learning to Understand - General Aspects of Using Self-Organizing Maps in Natural Language Processing. Proceedings of the CASYS'97, Computing Anticipatory Systems, D. Dubois (ed.), American Institute of Physics, Woodbury, New York, 1997, pp. 563-576.

Kohonen, Teuvo (1995) Self-Organizing Maps, Springer Series in Information Sciences, Vol. 30, 1995; Second edition, 1997

Legrady, George, Intersecting the Virtual and the Real: Space in Interactive Media Installations. T. Murray, (ed.), Digitality and the Memory of Cinema, Wide Angle, Vol 21, No.1, Ohio University School of Film, 2001, pp. 104-113

Legrady, George, Modular Structure and Image/Text Sequences: Comics and Interactive Media. A. Magnussen and H-C Christiansen, (eds.), Comics Culture: Analytical and Theoretical Approaches to Comics. Tusculanum Press, Copenhagen 2000, pp. 79-90

Legrady, George, Slippery Traces: The Postcard Trail. A.Sommer (ed.) Artintact 3, ZKM, Center for Art and Media, Karlsruhe, 1996, pp. 101-104

Legrady, George, the Equivalents II. H. von Amelunxen (ed.), B+B Arts International, Amsterdam, 1996, pp. 216-221

Steinheider, Brigitte, Legrady, George, Realizing a Digital Media Installation: Problems and Synergetic effects of an Interdisciplinary Collaboration. Proceedings of MTAC 2001, Multimedia technology and applications, UC Irvine, 2001, pp. 255-260