Exploratory Image Databases: Content-Based Retrieval

The scene is represented in an ink print that I have seen so many times I almost feel as though I was there. It is a quiet street on the outskirts of Paris; I like to imagine that the events took place a little bit out of the city, maybe on the otherbank of the Seine, in the area that is now known as la Defense. Nowadays, looking east, one would have a magnificent view of the Champs Elysees, with the Arc de Triomphe and, in the background, the Parisian skyline stamped against the French sky. Looking a bit south, you could see the metallic shape of the Eiffel tower, and, a bit north, you could maybe have a glimpse of the top of the Beaubourg, the machine/building. At the time of my ink print, though, you would not have seen any of those, because the events took place in the year 1769 and Napoleon Bonaparte, Georges Eiffel, and Renzo Piano were still dim possibilities of the future.

The boiler is hot. Assistants have been working on it the whole morning, and now it is full of eager steam; looking at it in the print, you can almost feel the pressure building inside. Nicholas Cugnot takes care of the final details, then climbs to the control post of the vehicle. It is a few tons of steel, wood and fire, mounted on four robust wheels, a carriage not unlike those that peasants and masons use for heavy loads, attached to a pair of oxen. But there are no oxen today--the machine is going to be pushed by the force of steam and the will of Nicholas Cugnot.

The machine starts. At first it moves very slowly; then, with the pure and native faith in progress that only an Eighteenth Century machine or inventor could have, it moves faster. And faster, and faster. Now I choose to think (the writer is a benign demiurge) that in that moment Cugnot felt very happy and that, at the same time, he felt a shadow of sadness: in a few minutes the great intellectual achievement of his life would be accomplished and nothing he would do in the years to come could reach the intensity of these moments.

Of course, we know now that things would go very differently and that there would be no "years to come." Cugnot probably had the time to realize it, in a glimpse of the wall of a country house growing closer and closer, before the impact and the explosion of the boiler would precipitate him into oblivion.

Nicholas Cugnot built the first working steam car, but not the first working pair of brakes.

Let me go back for a moment to Cugnot riding his engine (which actually he did many times before the accident, and not only once as I, with a slightly Gothic melodramatic penchant have chosen to imagine). Now I choose to imagine him looking at his machine with satisfaction, and thinking of the future. One day, he thought, machines like his, or better, would be all around France, doing the work of many animals; people could move faster, and more reliably. The streets of the city would not be full of the smell of horses, and the air would be clear (little did he know...).

I suppose that, even in their wildest dreams, people like Cugnot, Benz, and Diesel never imagined anything even remotely close to the consequences of their inventions. It would have been possible for them (maybe while sitting in front of a fireplace, after dinner, with a few friends, a glass of V.S.O.P.and a cigar, when the fantasy can run loose without worrying about details like plausibility) to imagine endless technological developments of their inventions. Why, one day you will have an engine so small that it will fit in your pocket, and that will drive you to 200 Km/h. What they could not do, what was beyond their foresight, and beyond that of everybody else, were the cultural and social consequences of the new technology.

Most of these pioneers imagined a society more or less like the one in which they lived, with maybe a few things that you could do better than before because of their inventions. They could probably have imagined that the food could have reached the grocery store down the block more quickly, at a lower price, and in greater variety. They could not have imagined that, with the advent of massive road transportation, the grocery store down the corner would disappear, driven out of business by the chain of big supermarkets on the outskirts of the city. A marvel made possible by the car.

Dreamers could have imagined people moving faster inside the cities thanks to the new cars, but they could not have imagined the emergence of the suburban society in America--people voluntarily leaving the cities to go and live on the periphery. They were used to a society in which the rich and the poor lived close to each other; it would have been hard for them to imagine that, thanks to the car, the difference between the rich and the poor would become geographic--in America, the rich living in the suburbs, the poor living in the inner cities, in Europe the other way around.

Every researcher working on information technology today is, in a very real sense, the modern counterpart of Cugnot or Benz; it is equally difficult today forthem to fully understand the consequences of their actions. Like Cugnot’s machine, today’s computers and communication networks are technological devices, and like Cugnot machines, they have just started to leave a faint mark on our social and cultural fabric; a mark that--there is no doubt--will deepen and radicalize in the decades to come.

If trying to understand and predict technological developments beyond the limited horizon of a few years is a hazardous enterprise, to try and understand the cultural changes that this technology will spark beyond the same limited horizon is nothing short of titanic.

This book is concerned with a very small and technical fragment of the current evolution of information technology, the study of computer systems for access and retrieval of images from large databases, with particular interest in retrieval based on the content of the images. Its limited scope notwithstanding, content-based image retrieval can unveil and clarify some crucial aspects of the current and future modalities of human communication. It may seem a sign of hubris for a researcher in a field to state that his field will be the "Otto cycle" of information technology, and it probably would be an overstatement to say it of content-based image retrieval with respect to the overall cultural impact of information technologies. The statement may be true, however, if one considers the potential impact of technical disciplines on the modalities of visual communication, and on their relation with linguistic discourse.

Thanks in part to the ease of image production, transmission and reproduction, we live in a very visual age. The iconosphere is around us, and is not going to lift anytime soon; the Twentieth Century started with the centrality of language as an object of philosophical study (with names like Saussure, Peirce, Wittgenstein), and is drawing to a close posing more and more insistently the question of images. From Barthes’s unforgettable analysis of the Pasta Pinzani advertisement, to the emergence of the field of pictorial semiotics, the cultural debate on the semiosis of image has characterized the second half of the century.

Information technology has remained largely oblivious of this debate. The reasons are many including, in part, a traditional division of intellectual activity in disciplines of analysis (study of the existent, e.g., science, linguistics,...) and disciplines of synthesis (modification of the existent, e.g., all technological disciplines). More immediately, the traditional problems of computer vision considered images as data from a "sensor" that could be used to solve a specific problem; human perception and culture can be used as a guide and an inspiration but, strictly speaking, there is no reason why a robot or an automatic quality control system should use the data from a camera the same way people use their eyes to "see." The only thing that matters is that enough information be extracted from the sensor allow solution of the problem. Social and cultural considerations on the semiotic status of images have little to do with the metier of computer vision because, ex hypothesis, an automaton is not a social participant in human activities, but a mere tool--little more than a smart hammer.

The social and cultural preoccupation of the role of images as a communication medium cannot be discarded, not even provisionally, once one starts working in image retrieval. Here, the relation between the image and its recipient (be it the user of the retrieval system, or the recipient of a message that the user is putting together) is of primary importance, and it is impossible to ignore it.

It is in the sense of recognition of this special relation that the statement that content-based image retrieval as a discipline is independent of image analysis and computer vision is not only defensible, but unavoidable. The recognition of this independence is important because it entails, despite all the points of contact and the many common methods, a different consideration and role for the entity that represents the starting point of both: the image.

Because of this "eye opening" relation between the user and the image, content-based image retrieval has, much more than computer vision, the potential to change our relation with and our understanding of images, their language, their meaning, and their social role as much, maybe, as the invention of the car changed our understanding of the city and of its relation with the countryside. One of the points this book is trying to make (maybe the point this book is trying to make) is that the relation between retrieval technology and image signification is a two-way street: Technology will help us gain a new understanding of image signification, and the semiotic study of images is necessary for anybody who wants to work in this area.

In this day and age, it is impossible to write a book like this without accounting for the presence of the Internet. In image databases, the Internet plays a double role. On one hand, it has the very technical function of giving many people access to a virtually unlimited supply of images, and forces database designers to consider the implications of a distributed reservoir of images and a distributed usership. More fundamentally, the Internet constitutes a large context in which images are placed. The Internet is not a sterile collection of mute and anonymous images: It is a complete, self-contained discourse in which images are embedded and, by virtue of this embedding, given meaning. This, I believe, is the real role of the Internet and the basis of its tremendous significance, much more than its convenience as a medium for accessing far-away images.