Content-Based Image Retrieval - CBIR (Art History)

A suggestion for an area of tools development voiced at the Methods Network Tools Development Workgroup (for a report see involved a method of carrying out automated image analysis. Taking inspiration from the TaPOR project which allows users to run a suite of text analysis tools against predefined or imported texts, it was suggested that a similar approach for image analysis might enable researchers to engage with images in new and interesting ways.

In terms of what the components of that suite of tools might be, content based image retrieval techniques would presumably feature. The use of automated searching across a body of material on the basis of shape, tone, colour, texture or spatial location has been around for a long time and is still a burgeoning research area with more papers appearing every year in scientific and technical forums. The application of these techniques has clear relevance in a number of fields including medicine, policing, military work, manufacturing and many other areas where there are prodigious numbers of images that have similar properties but also demonstrate measurable and definable differences. However, the challenge of using these techniques in the context of art historical research always seems to have faltered at what those working in the field have termed the ‘semantic gap’. This can be understood as the difference in the potential for searching for primitive, quantitatively definable features and the currently impossible task of automating searches for essence, meaning, emotion, personality, irony and any number of other states of being or modes of interpretation that are widely represented throughout the realm of art. At a recent Methods Network Expert Seminar, Kirk Martinez suggested that it would take multiple generations before computer science was able to get close to crossing the ‘semantic gap’ with automated retrieval techniques.

As with the application of all technologies, if a system that initially promises much fails to match the expectations of a majority of users within a reasonable time span, then the attention of those users will turn to other issues and arguably, this has been the case with the attitude of the art historical community to CBIR. It may be useful then to concede that high level semantic searching is not a realistic option for automated image retrieval and refocus attention on what can be gained by searching for primitive or ‘level 1’ features as Eakins and Graham refer to them in a JISC report from 2000.

At the Hamilton Kerr Institute, a system is being developed to search across an image store of around 5000 digitised cross section paint fragments using algorithmic similarity features. They are currently investigating the feasibility of using an open source programme developed by Thomas Deselaers ( and colleagues at Aachen University which will allow the user to query by example from a repository of images and will attempt to match that sample using various quantitative techniques.

Fig.7  Tamura Features Histogram (with kind permission from Eike Friedrich, Hamilton Kerr Institute)Fig.7 Tamura Features Histogram (with kind permission from Eike Friedrich, Hamilton Kerr Institute)

Fig.7 shows one of the histograms used by the FIRE system (Flexible Image Retrieval Engine) which attempts to quantify image data in a way that approximates to human perceptions of texture. In combination with other histograms relating to shape and colour, the system then attempts to retrieve similar images and in many cases, produces plausible and useful results (see fig.8).

Fig.8  Cross Section Images (with kind permission from Eike Friedrich, Hamilton Kerr Institute)Fig.8 Cross Section Images (with kind permission from Eike Friedrich, Hamilton Kerr Institute)

The majority of items in the retrieved dataset window have a clear relation to each other and the range of types of cross section that are available in the collection can be seen below them in a randomly generated selection of other examples.

The materials under scrutiny do of course, by their nature, lend themselves to this kind of analysis. The value of the sample depends on the clarity of the layers that can be seen within a necessarily limited outline and therefore slides will be archived that have clear definitions of banding or visibly embedded diverse materials. They are all roughly the same diminutive size because of the nature of the technique and they will often have a relatively flat section as one part of their silhouette that will represent the uppermost surface of the various layers and the one that would be visible to the viewer of the work of art were they standing in front of it.

The value to conservators of retrieving similar cross-section images from large digital libraries will be clear. Comparisons between samples from similar chronological periods will begin to build up a picture of the use of materials in certain regions at certain times and this will help not only with understanding how best to conserve the damaged or at risk item but may also help with attribution and verification issues. It is feasible that a trained eye could look at certain cross-sections and at a glance would be able to categorise the specimen as coming from a painted object from a particular country within a fairly specific time frame. They might also, by analysing the juxtaposition of physical material in the sample, be able to speculate on a number of other issues to do with the trade and availability of materials at certain points in history which, begins to impact upon areas of art historical research outside the circle of conservation studies.

A Tamura Features histogram featuring a jumble of arithmetic information cannot adequately represent later 17th century Flemish Painting (for instance), but it can provide important insights that may help to more clearly define that notion; and having accepted that CBIR techniques are not going to provide users with astonishing new techniques for subject based querying of image data, it is potentially the cumulative affect of applying various processes and methods that will allow art historians to extract the most benefit from adopting a technological approach.

Syndicate content