A new systematic approach to survey of architectural heritage:
applications of ground-based long range
remote stand-off imaging and multi-modal
spectroscopic investigation of wall paintings

Introduction

Current method of surveying and recording architectural heritage is mainly based on 3D laser scanning and colour photography, that is, recording what humans can see with their naked eye. However, such information is limited because different materials can sometimes give the same colour appearance. Increasingly, the study of material cultural is incorporated in historical research. Detailed scientific analysis of materials at the chemical level (molecular and chemical elements), as well as imaging in the spectral range beyond the visible (e.g. X-rays and infrared imaging) have been adopted in archaeology, art history and conservation. The challenge of conducting studies on architectural heritage is the size and height of the monuments, which precludes easy access to the whole site. Detailed investigation is very time consuming and require scaffolds or cumbersome mechanical devices to bring an instrument close to the area of interest for investigation. In this chapter, we will introduce a comprehensive approach to the survey of architectural heritage that includes not only the traditional 3D and colour imaging, but also the recording of materials such as the artist materials used for the wall paintings and construction materials etc. Since 2006, the Imaging and Sensing for Archaeology, Art history and Conservation (ISAAC) Lab at Nottingham Trent University has been developing remote stand-off imaging and spectroscopy instruments for detailed investigation of wall paintings and architectural heritage focussing on their material properties¹.

There is confusion in literature on what ‘remote’ or ‘ground-based remote sensing’ means. By ground-based remote stand-off imaging and spectroscopy, we mean the entire instrument including the probe and the control system is placed on stable ground to collect data at a spatial resolution similar to that collected at the usual close range distance (within 1 m) but at a stand-off distance of a few metres to tens of metres. Note that any camera can be used to image ceiling paintings from the ground at low resolution, but the essence of remote imaging here is to do so at high spatial resolution typical of close range operation within about 1 m. Similarly, there is confusion in literature on what multispectral and hyperspectral imaging means. Spectral imaging or imaging spectroscopy is the general term used to describe all techniques that collect a spectrally continuous set of images that allows the extraction of a spectrum per pixel. If they are used to measure reflectance, then reflectance spectral imaging is the precise term. If they are used to measure fluorescence, then it is UV laser induced fluorescence spectral imaging if the excitation source is a UV laser, or X-ray fluorescence (XRF) spectral imaging if the excitation source is an X-ray source.

In the review of Liang², multispectral and hyperspectral imaging were grouped together as one technique, spectral imaging, which gives reflectance spectra per spatial pixel within a 2D scene, as there is no difference fundamentally other than an arbitrary spectral resolution division. Unfortunately, multispectral imaging has been used more recently in the cultural heritage field, especially amongst conservators, to mean something entirely different (e.g. often it is a stack of images including unrelated techniques such as UV fluorescence image, UV reflectance image, reflectance images in R, G, B and near infrared). It would have been better to refer to such image stacks as multi-band imaging. In this chapter, we will not use the term multispectral imaging to avoid confusion. Grating based line scan systems used for high spectral resolution reflectance spectral imaging are these days exclusively referred to as hyperspectral imaging, probably because these instruments were first used in remote sensing and referred to as such before they were used in terrestrial applications. We will only use the term hyperspectral imaging here to refer to such systems and in all other cases no matter what the spectral resolution is, we will use the term spectral imaging.

The development of remote stand-off imaging and sensing instruments was initially inspired by an initiative led by the Dunhuang Research Academy to digitally preserve, using colour photography, wall paintings at the Mogao Caves, a UNESCO world heritage site along the ancient Silk Road. The Mogao cave temple complex is just outside the oasis town of Dunhuang in China. It has a total of 735 caves carved into the side of a cliff, amongst which are 492 painted cave temples with 45,000 m² of wall paintings that were constructed from the 4th to the 14th century. Throughout its history, the site was occupied by various empires and the wall paintings and inscriptions are a record of not only Buddhist art, but also cultural, technological and political history of the region for over 1000 years. Since 2011, the ISAAC Lab and the Dunhuang Research Academy have collaborated on the deployment of remote stand-off spectral imaging systems and complementary multi-modal non-invasive spectroscopy such as XRF and Raman to characterise the artistic materials on the wall paintings to understand what kind of pigment mixtures were used for painting the different colours and how they were painted, uncovering any faded writings or drawings to reveal the history of the cave temple, date of construction³ and assess the conservation state of the paintings.

In the following sections, we will introduce each ground-based remote stand-off imaging and spectroscopy technique using our work on the Mogao cave temple 465, the Cathedral Church of St Barnabas and the associated Convent of Mercy in Nottingham UK designed by the 19th century English Gothic Revival architect Augustus Pugin, as examples to illustrate how each technique was used or can be used to gather information about wall paintings.

Long range stand-off 3D spectral imaging for large area surveys

An automated remote spectral imaging system for high spatial resolution spectral imaging was designed and developed by the ISAAC Lab for efficient and convenient large scale survey of architectural monuments or wall paintings from the ground without the need for scaffolds or lifts to move the imaging system close to the area being examined while still being able to achieve the high spatial resolution (sub-millimetre to <100 microns) typical of close range operation⁴. Fig. 1a shows the system, PRISMS, in operation in Cave 465 of the Mogao cave temple complex. PRISMS system records a series of 10 spectral images that covers the visible and the near infrared spectral range (400 nm–850 nm) using a computer-controlled filter wheel that rotates through the 10 filters with central wavelength ranging from 400 nm to 850 nm at 50 nm intervals, each with a spectral bandwidth of 40 nm. The filter set can be changed to suit the needs of each research project and in the case of the Mogao Cave 465 project, the last filter was replaced with a filter centred at 880 nm with a spectral bandwidth of 70 nm to allow more light through and increase the speed of capture (fig. 1b). Each pixel records a point on the wall or ceiling painting through each of the 10 filters and hence collecting a reflectance spectrum per spatial position (fig. 1c). The camera has over a million pixels, and therefore, each field of view consists of over a million spectra. Broadly speaking, spectral reflectance contains the signature of a material. Since most pigments and dyes have very broad spectral features in the visible and near infrared spectral range of 400 nm–1000 nm, spectral resolution of 50 nm was found to be sufficient to capture all the spectral features for most historic pigments and dyes except for a handful of materials such as lac and madder dyes and cobalt based pigments⁵. The trade-off is between spectral resolution and in situ survey time, and the intention is to use PRISMS for a preliminary large scale survey for material classification which can then be followed up with detailed analysis using a suite of complementary spectroscopic techniques to confirm the identity of the materials. PRISMS was designed for high spatial resolution imaging at large distances (~0.1 mm at distance of 10 m), therefore, the field of view is small and a large number of image cubes need to be collected to survey a large area.

Given that PRISMS records spectral reflectance in the visible spectral range, an accurate colour image can be derived from the spectral images for any illumination spectrum (fig. 1c). This is different from that achievable by a RGB colour camera which records only the accurate colour for the illumination used during capture. Since colour imaging cannot decouple the illumination spectrum from the spectral reflectance of a material unlike spectral imaging, it cannot give an accurate colour representation of an object under a different illumination to that used when the RGB image was captured. PRISMS operation is fully automated from focussing to scanning an entire architectural interior. An additional feature was later added to allow simultaneous spectral imaging and 3D topographic mapping⁶. Once the focus position for each spectral image cube is recorded, the distance to the painting at the centre of each field of view can be determined, since there is a one-to-one relationship between object distance and focus position. The distance accuracy is about 5 mm at 10 m distance⁷. A rough 3D point cloud can be constructed from the 3D position per image cube as a by-product of spectral imaging without having to acquire extra measurements, which gives, for example, 3D positions every 3-4 cm at ~10 m distance (fig. 1d).

Automated machine learning assisted data processing

The automated spectral imaging instrument, PRISMS, greatly improves the speed of data collection, making it possible to survey an entire cave or architectural interior. The significant increase in data collection speed poses a new problem in data analysis, that is, traditional manual analysis of the data is no longer adequate. For example, PRISMS produces about 5000 spectral image cubes and over 5 billion spectra for a 10 m² area imaged at 10 m distance. An efficient automated data processing method based on machine learning algorithms was developed by the ISAAC Lab to deal with the ‘big data’⁸.

Revealing ‘hidden’ writing and drawing

Spectral imaging is often used to reveal faded or ‘hidden’ writings or drawings by examining the spectral images in the near infrared bands (i.e. 800-880 nm bands in this case). Fig. 2 shows an example of the infrared band at 880 nm showing the preparatory sketches which are not seen in the colour image or the bands in the visible spectral range. Statistical methods such as principal component analysis (PCA) and independent component analysis (ICA) can reveal even more of the ‘hidden’ information as illustrated in fig. 2⁹. Automatic processing of the spectral image cubes can reveal faded writings, some of which are not discernible by the naked eye.

For example, the Sanskrit dhāranī were found to be stamped on small pieces of paper and glued to the foot of the large Buddhas painted on each of the 4 sloped ceilings. These stamped sheets of prayers were typically used for consecration which means that palaeographic dating of the Nagari script allowed dating of the paintings to after the second half of the 12th century¹⁰.

Clustering of similar spectra

PRISMS collected over 5 billion spectra just for the east ceiling of Cave 465. An automated clustering code was developed to group similar spectra into clusters using a machine learning algorithm¹¹. It employs an artificial neural network based technique, self-organising maps (SOM), to automatically group the reflectance spectra into clusters such that billions of spectra were reduced to a few hundred unique reflectance spectra that corresponds to the mean cluster spectrum of each cluster. The clustering of the spectra was processed sequentially such that similar spectra from all the spectral image cubes collected from the cave temple are grouped into the same cluster (e.g. fig. 3) where the cluster map labels pixels from a cluster with the same false colour (fig. 3b). This can also be extended to other cave temples from the same site, such that all similar spectra in a number of caves are clustered together. This process greatly reduces the number of spectra that needs to be examined manually. For example, the over 5 billion spectra collected from the east ceiling of Cave 465 was reduced to a few hundred clusters, which meant that only the few hundred mean cluster spectra need to be examined for pigment identification. The clustering process can also be used to pick out the final sketches by selecting the cluster that corresponds to the final sketches drawn by carbon ink (fig. 3c). Traditionally such drawings are required in archaeological reports where draftsmen are required to sketch these manually. This part of the ceiling painting appears to have been painted on paper and glued onto the existing ceiling painting, which is not a common practice found at the Mogao cave temple complex. It is important to note that clustering here is based on the entire spectra unlike the common cluster analysis in archaeology where two principal components are plotted against each other to allow manual grouping of pixels.

Complementary long range stand-off spectroscopy and spectral imaging

While it is possible to separate the materials on the paintings into distinct clusters using PRISMS data, it is not always possible to identify the materials accurately using just PRISMS reflectance spectra alone. In analytical science, it is well understood that there is never one technique that can be used to identify all materials, and that a suite of complementary techniques is needed.

Remote stand-off reflectance hyperspectral imaging

While PRISMS is efficient for large area survey, its low spectral resolution precludes the identification of red dyes such as lac and madder which have characteristic narrow absorption lines (~5 nm) between 500 nm and 600 nm. Organic pigments degrade easily, and apart from the blue dye indigo, it has been a challenge to identify organic pigments at the Mogao site. An ISAAC Lab developed grating based line scan remote spectral imaging system with spectral resolution of 2.8 nm and spectral range of 400-1000 nm was used for targeted spectral imaging from the ground to search for these red organic dyes. Fig. 4 shows that a red dye, consistent with a scale insect dye such as lac, was identified at the bottom edge of the west ceiling of Cave 465. The upper parts of the cave are relatively free from human activities allowing better preservation of the organic painting materials.

Remote stand-off short wave infrared (SWIR) reflectance hyperspectral imaging

Spectral imaging in the visible and near infrared (400-1000 nm) probes the electronic transitions in molecules, which tend to have broad absorption bands that are not always specific enough for material identification (e.g. yellow and white pigments). By extending the spectral range to the Short Wave Infrared (SWIR; 1000-2500 nm), it can probe the overtones of molecular vibrations allowing clear identification of additional pigments such as azurite, malachite, atacamite and various white pigments with relatively high specificity¹². Different white pigments were used throughout the history of Mogao cave temple complex and the identification of the white pigments is important for the dating of the paintings¹³. A remote stand-off grating-based line scan SWIR spectral imaging system (fig. 5a) has recently been adapted to complement the visible and near infrared remote spectral imaging systems for architectural monuments¹⁴. Remote stand-off SWIR hyperspectral imaging is also demonstrated to be an effective method for mapping moisture and salt on buildings, which is one of the most damaging factors for the deterioration of built heritage¹⁵.

Remote stand-off Raman spectroscopy

Raman spectroscopy can identify not only the molecule but also the crystalline structure (e.g. it can distinguish between massicot and litharge), making it a technique for pigment identification with high specificity. However, Raman spectroscopy suffers from poor scattering efficiency (factor of more than a million lower than reflectance spectroscopy) and laser induced fluorescence can often mask the weak Raman spectra. ISAAC Lab has recently developed a remote stand-off Raman spectroscopy system using a 780 or 785 nm laser to assist pigment identification from the ground at distances of up to 15 m (fig. 5b)¹⁶. An automated daylight subtraction method was developed to allow the system to function indoors during the day. Fig. 6 shows the results from the ground-based remote stand-off Raman spectroscopy of the modern wall painting covering the original 19th century decoration in the Convent of Mercy adjacent to the Cathedral Church of Saint Barnabas in Nottingham, showing characteristic lines of rutile and copper phthalocyanine blue. Fig. 6b shows the complementary remote stand-off reflectance spectrum confirming the spectral characteristics of copper phthalocyanine blue. It is important to note that Raman spectroscopy is very difficult to perform on scaffolds as the integration time is relatively long because of its low scattering efficiency and the need to avoid laser induced damage¹⁷.

Remote Stand-off Laser Induced Fluorescence (LIF) Spectroscopy and LIF Spectral Imaging

While fluorescence can mask Raman signals, they are also a source of information. Typically, a UV laser is used in laser induced fluorescence spectroscopy (LIF) to efficiently excite fluorescence in materials at a well-defined monochromatic excitation wavelength. Remote stand-off LIF spectroscopy has a long history in its application to architectural monuments¹⁸, mainly for mapping of biodeterioration using either LIF analysis at a single point or through raster scan of a single point to survey an area¹⁹. ISAAC Lab has recently developed a remote stand-off LIF system to complement the other remote spectroscopic techniques for material identification (fig. 5c). Here we have developed a true LIF spectral imaging system where the excitation laser is expanded to illuminate an area and the thousands of LIF spectra along the line defined by the slit are collected simultaneously using a grating based line scan hyperspectral imaging system (same as the one introduced earlier for reflectance hyperspectral imaging) thus significantly increasing the efficiency of remote stand-off LIF area scan using the line scan system rather than a point based raster scan. LIF spectra tend to have broad spectral features, but it can still help in distinguishing materials.

For example, both vermilion and cadmium red oil paint have similar reflectance spectra (fig. 7a), but their LIF spectra are very different (fig. 7b). The laser-based techniques are non-invasive but require extra caution associated with laser safety.

Remote stand-off Laser Induced Breakdown Spectroscopy

X-ray fluorescence (XRF) spectroscopy identifies the chemical elements in a material. Portable XRF operating in air in close proximity to the material can typically detect elements with atomic number greater than about 13 or down to 12 with helium purging. XRF depth of penetration depends on the material and it is sometimes difficulty to determine whether a detected element is from the surface or deeper in depth. XRF is non-invasive, but since it uses X-rays, special precautions and regulations for health and safety need to be observed. In addition, it can only be effective when operated close to the material, which limits it to the investigation of the easily accessible lower parts of wall paintings.

All the spectral imaging and spectroscopy techniques discussed so far, apart from XRF, have their remote stand-off versions developed to serve the needs of material analysis for architectural monuments and sites. However, a micro-destructive technique, laser induced breakdown spectroscopy (LIBS), can replace XRF for analysis of elemental composition at a long distance. Remote stand-off LIBS is the only technique discussed so far that is invasive, but it is only micro-destructive as the focused laser beam ablates a small amount of the surface material forming a small crater of ~1 mm in diameter. With each laser pulse, the ablated material forms a plume from which a spectrum is recorded. The advantage of LIBS is that it can detect all the elements including the light elements that cannot be detected by XRF. Ground-based remote stand-off LIBS has been developed for architectural heritage to provide elemental analysis from a distance of a few to tens of metres²⁰. The destructive nature of LIBS can also be used to its advantage for layer-by-layer material identification, especially when combined with other complementary spectroscopic techniques such as Raman spectroscopy. In a recent paper, ISAAC Lab has demonstrated a ground-based remote stand-off LIBS for layer-by-layer pigment identification (fig. 5c) which can be used in situations where such micro-destructive techniques are allowed. For example, in some situations, to investigate upper parts of a wall where a small damage is unlikely to be noticed by a visitor standing on the ground or to investigate the original historical paint layer covered by a modern paint layer. Fig. 8a shows a mock up paint sample with paint layers typical of those found in wall paintings at the Mogao site but painted on a white gesso board.

The remote LIBS and Raman spectroscopy systems were setup at a distance of about 5 m and a Raman spectrum was recorded before each laser pulse (fig. 8b) while LIBS spectra were recorded at each laser pulse (fig. 8c) allowing layer by layer material identification. For example, the 2nd laser pulse shows that the elements, lead (Pb) associated with the red lead pigment and calcium (Ca) associated with the animal glue binder, while 785 nm laser induced Raman spectrum shows the presence of red lead and vermilion. The near infrared laser at 785 nm penetrated through the thin red lead layer to the vermilion layer. It is only by combining the information from LIBS and Raman, that we can arrive at the correct interpretation. A combined ground based remote stand-off LIBS, Raman and reflectance spectral imaging for chemical analysis of the stratigraphy of a whitewashed 19th century wall painting in the Unity Chapel of the Cathedral Church of Saint Barnabas in Nottingham revealed at least 7 paint layers painted through the history of the wall painting²¹.

Discussion and conclusions

A suite of ground based remote spectroscopy and spectral imaging instruments for operations at from 3 m to tens of metres has been developed for comprehensive survey of architectural heritage that collects not only the traditional 3D and colour images but also the material content of a monument at submm spatial resolution. A machine learning based data processing method has been developed in tandem to automatically process the large volume of high spatial resolution spectral imaging data to reveal ‘hidden’ or faded writings and drawings and cluster the pixels with similar spectra to produce chemical maps of the materials and to guide the targeted complementary remote spectroscopic measurements for material identification and thus ensuring comprehensive analysis. The workflow adopted is similar to that of close range multimodal spectral analysis of collections of objects²². The 3D remote spectral imaging system PRISMS for large area surveys, scans the entire site followed by machine learning processing to produce material cluster maps, which guides the areas to perform complementary ground based remote stand-off non-invasive spectroscopy such as Raman spectroscopy, XRF spectroscopy, SWIR reflectance spectroscopy and LIF spectroscopy for material characterisation. In order to assess the effectiveness of material characterisation when a mixture of materials is involved, a final check to make sure that the suite of non-invasive spectroscopic techniques has, for example, correctly identified the pigment mixture, a Kubelka-Munk fit can be used to model the light-matter interaction to verify if the measured reflectance spectrum can be the result of mixing the various pigments identified by the individual spectral techniques²³.

The full suite of ground based non-invasive remote reflectance spectral imaging systems (visible/near infrared and SWIR hyperspectral imaging systems), laser induced fluorescence (LIF) spectral imaging system and point based laser induced spectroscopy techniques (Raman spectroscopy) has recently been applied at two UNESCO sites, the Macedonian Royal Tombs at Vergina Greece (Tomb II) and the Stiftung Pro Kloster St. Johann at Mustair Switzerland as part of the IPERION HS MOLAB service, to perform material analysis to answer art historical and conservation questions.

In some situations, especially when an earlier painting is suspected to be covered over by a modern painting, the micro-destructive technique of ground-based remote stand-off LIBS combined with the other non-invasive spectroscopic techniques can potentially be used to reveal the material content of the earlier painting and obtain the full stratigraphy of the paint layers offering in situ stratigraphy analysis which is more convenient than traditional method of physical sampling which potentially requires scaffolding and is time consuming as the samples needs to be prepared in a laboratory. All the ground-based remote stand-off spectral imaging and spectroscopy systems described here are on offer through the European Research Infrastructure of Heritage Science (E-RIHS) services (https://catalogue.e-rihs.eu/catalogue?c[0]=United+Kingdom&p[0]=Molab) or through direct application to ISAAC Lab (https://www.isaac-lab.com/isaac-mobile-lab).

The advantage of remote stand-off spectroscopy and spectral imaging is not just in investigation of objects at inaccessible positions, but it provides flexibility and convenience in analysing even accessible objects as it does not require re-positioning of the instrument to investigate different objects in a room or a large object, because focussing and positioning are automated. In addition, if the long range instruments are positioned at close range to an object, microscopic spatial resolution can be achieved.