D-Lib Magazine

November/December 2016
Volume 22, Number 11/12

A Doomsday Scenario: Exporting CONTENTdm Records to XTF

Andrew Bullen
Illinois State Library
abullen@ilsos.net

DOI: 10.1045/november2016-bullen

Abstract

Due to the challenging state budget situation in Illinois, Andrew Bullen of the Illinois State Library was asked to explore how the existing CONTENTdm collections could be migrated to another platform should the State Library be unable to pay its bill for its existing services. Andrew chose XTF, a resource at hand and a logical candidate for undertaking the transfer, and methods that were already available that would allow rapid migration. He describes in detail the process of transferring three collections that represented a reasonable cross section of Illinois Digital Archives content from CONTENTdm to XTF, and offers an evaluation of his test migration approach and lessons learned.

1 Introduction

Residents of Illinois and Illinois state government employees are living in interesting budgetary times. As of this writing, the Governor and the Illinois State Legislature have yet to pass a budget for FY2016. There is a stopgap budget passed for part of FY2017; however, the budget only covers some expenses until the November 2016 elections. What happens after that is unknown. Naturally, we are concerned about being able to support existing services to Illinois libraries.

One of the main services that we provide to Illinois libraries is the Illinois Digital Archives (IDA), a repository for the digital collections of libraries and cultural institutions in the State of Illinois, based on CONTENTdm. In light of budget concerns, we thought it prudent to explore what would happen to the collections on IDA if we could not pay our subscription fee to OCLC and were forced to cancel the service. There are currently 156 disparate collections in IDA and more than 468,000 objects in these collections, a fairly substantial instance of CONTENTdm. How could we replace it if we had to cancel our CONTENTdm subscription and convert these collections to another platform? I was charged with investigating our options.

2 My Parameters

Simulating a budget crisis, I had to use resources we would have on hand. Similarly, I chose methods that would allow me to quickly migrate the collections onto a new platform. This limited my tools to PHP, Perl, and a previously installed instance of XTF. We use XTF as the platform for our digital collection of deposited electronic Illinois state documents. I am more comfortable with Perl than PHP so I defaulted to that as my primary programming language.

XTF is a useful choice for Illinois libraries as it is also used by the Chicago Collections Consortium. There is a large and dedicated users group in Chicago that could lend assistance or implementation advice if needed.

3 The Test Data

Since we are a hosted site, I asked CONTENTdm Support to tar together the images of three collections and place them on an FTP site. These collections, listed below, were selected because I was familiar with the metadata and they represented a reasonable cross section of IDA collections.

1. A simple collection of images (one to one relationship of metadata to object) — The Springfield Aviation Company Collection.
2. A simple collection of compound objects (a flat, regular, one to many collection of metadata to objects) — The Music of the First World War.
3. A complex collection of compound objects (variable, one to many collection of metadata to objects) — The Illinois State Library's General Collection.

OCLC staff provided me with three tarballs of these collections. I exported the metadata in each of these collections though the CONTENTdm administration site as custom XML files. I chose this method so I could edit the element names to make the resulting records easier to manipulate:

4 First Conversion

The Springfield Aviation Company Collection was my first target. I wrote a Perl program to parse the exported monolithic metadata file into separate XML records (Appendix One).

The resulting XML records were transformed into:

I then created the necessary indexing rules in XTF to create a viable index of the collection and indexed the XML records. The XTF instance can be found here. The thumbnails in the brief record display are found in the <cdmFileName> element. The JPEG icon file itself is found by simply adding "icon" to the <cdmFileName> value, icon29.jpg in the above example.

I opted to build Perl programs for the full record display. The full record is linked to the title element display in the brief record (Appendix Two). The information I pass to the full record Perl program is the XML record number and the collection name.

The full record is built in two parts. The first part is a simple text file, named {collection name}.txt, which translates the XML element names into more descriptive text. linl.txt consists of:

The sectionHead tag indicates a display header, intended to organize the information on the screen in a more logical manner. Below is a screen shot of the result. View the live record here.

Shown below is its IDA/CONTENTdm counterpart. View the live record here.

5 Second Conversion

My second test collection was The Music of the First World War. This collection, from the Pritzker Military Museum and Library, consists of flat, regular records of compound objects. Each record consists of five parts, a JPEG cover image, a PDF of a whole musical score, and MP3, AIFF, and MIDI copies of the digitized music.

CONTENTdm's export yielded these compound objects in two parts. The first part is the metadata itself and the second is an XML file that points to the actual digital objects. The metadata XML record looks like:

The <cpdfile> element points to an XML file that contains pointers to the actual digital objects:

For ease of display for this test simulation, I decided to convert the JPEG2000 formatted files into conventional JPEG files. Were I doing this in a real situation, I would install djatoka as a Tomcat webapp so that I properly handle JPEG2000 files (see Chute & Van de Sompel, Introducing djatoka, D-Lib Magazine, 2008). I converted the JPEG2000 files using Imagemagick for this test instance.

I opted to concatenate the two XML files into one file for ease of indexing and full record display. I wrote a Perl program that brought the two parts together. The resulting XML file looks like:

I can index the <aiffFile>, <mp3File>, etc. elements in XTF:

I can then link to the different parts of the compound object and make them available directly from the brief record:

6 Third Conversion

The third data set was by far the most challenging. It includes a wide variety of material from the Illinois State Library's own collection. Most of the items consist of complex compound objects, and like the second data set, multi-part XML records. The exported parent metadata record looks like:

The accompanying compound object XML file:

I admit to facing a quandary at this point. How could I search within each of these complex documents? My indexing instructions to XTF would account for keywords within the titles, descriptions, etc., but I would miss the terms within each document. One approach would be to export the metadata with the page-level metadata flag checked. The huge resulting file generally contains a page-level transcript element for each page of the compound object:

I could then concatenate the data from each <transcript> element for each item in the collection and place it into one single large <transcript> element in the XML record. Instead, I opted for a second approach. I concatenated the XML data from the two halves of the record into one XML file:

I then wrote a program that brachiated each XML record's <cpd> section, looking for the <pagefile> element, and built a concatenated PDF file from the individual JPEG files using ImageMagick. I also used ImageMagick to convert the JPEG2000 files into JPEGs, as needed. My Perl program used system calls to work with ImageMagick. It did not occur to me until after the fact that, yes, there is a Perl library designed to work with ImageMagick called PerlMagick. I would install and use PerlMagick if I had to do this again. I named the resulting PDF file with the record number of the XML record and added an element to reflect this: <pdfFile>131.pdf</pdfFile>. Building in the indexing rules in XTF allowed me to index the actual content of the created PDF file as well as the XML record.

Having a <cpd> page-level section in the XML record allows me to use the JavaScript library turn.js (free on Github) to load the component JPEG files into a flip book. The resulting screen is shown below. View the live record here.

I modified the two loop engine in my xtfFullRecord.pl program to search for a <cpd> section, and, if found, build up a string with each of the <pagefile> values:

which generates HTML code:

I used version 3 of turn.js. Version 4 offers a much larger array of features. As of this writing, licensing version 4 costs $45.00 USD.

7 Conclusion and Lessons Learned

This exercise was a fire drill. As such, there were a number of things I did not fully implement, such as faceted browsing and a proper, unified UX. The point of the exercise was to see if the data could be reasonably converted so I did not pay much attention to design elements. Were I to do this again, I would start with solid UX planning. I would also look at examples of all of the metadata in all of the collections being converted in order to plan what elements can be indexed and how faceted browsing could be implemented. The lack of these two steps resulted in wasted coding time and clumsier programs.

I estimate I spent a total of forty hours on this project, most of it learning how to manipulate the raw XML files and understanding how the structures worked together. Since this was intended as a fire drill, I did not turn to CONTENTdm's support for assistance — I puzzled out the pieces in isolation.

Each collection presented different challenges. Each conversion required custom programming. In planning for a conversion, in addition to UX concerns I would recommend reviewing the metadata elements of each collection and normalizing like elements as much as possible.

I think my approach to including the <cpd> section as part of the XML metadata record is potentially very useful, particularly if one uses the powerful features of version 4 of turn.js (for example, to build a table of contents using the <pagetitle> element). As I mentioned above, I would also install djatoka as a Tomcat webapp so that I could properly handle the component JPEG2000 files.

Were this a real situation, I might also consider using CONTENTdm's APIs as a data transfer medium. CONTENTdm has a number of useful APIs for directly accessing information contained in its indexes and files. I have been working on a cookbook of techniques for using the APIs to manipulate CONTENTdm collections. Please note that neither path — manipulating the XML directly or using CONTENTdm's APIs — is an easy one for conversion.

Finally, I only used part of XTF, and then only in its default mode. The Perl program for full record display, for example, could have been implemented in XSLT. Rowan Brownlee describes how to do this in his excellent tutorial, "Rowan Brownlee's Beginner's Guide to XTF". XTF, or whatever content management system one might use as a backend, should be configured and customized to properly accommodate the exported data set.

Appendix One

Appendix Two

About the Author

Andrew Bullen is the Information Technology Coordinator at the Illinois State Library (ISL). Among his other responsibilities is the technical management of the digital information systems at ISL. His CV can be found here.