ODI Leeds

Best Practices for PDF and Data

Best Practices for PDF and Data.

Earlier this year we shared news that we were working with Adobe and The ODI in London to investigate how PDFs could play nicely with data. Today we are sharing our report containing our findings, and making recommendations.

Download Best Practices for PDF and Data.pdf now. Or read the report as HTML on this page.

Our work and interviews have been extremely wide-ranging. We’ve heard from people in the open data community who recoil in terror at hearing the word PDF. We’ve heard from people whose jobs would not possible without it. We’ve used, improved, written, and published our own open-source software and tools to make more of the standard.

Our report is clear; data in spreadsheets should be released as spreadseheets or CSVs, not printed to PDF. Additional research shows that in most of the world this advice is being followed, but that the USA is doing less well.

But when it comes to publishing or creating documents, there is good reason for many to look again at the PDF format. It is an open standard, it can be manipulated with open-source tools, and our work opens the possibility for it to contain data in many forms.

Our report is available in the PDF standard that you all know and many of you love. Additionally, it makes use of PDF support for attachments to include structured data including a fully editable version of the report, the appendices, and our spreadsheet of use cases at the time of publication.

Report with attached files in PDFAttacher
The report includes data and other file attachments.

You can create similar PDFs using our open-source PDFAttacher software.

What's next?

The next steps that we set out in the report will be continued in collaboration with The ODI in London via the W3C Community Group on PDF and Open Data . Please join us in the W3C Community Group and add your use-cases to our open use-case spreadsheet.

This blog post was amended on 2017-10-24 to highlight a key message of our report, contained on pages 1, 4, 5, 6, and 16, that spreadsheets of data should be shared as spreadsheets or CSVs, not printed to PDF. View the original version of this blog post.

Funding

Our work and this report was funded by The Future Cities Catapult, ODILeeds' sponsors, and Adobe.

HTML version of the report.

Published on 2017-10-26

 

 

Best Practices for PDF and Data: Use Cases, Methods, Next Steps

Thomas Forth and Paul Connell, ODILeeds
The ODI

Summary

The Portable Document Format (PDF) has been a widely-preferred representation for reliable transmission of documents for decades. As it has become increasingly desirable to publish data underlying a document in addition to the document itself concerns have been raised about the suitability of PDF.

We have talked to a wide variety of participants, analysed in depth several use cases, and demonstrated how PDF can play a dual role of faithfully communicating human-readable documents and carry data.

In particular, we focused on the civil planning process in the UK, where PDFs are essential. We have created, documented, and tested, within the existing workflow, a method that improves PDFs within the existing standard and makes them work better with data. We have retained current strengths within existing workflows, adding the use of PDF’s ability to embed and archive data.

We show how our work is shaping the thinking of the UK government around the future of digital documents and suggest a number of other similar use cases.

Background

The rise of personal computing in the late 1980s and early 1990s broke the link between content and layout in documents. Even when styled for print, the same content would often look different in different settings. By combining layout, content, graphics, and fonts into one document, PDF bridged that gap. A PDF file always looked and printed the same.

Today there is a new divide. Documents and especially the charts, diagrams, infographics and arguments within them, are increasingly based on data that is open, available to the public, and changing. However, the data used to create the content within documents is difficult to extract and the original source can often be hard to find, especially as the document ages.

During the last six months, we have spoken at many events, talking with dozens of people, businesses, and communities to understand how well PDF works with these new types of document. We have heard that often PDFs work well, but that sometimes they don’t. In these cases, we have asked what should be done, collected the responses, and summarised them in this report.

The most negative opinions we have heard on how PDFs work with data are from people who regularly extract data represented as content within PDF documents -- for example, where a government or business has shared a spreadsheet by printing it to PDF. In this case an obvious solution is to publish spreadsheets not just PDFs, and this change in practice is well underway. Today on Data Mill North, one of the UK’s largest local government open data portals, there are ten times as many CSVs and spreadsheets as there are PDFs.

But in many cases, especially where printable documents are deeply-embedded into existing workflows, we have heard much more positive opinions on the role of PDF.

One such area is scientific publishing, where “the paper” — laid out for print and containing complex embedded figures, tables, and formulas — remains the standard unit of output. Considerable efforts have been made to move away from PDF as a standard within scientific publishing and none has succeeded. We examine what has made PDF so useful within scientific publishing and suggest how the document format could serve many of the use cases for which alternatives have been designed.

In particular we look at the role of reference managers and the methods they used to manage personal scientific libraries that are overwhelmingly a collection of PDFs.

Other areas where we have identified PDF playing an important role are those where documents serve legal purposes, such as in city planning, government, and courts of law. In these fields, it is critical that documents are faithfully reproduced on all devices and printers, both today and decades into the future. PDF remains well-liked and widely-used for this purpose. There is little appetite to replace PDF, and considerable interest in making it work better with data.

What are portable documents?

The Portable Document Format (PDF) is an ISO standard (ISO 32000-1:2008) defining a final-form document representation language in use for document exchange, including on the Internet, since 1993.

Those are the exact words of the Internet Engineering Task Force registration for the application/pdf media type. For the rest of this report I will refer to portable documents. By far the most widely-used format for portable documents is PDF; first created by Adobe in 1993, it is now an established open ISO standard.

I have found it important over the past six months to have a shared understanding of what a portable document is in every conversation. This is especially true when discussing the way that portable documents work with data.

Data can be represented in many ways; for example, tables can be laid out in different orientations, there are different types of graph, geographical data can be displayed in many forms, from a list of coordinates to heavily-styled maps, and mathematical formulas can be displayed in different ways. I call a representation of data, such that it can form part of a document, data-derived content.

Creating data-derived content from data is almost inevitably a lossy process; the original data cannot be easily or completely recovered.

Combining data-derived content with other content (text, images, etc.) and then adding style (fonts, colours, etc.) and layout (page size, position and scale of elements, etc.) creates a portable document.

Figure 1: To a reasonable approximation a portable document is the combination of content, layout, and style (including fonts).

The essential feature of a portable document is that it looks the same (with exceptions for accessible views or views on mobile) wherever and whenever it is viewed or printed. This is a basis for reliable communication, when a document is sent, published, archived, or distributed, as the basis for further discussion.

Creating a portable document from its components is almost inevitably a lossy process; the original editable content, style, and layout cannot always be easily or completely recovered.

The difference between extracting data from data-derived content and extracting content, style, and layout from a portable document is critical. During the preparation of this report we have developed the following aid to having these discussions.

Figure 2: Extracting content from a portable document is a different challenge to extracting data from a piece of data-derived content, such as a graph, table, or mathematical formula.

Current perceptions and usage of PDF and data

We asked a variety of people and organisations how they used PDFs today, how they felt PDFs currently work with data, and how PDFs could work for them better.

We have spoken with the UK and European open data community. This includes attending and presenting at The Open Data Camp in Cardiff, The Software Sustainability Institute’s annual conference in Leeds, The Future Cities Catapult Future of Planning series, several Open Data Manchester meetings, The ODILeeds showcase, and The Open Data Institute’s Friday lunchtime lecture, which was broadcast live on the web.

We have spoken with software developers, scientific publishers, print designers, and typographers.

Finally, we have spoken with government administrators, civil servants, city planners, lawyers, and accountants.

During these discussions, we heard common themes which we have, consequently, collated into three groups.

  1. We found a group of users that had negative opinions of PDF; not because of any major deficiency in the format, but because the documents that they were dealing with would have been more usefully shared as raw data, not only as a document. This group were often users of open data and developers of tools that extract data from PDFs.
  2. We found a group of users that valued PDFs but were concerned that it might not meet their future needs. Content extraction for search was a key interest and considerable efforts were being made to do this better. Member of this group had often tried to develop alternatives to PDF, but none had been widely adopted. This group were sceptical, but interested in the possibility of PDFs that might work better with data to solve their problems. This group included people who worked within libraries, archival, and print, especially scientific publication.
  3. We found a group of users for whom documents were essential to many workflows, with PDF almost always the preferred format. This group were frustrated by proposed alternatives to PDF, which often moved away from the concept of a document. They considered alternatives to documents poorly matched to their needs. They were very interested in the possibility of PDFs that could work better with data to solve their problems. This group frequently included individuals from business and government.

For each group, I have expanded on the opinions we heard below, including relevant additional information for context.

Group 1: Users of open data and developers of PDF data extraction tools.

At first, people that we spoke to about PDFs in the open data community were hostile. Their anecdotes included retyping data from an embedded table, tracing points from an embedded graph, removing line breaks from copied paragraphs of text, taking high resolution screenshots of embedded photographs, and struggling to follow links or get back to the original source of a document. A surprisingly large number had stories of spreadsheets or databases that they had requested the contents of being printed to PDF and sent to them, rather than released directly.

But when we spoke in more detail - explaining the structure of portable documents, as given in figure 1 and figure 2 - their opinions softened. Users who had received data that had been printed from a spreadsheet understood that the problem was not the PDF format itself, but rather the decision to print a spreadsheet to PDF instead of releasing the spreadsheet. This process, outlined in figure 3, makes the original data hard to recover.

Figure 3: Printing a spreadsheet to a PDF causes the content to undergo two lossy processes.

The solution here is simple; release the spreadsheet. However, in some cases, a PDF is all that people have. In this case you need to try and extract data from content.

To understand more about this process, I conducted a full interview with PDFTables, below.

PDFTables

PDFTables is a webservice that converts tables in PDFs into spreadsheets. To date it has converted nearly 60 million pages of PDF.

Figure 4: PDFTables uses computer vision to recognise tables in PDFs and extract data from them.

In the creation of data-derived content from data, the concept of a table is often lost. For this reason, PDFTables uses computer vision to recognises tables by their regular layout. There are many similar tools and they mostly work in the same way.

Because text is selectable and copy-able in PDFs, the content of cells in the output spreadsheet can be populated with high accuracy, though some mistakes are made in the assignment of data to cells, especially with complicated table structures.

The conversion of data to data-derived content is lossy, and recovering data by reversing the process is never guaranteed to be perfect. For this reason, PDFTables recommend that all customers try to find the original data source before using their tool.

Opinions about PDF, within the open data community, have been tainted by its misuse in the early days of open data release. The solution is to keep pushing for open data to be released in sensible formats. Where original sources can be located, they should replace current poor formats. Where they cannot, good tools such as PDFTables and PDF.io can extract data from tables.

But tables are just one form of data-derived content and many forms of data-derived content, such as document structure, graphs and mathematical formulas, are much harder to extract data from. Trying to do this, and often failing, are the challenges associated with our second group of users.

Group 2: PDF users within libraries, archival, and print: especially scientific publishing.

For better or worse, “the paper” is still the unit of scientific output against which scientific research careers are judged, and through which most scientific understanding is shared. Photocopies of journal pages from the library have largely been replaced by printed papers on the desks of professors. These have been, in turn, increasingly replaced by software libraries of PDFs with digital annotations. But the document, almost always a PDF portable document, is a constant.

We have spoken with a wide range of people working in science to gather their views on PDF, including library staff and archivists, researchers, and publishers. At first their concerns were similar to group 1; they found it frustrating that they cannot extract data, although their experience differed in two key ways.

  1. The documents that they were working with are increasingly complicated, with many different types of content and significant meaning embedded in the layout and style.
  2. They have made considerable efforts to replace the PDF within their workflow and failed.

The first point is best addressed by returning to figure 2. Extracting the content from a scientific paper is not that difficult and, as covered in the next section of this report, content extraction is increasingly faithful. But extracting content from a PDF does not get you back to the data that created that content.

There are a number of tools that can attempt to extract (or re-create) this data.

  • WebPlotDigitizer is a tool that can extract data from charts.
  • InftyReader is a tool that can extract or recreate a LaTeX representation of mathematical formulas from printed formulas.
  • PDFTables, as already described extracts data from tables.
  • ScienceBeam is a tool that tries to understand the context of text within a scientific publication. For example, which text is the title, which is the date, which is the document body, which is the abstract, etc…

All of these approaches, and most others for different types of data-derived content, use computer vision. They try to treat the data-derived content as a human would, using the information that is contained in layout and styling to understand the elements of the document. In all of these approaches, critically, even with access to the document from which the PDF was created — in Microsoft Word or Adobe InDesign format for example — the data would not be easily extractable.

Once this has been explained, this group of users was much less hostile to the PDF format. Indeed, we then heard many more positive opinions about PDF, including an opportunity to use existing under-used features of the format to improve workflows in science.

Everyone in science uses PDFs and almost everyone in science creates their own archive of PDFs in some way. When everything was in print, archival was simple. Things were printed, that copy stayed the same, and you stored it in a folder or a library. PDF, as a digital extension of print, initially led to digital archiving in the same way - you kept your PDFs in a folder.

As papers increasingly refer to other documents, extended methods documents, scientific models, and large raw datasets, archival has become harder. These extra files are usually stored in their own file formats, contained within custom document management systems, and made available for separate download from publishers’ websites.

When downloaded to users' computers they are usually stored either in folders with the PDF that they refer to, or within reference management systems, provided by a range of software providers.

Often the only link between a scientific paper and the data that is increasingly an essential part of that paper is a URI embedded in the PDF or on the publisher's page from which the PDF was originally downloaded. Fears of these links dying, link rot (explained below), is a concern for archivists and there is some interest in the role that attaching associated files and data to PDFs could play in solving this problem, especially if the contents of these files remained easily searchable.

Link Rot

Links between documents, between documents and data, and within data itself are hugely valuable. The world-wide web and its popularisation of the uniform resource locator (URL) — a subset of the uniform resource identifier (URI) — was a hugely important step in demonstrating the value of the link to everyone.

The problem with links is that they rot. Over time the destinations that they point to disappear. Indeed, some of the links in this report have died during its preparation.

Figure 5: Link rot has already affected this report, with permalinks to Adobe Blogs about PDFs and data no longer working.

Current best estimates are that around 5% of links in scientific papers die per year. This is of huge concern to people who manage and archive documents. If a portable document relies on a link that can die, this reduces how well it meets one of its key criteria for success.

Group 3: PDF users within business and government.

Our final grouping of users was the most positive. We especially had very positive reactions with participants in The Future Cities Catapult’s Future of Planning series when discussing the role of PDF in the context of planning, legal, and strategy work within Southwark council, Bradford council, and Leeds council.

The common trait for all of the users in this group was that documents played a critical role in how they worked. Often this was for legal reasons; for example, where documents were how they collected consultation responses, defined laws, or conducted and documented court proceedings. These users valued that PDFs could be trusted to faithfully reproduce on all devices and printers, both today and decades into the future. They valued features of the PDF such as password protection and tamper-resistance. And they valued features of the PDF, both technical and legal, that make contents hard to extract in bulk, including a recent UK legal ruling that a PDF may enjoy database rights in EU law.

It should also be noted that this group of users had two concerns about PDFs. Many thought that PDF were a closed standard, controlled by Adobe, and almost always produced by Microsoft’s Office tools. Since they were increasingly being pushed to avoid reliance on proprietary software, this was a concern to them. Many were also keen to point out that any improvement to PDFs needed to fit within their existing workflows. A key part of this was that almost all the users in this group used Windows, often Windows 7. Both of these concerns suggest simple solutions. First, to better communicate that PDF is a broadly implemented, open ISO standard. And second, to ensure that any new workflows developed with PDFs work well on the systems in use today.

Examples of working with PDFs and data, and new possibilities

During this project, we’ve tried lots of software and written our own software to open, create, and manipulate PDFs. We have also tested lots of processes and workflows for working with and managing PDFs. Here I present a selection of them, split into themes.

These examples help to show how PDFs are currently used with data. They document good examples, previous efforts, and some dead-ends, in the hope of guiding future work.

Extracting content from PDFs

Most people have had an experience where they extracted some text, a photograph, a graph, or a table from a PDF. This is the process marked as “content extraction from PDF” in figure 2 and we’ve heard from a lot of people that found it difficult.

To explain why this is process can be difficult we need to make clear that creating a portable document is a lossy process. Some editability of original content is almost inevitably lost.

In order to understand the extent to which editability is lost and to see how good the existing tools for extracting content from PDFs are, I conducted a trial of both free and paid software. The results are published on my blog and show that excellent tools, both free and paid, exist to extract content from PDFs.

But there is an alternative to content extraction, offered by something called Hybrid PDF. A hybrid PDF file contains both the original document and a portable version, as described below.

Hybrid PDFs

When you export a document to PDF from LibreOffice or OpenOffice you have the option of creating a Hybrid PDF. That means that the original file, in the Open Document Format (ODF), is embedded into the PDF.

Figure 6: Hybrid PDFs can be created using Export as PDF… in LibreOffice and OpenOffice.

If you open a Hybrid PDF created in this way in any PDF reader, you get a faithful reproduction of the original: that’s the PDF part. But if you open the file in LibreOffice or OpenOffice, you get the original document as it was before you exported it to PDF.

Hybrid PDFs highlight the limit of content extraction. Even though the content is perfectly extractable, it does not mean that data can be extracted from data-derived content within the document. For example, a chart in an ODF document is still a chart and not the original data.

Other problems with hybrid PDFs are that, even though ODF files are widely readable, LibreOffice and OpenOffice are the only pieces of software that know what to do with hybrid PDFs. If you open a Hybrid PDF almost anywhere else there’s no indication that an ODF file is attached or embedded. The file extension of a Hybrid PDF is the same as a normal PDF, there’s no indication in the metadata that you’re looking at a hybrid PDF, and it’s surprisingly difficult to tell whether a PDF is hybrid or not.

Hybrid PDFs have been tried and have largely failed as a method for making content perfectly recoverable from portable documents. In our PDF for Planners example later in this report, we will argue that better alternatives exist today.

Before continuing, it is worth noting some other efforts around extracting data from PDFs which we have investigated and may be of interest to readers.

CloudTrade

CloudTrade are a company that helps small businesses move towards electronic invoicing. Their process relies on the legacy PDF documents that most small businesses, and even large businesses use. It keeps invoices human-readable while processing the contents digitally.

Their system takes advantage of the fact that PDFs can contain structured content. Each text field in a PDF can be given a name, an ID, etc… as in HTML. Because CloudTrade knows what names each invoicing software package gives to each field in the PDFs it creates, it can extract them directly into a database or a standard format for e-Invoicing and purchasing, such as EDI.

CloudTrade’s approach shows how the structured content features of the PDF format can enable accurate data extraction and it might be a useful template for other applications.

Portable data documents: attaching data to PDFs with PDFBox

"The Apache PDFBox® library is an open source Java tool for working with PDF documents. This project allows creation of new PDF documents, manipulation of existing documents and the ability to extract content from documents."

The above is an exact self-description of the Apache PDFBox project that produces one of the most complete open-source toolkits for working with PDFs. Of particular interest to this report, PDFBox allows the attachment of files to PDF documents. Since files can contain data, this means that we can attach data to PDF documents.

The possibility of attaching files containing data extends the possibility of what a portable document is, as below in figure 7.

Figure 7 : PDFs support the attachment and extraction of data. This is a lossless process.

Attachments are a long-supported and well documented feature of PDF. Adobe have written about it in blogs and many existing PDF readers support attachments. PDFBox contains excellent documentation for working with attachments, and the open-source PDFData project implements many of them.

We investigated support for PDF attachments in 9 popular packages for reading and manipulating PDFs. The results are below in table 1.

Of particular importance are the web browsers: Google Chrome, Mozilla Firefox, and Microsoft Edge. These browsers increasingly open PDFs themselves instead of passing them on to their host operating system’s default PDF reader. And of these browsers, only Firefox currently supports PDF Attachments, via Mozilla’s open source PDF.js platform.

The PDF.js platform offers an opportunity to quickly bring PDF attachment support to all browsers. There is already an implementation as a Google Chrome extension which adds PDF attachment support to Google Chrome, and the web version of PDF.js allows Microsoft Edge to do the same.

Table 1: Software for reading PDFs and how they handle PDF attachments. *these are frameworks used by other tools, not end-user software.

A final crucial point to make about PDF attachments is that unlike the process of creating data-derived content from data — or creating portable documents from content, layout, and style — attaching and extracting files containing data from PDFs is a lossless process. You get out exactly what you put in.

A Clearer Plan, PDF for Planners, and PDFAttacher

Once we understood the possibilities of PDF attachments as above, it became clear that they provided a solution to many of the problems experienced by group 3 earlier in this report. Specifically, for town planners, PDF attachments would allow them to keep using their existing document-reliant workflow for planning while incorporating more of the data that they increasingly use to engage with citizens.

As shown in figure 8, the town planning system within England, as in most of the world, is reliant on documents. PDF is by far the most common format for these documents and the ability to print and archive them is essential to how the system works.

Figure 8: Town planning in England, and in most of the world, creates and requires a lot of documents.

And yet town planners increasingly make their decisions using open data, using it to engage citizens and gather consultation responses. Figure 9 shows A Clearer Plan, a tool that we have built to help town planners explain their decisions and reasoning using open data.

Figure 9: Town planning is increasingly justified by open data on things like bus routes and flood risk.

A Clearer Plan also serves as a tool for gathering citizen opinions on planning applications and decisions near to them. By entering their postcode, a citizen is shown the situation in their area. This highlights the features that commonly lead to opposition to development, including flood risk, demography, school places, and public transport access.

The essential innovation of A Clearer Plan is shown in figure 10. The tool produces a document as an output, which can then be edited by a citizen and submitted to the planning system. Crucially, this output — something that we call PDF for Planners — contains the data that was used to create the document.

Figure 10: A Clearer Plan creates planning documents from open data, embedding data into PDFs it creates.

In this example, a file containing json format data is attached to the PDF document produced by A Clearer Plan using PDFAttacher; an open-source tool that I have built to read, add, and delete attachments from PDFs. This is a small single-purpose tool for Windows, powered by PDFBox. It was necessary to prove that the workflow we have developed can be implemented within existing workflows, without the purchase of new software.

Figure 11: Data attached to PDFs can be read, added to, and deleted in PDFAttacher (left) and read using many tools, including Acrobat Reader (right).

We have presented our work on A Clearer Plan and PDF for Planners to over a hundred people interested in English and Welsh planning, as part of The Future Cities Catapult Future of Planning series of events. We have trialled it within an existing planning workflow, and we have released the software on GitHub so that it serves as documentation of the process of working in a new way with PDFs and data.

The impacts that this has already had are very encouraging, forming a key component of the fourth recommendation in the next steps section at the end of this report.

PDFs in science

Scientific researchers have hundreds of PDFs on their computers and most use reference management software to help organise them. A selection of these is maintained on Wikipedia’s List of Reference Managers with well-known ones including Mendeley and Zotero.

The core feature of a reference manager is to assist with inserting citations into new documents and managing the creation and formatting of the references section at the end of the paper. But most reference managers also perform four additional tasks. They,

  1. Extract and improve metadata, such as authors, keywords, publication year, and institution so that they are searchable.
  2. Allow annotations and notes to be made on a PDF.
  3. Allow files, such as models, raw data, and appendices to be associated with a PDF.
  4. Organise PDFs so that they are consistently named and the information in points 1, 2, and 3 is sharable across a research group.

In a full blog post I have examined each in detail. I include only a summary here.

Currently all four tasks are almost always completed in proprietary and incompatible ways by each reference management software. This makes switching between reference managers hard, and reduces the chance that annotations and attached files will remain accessible in the future. There is a considerable opportunity to use the existing features of the PDF format to improve this.

  • Metadata for scientific papers, often downloaded from online repositories such as PubMed, CrossRef, or ArXiv, could be embedded into PDFs using the open ISO standard, XMP.
  • Annotations and notes are already full-supported within the PDFs format and none of the reference managers implements features beyond the standard. Annotations and notes could be embedded into PDFs rather than stored separately.
  • As already shown, PDFs support attached files and could contain the extra data currently available separately as supplementary information and weakly linked within reference managers.

Representing a scientific paper, and all of its associated metadata, comments, and supplementary information in a single PDF file would simplify scientific library management considerably. Since PDF contents are searchable, shareable, and sync-able using any of the file-syncing services that we almost all already use, it would fit neatly into existing workflows. Switching or deleting a reference manager would not mean losing notes, annotations, and attached files. All it needs is for the authors of scientific reference managers to embrace the existing power of the PDF format.

Next steps

PDF as a document format has many desirable features which make it a great choice for many applications, and difficult to replace. The demand for communicating data as well as documents has cast PDF as a barrier and a source of frustration. Features already exist within the PDF open standard to overcome these problems. We suggest the following actions to achieve this.

Build a community of users and developers

PDF, as an open standard, now belongs to the community of users. The work reported here has examined a few workflows in depth. Additional work can advance in an open community with a focus on workflows where PDF is essential. The W3C PDF Open Data Community Group was founded to address these problems, and should serve as a venue for many of these next steps.

Explain better what portable documents are and why they matter.

Although we use them all the time, most people do not understand what a portable document is. We have found that the diagram we presented in figure 1 is extremely useful in explaining this.

Once people understand that, for example, it was the decision to print a spreadsheet as a PDF which is the cause of problems — and not the fault of the PDF format itself — their opinion of PDF changes. Supporters of the PDF format should support their use where they add value, and instead promote alternatives where they don’t.

However, the printed spreadsheet example is simple and in many more complicated cases the PDF remains extremely important. In scientific publishing, people have spent considerable effort developing alternatives to PDFs. They have largely failed. The huge variety of data-derived content in a scientific paper makes it almost impossible to define a single, widely-adopted structured data format; one that stores the content of the paper and can also reliably reproduce the document for use within existing scientific workflows.

In the absence of such a format, PDF will continue to play a role in scientific publications. There is considerable scope within the existing format to extend its use to more frequently carry raw data, to carry XMP-format metadata, to carry annotations and notes, and in the future to carry a structured-data representation of the paper if such a format is ever standardised. Both would facilitate archiving and search — two functions that are large parts of the work that many PDF users do.

Develop use of other features of PDF for associating data

There are several other ways of including data in a PDF file, and more in development. These all have pros and cons, and are appropriate (or not) for different kinds of data. Attaching data, as CSV or JSON-LD is the simplest and likely the easiest to implement, and so we have started with that.

But some data really belongs in the PDF metadata represented by XMP – data about the document as a whole, such as Title, Author, Copyright. Other data might be stored as an XMP attribute.

Other file formats (images, video) also support XMP, and the principles used here for PDF might help with the inclusion of “data” in these file formats too.

In some cases, the “data” is just the plain text of the document, or the text of the table of contents. Users that want the text of a text document to be extractable can achieve this using the existing accessibility features of PDFs and by insuring that all strings of characters have ToUnicode mappings.

Further to our work on attachments to a PDF document, PDF can associate files not only with the document as a whole, but also with individual components, regions, structures. This might be more appropriate, especially for documents with multiple data sets.

PDF has a standard profile, PDF/A3, which might be more appropriate for many applications.

Create examples of PDFs with attached and embedded data that approach 5-star standard.

The 5 ★ OPEN DATA site currently uses PDF as its example of a one-star open data, with other examples going on a five-step scale. We think that with the work in this report, we may have already elevated the format to three stars.

Specially, with attached data, the second star (make it available as structured data) is probably earned. Making these documents available on the web in a non-proprietary format (such as PDF) earns a third star. There are possibilities to earn a fourth and fifth star by using fragment identifiers (for inbound links) and additional metadata (to turn values into links) to fulfil the requirement to “use URIs to denote things” and “link your data to other data to provide context”.

We have published an example of what we consider to be three-star open data in PDF format on Data Mill North and suggest pursuing the recommended process to have this recognised. Doing so would help with the next suggestion.

Promote that PDF is an open ISO standard with excellent open source tools.

Users, developers, and governments are increasingly cautious about committing to closed technical solutions. The UK government’s digital service manual warns to, “avoid contracts that lock you into proprietary software”. In addition, openness was a key concern among the first two groups of users described in this report.

Openness is not a problem for PDF as it is an open ISO standard, with excellent tools for manipulation — both proprietary and open. But we have found that many people are not aware of this.

To break the myth that PDF is an Adobe format that locks users into Adobe tools, we suggest promoting the diversity of the ecosystem. Adobe’s existing partnership with Microsoft is such an example. The W3C community group that we refer to in our next suggestion is another.

In particular, we suggest extra promotion of the work that Apache PDFBox are doing with PDF in their work to explore the future of documents. Their tool is open-source and well-documented which should reassure people and organisations considering improving existing workflows that rely on PDFs that they always have an option to use open tools, even if they choose to use non-open software where it meets their needs better.

Publicly document how PDFs with data would solve real problems.

Throughout this project we have listened to people who use and create PDFs, asking them what could be improved. We have identified a significant number of potential use cases and selected one, PDF for Planners, to build, test, and document. But there are many other potential use cases that would benefit from similar investigation, documentation, and discussion. We have described the possibilities of PDFs in science and highlighted the existing work of CloudTrade, but there are many more examples out there that we haven’t found yet.

We suggest that investigating, documenting, and discussing use cases should now become the primary initial objective of the W3C PDF and Open Data Community Group. Additionally, it should be made clear that all data, not just open data, can be considered in the use cases.

The aim of this process should be to create a list of well-defined use-cases, with use-case owners for each. We have started a spreadsheet to gather these use cases, which we will promote from this report and the W3C community group. An example to follow in this regard is the output from the W3C Working Group for CSV on the Web: Use Cases and Requirements.

Build on existing government interest for improved PDFs.

Our work with PDF for Planners has been done in collaboration with UK government bodies and generated significant interest within UK government departments. In particular we have shared our work with The Department for Communities and Local Government (DCLG). We have discussed our work in detail with the Director of Data Projects at DCLG and with politicians and we were please to see some of these ideas included in the successful manifesto in the 2017 general election, committing to “set the standards to digitise the planning process”.

There is a clear desire within DCLG to incorporate much more data within the English planning system. Investigations that they have helped fund on The Future of White Papers make this clear, while recognising the requirement for documents for the current system to work.

There is a clear opportunity, proven by our work with PDF for Planners, to promote PDFs as a preferred document format as the English planning system is updated. We should continue to explore this opportunity and others.

Funding

Our work on this project and in preparing this report was supported financially by The UK Government’s Future Cities Catapult as part of its Future of Planning series, by Adobe, and by The Sponsors of ODILeeds.

Attachments

The PDF version of the report contains attachments. These are reproduced here as links.