51

Is that even possible!?!

I have a bunch of legacy reports that I need to import into a database. However, they're all in pdf format. Are there any R packages that can read pdf? Or should I leave that to a command line tool?

The reports were made in excel and then pdfed, so they have regular structure, but many blank "cells".

Tomas
  • 57,621
  • 49
  • 238
  • 373
Justin
  • 42,475
  • 9
  • 93
  • 111
  • 2
    Taking a glance at CRAN, there doesn't appear to be any library that does that. You might be better off using another language that has such libraries (Perl and Python, for example, both have them), grabbing the data that you need, and then writing it to a file that can be read by R. –  Feb 07 '12 at 23:51
  • 1
    @JackManey Thanks, that's what I thought. There is `readPDF` in the `tm` package (text mining), but it isn't exactly user friendly and I think it uses the command line utility `pdftotext` under the hood anyway. – Justin Feb 07 '12 at 23:56
  • 4
    You have my sympathies. Maybe some day we'll live in a world where all data is available as data! – Ari B. Friedman Feb 08 '12 at 00:21
  • 1
    @gsk3 (+1) I appreciate the condolences... I spend most of my days wishing that. And since people are paying attention and I didn't look hard enough... (http://stackoverflow.com/questions/3852354/extracting-text-data-from-pdf-files) confirms my suspicions. – Justin Feb 08 '12 at 00:29
  • 1
    There is also the [grImport](http://www.warwick.ac.uk/statsdept/user-2011/TalkSlides/Contributed/18Aug_1400_KaleidIIIb_2-Murrell.pdf) package, which can read PDF files, but it is designed to extract vector graphics -- the text will also be there, but perhaps not in a very useable form. – Vincent Zoonekynd Feb 08 '12 at 00:57
  • I've never had success with `tm::readPDF`, but managed a work-around using `pdftotext` in my `R` workflow like this: http://stackoverflow.com/a/19926301/1036500 – Ben Jul 02 '14 at 18:25

5 Answers5

31

So... this gets me close even on a fairly complex table.

Download a sample pdf from bmi pdf

library(tm)

pdf <- readPDF(PdftotextOptions = "-layout")

dat <- pdf(elem = list(uri='bmi_tbl.pdf'), language='en', id='id1')

dat <- gsub(' +', ',', dat)
out <- read.csv(textConnection(dat), header=FALSE)
zx8754
  • 52,746
  • 12
  • 114
  • 209
Justin
  • 42,475
  • 9
  • 93
  • 111
  • I am running into problems which I do not know how to solve. The following line `dat <- pdf(elem = list(uri='C:/Users/Farrel/Downloads/bmi_tbl.pdf'), language='en', id='id1')` produces the following error `Error in file(con, "r") : cannot open the connection In addition: Warning message: In file(con, "r") : cannot open file 'C:\Users\Farrel\AppData\Local\Temp\RtmpegXWQ3\pdfinfo57c9716105': No such file or directory`. – Farrel Nov 29 '13 at 15:44
  • it does not seem to work for me. I want to extract some text from that. let me know how I can do it. – Arun Raja Mar 23 '15 at 04:48
  • it shows an error unused argument(PdftotextOptions = "-layout") Calls – Moby M Aug 03 '17 at 09:45
23

Just a warning to others who may be hoping to extract data: PDF is a container, not a format. If the original document does not contain actual text, as opposed to bitmapped images of text or possibly even uglier things than I can imagine, nothing other than OCR can help you.

On top of that, in my sad experience there's no guarantee that apps which create PDF docs all behave the same, so the data in your table may or may not be read out in the desired order (as a result of the way the doc was built). Be cautious.

Probably better to make a couple grad students transcribe the data for you. They're cheap :-)

Carl Witthoft
  • 20,573
  • 9
  • 43
  • 73
  • 1
    I wish! Some of us don't have grad students to do our bidding. And I'm too low on the totem pole to hire interns (read lackeys). But good advice! – Justin Feb 08 '12 at 02:44
  • @CarlWitthoft I'll accept your answer! Particularly the last line. – Justin Feb 09 '12 at 15:41
  • 4
    Humans are lousy. I know because I am one and I know lots of others. They excel at three things: solving novel problems; creativity (music, arts and literature); and interpersonal emotional support or persuasion. They can not be relied upon to transcribe. – Farrel Nov 29 '13 at 15:19
13

The current package du jour for getting text out of PDFs is pdftools (successor to Rpoppler, noted above), works great on Linux, Windows and OSX:

install.packages("pdftools")
library(pdftools)
download.file("http://arxiv.org/pdf/1403.2805.pdf", "1403.2805.pdf", mode = "wb")
txt <- pdf_text("1403.2805.pdf")

# first page text
cat(txt[1])

# second page text
cat(txt[2])
Ben
  • 41,615
  • 18
  • 132
  • 227
  • I like this package. – Yimihua Sep 13 '16 at 18:47
  • You may also find https://github.com/ropenscilabs/tabulizer useful for extracting data from tables in PDF files – Ben Sep 13 '16 at 20:44
  • 1
    @Ben this worked first time for me; great answer. btw: the French phrase is "du jour" which translates to "of the day", not "de jour" meaning "of day". Sorry to be pedantic :-) – hackR Dec 15 '16 at 20:50
  • @hackR Merci beaucoup pour votre commentaire ;) – Ben Dec 16 '16 at 00:17
6

You can also (now) use the new (2015-07) Rpoppler pacakge:

Rpoppler::PDF_text(file)

It includes 3 functions (4, really, but one just gets you a ptr to the PDF object):

  • PDF_fonts PDF font information
  • PDF_info PDF document information
  • PDF_text PDF text extraction

(posting as an answer to help new searchers find the package).

hrbrmstr
  • 77,368
  • 11
  • 139
  • 205
4

per zx8754 ... the following works in Win7 with pdftotext.exe in the working directory:

library(tm)
uri = 'bmi_tbl.pdf'
pdf = readPDF(control = list(text = "-layout"))(elem = list(uri = uri),
                                                language = "en", id = "id1")
Paul McGee
  • 102
  • 2
  • 7