Python - read parquet file without pandas

Question

Currently I'm using the code below on Python 3.5, Windows to read in a parquet file.

import pandas as pd

parquetfilename = 'File1.parquet'
parquetFile = pd.read_parquet(parquetfilename, columns=['column1', 'column2'])

However, I'd like to do so without using pandas. How to best do this? I'm using both Python 2.7 and 3.6 on Windows.

What about the pyarrow.parquet library? [link](https://arrow.apache.org/docs/python/parquet.html) As a matter of curiosity, why don't you want to use pandas? — Uncle Ben Ben, Jun 22 '18 at 12:45
@user2413548, I've already tried this out. But I can't seem to install parquet on both 2.7 and 3.6. — inquisitiveProgrammer, Jun 22 '18 at 13:13
@UncleBenBen pandas is a bulky module and difficult to install on aws lambda, hence I'm looking for a workaround. — inquisitiveProgrammer, Jun 22 '18 at 13:15
@inquisitiveProgrammer did you try this ? https://stackoverflow.com/a/45147779/1552998 — stormfield, Apr 03 '19 at 23:56

score 1 · Answer 1 · answered Jul 08 '20 at 00:26

You can use duckdb for this. It's an embedded RDBMS similar to SQLite but with OLAP in mind. There's a nice Python API and a SQL function to import Parquet files:

import duckdb

conn = duckdb.connect(":memory:") # or a file name to persist the DB

# Keep in mind this doesn't support partitioned datasets,
# so you can only read one partition at a time
conn.execute("CREATE TABLE mydata AS SELECT * FROM parquet_scan('/path/to/mydata.parquet')")

# Export a query as CSV
conn.execute("COPY (SELECT * FROM mydata WHERE col = 'val') TO 'col_val.csv' WITH (HEADER 1, DELIMITER ',')")

Python - read parquet file without pandas

1 Answers1