CSV Imports into Google Cloud BigQuery

If you have a large Postgres database, consider trying Google Cloud BigQuery for OLAP-style reporting queries.

database> COPY (<query_to_denomralize_report_data>) TO '<dest_filename>' DELIMITER ',' CSV HEADER;

Then upload it to GCS:

$ gsutil cp <dest_filename> gs://<dest_bucket>/<dest_filename>

Then, follow Loading Data into BigQuery (with Google Cloud Storage). For this to work, you’ll have to define the schema and skip the first leading row.

My most recent load job took 4 seconds, loading a very large CSV file. Query outputs can be saved to Google Sheets or other BigQuery tables.

For those of you who are looking for something similar on AWS, check out Amazon Athena.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s