A Python sample project that extracts Shopify data using the Shopify GraphQL Admin API and loads it into Google BigQuery.
The project demonstrates how to build a lightweight ELT pipeline without commercial integration tools. It was created as a reference implementation for developers who want to understand how a custom data pipeline compares with managed ETL/ELT platforms such as Skyvia, Fivetran, or Airbyte.
- Extracts Shopify data using GraphQL
- Supports cursor-based pagination
- Loads Customers, Companies, and Orders
- Automatically creates BigQuery tables
- Stores the complete Shopify object in a BigQuery
JSONcolumn - Preserves nested arrays and objects without flattening
- Uses environment variables for configuration
- Written using Python with minimal dependencies
Shopify
│
▼
GraphQL Queries (.graphql files)
│
▼
ShopifyClient
│
▼
Python Dictionaries
│
▼
BigQueryClient
│
▼
BigQuery Tables
Below are the BigQuery Tables used in this pipeline:
Each table contains four columns:
| Column | Type | Description |
|---|---|---|
| shopify_id | STRING | Shopify Global ID |
| extracted_at | TIMESTAMP | Pipeline execution timestamp |
| api_version | STRING | Shopify API version used |
| payload | JSON | Complete Shopify object |
Below is a sample using the customers table:
Instead of flattening every Shopify object into relational columns, this project stores the complete payload in a BigQuery JSON column.
Advantages:
- Keeps the original Shopify structure
- New Shopify fields require only GraphQL query changes
- Supports nested objects and arrays naturally
- Ideal for a Raw Data layer
Analysts can later create views or reporting tables that flatten only the fields needed.
Example:
SELECT
JSON_VALUE(payload, '$.firstName') AS first_name
FROM customers;Arrays can be expanded using UNNEST().
shopify_bigquery/
│
├── config.py
├── shopify_client.py
├── bigquery_client.py
├── query_loader.py
├── pipeline.py
├── requirements.txt
├── .env.example
│
├── graphql/
│ ├── customers.graphql
│ ├── companies.graphql
│ └── orders.graphql
│
└── README.md
- Python 3.11 or later
- Shopify Admin API access token
- Google Cloud Project
- BigQuery Dataset
- Service Account JSON key
Clone the repository.
git clone https://github.com/yourname/shopify-bigquery.git
cd shopify-bigquery
Create a virtual environment.
python3 -m venv venv
Activate it.
Linux/macOS
source venv/bin/activate
Windows
venv\Scripts\activate
Install dependencies.
pip install -r requirements.txt
The pipeline requires a Shopify Admin API access token.
- Go to Settings → Apps → Develop Apps → Build Apps in Dev Dashboard → Create App.
- Create a new app and provide API scopes (for example, read_customers, read_orders, and any others your queries require).
- Note the generated Client ID and Client Secret.
During the development of this project, the access token was obtained using:
curl -X POST "https://<your_store>.myshopify.com/admin/oauth/access_token" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials" \
-d "client_id=<your_client_id>" \
-d "client_secret=<your_client_secret>"The response contains the access token:
{
"access_token": "shpat_xxxxxxxxxxxxxxxxx"
}Add the token to your .env file
Note: Shopify's authentication workflow may differ depending on the type of app, store, or API version. The steps above are the workflow used to develop and test this sample project.
Create a .env file.
SHOP_NAME=your-store-name
SHOPIFY_ACCESS_TOKEN=shpat_xxxxxxxxxxxxxxxxx
SHOPIFY_API_VERSION=2026-07
GOOGLE_APPLICATION_CREDENTIALS=/path/to/bigquery-key.json
GOOGLE_CLOUD_PROJECT=your-project-id
BIGQUERY_DATASET_ID=your-dataset
I've provided a .env.example where you can change the values and rename the file to .env.
GraphQL queries are stored separately under the graphql folder.
Example:
graphql/customers.graphql
graphql/orders.graphql
graphql/companies.graphql
Keeping queries outside the Python code makes maintenance easier when Shopify adds or changes fields.
Here's a sample from the customer.graphql:
python pipeline.py
The pipeline will:
- Load the GraphQL query
- Retrieve every page of data from Shopify
- Create the BigQuery table if it doesn't exist
- Insert all rows
- Print progress information
Below is the output coming from my test Shopify account:
Example:
Customer name
SELECT
JSON_VALUE(payload, '$.firstName') AS first_name,
JSON_VALUE(payload, '$.lastName') AS last_name
FROM customers;
Order prices
SELECT
JSON_VALUE(payload, '$.name') AS order_number,
JSON_VALUE(payload,
'$.currentTotalPriceSet.shopMoney.amount') AS total
FROM orders;
Order line items
SELECT
JSON_VALUE(payload, '$.name') AS order_number,
JSON_VALUE(item, '$.title') AS product,
JSON_VALUE(item, '$.quantity') AS quantity
FROM orders,
UNNEST(
JSON_QUERY_ARRAY(payload, '$.lineItems.nodes')
) AS item;
This sample intentionally stores the complete Shopify object instead of flattening it.
Reasons:
- Preserve the original API response
- Keep the ingestion pipeline simple
- Avoid schema changes when Shopify evolves
- Allow downstream modeling for analytics
Many production data platforms follow a similar Raw → Curated architecture.
Possible enhancements include:
- Incremental loading
- Logging
- Retry logic
- Parallel extraction
- Command-line arguments
- Automatic schema exploration
- Unit tests
- GitHub Actions CI
- Support for additional Shopify objects
This project is intended as an educational sample demonstrating one possible implementation of a Shopify to BigQuery pipeline.
Production deployments should consider authentication management, monitoring, retry policies, incremental synchronization, API rate limits, and operational logging.
MIT License