BigQuery

Connect to Google BigQuery using the REST API authenticated by a Google Cloud service account JSON key.

Prerequisites

Before creating a connection, you need:

1. A Google Cloud project with the BigQuery API enabled
2. A service account scoped to read-only BigQuery access
3. A JSON key downloaded for that service account
4. The location (multi-region or region) your datasets live in

Network access

Reeflow connects to BigQuery over the public internet via https://bigquery.googleapis.com. If you have a VPC service control perimeter, allow Reeflow’s egress IPs.

Creating a service account

For least-privilege access, create a dedicated service account for Reeflow rather than reusing one tied to other workloads. Google’s service accounts overview covers this in more depth.

1. Sign in to the Google Cloud console and pick your project
2. Navigate to IAM & Admin, then Service Accounts
3. Click Create service account and give it a descriptive name (e.g. reeflow-reader)
4. On the permissions step, grant BigQuery Job User (roles/bigquery.jobUser). Click Done to finish creating the service account.
5. Navigate to BigQuery, open your analytics dataset, click Sharing then Permissions, and add the service account as BigQuery Data Viewer (roles/bigquery.dataViewer). Repeat for each dataset Reeflow should access.

The equivalent gcloud commands:

Service account nameProject IDDataset

# Create the service account
gcloud iam service-accounts create reeflow-reader \
--display-name="Reeflow Reader" \
--project=my-gcp-project

# Grant job user at the project level (required for running queries)
gcloud projects add-iam-policy-binding my-gcp-project \
--member="serviceAccount:reeflow-reader@my-gcp-project.iam.gserviceaccount.com" \
--role="roles/bigquery.jobUser"

# Grant data viewer on your analytics dataset
bq add-iam-policy-binding \
--member="serviceAccount:reeflow-reader@my-gcp-project.iam.gserviceaccount.com" \
--role="roles/bigquery.dataViewer" \
my-gcp-project:analytics

Granting access to all datasets

For a simpler setup that automatically includes future datasets, grant roles/bigquery.dataViewer at the project level instead of per-dataset. Add it alongside roles/bigquery.jobUser in step 4 of the console flow, or run:

gcloud projects add-iam-policy-binding my-gcp-project \
  --member="serviceAccount:reeflow-reader@my-gcp-project.iam.gserviceaccount.com" \
  --role="roles/bigquery.dataViewer"

Generating a JSON key

Reeflow authenticates with BigQuery using a service account JSON key. The key is signed by Google and used to mint short-lived OAuth2 access tokens for each request.

1. In the Service Accounts list, click the service account you just created
2. Open the Keys tab
3. Click Add key, then Create new key
4. Choose JSON and click Create
5. Save the downloaded JSON file. Google will not show it again

Treat the key like a password

The JSON key grants the same access as the service account it belongs to. Store it in a password manager, do not commit it to source control, and rotate it if it leaks.

Tip

Service account keys do not expire by default. To rotate, generate a new key, paste it into Reeflow, then delete the old key from the Google Cloud console.

Configuration

When creating a BigQuery connection in Reeflow, provide the following:

Field	Description
Service account key	The full JSON key downloaded above. Paste it, drop the file, or browse to it. Reeflow stores it encrypted at rest.
Project ID	Your Google Cloud project identifier (e.g. my-gcp-project). Auto-filled from the service account key. Edit this only if you want to bill queries to a different project than the one the key was created for.
Location	The BigQuery multi-region (US, EU) or region (us-central1, europe-west2, etc.) where your datasets live. Queries can only join data within a single location.
Default dataset	Optional. Unqualified table references are resolved against this dataset.

Creating a connection

Create a BigQuery connection

Add a BigQuery project as a data source in Reeflow.

Navigate to Connections in the main navigation, then click New Connection.

Enter a Name for the connection and an optional Description.

Select BigQuery as the connection type.

Paste the Service account key JSON (or drop the file). Reeflow reads the project_id from the key and auto-fills the Project ID field. Edit the project ID only if you want queries billed to a different project.

Once credentials are filled, pick the Location your datasets live in and, optionally, a Default dataset for unqualified table references.

Click Test Connection to verify your credentials are correct.

Click Create Connection to save. The connection appears in your connections list.

1

2

3

4

5

6

7

Navigate to Connections in the main navigation, then click New Connection.

Enter a Name for the connection and an optional Description.

Select BigQuery as the connection type.

Paste the Service account key JSON (or drop the file). Reeflow reads the project_id from the key and auto-fills the Project ID field. Edit the project ID only if you want queries billed to a different project.

Once credentials are filled, pick the Location your datasets live in and, optionally, a Default dataset for unqualified table references.

Click Test Connection to verify your credentials are correct.

Click Create Connection to save. The connection appears in your connections list.

PreviousNext

Related

• Connections overview
• Connection and table security