Google BigQuery

Overview

The BigQuery asset synchronization feature enables direct import of asset inventory data from Google BigQuery into RegScale. This integration supports:

Table Mode: Import from a specific BigQuery table
Query Mode: Execute custom SELECT queries and import results
Dry-Run Mode: Count assets, inspect schema, and plan parallel processing
Pagination: Process large datasets in chunks via --offset and --limit for distributed execution

Features

Dual Mode Support — Import from existing BigQuery tables (table mode) or execute custom SQL queries for filtered imports (query mode)
Parallel Processing — Dry-run mode provides row counts and pagination recommendations; offset/limit pagination enables distributed job execution
Security — SQL injection protection (only SELECT queries allowed); query validation blocks dangerous operations (DELETE, UPDATE, DROP, etc.); GCP authentication via service account or Application Default Credentials
Flexibility — Case-insensitive column matching; optional fields with sensible defaults; CLI overrides or init.yaml fallbacks for all BigQuery parameters
Enterprise Ready — Reuses proven code from FedRAMP POAM BigQuery integration; comprehensive error handling and logging; built on the ScannerIntegration framework

Prerequisites

1. GCP Credentials

Configure GCP authentication using one of these methods:

Option A: Service Account JSON File (Recommended for CI/CD)

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

Option B: Application Default Credentials (Recommended for local development)

The CLI does not find Application Default Credentials on its own. After you log in, set GOOGLE_APPLICATION_CREDENTIALS to the file that the login writes:

gcloud auth application-default login
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/application_default_credentials.json"

That path is for Linux and macOS. On Windows the file is %APPDATA%\gcloud\application_default_credentials.json. If CLOUDSDK_CONFIG is set, the file is in that folder.

Option C: init.yaml Configuration

# Base64-encoded credentials (recommended)
gcpCredentialsJsonBase64: "eyJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsIC4uLn0="

# Or raw JSON (may have YAML escaping issues)
gcpCredentialsJson: '{"type": "service_account", ...}'

# Or the path to a service account JSON file
gcpCredentials: "/path/to/service-account.json"

If GOOGLE_APPLICATION_CREDENTIALS is set, the CLI uses it and ignores these keys. If not, the CLI tries the three keys in the order shown and uses the first one with a valid value. It logs a warning for each key it skips because the value is not valid. If no key is valid, the run fails before it reads BigQuery.

Generate base64-encoded credentials:

cat service-account.json | base64

2. BigQuery Permissions

The service account needs these IAM permissions:

bigquery.tables.get — Read table metadata
bigquery.jobs.create — Execute queries
bigquery.tables.getData — Read table data

IAM Role: roles/bigquery.dataViewer (includes all required permissions)

3. RegScale CLI Installation

pip install regscale-cli
regscale init  # Configure RegScale credentials

Quick Start

1. Import from BigQuery Table

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type table \
  --bigquery-project my-project \
  --bigquery-dataset inventory \
  --bigquery-table assets

2. Import with Custom Query

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type query \
  --bigquery-project my-project \
  --bigquery-query "SELECT * FROM inventory.assets WHERE status = 'Active' AND environment = 'Production'"

3. Dry-Run to Count Assets and Inspect Schema

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type table \
  --bigquery-project my-project \
  --bigquery-dataset inventory \
  --bigquery-table assets \
  --dry-run

Usage Modes

Table Mode

Import all data from a specific BigQuery table.

Required Parameters:

--source-type table
--bigquery-project — GCP project ID (or set gcpBigQueryProject in init.yaml)
--bigquery-dataset — BigQuery dataset ID (or set gcpBigQueryDataset in init.yaml)
--bigquery-table — BigQuery table name (or set gcpBigQueryAssetTable in init.yaml)

Example:

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type table \
  --bigquery-project production-project \
  --bigquery-dataset asset_inventory \
  --bigquery-table current_assets

When to Use:

You have a dedicated asset table in BigQuery
No filtering or transformation needed
Fastest option for full table imports

Query Mode

Execute a custom SELECT query and import the results.

Required Parameters:

--source-type query
--bigquery-project — GCP project ID
--bigquery-query — SQL SELECT query

Example:

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type query \
  --bigquery-project production-project \
  --bigquery-query "
    SELECT
      hostname as name,
      asset_id as assetId,
      type as assetType,
      description,
      status,
      ip as ipAddress,
      location,
      owner
    FROM inventory.assets
    WHERE status = 'Active'
      AND last_seen > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
  "

When to Use:

Need to filter assets (by status, date, location, etc.)
Need to join multiple tables
Need to transform column names or values
Want to import a subset of assets

A filtered query is still a full run, so it retires the plan's other BigQuery assets. Read Assets That a Run Does Not Return first.

Query Guidelines:

Only SELECT queries are allowed (security protection)
Use column aliases to match required schema (e.g., hostname as name)
Enclose queries in double quotes
Use proper SQL escaping for string literals within the query

Dry-Run Mode

Report the row count and a pagination recommendation without importing data. In table mode, dry-run also displays the BigQuery table schema.

The row count comes from a different source in each mode:

In table mode, --dry-run reports the table's row-count metadata (numRows). It does not read the rows. The metadata does not count rows that are still in the streaming buffer, so a table that receives streaming inserts can show fewer rows than the sync reads.
In query mode, --dry-run runs SELECT COUNT(*) over your query.

A view or an external table stores no rows of its own, so its row-count metadata can be 0 or absent. If it is 0, a table-mode dry run reports 0 rows. If it is absent, the dry run counts the rows with SELECT COUNT(*) instead. The sync itself reads every row of the view or external table. To count the rows of a view, run the dry run in query mode, for example with --source-type query --bigquery-query "SELECT * FROM inventory.assets_view".

Parameters:

Add --dry-run flag to any table or query mode command

Example:

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type table \
  --bigquery-project production-project \
  --bigquery-dataset inventory \
  --bigquery-table assets \
  --dry-run

Output:

[dry-run] BigQuery source: assets
[dry-run] Total rows: 25000
[dry-run] Schema:
  - name (STRING, NULLABLE)
  - assetId (STRING, REQUIRED)
  - assetType (STRING, NULLABLE)
  - status (STRING, NULLABLE)
  - ipAddress (STRING, NULLABLE)
[dry-run] Recommended pagination: 5 pages of 5000 rows

When to Use:

Before importing large datasets
Planning parallel job execution with the orchestrator/job splitter
Estimating processing time and costs
Validating BigQuery connectivity and schema

Pagination and Parallel Processing

For large datasets, use pagination to split the import into parallel jobs. The --offset and --limit flags push pagination directly to BigQuery SQL, so only the requested slice is fetched.

Parameters

--offset INTEGER — Number of rows to skip (default: 0)
--limit INTEGER — Maximum rows to process (default: unlimited)

Single Job with Pagination

Process a specific chunk of data:

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type table \
  --bigquery-project my-project \
  --bigquery-dataset inventory \
  --bigquery-table assets \
  --offset 1000 \
  --limit 1000

Parallel Jobs with Shell Script

#!/bin/bash
# parallel-import.sh

TOTAL_ASSETS=10000
CHUNK_SIZE=1000
NUM_JOBS=$((TOTAL_ASSETS / CHUNK_SIZE))

for i in $(seq 0 $((NUM_JOBS - 1))); do
  OFFSET=$((i * CHUNK_SIZE))

  regscale bigquery sync_assets_bq \
    --regscale_ssp_id 12345 \
    --source-type table \
    --bigquery-project my-project \
    --bigquery-dataset inventory \
    --bigquery-table assets \
    --offset $OFFSET \
    --limit $CHUNK_SIZE &

  echo "Started job $((i + 1))/$NUM_JOBS (offset: $OFFSET)"
done

wait
echo "All parallel jobs completed!"

Parallel Jobs with a Kubernetes Indexed Job

apiVersion: batch/v1
kind: Job
metadata:
  name: bigquery-asset-import
spec:
  completions: 10          # TOTAL_ASSETS / CHUNK_SIZE
  parallelism: 5
  completionMode: Indexed
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: regscale
          image: regscale/regscale-cli:latest
          command: ["/bin/sh", "-c"]
          args:
            - |
              CHUNK_SIZE=1000
              OFFSET=$((JOB_COMPLETION_INDEX * CHUNK_SIZE))
              regscale bigquery sync_assets_bq                 --regscale_ssp_id 12345                 --source-type table                 --bigquery-project my-project                 --bigquery-dataset inventory                 --bigquery-table assets                 --offset "$OFFSET"                 --limit "$CHUNK_SIZE"

BigQuery Schema Requirements

Required Columns

Your BigQuery table or query must include a name column and an assetId column. If either column is missing, the run fails with Missing required columns before it writes anything. Two cases fail differently:

A paginated run (any --limit, or an --offset above 0) sorts on assetId, so without that column BigQuery fails the run with Unrecognized name: assetId.
A source that returns no rows is not checked, and the run creates no asset. A full run over it still retires every BigQuery asset of the security plan. See Assets That a Run Does Not Return.

Column	Type	Asset field	Rule	Example
`name`	STRING	`name`	The asset name. An empty value falls back to the row's `assetId`.	`web-server-01`
`assetId`	STRING	`otherTrackingNumber`	The asset's key, stored in the asset's Other Tracking Number. Findings and later syncs match on it. A row with an empty `assetId` is skipped and counted as a failure. A run where no row has an `assetId` value exits 1.	`asset-12345`

Give every row its own assetId. RegScale matches the key in any case, so rows that share an assetId, or that differ only in case (asset-1 and ASSET-1), update one asset. A paginated run sorts on assetId, so rows that share a value can land in two slices or in none.

Optional Columns

The sync reads these columns when they are present:

Column	Type	Asset field	Rule	Example
`assetType`	STRING	`assetType`	Sent as written. RegScale stores it by the asset type mapping below. Empty: `Other`.	`"Physical Server"`
`status`	STRING	`status`	Mapped by the status mapping below. Empty: Active (On Network).	`"Active"`
`owner`	STRING	`assetOwnerId`	The asset owner's RegScale user id (a GUID). Any other value, for example a person's or a team's name, is logged as a warning and not sent. Empty or not sent: a new asset gets the user who runs the job, and an existing asset keeps its stored owner. A GUID that names no RegScale user is refused, and the run fails, if Asset Owner is a required field. Otherwise RegScale saves the asset without an owner.	`"a1b2c3d4-…"`
`description`	STRING	`description`	Copied as written. Empty: not set.	`"Production web server"`
`ipAddress`	STRING	`ipAddress`	Copied as written. Empty: not set.	`"192.168.1.100"`
`macAddress`	STRING	`macAddress`	Copied as written. Empty: not set.	`"00:1B:63:84:45:E6"`
`manufacturer`	STRING	`manufacturer`	Copied as written. Empty: not set.	`"Dell"`
`model`	STRING	`model`	Copied as written. Empty: not set.	`"PowerEdge R740"`
`serialNumber`	STRING	`serialNumber`	Copied as written. Empty: not set.	`"CN7475171B0123"`
`location`	STRING	`location`	Copied as written. Empty: not set.	`"DC1-Rack-42"`

The sync removes leading and trailing spaces from every value. A value that is only spaces counts as empty.

A "not set" column sends no value. A new asset leaves the field empty. An existing asset keeps its stored value, so if you clear a cell in BigQuery, the next sync does not clear the field in RegScale.

The reader also declares tags and customFields as optional columns, but the sync does not map them. Neither column reaches the asset.

Every imported asset gets scanning tool BigQuery and category Hardware, and is marked as virtual (bVirtual).

Asset Type Mapping

The CLI sends assetType as written. RegScale then stores one of its asset types:

A value that names a RegScale asset type, in any case, is stored as that type. Leading and trailing spaces are removed, and a run of spaces counts as one space. For example, physical SERVER is stored as Physical Server. A missing space, a hyphen or an underscore does not match: PhysicalServer and physical-server are stored as Other.
The aliases in the table below are stored as the type they name.
Any other value, and an empty value, is stored as Other.

Source value	Stored asset type
`server`	Physical Server
`workstation`	Desktop
`vm`	Virtual Machine (VM)
`virtual machine`	Virtual Machine (VM)
`router`	Network Router
`switch`	Network Switch

The aliases match by the same rule. The RegScale asset types are: Physical Server, Virtual Machine (VM), Appliance, Network Router, Network Switch, Firewall, Desktop, Laptop, Tablet, Phone, Container, Database, Cloud Storage, Load Balancer, Serverless Function, Kubernetes Cluster, Application, Operating System, Firmware, Utility and Other.

Status Mapping

The sync matches status in any case, with leading and trailing spaces trimmed, and maps it to a RegScale asset status:

Source value	RegScale status
`Active`	Active (On Network)
`Inactive`	Off-Network
`Decommissioned`	Decommissioned
anything else, or empty	Active (On Network)

Column Name Matching

Column names must match the names above exactly, except for case. The match is case-insensitive: name, Name and NAME all match the name column, and assetId, AssetID and ASSETID all match the assetId column.

Columns with other names are ignored, even when they hold the same data. For example, asset_id does not match assetId. A table that uses different names, for example a FedRAMP inventory workbook's unique_asset_identifier or ipv4_or_ipv6_address, needs a view or a query-mode --bigquery-query that renames them:

SELECT
  unique_asset_identifier AS assetId,
  asset_name AS name,
  ipv4_or_ipv6_address AS ipAddress,
  location
FROM inventory.fedramp_workbook

A view works as the table: set --bigquery-table to the view. A table-mode dry run on a view can report 0 rows. See Dry-Run Mode.

Assets That a Run Does Not Return

A run with no --limit and no --offset above 0 is a full run. At the end of a full run, RegScale retires every BigQuery asset of the security plan that the run did not return. A retired asset gets status Inactive. If a later run returns the asset again, the asset takes the status of that row.

A full run that returns no rows retires every BigQuery asset of the plan. The CLI does not retire assets after a full run in which an asset failed to process, a request was not fully applied, or a row had no assetId value. With debug logging on, the log says why.

This rule applies to every full run, including a query-mode run that filters rows. Before you run a full sync, make sure that:

the source returns every asset that the plan must keep, not a filtered subset.
no other BigQuery table or query syncs into the same security plan. Two full runs from different sources into one plan retire each other's assets.

A paginated run (any --limit, or an --offset above 0) retires nothing, so the slices of a split run do not retire each other's assets. But if you only ever sync a source in slices, an asset that leaves the source stays Active. Run a full sync from time to time to retire it.

Example BigQuery Table Schema

CREATE TABLE inventory.assets (
  -- Required
  name STRING NOT NULL,
  assetId STRING NOT NULL,

  -- Optional
  assetType STRING DEFAULT 'Other',
  description STRING,
  status STRING DEFAULT 'Active',
  ipAddress STRING,
  macAddress STRING,
  manufacturer STRING,
  model STRING,
  serialNumber STRING,
  location STRING,
  owner STRING,

  -- Metadata (ignored by import)
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP(),
  updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP()
)
PARTITION BY DATE(created_at)
CLUSTER BY status, location;

Configuration

Command-Line Parameters

Parameter	Short	Required	Default	Description
`--regscale_ssp_id`	`-id`	Yes	—	RegScale Security Plan ID to sync assets to
`--source-type`	—	No	`table`	Source type: `table` or `query`
`--bigquery-project`	—	No	init.yaml	GCP project ID
`--bigquery-dataset`	—	No	init.yaml	BigQuery dataset ID (table mode)
`--bigquery-table`	—	No	init.yaml	BigQuery table name (table mode)
`--bigquery-query`	—	No	—	SQL query (query mode)
`--dry-run`	—	No	`false`	Count assets and display schema without importing
`--offset`	—	No	`0`	Rows to skip (pagination)
`--limit`	—	No	None	Max rows to process

CLI flags override init.yaml values. If a CLI flag is not provided, the integration falls back to the corresponding GcpVariables key.

init.yaml Configuration

Add BigQuery configuration to your init.yaml:

# GCP Authentication (choose one method)
gcpCredentialsJsonBase64: "eyJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsIC4uLn0="

# BigQuery Configuration (optional — can override with CLI parameters)
gcpBigQueryProject: "my-production-project"
gcpBigQueryDataset: "asset_inventory"
gcpBigQueryAssetTable: "current_assets"

init.yaml Key	CLI Override	Description
`gcpBigQueryProject`	`--bigquery-project`	GCP project ID
`gcpBigQueryDataset`	`--bigquery-dataset`	BigQuery dataset ID
`gcpBigQueryAssetTable`	`--bigquery-table`	BigQuery table name
`gcpCredentialsJsonBase64`	—	Base64-encoded service account JSON
`gcpCredentialsJson`	—	Raw service account JSON
`gcpCredentials`	—	Path to a service account JSON file

Environment Variables

# GCP Authentication
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

# RegScale Configuration
export REGSCALE_URL="https://my-instance.regscale.com"
export REGSCALE_API_TOKEN="your-api-token"

Examples

Example 1: Simple Table Import

Import all assets from a table:

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type table \
  --bigquery-project production-project \
  --bigquery-dataset inventory \
  --bigquery-table assets

Example 2: Filtered Query Import

Import only active production assets:

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type query \
  --bigquery-project production-project \
  --bigquery-query "
    SELECT
      hostname as name,
      id as assetId,
      type as assetType,
      ip as ipAddress,
      location,
      owner
    FROM inventory.assets
    WHERE status = 'Active'
      AND environment = 'Production'
      AND last_scan > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  "

Example 3: Multi-Table Join

Import assets with vulnerability counts:

regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type query \
  --bigquery-project production-project \
  --bigquery-query "
    SELECT
      a.hostname as name,
      a.id as assetId,
      a.type as assetType,
      a.ip as ipAddress,
      a.location,
      a.owner,
      CONCAT(a.description, ' (', CAST(COUNT(v.id) AS STRING), ' vulnerabilities)') as description
    FROM inventory.assets a
    LEFT JOIN vulnerabilities.current v ON a.id = v.asset_id
    WHERE a.status = 'Active'
    GROUP BY a.hostname, a.id, a.type, a.ip, a.location, a.owner, a.description
  "

Example 4: Paginated Import

Import assets in chunks of 1000:

# Job 1
regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type table \
  --bigquery-project production-project \
  --bigquery-dataset inventory \
  --bigquery-table assets \
  --offset 0 \
  --limit 1000

# Job 2
regscale bigquery sync_assets_bq \
  --regscale_ssp_id 12345 \
  --source-type table \
  --bigquery-project production-project \
  --bigquery-dataset inventory \
  --bigquery-table assets \
  --offset 1000 \
  --limit 1000

Example 5: Using init.yaml Defaults

With gcpBigQueryProject, gcpBigQueryDataset, and gcpBigQueryAssetTable configured in init.yaml:

# Minimal command — all BigQuery params from init.yaml
regscale bigquery sync_assets_bq --regscale_ssp_id 12345

Troubleshooting

Issue: "Missing required columns: name, assetId"

Cause: BigQuery table/query doesn't include required columns.

Solution:

Ensure your table has name and assetId columns
Use column aliases in query mode: hostname as name, id as assetId

Issue: "None of the N BigQuery row(s) has a value in the key column assetId"

Cause: The source has an assetId column, but every value in it is null, empty or only spaces. The run exits 1 and writes nothing.

Solution:

Make sure that the column you alias as assetId holds the asset's key on every row
Filter out rows without a key, for example WHERE assetId IS NOT NULL AND TRIM(assetId) != ''

If only some rows have no key, the run writes the other rows, warns Asset has no identifier, skipping for each skipped row, logs the skipped count and up to 10 row names at ERROR, and marks no asset Inactive. In a split run, each child judges its own slice, and null keys sort into the first slices.

Issue: "Query contains dangerous operation: DELETE"

Cause: Query includes non-SELECT operations (security protection).

Solution:

Only SELECT queries are allowed
Remove DELETE, UPDATE, DROP, INSERT operations

Issue: "BigQuery table or dataset not found"

Cause: Invalid project, dataset, or table name.

Solution:

Verify project ID: gcloud config get-value project
List datasets: bq ls --project_id=my-project
List tables: bq ls --project_id=my-project dataset_name

Issue: "Insufficient permissions to access BigQuery"

Cause: Service account lacks required IAM permissions.

Solution:

# Grant BigQuery Data Viewer role
gcloud projects add-iam-policy-binding my-project \
  --member="serviceAccount:[email protected]" \
  --role="roles/bigquery.dataViewer"

Issue: "Query exceeded timeout (600s)"

Cause: Query takes too long to execute.

Solution:

Use --limit to process fewer rows
Add WHERE clause to filter data
Optimize query with indexes

Issue: "No assets found in BigQuery"

Cause: Empty result set.

Solution:

Check query WHERE clauses
Verify table has data: SELECT COUNT(*) FROM dataset.table
Use --dry-run to count rows

Issue: "Failed to initialize BigQuery client"

Cause: Invalid GCP credentials.

Solution:

# Use Application Default Credentials through the environment variable
gcloud auth application-default login
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.config/gcloud/application_default_credentials.json"

# Or set service account
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/key.json"

Performance Optimization

1. Use Partitioned Tables

CREATE TABLE inventory.assets
PARTITION BY DATE(created_at)
CLUSTER BY status, location
AS SELECT * FROM source_table;

2. Query Optimization

-- Bad: Scans entire table
SELECT * FROM inventory.assets;

-- Good: Uses partition pruning
SELECT * FROM inventory.assets
WHERE DATE(created_at) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY);

3. Parallel Processing

For 100,000 assets:

Dry-run to get count
Split into 20 jobs of 5,000 assets each
Run jobs in parallel (CI/CD, shell scripts)

4. Query Performance

-- Use WHERE clauses
SELECT * FROM inventory.assets WHERE status = 'Active';

-- Use LIMIT for testing
SELECT * FROM inventory.assets LIMIT 100;

-- Use column selection (not SELECT *)
SELECT name, assetId, assetType, location FROM inventory.assets;

Architecture

The integration is built on the RegScale ScannerIntegration framework:

regscale/integrations/commercial/bigquery/
  __init__.py           # Package init
  cli.py                # Click CLI commands (sync_assets_bq)
  scanner.py            # BigQueryScannerIntegration (ScannerIntegration subclass)

regscale/integrations/scanner/
  bigquery_source.py    # AssetBigQueryDataSource (table/query, pagination, dry-run)
  exceptions.py         # BigQuery-specific exception hierarchy

regscale/integrations/commercial/gcp/
  auth.py               # ensure_gcp_credentials() — service account / ADC setup
  variables.py          # GcpVariables — init.yaml configuration keys

Data flow: BigQuery table/query -> AssetBigQueryDataSource -> DataFrame -> _row_to_asset() -> IntegrationAsset -> ScannerIntegration.sync_assets() -> RegScale API

Support

For issues or questions:

GitHub Issues: https://github.com/RegScale/regscale-cli/issues
Documentation: https://docs.regscale.com
Support: [[email protected]](mailto:[email protected])

Overview

Features

Prerequisites

1. GCP Credentials

2. BigQuery Permissions

3. RegScale CLI Installation

Quick Start

1. Import from BigQuery Table

2. Import with Custom Query

3. Dry-Run to Count Assets and Inspect Schema

Usage Modes

Table Mode

Query Mode

Dry-Run Mode

Pagination and Parallel Processing

Parameters

Single Job with Pagination

Parallel Jobs with Shell Script

Parallel Jobs with a Kubernetes Indexed Job

BigQuery Schema Requirements

Required Columns

Optional Columns

Asset Type Mapping

Status Mapping

Column Name Matching

Assets That a Run Does Not Return

Example BigQuery Table Schema

Configuration

Command-Line Parameters

init.yaml Configuration

Environment Variables

Examples

Example 1: Simple Table Import

Example 2: Filtered Query Import

Example 3: Multi-Table Join

Example 4: Paginated Import

Example 5: Using init.yaml Defaults

Troubleshooting

Issue: "Missing required columns: name, assetId"

Issue: "None of the N BigQuery row(s) has a value in the key column assetId"

Issue: "Query contains dangerous operation: DELETE"

Issue: "BigQuery table or dataset not found"

Issue: "Insufficient permissions to access BigQuery"

Issue: "Query exceeded timeout (600s)"

Issue: "No assets found in BigQuery"

Issue: "Failed to initialize BigQuery client"

Performance Optimization

1. Use Partitioned Tables

2. Query Optimization

3. Parallel Processing

4. Query Performance

Architecture

Related Documentation

Support