Skip to main content
Quality is a companion catalog, providers_quality. It is not a fifth data pool. Provider tables in providers hold the records. Quality holds measurements about those tables: quality checks, statistical process control (SPC), shape statistics, and health. Read the latest results across the Workspace, or the history of one table: Only main Provider tables have quality results. Identity tables (providers.identity.*), artifacts, and parables do not. The Provider Pools Reader, Member, Member with All Data Access, Data Access, and Workspace Administrator roles can query this catalog. Each row describes one Provider table’s rows, so the controls that narrow that table also apply here (Rows you can read):
  • If a Provider Policy hides or narrows a table for you, you get none of its summary rows, and its history tables read as empty.
  • If Provider Permissions narrow a table for you, the same applies. While the Workspace enforces Provider Permissions, that covers each table whose Provider says who can open each row, unless a Policy rule gives your role All rows on it.
  • People with All data access get every row.
Run the SQL below over Flight SQL. Query the pools has the clients for each language.

What each measurement is

Quality checks

A quality check compares a Provider table with a threshold from its stream’s quality settings. Each row is one check on one table. SKIPPED is not a pass. It means that the check did not evaluate. Checks run at two times. The lifecycle column tells you which: A promote pass can exist beside duplicates across loads. Use the latest cadence result to judge the whole table. Only an observed dupe_rate of zero proves that a key is unique. An event or history table can hold several records for one entity, so check the table’s grain before you call a repeated ID a defect.

Statistical process control

SPC is a Shewhart control chart on series that Parable tracks, such as row count and the null percentage of a column. It adds to the quality checks. It does not replace them. Tracked series kinds include TABLE_ROW_COUNT, COLUMN_NULL_PCT, COLUMN_VALIDATION_FAILURE_PCT, COLUMN_DISTINCT_COUNT, and COLUMN_AVG.

Shape and health

Shape statistics record measurements such as row count, column count, and size in bytes. Distinct counts are approximate. A very large table is measured from a sample of its files. Its sampled_row_count is then the number of rows read, and its distinct_count is null. Health is the catalog’s state for a table: its state, the reason, and its watermark.

Latest results across the Workspace

Start here. Each summary table holds the current result for each Provider table. connector and table_name are the schema and table of the Provider table’s SQL address. The row for providers.google.users has connector = 'google' and table_name = 'users'. Failed checks for Google Workspace:
observed is the measured rate or count. threshold is the configured limit. comparator is how Parable compared them. Every check that did not pass:
SPC series with a warning, a control-limit breach, or a recorded violation:
Largest Provider tables by row count:
Tables with stale watermarks:
When each Provider’s tables last changed. Filter table_commits on connector, so the query reads only those Providers’ history:
A compaction or a VACUUM also commits a version, so this time can be newer than the data. To read when a connection’s newest load arrived, read workspace.stream_data.last_data_at.

History for one table

Each suffix holds one kind of result: Recent checks for Google Workspace directory users:
How row-count SPC for that table moved:
History tables keep every run. They can also have result_id, run_id, evidence_json for checks, and violations_json for SPC, when you need the detail behind a status.
A history table can hold a status in mixed case, such as Fail or OutOfBounds, or in uppercase, such as FAIL and OUT_OF_BOUNDS. Summary tables always use uppercase. If a history query returns no rows, look at the values in the table before you filter on one casing.

Read quality over the Workspace API

The Workspace API returns the same latest results as JSON:
GET /api/v2/catalog/table-stats gives each main table one quality status from its latest promote result and its latest cadence result. If either result is FAIL, the status is FAIL. Otherwise, if either is PASS, the status is PASS. Otherwise it is SKIPPED. qualityCheckedAt is the time of the run that gave the status, the newer one if both gave it. So a newer promote pass does not hide an older cadence failure. The same rule marks a table degraded in providers_quality.summary.table_health. The quality, SPC and shape operations return one page at a time, in Tap order:
  • limit is 20 by default and 100 at most. A page also stops before the whole answer passes 40,000 characters, but it always holds one result.
  • tenantConnectorTapId keeps only the results of one Tap.
  • nextOffset in the answer is the offset of the next page. The last page has no nextOffset.
  • driftingOnly=true on the SPC operation keeps only the series in WARNING or OUT_OF_BOUNDS, and only the tables that have one.
The API Reference has the response of each operation.