providers_quality. It is not a fifth data
pool. Provider tables in providers hold the records.
Quality holds measurements about those tables: quality checks, statistical
process control (SPC), shape statistics, and health.
Read the latest results across the Workspace, or the history of one table:
Only main Provider tables have quality results. Identity tables
(
providers.identity.*), artifacts, and
parables do not.
The Provider Pools Reader, Member, Member with All Data Access, Data Access,
and Workspace Administrator roles can query this catalog. Each row describes
one Provider table’s rows, so the controls that narrow that table also apply
here
(Rows you can read):
- If a Provider Policy hides or narrows a table for you, you get none of its summary rows, and its history tables read as empty.
- If Provider Permissions narrow a table for you, the same applies. While the Workspace enforces Provider Permissions, that covers each table whose Provider says who can open each row, unless a Policy rule gives your role All rows on it.
- People with All data access get every row.
What each measurement is
Quality checks
A quality check compares a Provider table with a threshold from its stream’s quality settings. Each row is one check on one table.SKIPPED is not a pass. It means that the check did not evaluate.
Checks run at two times. The
lifecycle column tells you which:
A
promote pass can exist beside duplicates across loads. Use the latest
cadence result to judge the whole table. Only an observed dupe_rate of
zero proves that a key is unique. An event or history table can hold several
records for one entity, so check the table’s grain before you call a
repeated ID a defect.
Statistical process control
SPC is a Shewhart control chart on series that Parable tracks, such as row count and the null percentage of a column. It adds to the quality checks. It does not replace them.
Tracked series kinds include
TABLE_ROW_COUNT, COLUMN_NULL_PCT,
COLUMN_VALIDATION_FAILURE_PCT, COLUMN_DISTINCT_COUNT, and COLUMN_AVG.
Shape and health
Shape statistics record measurements such as row count, column count, and size in bytes. Distinct counts are approximate. A very large table is measured from a sample of its files. Itssampled_row_count is then the
number of rows read, and its distinct_count is null.
Health is the catalog’s state for a table: its state, the reason, and its
watermark.
Latest results across the Workspace
Start here. Each summary table holds the current result for each Provider table.connector and table_name are the schema and table of the Provider table’s
SQL address. The row for providers.google.users has connector = 'google'
and table_name = 'users'.
Failed checks for Google Workspace:
observed is the measured rate or count. threshold is the configured
limit. comparator is how Parable compared them.
Every check that did not pass:
table_commits on
connector, so the query reads only those Providers’ history:
workspace.stream_data.last_data_at.
History for one table
Each suffix holds one kind of result:
Recent checks for Google Workspace directory users:
result_id, run_id,
evidence_json for checks, and violations_json for SPC, when you need the
detail behind a status.
A history table can hold a status in mixed case, such as
Fail or
OutOfBounds, or in uppercase, such as FAIL and OUT_OF_BOUNDS. Summary
tables always use uppercase. If a history query returns no rows, look at
the values in the table before you filter on one casing.Read quality over the Workspace API
The Workspace API returns the same latest results as JSON:GET /api/v2/catalog/table-stats gives each main table one quality status
from its latest promote result and its latest cadence result. If either
result is FAIL, the status is FAIL. Otherwise, if either is PASS, the
status is PASS. Otherwise it is SKIPPED. qualityCheckedAt is the time of
the run that gave the status, the newer one if both gave it. So a newer
promote pass does not hide an older cadence failure. The same rule marks a
table degraded in providers_quality.summary.table_health.
The quality, SPC and shape operations return one page at a time, in Tap
order:
limitis 20 by default and 100 at most. A page also stops before the whole answer passes 40,000 characters, but it always holds one result.tenantConnectorTapIdkeeps only the results of one Tap.nextOffsetin the answer is theoffsetof the next page. The last page has nonextOffset.driftingOnly=trueon the SPC operation keeps only the series inWARNINGorOUT_OF_BOUNDS, and only the tables that have one.