connector.taps[]:
schema, dataKind, deletionSemantics,
identityDirectory, and responseExtraction.primaryKey. Your Workspace
keeps its own copy of each schema as outputSchema. On that copy you can hide
fields from SQL, re-point the ordering field, and re-declare identity roles,
but not change the keys. See
Change a Tap schema in your Workspace.
Tap schemas lists every annotation.
From annotations to a table
The Identity family also holds Teams, primary memberships and reporting
lines. A Workspace has it when one of its Taps is a directory or lists Teams.
Without such a Tap, a query that names its tables reads no rows. Team
matching can use names, but people still match only by exact identity keys.
See Teams and account flags
for mark values, ranking, activity and classification.
responseExtraction.primaryKey is how a sync recognizes a fetched record. It
is almost always id. The Provider Pool merges a main table on one column:
- The first field marked
x-transformDedupKey, by field name. - Else the output field of the Tap’s synthetic key.
- Else
responseExtraction.primaryKey. - Else a field named
id.
primaryKey.
Names in SQL
The address of a table isproviders.{connector}.{table}. {connector} is
the plugin’s slug, with each - written as _. {table} is the Tap name.
An array of objects can become a child table named {tap}__{path}. GitHub
issue labels, for example, are in providers.github.issues__labels. Not every
nested object becomes a child table, so check the names in
catalog discovery
before you query.
Column names are the schema field names in lowercase. A flattened object path
uses __ between its parts:
SQL lowercases a name without quotes, so
SELECT updatedAt finds
updatedat. A quoted "updatedAt" does not match the column. A struct column keeps the
source names of its fields. Write profile."userType" or
profile['userType'] to read one.
Dedup keys
A mainsnapshot or changelog table has one current row for each dedup
key. Linear issues and
GitHub pull_requests both key on id. You can join or filter on id and
expect one row for each issue or pull request.
A deleted record can stay in the table. Each row has state columns such as
_parable_deleted and _parable_observed. To read current state, filter on
_parable_deleted = false. The
Provider Pool reference
lists every state column.
A child table holds one row for each element of the parent’s array. It
merges on parent_pk and ordinal: the parent’s key as text, and the
element’s position in the array. The parent ID repeats on each child row. Use
the complete logical key from the catalog. See
Keys and relationships
for how to join a child table to its parent.
A Tap with child tables also has two views for each child table: a
{child}_with_parent view that joins each child row to its parent row, and a
{parent}_nested view that puts each child table back into its parent row as
an array. GitHub issue labels are in
providers.github.issues__labels_with_parent. On an event Tap that stores
an event again for each load that reads it, the views pair a child row with
the parent row of the same load, on dt and _parable_seal_id. The
Provider Pool reference
shows how to keep one load’s rows.
Do not use SELECT DISTINCT id to clean up a main table. Parable already
merged its rows. If you see two rows with the same id, you are probably
reading a child table or an event table.
Ordering keys
Each table has at most onex-transformOrdering field. When a sync writes a
newer record for the same dedup key, the row with the greater ordering value
is the one you query. When the Tap marks no ordering field, its
sync.cursorField is the ordering column, if the schema has that field. A
child table carries the ordering column of its root row.
Linear issues order on updatedAt (SQL updatedat). GitHub pull_requests
order on updated_at. Directory Taps such as Linear users also order on
updatedAt, so a renamed user replaces the earlier row.
dataKind says how Parable treats those versions:
deletionSemantics tells you how a deletion in the Provider shows in the
table:
Foreign keys
A field markedx-transformForeignKey references a row of another Tap of the
same Provider. When the field is an object, the table has one <field>_id
column with the referenced row’s ID, not a copy of the row. Notion
pages.created_by is the column created_by_id. Join it to the other Tap’s
table for the rest:
user, the
column keeps its own name. A field whose type is the Tap’s own record type
becomes a <field>_id column too, such as a comment’s parent_id. The catalog
lists these columns as foreign keys.
Event tables
Anevent table is laid out by its event time: the field marked
x-semantic-role: event_time, or else a timestamp ordering column. Each write
is sorted by event time, so a query for one day skips the files of other
days. A table that Parable creates this way is partitioned by
_parable_event_date, the UTC day of the event time. Filter on it to read
whole days:
_parable_event_date is null for a row with no event time. snapshot and
changelog tables do not have the column.
A child table of an event table carries its parent’s event time under the same
name and in the same type, when that field is also the tap’s ordering field. So
chrome_activity__events has its own id__time, and a read can filter on it as
it filters on the parent.
Identity keys
Identity annotations mark which columns are about a person or account, not which row is unique. Linearusers shows every identity annotation on one
Tap:
id is both the dedup key of the row and the Linear account ID. email joins
that person to Google primaryemail or to workspace.users.email.
gitHubUserId is the ID of the same person’s GitHub account. Its annotation
names the system (github) and what the value equals (id), so Parable joins
the Linear user and the GitHub member with that ID into one identity. A bare
"x-transformExternalUserId": true names no system, and Parable joins
nothing through it. name labels the person. It never joins.
Taps with identityDirectory: true, such as Linear users, GitHub
members, and Google users, feed the providers.identity tables. These
tables have rows only after at least one directory Tap has synced.
Do not join the
id columns of two Providers to each other and expect a
person match. Join on an email, or go through the identity tables.
account_id_type is id, uuid, or email. match_method records how
Parable linked the account. Every current method is exact, and Parable never
links accounts on a similar email or an equal name. Identity rows are
best-effort: service accounts and shared mailboxes get identities too.
An identity ID can change when Parable splits an identity. To keep a stable
key for an account, keep its connector_tap_id and account_id. The
Provider Pool reference
lists every match method.
Worked tables
Linearissues (dataKind: changelog): one row for each id. Order on
updatedat. Nested references such as assignee and team stay on the issue
row as flattened columns, such as assignee__email and team__key. They are
not a second grain. Filter WHERE id = ... for one issue.
GitHub pull_requests (dataKind: changelog): one row for each id.
Order on updated_at. The sync copies full_name onto each pull request, so
you can filter by repository without a join to repositories.
GitHub issues (opt-in, dataKind: changelog): labels land in
providers.github.issues__labels, one row for each label on each issue. The
issue’s key repeats in parent_pk.
Google users (dataKind: snapshot, identityDirectory: true): one row
for each directory id, which is also the account ID. primaryemail is the
column to join people on. When a user leaves the directory, the next full read
marks the row _parable_deleted = true.