issues, GitHub pull_requests, or Slack
channels. The Linear plugin, for example, has Taps from issues and
projects to comments and workflow_states.
Each Tap lands as its own set of tables in the
Provider Pool.
The Provider Plugin describes how Parable reads each Tap. Read it in
connector.taps of one Provider connection. You do not set the transport,
the pagination, or the sync mode, because they follow how the Provider’s API
works. How a sync reads a Provider follows these
fields through a Linear query and a GitHub parent-child read.
A Tap looks like this (other fields are not shown):
Transport
Each Tap has one or more transport options. An option is one way to call the Provider:
A sync uses one option for each Tap. A
rest option wins over a graphql
option. Among options of the same kind, the lowest priority wins, and 1 is
highest. A sync does not try the next option when the first one fails.
GitHub Taps use rest. Linear Taps use graphql.
Pagination
Providers return results in pages in different ways. The transport option’spagination.type tells a sync how to request each page until the Provider has
no more:
You do not choose this. It lets a GitHub Tap and a Linear Tap use the same
sync engine with two different paging rules.
Sync mode
sync.mode says how much each run reads:
Each run reads a fixed time window. After the first sync of a new connection,
scheduled runs usually read the previous complete UTC day. Google Cloud Billing reads hourly export windows.
Sync windows says when the first sync
runs and what it reads.
sync.incrementalParams.filterType says how the window goes to the Provider:
Some Providers use a job that a sync creates, polls, and then reads
(
sync.asyncQuery). You see that block on the Tap when the Provider works
that way.
Record shape
schema on the plugin’s Tap is the shape of the records it emits. A sync
checks each record against it. Annotations on that schema, such as
x-transformDedupKey, x-transformOrdering, and the identity annotations,
decide how each row lands in the Provider Pool.
Tap schemas lists every annotation, and
Tap tables shows the tables that result.
Template Taps
A template Tap is a plugin Tap that hasparameters. It tells Parable
how to read one table from a database Provider, such as Snowflake, but it
does not name the table. A template Tap has no tables of its own. It runs
only through the custom Workspace Taps that fill it. Each custom Tap
supplies the table, the key, and the cursor in config.source.
parameters is a Parable input schema. Each input field names one value
that the request reads as a {tap.NAME} placeholder. A placeholder can be
anywhere in transportOptions or sync, for example in a Snowflake
requestBody.statement:
custom Tap:
Each parameter must have an
x-validatePattern that starts with ^ and ends
with $, and the two anchors must hold the whole pattern. Put an alternation
inside a group, as ^(?:a|b)$. The value comes from a Workspace, so the
pattern stops a value that could change the request. For columns, the pattern applies to each column.
A write of a plugin Tap fails with a field error when:
A template Tap never runs on its own. Tap selection never turns it on, even
when
ingestionConfig.enabledTaps names it, and a Workspace override of it
fails with WA-TCT-038.
Definition rules
lists these rules with the other plugin rules.
Staff set parameters on POST /api/connectors/{connectorId}/taps
and change it on PATCH /api/connectors/{connectorId}/taps/{tapId}.
null on the update removes it, and the Tap is then an ordinary Tap. Every
answer that returns the plugin Tap includes parameters. Research does not
write template Taps, so a promotion keeps the parameters of a stored Tap. A
rollback to a revision that has no parameters also keeps the stored value.
Reader text
Each Tap carries text for the people who read its data:
Each field of the Tap’s schema has its own text in
title, description,
and x-purpose. Reader text never changes the stored data.
Required access
requiredAccess says what the Provider credential must allow before the Tap
can read its data. Parable derives it from the plugin. You cannot set it.
GitHub pull_requests needs the repo scope:
For an operation that requires no additional OAuth scopes, set
sync.scopesNotRequired to true and leave sync.scopes empty. Its
requiredAccess keeps the oauth_scope model with empty values. Set it only
when the Provider’s documentation states that the operation needs no scope. An
operation whose scope the source does not state is not proof. Missing scope
information still fails catalog validation (scopesRequired). A write that sets
the declaration beside nonempty scopes fails with scopesNotRequiredConflict.
Research does not set the declaration. An author sets it by hand.
requiredAccess is null when the plugin offers more than one authentication
strategy, because each strategy can carry different access.
Sensitive Taps
Some Taps read sensitive data: pay and payroll, tax IDs, birth dates, sex, home contact details, bank accounts, or free-text notes. Each kind has its own Tap, and each such Tap is off by default (enabledByDefault: false).
The plugin’s main authentication strategy asks only for the access that the
default Taps need. A second strategy of the same type, with its own label,
adds the access of the sensitive Taps. Justworks, for example, offers
“Justworks OAuth 2.0 (company OAuth application)” and “Justworks OAuth 2.0
with pay, payroll and personal data”. Because such a plugin has two
strategies, its Taps show requiredAccess: null. To read a sensitive Tap:
- Connect with the strategy that adds the sensitive access, or change the connection to it.
- Set a Provider Policy on each table of the sensitive Tap. A table that no Provider Policy names is open to every reader who can read Provider data.
- Turn the Tap on (see Turn Taps on and off).
Taps in your Workspace
Your Workspace keeps its own record for each Tap of a connection. A new connection starts with the plugin’s default Taps (enabledByDefault: true)
and the parent Taps that they need.
Each Workspace Tap has these fields:
Turn Taps on and off
The connection’singestionConfig.enabledTaps decides which plugin Taps a
sync runs:
- When
enabledTapsis absent, a sync runs the Taps withenabledByDefault: true. - When
enabledTapsis a list, a sync runs the Taps that it names. The list replaces the defaults. An empty list runs no Tap. - A sync also runs the parent Taps of each Tap that runs.
- A Workspace Tap with
enabled: falsedoes not run, even whenenabledTapsnames it. - A template Tap (a Tap that declares
parameters) never runs on its own, even whenenabledTapsnames it. It gets nooverrideWorkspace Tap, and the catalog serves no table for it. Only itscustomWorkspace Taps run. A template Tap must haveenabledByDefault: falseandidentityDirectory: false. - When the plugin changes a Tap’s
enabledByDefault, Parable turns that Tap on or off in each connection that has noenabledTapslist. A Workspace Tap that you turned off stays off. Demo Workspaces keep their Taps. A Tap stays off when its tables would get the SQL name of a custom Tap’s, or when its parent Tap is off.
ingestionConfig, with
enabledTaps listing every Tap that you want, to
POST /api/connectors/configure. The ingestionConfig that you send
replaces the stored one, so include your other settings, such as orgId.
enabled to false on its Workspace Tap:
enabled: false stops a Tap. enabled: true does not add a Tap that
enabledTaps leaves out. A child Tap needs its parent. Before you disable a
parent Tap, disable its child Taps (WA-TCT-025, WA-TCT-026).
The catalog serves only enabled Taps, so a disabled Tap has no table there
until you enable it again.
Two enabled Taps of one Provider in a Workspace cannot have tables or views
with the same SQL name, on one connection or on two. For example, orders and
orders_ get the same name. A create, a rename, a schema change or an enable
that causes this fails with WA-TCT-036, and nothing is stored. The error
names the other Tap.
The same rule applies when Parable adds a Provider’s Taps to your Workspace:
when the Provider gets a new Tap, and when you save a connection’s
enabledTaps. If a Tap would get the SQL name of an enabled custom Tap in
your Workspace, Parable does not add it. It adds the other Taps and gives no
error. To get the missing Tap, rename or disable the custom Tap, then save
the connection again.
Change a Tap schema in your Workspace
TheoutputSchema of a Workspace Tap of kind override is a full copy of
the plugin’s Tap schema. It keeps the same fields and types as the plugin’s
schema (WA-TCT-022). You can change only annotations, and only some of
them.
x-exclude and x-transformEmbed: false stay on your copy when Parable
changes the plugin’s Tap. A re-pointed ordering, partition date, fingerprint
input, or identity role lasts until Parable next changes the plugin’s Tap.
Your copy then takes the plugin’s annotations again.
Team, bot and external marks inherit the plugin’s schema on fields the
layout retains. Your copy can re-declare them. The manager mark always
follows the plugin’s schema. See
Teams and account flags.
Parable refuses these writes:
A
WA-TCT-021 message names the rule and the table, such as
outputSchema violates promote directive rules: <rule> (root table): <message>.
A WA-TCT-035 error lists the fields in its details. Send every x-access*
value as the plugin’s copy holds it.
Provider Permissions
explains why.
A change to outputSchema can need a rebuild of the Tap’s stored tables, for
example a new ordering field. When a Tap schema changes
lists the classes of change and how to approve a rebuild.
config also has keys that only Parable writes: materializedDefault,
systemDisabled, catalogSlug, storageUri, and source. A write that sets
one of them fails with WA-TCT-029. config.selectedFields works only on Salesforce
Taps (WA-TCT-024), and never on an Artifact Tap (WA-TCT-028).
Custom Taps on a database connection
A database Provider, such as Snowflake, has a template Tap. A template Tap holds how to reach the database and how to send a query, but it reads no table of its own. Its request has{tap.*} placeholders, and its
parameters give a pattern for each one. A custom Tap (kind: custom)
on a connection of that Provider reads one table. Its config.source names
the table, the key and the cursor. The root fields of its outputSchema are
the columns that it reads. Your catalog serves it under its own name.
Only Parable staff create custom Taps for now. The operation needs the staff
permission admin.tenant.connectors.taps.custom. A Workspace Administrator
gets 403.
- The Tap in
connectorTapIdis a template Tap of the connection’s Provider. If not, the request fails withWA-TCT-038. - Each value that fills a
{tap.*}placeholder matches the pattern of its parameter. For{tap.columns}, each root field ofoutputSchemamust match. If one does not match, the request fails withWA-TCT-037, and the details name the parameter. - The source fills each placeholder that the template reads. The key
columns,
key.syntheticFieldandcursorColumnare root fields ofoutputSchema. A key of more than one column names asyntheticField. The key field (the one key column, orsyntheticField) is the one root field withx-transformDedupKey.refreshLookbackDaysandmaxWindowDaysneed acursorColumn. If one of these fails, the request fails withWA-TCT-038. - The
outputSchemaplans with the source’s key and cursor. - The Tap’s catalog tables do not get the SQL name of another enabled Tap of
the Provider in your Workspace (
WA-TCT-036).
access is optional. A custom Tap with no access stores
{"roles": ["resources"]}. Only people with All data access read its table
until a Provider Policy opens it. See
A table that only All data access reads.
PATCH and DELETE work on a custom Tap as on any Workspace Tap. A change
of config.source, config.access or outputSchema needs the same staff
permission and runs the same checks. A config write that leaves out source
or access keeps the stored values. A change of source.mode keeps the
Tap’s name, so the table keeps its SQL name. config.source and
config.access on any other Tap fail with WA-TCT-038 and WA-TCT-028.
Manage Taps with the API
The list returns up to 250 Taps on each page. Use
limit and offset to read
more.
The read of one Tap returns its whole outputSchema. The schema of a large
Provider can be more than 50,000 characters. To read it a page at a time,
send fieldLimit (1 to 1,000), fieldOffset, or both. The answer then holds
one page of the fields of outputSchema: at most fieldLimit fields, and
fewer than 40,000 characters in the whole answer, but always one field. The
fields of the root type come first, then those of each other type by name. A
type is on each page that holds one of its fields, and the enums are on every
page. Send nextFieldOffset back as fieldOffset for the next page. The last
page has no nextFieldOffset. A fieldLimit outside 1 to 1,000 or a negative
fieldOffset fails with 400.
PATCH can change enabled, name, description, outputSchema,
config, and canonicalPersonSource. It cannot change the connection, the
plugin Tap, the sync mode, the pagination, or the transport. A Workspace Tap
record does not add a Tap to the Taps that a sync runs. enabledTaps does.
The API Reference has the full request and response
shapes.