connector: Taps, traversal, rate limits, addressing, and the
ingestion settings schema. That object is the map that a sync follows when it
reads the Provider.
connector.taps, connector.traversal, connector.defaultBaseUrl,
connector.addressingConfig, connector.rateLimits, and
connector.ingestionConfigSchema.
What you set and what you read
The connection holds the values that are yours. The plugin holds everything that says how to read the Provider.
Change
ingestionConfig and parentDataFilters with
POST /api/connectors/configure. See
Change a connection.
GitHub’s ingestionConfig requires orgId. Linear needs no ingestion
settings. Neither one lets you change pagination or GraphQL queries.
Sync windows
Each sync run reads a fixed time window, fromwindow_start up to
window_end. A retry reads the same window again.
- When Parable accepts a connection’s credentials, and the plugin’s
availability is
executableor later, it starts a first sync. The first sync reads the 90 whole UTC days before the day Parable accepts them. Its window ends at UTC midnight of that day. The first scheduled sync runs 24 hours later and reads that day. Atauthorized, no sync starts (Availability). - An
oauth2_refreshconnection gets its token when a person completes the Provider’s consent screen. Parable checks the token then, and the rule above applies to it. See Strategies that use a consent screen. - If you save the credentials again, or complete consent again, Parable does not start a second 90-day sync while the first one is queued, running, or finished. When the first sync failed or was canceled, the next valid credential save starts a new one.
- A valid check of the stored credentials (
POST /api/connectors/validate) starts the first sync only when the connection never had one, for example because the check at save time failed. A check never replaces a first sync that failed or was canceled. See Check credentials. - After the first sync, scheduled reads use the previous complete UTC day by default. Google Cloud Billing uses the previous complete UTC hour and a one-day overlap for late exports. A provider can declare an hourly enrollment default with a hidden integer field named
defaultScheduleIntervalHoursand a schema default of1in its ingestion configuration. The scheduler and the ingestion plan both use the interval on the saved schedule. Credential rotation preserves that interval and its next due time. - Turning a Tap on later starts no backfill. The Tap starts with the next scheduled sync.
- A backfill can read only some parents. Send
parentDataFilterson theBACKFILLrequest, in the same shape as the connection’s filter. Each key must be the parent Tap of a traversal edge. When the connection also filters that key, the backfill reads only the values both allow, matched case-insensitively. The connection does not change, so the next scheduled sync reads every parent it names. Each seal below a parent the filter narrows is a fragment: promotion deletes no row by absence, and coverage gives the seal no credit. A filtered backfill never counts as the first sync. The request is refused withINVALID_BACKFILL_PARENT_FILTERSwhen it filters above a Tap with access facts, filters a key a Tap injects into its request, or is the first sync. - A Provider that sends data to Parable gets no first sync and no schedule. See Providers that send data.
A sync run
A sync skips a disabled Tap. A parent Tap with no records produces no child requests.Resolve the URL
A sync builds every request from three parts:defaultBaseUrl. The active authentication strategy’s value wins. Otherwise the plugin’s value applies. Linear useshttps://api.linear.app. GitHub useshttps://api.github.com.addressingConfig, when the host itself has variables.scheme,host, andbasePathcan hold{var}placeholders. Each variable names asource:user_input(fromingestionConfig),oauth_response(from the token response, such as a JiracloudId), orstatic. The active strategy’saddressingConfigwins over the plugin’s.- The Tap’s path template. A REST
restConfig.endpoint, such as/orgs/{orgId}/reposor/repos/{full_name}/pulls, gets its values fromingestionConfigor from a parent record.
GET https://api.github.com/orgs/{orgId}/repos. The {orgId} value comes
from the ingestionConfig that you sent when you connected.
Walk Taps in order
connector.traversal is the graph:
A child Tap has exactly one parent. On an edge:
- A sync reads
parentKeyFieldfrom each parent record and puts it in the child’s endpoint. templateVariablenames the placeholder when it differs from the field name.parentContextFieldscopies more parent fields onto each child record. GitHub copiesfull_nameonto pull requests, so each row keeps its repository.parentEligibilitydrops a parent before a sync plans any child request for it.parentsPerRequestputs several parents in one child request, which lowers the number of child requests.
perParentAuth.method: request_header, a sync keeps the connection’s own
credential and adds the header headerName, whose value is the scalar at
valueField on the parent record. Dropbox files sends
Dropbox-API-Select-User from the parent team member’s
profile.team_member_id, so it reads each member’s files with the team
credential. Such a Tap has exactly one parent. A sync skips a parent that
has no value at valueField, and records the skip.
GitHub is a tree. repositories, members, and teams are roots.
repositories.full_name feeds issues, pull_requests,
pull_request_comments, commits, branches, and releases.
branches.name feeds commits_by_branch, and commits_by_branch.sha feeds
commit_diffs. teams.slug feeds team_members.
Linear is flat. edges is empty, so no Tap waits for another. Each
GraphQL query runs on its own.
One Tap: request, page, extract
Transport
A sync uses one entry of the Tap’stransportOptions:
- A
restoption wins over agraphqloption. - Among options of the same kind, the lowest
prioritywins.1is highest. - A sync does not try the next option when the first one fails.
A sync that reads a Provider runs only
rest and graphql options. You can
see file and custom on Providers that send data to Parable.
REST placeholders get their values from ingestionConfig or from the parent
record on a traversal edge. GraphQL placeholders usually live in variables.
A query value consisting of one ingestion configuration placeholder preserves
an array value. The HTTP client encodes its items as repeated query parameters.
This supports the OpenAPI default form encoding for arrays. A placeholder inside
a larger string remains string substitution. JSON request bodies also preserve
an array bound to a complete placeholder.
Time window
sync.mode is incremental or full. An incremental Tap names a
cursorField, the Provider’s change time, such as Linear updatedAt or
GitHub updated_at. A full Tap reads everything on each run.
sync.incrementalParams.filterType says where the run’s window goes:
If an incremental Tap has no
incrementalParams, its request carries no time
filter.
Other fields change the window:
maxWindowDayssplits a long window into shorter requests, for Providers that refuse long ranges.refreshLookbackDaysmoves the start of each request back by that number of days. Each run then reads that overlap again, for records that change after their change time. The seal and coverage keep the work-unit window.maxLookbackDayssays how many days of history the Provider keeps. The first 90-day sync can ask for older data than the Provider keeps.
sync.asyncQuery. A sync creates a job, polls
statusEndpoint until it reports successValue, and then pages
recordsEndpoint. Microsoft Purview audit logs work this way.
Pagination
pagination.type tells a sync how to continue until the Provider has no more
records. Taps lists each type.
For cursor pagination, pagination.cursor names cursorPath and
hasMorePath, which are JMESPath expressions into the response. It also
names where the next request carries the cursor: cursorParam (a query
parameter) or cursorBodyPath (a body path). A GraphQL cursor goes into
variables.after unless the plugin names another path. cursorIsFullUrl
is for Providers that return the next request as an absolute URL, such as an
OData @odata.nextLink.
Page-number and offset pagination have two more fields:
A Provider can refuse a cursor that expired during the read. Then a sync
reads that parent again from the first page, with the same window. It does
this for three kinds of cursor:
- A Microsoft Graph next link with
$skiptoken=, refused with400or410. - A signed token whose expiry time has passed, such as Gong’s, refused with
400or410. - A GitLab keyset next link with
page_token=, refused with404.
IN-PG-005. The refusal of any other cursor fails the read. A page that
repeats the keys of the page before it ends the read. That read cannot show
that it saw every row, so it is a gap like a changed total: a full load seals
fragment, and a full by_absence tap that carries access facts fails with
IN-PG-006 so the next execution reads it again.
Records
responseExtraction turns a response body into records:
A sync checks each record against the Tap’s
schema and stores it in the
Provider Pool. A value that does not match the schema is
stored as null, and the row records the reason. A record without its key
goes to the providers_quarantine catalog instead. Uniqueness in the
Provider Pool comes from the schema’s x-transformDedupKey fields. See
Tap tables.
Two worked paths
Linear issues (GraphQL, flat)
The Tap is a root. Its transport is GraphQL POST /graphql. The query asks
for issues(first: $first, after: $after, filter: { updatedAt: { gt: $since } }).
Static variables set first to 100.
- The sync writes the window start into
variables.since(filterType: body_filter,bodyFilterPath: variables.since). - It sends the query.
- It extracts
data.issues.nodes. Theidof each node is the primary key. - If
data.issues.pageInfo.hasNextPageis true, it putspageInfo.endCursorintovariables.afterand sends the query again.
GitHub pull_requests (REST, child of repositories)
repositories is a root: GET /orgs/{orgId}/repos. GitHub returns the next
page in the Link header. The body is an array (responsePath: "@").
Each repository record has full_name. The edge
repositories -> pull_requests puts that value into
GET /repos/{full_name}/pulls?per_page=100&state=all&sort=updated&direction=desc.
This Tap sends no time filter.
The body is again an array, and the next page is again in the Link header.
The sync copies full_name onto each pull request record.
pull_request_comments is also a child of repositories. It reads
GET /repos/{full_name}/pulls/comments for each repository and sends the
window start as since.
Rate limits, retries, and refusals
connector.rateLimits, and a Tap’s own rateLimits, set how fast a sync
calls the Provider. admissionKind is one of two values:
fixed:requestsPerSecond,maxConcurrent, andretry. Linear allows 1.3 requests per second and 8 at once, and honorsRetry-After.provider_budget: aproviderBudgetPolicythat follows the Provider’s own quota. GitHub starts at 1.25 requests per second. It reads GitHub’s rate-limit headers and can rise to 15 requests per second when GitHub reports enough quota.
Refusals of one parent
When the Provider refuses a request for one parent record, such as a404
for one repository, the sync records that refusal and reads the other
parents. The Tap fails only when it reads none of its parents. A refusal of a
root request fails the read.
These answers also stay with one parent. The sync records the parent as
failed, reads the other parents, and then fails the Tap, so that Parable
reads the window again:
- A server error that outlasts its retries.
- A timeout while the sync waits for the Provider’s answer, after its retries. A Provider that answers a server error slowly can time out on the last retry.
- A
429that outlasts its retries, when the Tap reads each parent with its own credential (perParentAuth).
failed. A 429 still fails
the last retry. So does a full load of a Tap whose deletionSemantics is
by_absence, because promotion would read the failed parents’ rows as
deleted.
Answers that end the read
Some answers are not about one parent. They end the read of the Tap at once, and the window does not count as read:- A
401on the connection’s shared credential. - A
429on the shared credential that outlasts its retries, because every later parent uses the same quota. - A request that does not reach the Provider: a connection error, a connection timeout, or a DNS failure.
- A refusal that arrives after the parent already returned rows, so a table never keeps half of a parent. A retry reads that parent again from the start.
- An error inside a
200answer: a GraphQLerrorsentry, a credential or permission error in the body, or an error value that the plugin does not list. Parable retries a transient error first. An error value that names one parent, such as Slack’schannel_not_found, refuses that parent only.
errorBodyRules match one status code and one value at a JMESPath.
retryValues make the answer a retry. emptyValues make it an empty page,
not a refusal. Gong, for example, answers 404 with “No calls found
corresponding to the provided filters” when a window has no calls.
See what a sync refused
Each parent that the Provider refused to a sync attempt is a row ofworkspace.provider_refusals. A row names the Tap (platform_tap_name), the
parent (<key field>=<value>), the outcome (skipped or failed), a
category such as resource_absent or access_denied, the HTTP status, the
IN-* error code, the Provider’s own reason, and the number of rows read
before the error.
See which days landed
workspace.stream_coverage shows which UTC days of each stream the sync
covered, for every day since the stream was enabled. A row names the day,
whether it is covered, the cause of a gap (no_seal or partial_seal),
whether the day’s data reached the Provider Pool, and when it was computed.
A backfill or a tap reset that changes an older day changes its row at the
next refresh.
Providers that send data
Some Providers send their data to Parable instead. A sync does not read them, and nothing on this page applies to them. See Providers that send data.Step-by-step setup for each Provider (OAuth apps, API keys, and scopes) is
in the Provider Plugin guides.