Skip to main content
The Provider Plugin is read-only. Read one Provider connection to get its plugin in connector: Taps, traversal, rate limits, addressing, and the ingestion settings schema. That object is the map that a sync follows when it reads the Provider.
Look at connector.taps, connector.traversal, connector.defaultBaseUrl, connector.addressingConfig, connector.rateLimits, and connector.ingestionConfigSchema.

What you set and what you read

The connection holds the values that are yours. The plugin holds everything that says how to read the Provider. Change ingestionConfig and parentDataFilters with POST /api/connectors/configure. See Change a connection. GitHub’s ingestionConfig requires orgId. Linear needs no ingestion settings. Neither one lets you change pagination or GraphQL queries.

Sync windows

Each sync run reads a fixed time window, from window_start up to window_end. A retry reads the same window again.
  • When Parable accepts a connection’s credentials, and the plugin’s availability is executable or later, it starts a first sync. The first sync reads the 90 whole UTC days before the day Parable accepts them. Its window ends at UTC midnight of that day. The first scheduled sync runs 24 hours later and reads that day. At authorized, no sync starts (Availability).
  • An oauth2_refresh connection gets its token when a person completes the Provider’s consent screen. Parable checks the token then, and the rule above applies to it. See Strategies that use a consent screen.
  • If you save the credentials again, or complete consent again, Parable does not start a second 90-day sync while the first one is queued, running, or finished. When the first sync failed or was canceled, the next valid credential save starts a new one.
  • A valid check of the stored credentials (POST /api/connectors/validate) starts the first sync only when the connection never had one, for example because the check at save time failed. A check never replaces a first sync that failed or was canceled. See Check credentials.
  • After the first sync, scheduled reads use the previous complete UTC day by default. Google Cloud Billing uses the previous complete UTC hour and a one-day overlap for late exports. A provider can declare an hourly enrollment default with a hidden integer field named defaultScheduleIntervalHours and a schema default of 1 in its ingestion configuration. The scheduler and the ingestion plan both use the interval on the saved schedule. Credential rotation preserves that interval and its next due time.
  • Turning a Tap on later starts no backfill. The Tap starts with the next scheduled sync.
  • A backfill can read only some parents. Send parentDataFilters on the BACKFILL request, in the same shape as the connection’s filter. Each key must be the parent Tap of a traversal edge. When the connection also filters that key, the backfill reads only the values both allow, matched case-insensitively. The connection does not change, so the next scheduled sync reads every parent it names. Each seal below a parent the filter narrows is a fragment: promotion deletes no row by absence, and coverage gives the seal no credit. A filtered backfill never counts as the first sync. The request is refused with INVALID_BACKFILL_PARENT_FILTERS when it filters above a Tap with access facts, filters a key a Tap injects into its request, or is the first sync.
  • A Provider that sends data to Parable gets no first sync and no schedule. See Providers that send data.

A sync run

A sync skips a disabled Tap. A parent Tap with no records produces no child requests.

Resolve the URL

A sync builds every request from three parts:
  1. defaultBaseUrl. The active authentication strategy’s value wins. Otherwise the plugin’s value applies. Linear uses https://api.linear.app. GitHub uses https://api.github.com.
  2. addressingConfig, when the host itself has variables. scheme, host, and basePath can hold {var} placeholders. Each variable names a source: user_input (from ingestionConfig), oauth_response (from the token response, such as a Jira cloudId), or static. The active strategy’s addressingConfig wins over the plugin’s.
  3. The Tap’s path template. A REST restConfig.endpoint, such as /orgs/{orgId}/repos or /repos/{full_name}/pulls, gets its values from ingestionConfig or from a parent record.
For example, a GitHub repositories request is GET https://api.github.com/orgs/{orgId}/repos. The {orgId} value comes from the ingestionConfig that you sent when you connected.

Walk Taps in order

connector.traversal is the graph: A child Tap has exactly one parent. On an edge:
  • A sync reads parentKeyField from each parent record and puts it in the child’s endpoint.
  • templateVariable names the placeholder when it differs from the field name.
  • parentContextFields copies more parent fields onto each child record. GitHub copies full_name onto pull requests, so each row keeps its repository.
  • parentEligibility drops a parent before a sync plans any child request for it.
  • parentsPerRequest puts several parents in one child request, which lowers the number of child requests.
A child Tap can also send a request header from its parent record. With perParentAuth.method: request_header, a sync keeps the connection’s own credential and adds the header headerName, whose value is the scalar at valueField on the parent record. Dropbox files sends Dropbox-API-Select-User from the parent team member’s profile.team_member_id, so it reads each member’s files with the team credential. Such a Tap has exactly one parent. A sync skips a parent that has no value at valueField, and records the skip. GitHub is a tree. repositories, members, and teams are roots. repositories.full_name feeds issues, pull_requests, pull_request_comments, commits, branches, and releases. branches.name feeds commits_by_branch, and commits_by_branch.sha feeds commit_diffs. teams.slug feeds team_members. Linear is flat. edges is empty, so no Tap waits for another. Each GraphQL query runs on its own.

One Tap: request, page, extract

Transport

A sync uses one entry of the Tap’s transportOptions:
  • A rest option wins over a graphql option.
  • Among options of the same kind, the lowest priority wins. 1 is highest.
  • A sync does not try the next option when the first one fails.
A sync that reads a Provider runs only rest and graphql options. You can see file and custom on Providers that send data to Parable. REST placeholders get their values from ingestionConfig or from the parent record on a traversal edge. GraphQL placeholders usually live in variables. A query value consisting of one ingestion configuration placeholder preserves an array value. The HTTP client encodes its items as repeated query parameters. This supports the OpenAPI default form encoding for arrays. A placeholder inside a larger string remains string substitution. JSON request bodies also preserve an array bound to a complete placeholder.

Time window

sync.mode is incremental or full. An incremental Tap names a cursorField, the Provider’s change time, such as Linear updatedAt or GitHub updated_at. A full Tap reads everything on each run. sync.incrementalParams.filterType says where the run’s window goes: If an incremental Tap has no incrementalParams, its request carries no time filter. Other fields change the window:
  • maxWindowDays splits a long window into shorter requests, for Providers that refuse long ranges.
  • refreshLookbackDays moves the start of each request back by that number of days. Each run then reads that overlap again, for records that change after their change time. The seal and coverage keep the work-unit window.
  • maxLookbackDays says how many days of history the Provider keeps. The first 90-day sync can ask for older data than the Provider keeps.
A request never reads past the end of its window. A forward-looking Provider, such as a calendar, returns only records inside the window. Some Taps use sync.asyncQuery. A sync creates a job, polls statusEndpoint until it reports successValue, and then pages recordsEndpoint. Microsoft Purview audit logs work this way.

Pagination

pagination.type tells a sync how to continue until the Provider has no more records. Taps lists each type. For cursor pagination, pagination.cursor names cursorPath and hasMorePath, which are JMESPath expressions into the response. It also names where the next request carries the cursor: cursorParam (a query parameter) or cursorBodyPath (a body path). A GraphQL cursor goes into variables.after unless the plugin names another path. cursorIsFullUrl is for Providers that return the next request as an absolute URL, such as an OData @odata.nextLink. Page-number and offset pagination have two more fields: A Provider can refuse a cursor that expired during the read. Then a sync reads that parent again from the first page, with the same window. It does this for three kinds of cursor:
  • A Microsoft Graph next link with $skiptoken=, refused with 400 or 410.
  • A signed token whose expiry time has passed, such as Gong’s, refused with 400 or 410.
  • A GitLab keyset next link with page_token=, refused with 404.
A read restarts at most three times. The next refusal fails the read with IN-PG-005. The refusal of any other cursor fails the read. A page that repeats the keys of the page before it ends the read. That read cannot show that it saw every row, so it is a gap like a changed total: a full load seals fragment, and a full by_absence tap that carries access facts fails with IN-PG-006 so the next execution reads it again.

Records

responseExtraction turns a response body into records: A sync checks each record against the Tap’s schema and stores it in the Provider Pool. A value that does not match the schema is stored as null, and the row records the reason. A record without its key goes to the providers_quarantine catalog instead. Uniqueness in the Provider Pool comes from the schema’s x-transformDedupKey fields. See Tap tables.

Two worked paths

Linear issues (GraphQL, flat)

The Tap is a root. Its transport is GraphQL POST /graphql. The query asks for issues(first: $first, after: $after, filter: { updatedAt: { gt: $since } }). Static variables set first to 100.
  1. The sync writes the window start into variables.since (filterType: body_filter, bodyFilterPath: variables.since).
  2. It sends the query.
  3. It extracts data.issues.nodes. The id of each node is the primary key.
  4. If data.issues.pageInfo.hasNextPage is true, it puts pageInfo.endCursor into variables.after and sends the query again.
No parent records feed this Tap. The next Linear Tap is a separate query.

GitHub pull_requests (REST, child of repositories)

repositories is a root: GET /orgs/{orgId}/repos. GitHub returns the next page in the Link header. The body is an array (responsePath: "@"). Each repository record has full_name. The edge repositories -> pull_requests puts that value into GET /repos/{full_name}/pulls?per_page=100&state=all&sort=updated&direction=desc. This Tap sends no time filter. The body is again an array, and the next page is again in the Link header. The sync copies full_name onto each pull request record. pull_request_comments is also a child of repositories. It reads GET /repos/{full_name}/pulls/comments for each repository and sends the window start as since.

Rate limits, retries, and refusals

connector.rateLimits, and a Tap’s own rateLimits, set how fast a sync calls the Provider. admissionKind is one of two values:
  • fixed: requestsPerSecond, maxConcurrent, and retry. Linear allows 1.3 requests per second and 8 at once, and honors Retry-After.
  • provider_budget: a providerBudgetPolicy that follows the Provider’s own quota. GitHub starts at 1.25 requests per second. It reads GitHub’s rate-limit headers and can rise to 15 requests per second when GitHub reports enough quota.
These limits explain why a large first sync takes time.

Refusals of one parent

When the Provider refuses a request for one parent record, such as a 404 for one repository, the sync records that refusal and reads the other parents. The Tap fails only when it reads none of its parents. A refusal of a root request fails the read. These answers also stay with one parent. The sync records the parent as failed, reads the other parents, and then fails the Tap, so that Parable reads the window again:
  • A server error that outlasts its retries.
  • A timeout while the sync waits for the Provider’s answer, after its retries. A Provider that answers a server error slowly can time out on the last retry.
  • A 429 that outlasts its retries, when the Tap reads each parent with its own credential (perParentAuth).
The parents that finished are kept for the retry. This is also true when a failed parent returned rows before it failed, for example a server error on its second page. The sync drops those rows, and the retry reads only that parent again, from its first page. The last retry has no retry after it. If every failed parent of the last retry failed with a server error or a read timeout, the sync saves the other parents without the rows of the failed parents. The failed parents stay on record as failed. A 429 still fails the last retry. So does a full load of a Tap whose deletionSemantics is by_absence, because promotion would read the failed parents’ rows as deleted.

Answers that end the read

Some answers are not about one parent. They end the read of the Tap at once, and the window does not count as read:
  • A 401 on the connection’s shared credential.
  • A 429 on the shared credential that outlasts its retries, because every later parent uses the same quota.
  • A request that does not reach the Provider: a connection error, a connection timeout, or a DNS failure.
  • A refusal that arrives after the parent already returned rows, so a table never keeps half of a parent. A retry reads that parent again from the start.
  • An error inside a 200 answer: a GraphQL errors entry, a credential or permission error in the body, or an error value that the plugin does not list. Parable retries a transient error first. An error value that names one parent, such as Slack’s channel_not_found, refuses that parent only.
A plugin can also read the body of an error status to decide what it means. errorBodyRules match one status code and one value at a JMESPath. retryValues make the answer a retry. emptyValues make it an empty page, not a refusal. Gong, for example, answers 404 with “No calls found corresponding to the provided filters” when a window has no calls.

See what a sync refused

Each parent that the Provider refused to a sync attempt is a row of workspace.provider_refusals. A row names the Tap (platform_tap_name), the parent (<key field>=<value>), the outcome (skipped or failed), a category such as resource_absent or access_denied, the HTTP status, the IN-* error code, the Provider’s own reason, and the number of rows read before the error.
A retry adds its own rows and keeps the earlier attempt’s rows, so a refusal that a later retry cleared still counts.

See which days landed

workspace.stream_coverage shows which UTC days of each stream the sync covered, for every day since the stream was enabled. A row names the day, whether it is covered, the cause of a gap (no_seal or partial_seal), whether the day’s data reached the Provider Pool, and when it was computed. A backfill or a tap reset that changes an older day changes its row at the next refresh.

Providers that send data

Some Providers send their data to Parable instead. A sync does not read them, and nothing on this page applies to them. See Providers that send data.
Step-by-step setup for each Provider (OAuth apps, API keys, and scopes) is in the Provider Plugin guides.