> ## Documentation Index > Fetch the complete documentation index at: https://developers.askparable.com/llms.txt > Use this file to discover all available pages before exploring further. # How ingestion walks a source > Read the connector definition to see how a sync resolves the URL, walks taps in order, pages the API, and lands records. The connector definition is read-only. Every instance you list includes the nested `connector` object: taps, traversal, rate limits, addressing, and the schema for `ingestionConfig`. That object is the map ingestion follows when it talks to the source. ```bash theme={null} curl "https://api.askparable.com/api/vendors/tenant-connector-instances" \ -H "Authorization: Bearer $PARABLE_API_TOKEN" ``` Look at `connector.taps`, `connector.traversal`, `connector.defaultBaseUrl`, `connector.addressingConfig`, `connector.rateLimits`, and `connector.ingestionConfigSchema`. ## What you set vs what you inspect Two blobs on the instance are yours. Everything that describes *how* to walk the API lives on the definition. | You send | What ingestion does with it | | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `authenticationConfig` | Builds the credential for every request. See [Auth strategies](/connectors/auth). | | `ingestionConfig` | Fills URL and path variables (`orgId`, instance host, team ids). Some connectors also accept `parentDataFilters` here as an allowlist. Shape is `connector.ingestionConfigSchema`. | | Enabled [taps](/connectors/taps) | Chooses which streams in `connector.taps[]` actually run. | GitHub's `ingestionConfig` requires `orgId`. Linear's is an optional `baseUrl`. Neither lets you change pagination or GraphQL queries. ## A sync run ```mermaid theme={null} flowchart TD I["Your instance: credentials + ingestionConfig"] --> U["Resolve base URL"] U --> R["Root taps in connector.traversal"] R --> T["For each enabled tap"] T --> P["Pick transport by priority"] P --> W["Inject incremental window if the tap is incremental"] W --> Q["Request, page until exhausted"] Q --> E["Extract records, land in the lake"] E --> C["Child taps: one request per parent record"] C --> T ``` Disabled taps are skipped. Child taps whose parent was skipped or empty do not run. ## Resolve the URL Ingestion builds every request from: 1. **`defaultBaseUrl`** on the connector (or on the active auth strategy, which wins when set). Linear is `https://api.linear.app`. GitHub is `https://api.github.com`. 2. **`addressingConfig`** when the host itself is templated. `scheme`, `host`, and `basePath` can contain `{var}` placeholders. Each variable lists a `source`: `user_input` (from `ingestionConfig`), `oauth_response` (from the token payload, for example a Jira `cloudId`), or `static`. 3. **Path templates on the tap.** REST `restConfig.endpoint` values such as `/orgs/{orgId}/repos` or `/repos/{full_name}/issues` are filled from `ingestionConfig` or from a parent record, depending on the placeholder. A GitHub repositories request is therefore `GET https://api.github.com/orgs/{orgId}/repos` with `{orgId}` taken from the `ingestionConfig` you sent when you connected. ## Walk taps in order `connector.traversal` is the graph: | Field | Meaning | | --------------- | -------------------------------------------------------------------------------------- | | `rootTaps` | Streams with no parent. These run first. | | `edges` | Parent to child. Each edge names `from`, `to`, and `parentKeyField`. | | `configFanouts` | A list-valued `ingestionConfig` field that fans out a root tap: one request per value. | On an edge, `parentKeyField` is read from each parent record and substituted into the child's endpoint. `templateVariable` overrides the placeholder name when it differs from the field. `parentContextFields` copies extra parent fields onto each child record (GitHub copies `full_name` onto pull requests so later taps still know the repo). `parentEligibility` can drop a parent before any child request is planned. **GitHub** is a tree: `repositories`, `members`, and `teams` are roots. `repositories.full_name` fans out `issues`, `pull_requests`, `commits`, `branches`, and `releases`. `pull_requests.number` then fans out `pull_request_comments`. **Linear** is flat: every tap is a root and `edges` is empty. Each GraphQL query is independent; issues do not have to be listed before comments. ## One tap: request, page, extract For each enabled tap, ingestion picks the first usable entry in `transportOptions` (`priority` **1** is highest). That option is one way to call the source. ### Transport | `apiType` | Config block | What it contains | | ------------------------------ | ------------------ | ----------------------------------------------------------------------- | | `rest` | `restConfig` | `method`, `endpoint`, `queryParams`, `headers`, optional `requestBody`. | | `graphql` | `graphqlConfig` | `endpoint`, `query`, `variables`, optional `operationName`. | | `database` / `file` / `custom` | Connector-specific | Direct extract or a custom transport. | REST placeholders come from `ingestionConfig` or from the parent record on a traversal edge. GraphQL placeholders usually live in `variables` and are updated the same way (and by pagination / incremental injection below). ### Incremental window `sync.mode` is `incremental` or `full`. Incremental taps keep a watermark on `sync.cursorField` (for example Linear `updatedAt`, GitHub `updated_at`). The first run uses a lookback window; later runs start from the last watermark (minus a buffer for late-arriving rows). `maxLookbackDays` caps that first window when the source retains only recent data. How the window is *sent* is `sync.incrementalParams.filterType`: | `filterType` | Where the window goes | | ---------------- | ---------------------------------------------------------------------------------- | | `simple` | Query parameters (`sinceParam` / `untilParam`). GitHub issues uses `since`. | | `query_template` | A search string with `{window_start}` / `{window_end}` (Gmail `q`). | | `body_filter` | A field in the JSON body. Linear issues writes the watermark to `variables.since`. | `maxWindowDays` splits a long range into sub-windows when the source rejects large spans. `defaultLookaheadDays` extends the end past "now" for forward-looking APIs such as calendars. If `incrementalParams` is absent, the watermark is still stored from `cursorField`, but the source request may not receive a time filter. Some taps use `sync.asyncQuery`: POST to create a job, poll `statusEndpoint` until `successValue`, then page `recordsEndpoint`. You will see that block when the source works that way (audit-log exports, for example). ### Pagination The transport option's `pagination.type` is how ingestion walks pages until the stream is exhausted. Details of each type are on [Taps and streams](/connectors/taps#pagination). Cursor pagination also names `cursorPath` / `hasMorePath` (JMESPath into the body) and either `cursorParam` (query string) or `cursorBodyPath` (POST body, including GraphQL `variables.after`). `cursorIsFullUrl` is for APIs that return the next request as an absolute URL (OData `@odata.nextLink`). ### Records `responseExtraction` turns a response body into rows: | Field | Meaning | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `responsePath` | JMESPath to the array of records. `@` means the body is already an array. | | `responsePathFallback` | Alternate path when the primary path is null. | | `primaryKey` | Field used to identify a fetched record (`id` on both GitHub and Linear). Lake uniqueness is the tap schema's `x-transformDedupKey` set; see [tap projection](/connectors/tap-projection). | | `responseFormat` | `json` (default) or `jsonl` for newline-delimited bodies. | | `syntheticPrimaryKey` | Hash of named fields when the source has no stable id. | Those rows are validated against the tap `schema` and land in the [providers](/data/providers) pool. Malformed source records stop at the boundary instead of landing in the lake. ## Two worked paths ### Linear `issues` (GraphQL, flat) The tap is a root. Transport is GraphQL `POST /graphql`. The query asks for `issues(first: $first, after: $after, filter: { updatedAt: { gt: $since } })`. Static variables set `first` to 100. 1. Ingestion writes the watermark into `variables.since` (`filterType: body_filter`, `bodyFilterPath: variables.since`). 2. It POSTs the query. 3. It extracts `data.issues.nodes`. Each node's `id` is the primary key; `updatedAt` advances the watermark. 4. If `data.issues.pageInfo.hasNextPage` is true, it puts `pageInfo.endCursor` into `variables.after` and POSTs again. No parent fan-out. The next Linear tap is a separate query. ### GitHub `pull_requests` (REST, child of `repositories`) `repositories` is a root: `GET /orgs/{orgId}/repos`, `Link` header pagination, body is an array (`responsePath: "@"`). Each repository record has `full_name`. The edge `repositories -> pull_requests` substitutes that into `GET /repos/{full_name}/pulls?per_page=100&state=all&sort=updated`. Incremental runs also send `since` from the last `updated_at` watermark. The JSON body is again an array. GitHub's next page is the `Link` header, not a cursor field. `full_name` is copied onto each PR record so `pull_request_comments` can call `/repos/{full_name}/pulls/{number}/comments` without looking the repo up again. ## Rate limits, retries, and skipped parents `connector.rateLimits` (and optional per-tap `rateLimits`) cap requests per second and max concurrent calls. `retry.respectRetryAfter` honors the source's `Retry-After` when it is present. GitHub's definition sits near 1.4 requests/second, so a large backfill is paced on purpose. On fan-out, `skipOnHttpStatus` (for example `[403, 404]`) skips that parent and continues. `errorBodyRules` can reclassify a status by inspecting the body when the source uses one code for several outcomes. `perParentAuth` switches the OAuth subject per parent record (Google Workspace impersonation). You do not configure these; they explain why one repo 404s and the rest of the sync finishes. ## Allowlists in `ingestionConfig` Some connectors accept `parentDataFilters` inside `ingestionConfig`. When the tap's `restConfig` has `parentDataFilterQueryInject` or `parentDataFilterBodyInject`, those values are written into `$filter`, a repeated query param, or the JSON body so the source only returns matching rows. If the filter key is absent, the tap runs workspace-wide. If the key is present but empty or the field name does not match, the tap is skipped. Step-by-step connector setup (OAuth apps, API keys, scopes) lives in the [product documentation](https://docs.parable.work).