AI - Parable Developers

ponder() and embed() are SQL functions on the same Flight SQL connection you use for the lake. Each row that reaches the function is one model call (ponder) or one embedding (embed). Named arguments use =>. Keep a LIMIT on the input. These calls are billed and capped.

ponder()

ponder() runs an LLM once per row and returns a typed column. Required named arguments: provider, name, variant, version, instruction, output, and inputs. inputs must be a named_struct so each field has a prompt label. instruction and output are constant strings (they are part of the plan). server => 'vertex' is the production path. output is one of Utf8, Int64, Float64, Boolean, Date32, Timestamp, or Struct. schema is required when output is Struct, and is a JSON Schema object as a string. Classify Linear issue titles:

SELECT
  id,
  title,
  ponder(
    provider => 'google',
    name => 'gemini',
    variant => '2.5',
    version => 'flash',
    server => 'vertex',
    instruction => 'Classify the issue. Reply with one of: bug, feature, question, other.',
    output => 'Utf8',
    inputs => named_struct('title', title, 'body', description)
  ) AS category
FROM providers.linear.issues
WHERE title IS NOT NULL
LIMIT 20

Extract a struct. The return value is a struct of your schema fields plus an injected meta object (status, error, token usage, latency). meta is reserved; do not put it in schema.

SELECT
  id,
  classified.category,
  classified.reason,
  classified.meta.status,
  classified.meta.error
FROM (
  SELECT
    id,
    ponder(
      provider => 'google',
      name => 'gemini',
      variant => '2.5',
      version => 'flash',
      server => 'vertex',
      instruction => 'Extract a category and a one-sentence reason.',
      output => 'Struct',
      schema => '{"type":"object","properties":{"category":{"type":"string"},"reason":{"type":"string"}},"required":["category","reason"]}',
      inputs => named_struct('title', title)
    ) AS classified
  FROM providers.linear.issues
  LIMIT 20
) AS classified_rows

meta.status is ok (every required field present), partial (parsed, some required fields missing; meta.error names them), or error (the call failed). Primitive outputs (Utf8 and the rest) return NULL on failure; the rest of the query still runs. Pass model => some_column instead of inline provider / name / variant / version when the model config is already a column. Do not mix model with those inline arguments. Optional: date (default latest), effort, tokens, request_timeout_seconds.

embed()

embed() returns a unit-norm FixedSizeList<Float32> vector per row. The only required argument is input. Omitted model coordinates use the workspace default (Google gemini-embedding version 001, 768 dimensions, server vertex). purpose defaults to document. NULL or empty input returns NULL. A failed provider call NULLs that cell; the query continues.

SELECT
  id,
  title,
  embed(input => title) AS vec
FROM providers.linear.issues
WHERE title IS NOT NULL
LIMIT 50

Use purpose => 'query' for search strings and purpose => 'document' for stored text. Vectors are only comparable when they share provider, name, version, and dimensions. server is routing only; changing it does not change the vector identity.

SELECT
  embed(
    provider => 'google',
    name => 'gemini-embedding',
    version => '001',
    server => 'vertex',
    purpose => 'query',
    input => 'issues about billing'
  ) AS query_vec

Other purpose values: similarity, classification, clustering. A Plot that selects embed(...) materializes the vectors. The Plot table exposes the vector column; a Lance sibling parables.{parable}.{plot}__{column} holds the same vectors for ANN search.

Run these over Flight SQL the same way as any other lake query. Clients are on Querying the lake.

Getting Started

SDKs

Connectors

Parable Data Engine

AI

ponder()

embed()

​ponder()

​embed()

ponder()

embed()