Skip to main content

Pipelines

A pipeline is the unit of work in Whet. It defines where the content comes from (source), what the AI does with it (processing), and what it delivers (output). It is a re-runnable contract: the same pipeline runs multiple times over time and produces new artifacts each time.

The mental model​

┌──────────┐    ┌──────────────┐    ┌───────────┐
│  Source  │ →  │  Processing  │ →  │  Output   │
└──────────┘    └──────────────┘    └───────────┘
  3 layers       LLMs + jobs        prose | struct

Each pipeline persists its configuration (source, processing, output) and produces runs. Each run produces one or more artifacts.

The three ingestion layers​

Whet is not married to any platform. Each pipeline can take content via one of these three layers:

Layer 1 — Official API​

Bring your own API key from the platform. Whet respects ToS and rate limits.

Best for: compliance-first teams, paid APIs you already have, RSS feeds, internal sources.

Example: an RSS feed from a corporate blog, a paid endpoint from a research platform.

Layer 2 — Managed ingestion​

Bring your own burnable accounts and session credentials. Whet runs the credential pool, rotation, and FSM inside the binary on your machine.

Best for: operators with established niches and accounts they can put at risk.

The client operates the ingestion adapter on their machine. The client assumes the legal risk. Whet only provides the orchestration tooling.

Layer 3 — Manual paste (roadmap)​

Paste a URL, a thread, an email, or an OCR. Zero infrastructure.

Best for: content behind a paywall, internal docs, anything that is not on the open web or that you do not want to automate.

This is the universal layer — it works with any source.

:::caution Product status Today the CLI / Agent API only support two sources: social (managed ingestion of a platform handle) and webpage (public URL). Manual paste of threads, emails, and OCR is not implemented: for content behind a paywall, export it to a public URL or paste it into the style_reference field when you trigger a start_draft from MCP. It is the best approximation in the meantime. :::

Processing​

The middle block. This is where the following happens, depending on the pipeline config:

  • Decode — the LLM extracts patterns (hook, tone, format, triggers) from the input.
  • Aggregate — if there are several inputs, it joins them and computes niche metrics.
  • Synthesize — combines insights from heterogeneous sources into a single narrative.
  • Riff — generates a new draft grounded in what was decoded.
  • Refine — takes an existing artifact and produces a child variant (shorter, change tone, match voice, more punchy).

LLMs are swappable per pipeline: Anthropic, OpenAI, Gemini. Each pipeline defines which one to use.

Output​

What the pipeline delivers:

  • Prose — publishable drafts, with or without citations.
  • Structured — KV / JSON with the extracted patterns.
  • Dashboard — niche aggregations (top hooks, tone distribution, leaders).

Scope​

When the pipeline runs:

ScopeFrequencyTypical use
per_postevery time a new input appearscompetitor decode
daily_digestonce a day, aggregating the day's contentniche dashboard
weekly_digestonce a weeksummary reports

Refinement​

Every artifact can spawn child variants. Genealogy is preserved: the child artifact stores parent_id, and you can navigate the entire tree.

The available refinement actions are:

  • shorter — trim while keeping intent.
  • change_tone — move from analytical to punchy (or vice versa).
  • match_voice — adjust to the operator's voice profile.
  • more_punchy — add friction/edge.

Example pipeline​

# Track a competitor handle, draft per post, analytical tone
whet pipelines new \
  --source social \
  --handle @growth_dr \
  --kind prose \
  --scope per_post \
  --tone analytical \
  --auto-fanout

Result:

  • Source layer: managed ingestion (operator's session credentials).
  • Processing: decode → riff with Anthropic Claude.
  • Output: prose artifacts, one per new post from the handle.
  • Scope: per_post.
  • Refinement loops: available from the workbench and the MCP.

Multi-source synthesis (roadmap)​

A pipeline that takes multiple sources and combines them into a single draft with citations is one of the roadmap directions. The idea is:

Sources:
  - Layer 2 · @growth_dr (managed ingestion)
  - Layer 2 · @bootstrap_fi (managed ingestion)
  - Layer 3 · blog.example.com/pricing-anatomy (manual paste)

Processing: synthesize → riff
Output: 1 prose draft with [1] [2] [3] citations

Today the CLI/Agent API create pipelines with a single source. The UI shows the concept and the landing's Decoder demonstrates it, but the wire that combines multiple sources into one pipeline is in development. For feedback or specific use cases, open an issue or mention it on the discovery call.

See also​