Pipeline components

Overview

A pipeline component is a reusable fragment of a pipeline — a few nodes, the edges between them, the ports it exposes at its edge and the parameters it takes — published as a numbered version and referenced by as many pipelines as you like. Forty repositories that build the same way stop holding forty copies of the same three steps: you publish rust-build once on the group that owns them all, each repository references it, and improving the shared part is one edit and one publish instead of forty merge requests.

The version number is the contract. Publishing checks the change you made against the last version you published and refuses a number that understates it, so a pipeline tracking v1.* can pick up your patch releases without you promising by hand that nothing broke. When something does need to break, you deprecate the old version, see exactly which pipelines are still on it, and move them across in one sweep.

Quick start

Before you start

How it works

Three things, not one

Editing the draft cannot break anything, because nothing references a draft. Only a published version is referenceable, and only publishing makes an edit visible to a consumer.

A component has no stages of its own

A pipeline has stages. A component has stage slots — a short, ordered list of positions with names that mean something inside the component — and the pipeline that uses it decides which of its own stages each slot lands in.

Every slot the body declares must be mapped, and the stages you map them to have to run in the same order the slots are declared in. Several slots may share one stage; an inversion is refused with a message naming both slots, which is a much better error than the one you would get after the fragment had been flattened into nodes you never wrote.

Parameters replace whole values

A component declares typed parameters, and the body refers to one by putting {"$param": "name"} where the value belongs:

The instance that uses the component supplies image_tag, and expansion writes the value in whole. There is no string interpolation and no expression language: a reference occupies an entire value slot or it is not a reference. To build a string out of parts, use $join, whose arguments may themselves be references:

Two escapes are worth knowing. {"$literal": <value>} emits its value verbatim, which is how a configuration object that genuinely needs a key called $param gets one. And an object carrying $param alongside other keys is not a reference at all and passes through untouched — the rule is one key, or nothing.

A parameter may declare allowed_values, a closed set the supplied value must belong to. Adding a member to that set is a minor change; introducing the set, or removing a member from it, is a major one, because a consumer whose value has just left the set stops validating.

Ten node settings are scalars rather than JSON documents and so cannot hold a reference object: the connection a node authenticates with, its CPU, its memory, its timeout, the environment it deploys to, the environment whose variables it reads, the service it acts on, whether its failure is tolerated, and how many times and how long apart it retries. A component node names the parameter for each of those in a param_slots block instead. It is the same substitution at the same moment, which is what lets one deploy component ship to staging in one place and production in another rather than being published twice. A component that builds and then deploys names the same parameter in its build's variables_from and its deploy's environment, so each instance builds with the settings of the environment it deploys to. The capability a node calls and the provider version it pins are never parameterised, because those two decide which code runs and a version whose requirements are unknowable cannot be checked at all.

A value that fits the parameter's declared type can still be one the setting cannot hold — production! is a string, and it is not a valid environment name. That is refused when the pipeline is saved, naming the setting and the value. The platform never quietly declines to apply one: a deploy node silently left without its environment would lose that environment's protection and its approval step, and would look deliberate.

Read the consuming pipeline's variables

A body's configuration may read a variable with {"$var": "NAME"}, exactly as a pipeline step's can (see Read a variable in a step's configuration). Expansion leaves it untouched, so it reads the variables of the step it becomes in the pipeline that uses the component — that step's environment, its repository and its groups — not anything of the component's own.

That is what keeps a deploy component small. Rather than a parameter for every bucket, distribution and endpoint, it takes one parameter for the environment, spends it on the node's environment setting, and reads the values the environment already holds:

A pipeline that sets target to production syncs to production's bucket, and a second instance set to staging syncs to staging's, from one published version.

Every version lists the names its body reads as variable_requirements — what a consuming step has to be able to see. Each is checked against the consuming pipeline when one of its runs starts, and a missing one refuses the run naming your instance and the variable. Publishing a version that reads a name the previous version did not is a major change; reading fewer is a minor one.

A pipeline using a component may also hand a variable to a parameter, by writing {"$var": "NAME"} as the parameter's value in the instance's params. The variable is read from the consuming step, like any other, and it is that pipeline's choice rather than part of the component's variable_requirements. The parameter has to be able to receive one, and is refused with the violation component_param_var_refused otherwise:

• It is declared string. A variable is always a string, and nothing converts one into a number or a true-or-false value.
• It declares no allowed_values. A fixed set cannot be checked against a value nobody knows until the run starts.
• The component does not spend it on a node setting in param_slots, such as environment or connection. Those settings decide how the run is put together, which happens before any variable is read.

A parameter's default cannot read a variable. A default is the component author's choice, so write the {"$var": …} in the body where the value is used instead.

Version references, and which ones float

An instance names the version it wants in one of four forms:

The last three float: they re-resolve every time the pipeline is written or run, so a publish reaches them without anybody editing anything. A floating reference only ever selects an active version — never a deprecated or sunset one — which is what makes deprecating a safe way to retire something. An exact reference is immune to a publish and is the right choice when you need the same bytes every time.

The compatibility gate

Publishing computes what moved between the draft and the newest version you have not sunset, works out the lowest number that change permits, and refuses anything lower. The contract it compares is exactly the parameter declarations, the boundary ports and the variables the body reads — not behaviour.

Retirement has three states

A version is never deleted. Sunsetting is the only state that breaks a pipeline still pinned to the version, and it does so loudly: publishing that pipeline is refused, and a run triggered from it fails to start, both with a message naming the instance, the component and the version. A draft of it still saves, carrying the violation.

What happens when a pipeline uses one

Referencing a component adds no new mechanism to a pipeline. Before anything else runs, each instance is replaced by the nodes and edges of the version it resolves to, and everything downstream — structural checks, port wiring, configuration validation, environment and service binding — runs on a flat graph that no longer mentions components at all.

Two visible consequences. Node identifiers inside the component are prefixed with the instance's, so a node called build inside an instance called checkout becomes checkout--build, displayed as Checkout / Build and badged with the component and the version it resolved to. And a component's boundary outputs become candidates for automatic wiring in the pipeline around it, so adding an instance can make a previously unambiguous connection ambiguous. That is the rule working; the remedy is an explicit edge.

How to

Every task below exists twice — once for a component a group owns and once for a component a repository owns. The steps, the fields and the refusals are identical; only the owner in the path changes. Each task shows one scope, and the Reference section lists both.

Find the components you can use here

The listing for a scope answers what that scope may reference: its own components plus every parent group's, with the nearest declaration of a name winning. search narrows by name and description, and own_scope_only drops the inherited ones when you want to see only what you own.

One thing the listing does not show you: a component owned by a group you cannot read is dropped from it. That filter is on the picker alone — a pipeline in the subtree can still reference the component, and you can still open it by its id through this scope's own path. If somebody points you at a component you cannot find in the list, ask them for its id, or ask for read access to the group that owns it.

At group scope the same listing walks up from that group, so a root group sees only its own.

Create a component

A component is created with its name, an optional description, and a draft body. The name is lowercase letters, digits and hyphens, up to 100 characters, may not start or end with a hyphen, and is unique within the scope that owns it. The body has to be complete enough to stand on its own — at least one stage slot and at least one node — and is validated as you save it, not only when you publish. Nothing can reference the component until you publish a version, so creating one is safe in every sense.

Put it on the group that owns the repositories that will use it. A component on a repository is only ever usable by that repository, which is right for a fragment that encodes something local and wrong for anything you expect to share.

For something genuinely local to one repository, create it there instead.

Three node settings are refused inside a component body, and the refusal happens when you save the body so the person who wrote the field sees the error:

• Path filters (changes) — the paths would be matched against a repository the component author does not own, and a skipped internal node propagates its skip outward through the boundary into nodes that have nothing to do with the component.
• Approvals (approval) — naming an approver needs standing on it — Maintainer on a group whose role you name, authority over a team you name — so a team using your component would fail its own pipeline save on an approver it has never heard of. A cross-team gate belongs on an environment.
• Resource groups (resource_group) — a resource-group key is a lock shared across the namespace. One component used forty times would quietly serialise forty pipelines against a key none of their authors chose.

An environment and a service are allowed, and are re-resolved and re-authorized against the consuming repository after expansion, so a repository that may not deploy to production still cannot.

Edit the draft

The draft is a full replacement: send the name, the description and the body you want, not a patch. Published versions are untouched, so an edit here changes only what your next publish would capture and cannot break a running pipeline — and that holds for a rename too, because an instance stores the component's id and never its name.

A draft still has to be a well-formed body. A save that names a stage slot the body does not declare, repeats an identifier, refers to a parameter nothing declares or carries a field a component may not hold is refused with 400 PIPELINE_VALIDATION_FAILED and every violation listed, so a stored draft is always one a publish would accept.

Review a component before you use it

Reading a component gives you its draft body and its published versions, newest first. This is what to read before writing an instance: the parameters you have to supply, the ports you can wire, the variables each version reads — its variable_requirements, which the step it becomes must be able to see — and which versions are still active.

Check what a publish would do

Publishing takes a dry_run that validates the draft, checks the number you propose against the compatibility gate, and reports the blast radius — how many consumers wrote a floating reference rather than an exact one, and whether a sample of those still expands and validates against the draft. Nothing is written.

A dry run still needs the number you mean to publish, and it is judged exactly as the real publish would judge it. Propose one that understates the change and you get the refusal rather than a report — naming the step the contract moved by and the lowest number the gate permits. Propose one that is high enough and the report comes back carrying minimum_version, the lowest you could have used.

Read the floating count as how many consumers can move without being asked, not as how many will land on this number: a consumer tracking v1.* is counted whether you are publishing v1.4.3 or v2.0.0. Once the version exists, upgrade impact answers the narrower question.

Run it before every publish that is more than a typo. The sample covers up to 10 consuming pipelines; a failure in the sample is a strong signal, and a clean sample is not a proof for the other thirty.

Publish a version

A publish freezes the current draft under a version number. The number must be at or above what the compatibility gate permits, and the same number is never published twice: a retried publish that already succeeded answers 409, deliberately, because a silent success would mean a version's contents had changed.

List and read published versions

Versions come back newest first, ordered by version number rather than alphabetically, so v0.10.0 sits above v0.3.0. Filter by lifecycle_state to see only the active versions, only the deprecated ones or only the sunset ones; leave it off and you get all three.

Reading one version gives you the body it froze — the definitive answer to what does v1.2.0 actually do, which the draft cannot answer once it has moved on.

Reference a component from a pipeline

A component instance is a node in your pipeline. It carries the component's id, an instance identifier of up to 20 characters that is unique among the pipeline's nodes, the version reference you want, a map from every stage slot to one of your stages, and the parameters the body declares:

Wire into it with ordinary edges: the target is the instance and the port is one the component exposes as a boundary input. Wire out of it the same way, from a boundary output. A component is a node with ports, and there is no second wiring mechanism to learn.

A string parameter may take one of your variables instead of a written value — "params": { "bucket": { "$var": "WEB_S3_BUCKET" } } — which is read from your step when the run starts. Read the consuming pipeline's variables says which parameters can.

You may only reference a component whose owner is your repository or one of its parent groups. A component elsewhere is refused even if you have its identifier: publishing the pipeline comes back 400 PIPELINE_VALIDATION_FAILED with a violation naming your instance and saying the component does not exist here, and a draft save keeps the violation on the draft rather than blocking it. It reads the same whether the component is missing or merely out of your chain, because a scope you cannot see must not become discoverable by guessing.

Saving and publishing pipelines is covered in Pipelines; the only thing components add to it is that expansion runs first, so a violation inside a component is reported against your instance with the inner node named in the message.

See which pipelines use a component

The consumer list is the answer to what will this break. Each row is one reference: the pipeline, the repository, the instance identifier, the reference its author wrote, and the version that reference resolved to at the last write.

Rows for repositories you cannot read are dropped after the page has been fetched, so a page can come back shorter than the size you asked for while a next page is still waiting. A group-owned component consumed across a subtree you only partly administer therefore shows you fewer consumers than exist, which is deliberate.

Measure how far consumers are from a version

Upgrade impact answers the question that comes after a publish: who has still not moved. It measures every consumer against one version — the newest published one by default — and sorts them into three dispositions, because there are three different things to do about them.

Every reference is re-resolved as the report is built rather than read from what the last pipeline write saw, because a floating reference moves without its pipeline being written and that drift is exactly what the report exists to surface. Up to 10 consumers are also re-expanded against the target and re-validated, so you learn about a break before you cause it.

The report covers up to 500 consumers and says truncated when there are more, and its counts describe what it covered and what you are allowed to read rather than the whole population. Read the consumer list, which pages, when you need every one of them.

Move consumers onto a different reference

A re-pin rewrites the version reference in consuming pipelines. Give the reference to move everyone to; optionally give the reference to move from, so only instances written exactly that way are touched, and a list of pipelines to restrict the sweep to.

Run it with dry_run first. A dry run performs every read, every rewrite and every validation and writes nothing, so its report is an honest answer to would this break anything.

Three things about the result are worth expecting rather than discovering:

• Skips are normal and every one is named. A pipeline is left alone when you cannot manage its repository (forbidden), when you cannot read it at all (unreadable), when it already uses the target reference (already_pinned), when the rewritten definition would stop validating (validation_failed, carrying the message), or when it has been deleted since the list was built (pipeline_missing).
• Nothing is written for a pipeline that would not validate. Each pipeline is validated exactly as an ordinary save validates it, and a refusal becomes a row rather than a broken pipeline.
• A sweep covers up to 500 consumers. Past that the report says truncated. Every pass reads from the beginning of the consumer list, and a source reference or an explicit list of pipelines narrows what the sweep rewrites rather than which rows it reads — so after a truncated pass, run the sweep again and read the consumer list, which pages properly, to see what is still outstanding.

There is no transaction across pipelines, on purpose: they live in different repositories, and a partial sweep that reports exactly what it did beats one that rolls back thirty-nine successful writes because the fortieth failed.

Retire a version safely

Deprecating a version removes it from every floating track without disturbing anyone pinned to it exactly. It still runs. Consumers tracking v1.* move off it at their next resolution; consumers pinned to v1.4.2 stay where they are until somebody moves them. This is the way to retire a version.

A floating reference only ever selects an active version, so deprecating the newest one leaves latest reaching the next active version rather than resolving to nothing.

A pipeline pinned exactly to a version you have deprecated still saves and still runs. Saving it reports a warning naming the instance, the component and the version — which is the signal to move it, not a refusal.

If you deprecated something in error, put it back:

Stop a version from running at all

Sunset the version you need gone. The version is never deleted, so its body stays readable for anyone auditing what ran — but every pipeline pinned to it is refused from that moment, and nothing about that resolves itself: each one has to be edited onto a version that still resolves. Sunset is final: restoring or deprecating a sunset version is refused with 409, so the way back is to publish a new version.

Delete a component

A delete answers 409 while any pipeline still references the component, at any version. Read the consumer list, remove the instances from those pipelines, then delete. The guard is a live constraint rather than a check the request performs first, so a pipeline that starts referencing the component while you are deleting it still wins.

Reference

Components

Versions

Consumers

Limits

Two caps are metered and enforced. A request that would exceed one is refused with 429 TOO_MANY_REQUESTS and writes nothing — there is no partially created component and no half-published version.

Components are counted per scope rather than per namespace: a group holding a hundred shared components and a repository holding a hundred of its own are two separate counts against two separate ceilings. Versions are counted per component because a version is immutable and never reused, so without a ceiling a publish loop grows without bound.

Either cap can be raised for one namespace. An Owner of the root group requests an increase from the group's billing settings; the ceiling on a raised component or version limit is 50 000. Deleting a component releases its count; a version is never deleted, so the version count only falls when the whole component goes.

A component body has fixed structural bounds, and they apply on every plan. These are shape limits rather than metered quotas — exceeding one is 400, not 429.

Two operational bounds are worth planning around. A re-pin sweep and an upgrade-impact report each examine up to 500 consumers and say truncated when there are more; both read from the beginning of the consumer list and neither is narrowed by a filter, so the read that reaches every consumer is the consumer listing, which pages properly. And re-expansion is sampled: a publish's blast radius and an upgrade-impact report each re-validate up to 10 consuming pipelines — enough to catch a systematic break, not enough to prove its absence. Every listing pages at 20 by default and 100 at most.

Troubleshooting