Multi-File Pipelines

A Bridge Town model can contain multiple Python model files that compose a multi-step financial pipeline — for example, revenue → expenses → summary, where each file reads the outputs of the file before it. This page explains how to wire them together using the PIPELINE list in run.py, how the branch-scoped /upstream mount carries intermediate outputs between files, and where Google Sheets fit (spoiler: they are external I/O, not intra-run transport).

Two kinds of data paths

Path	What it carries	Scope	Access
/data/	CSV/Excel/Google Sheet snapshots	Model-wide, all runs	Read-only
/upstream/	Intermediate outputs from earlier pipeline steps	This run, this branch	Read-only after upstream file writes

/data/ is for external inputs. It holds immutable Parquet snapshots produced by ingest_file_upload or refresh_google_sheet_snapshot. Every run — whether a single file or the whole model — can read from /data/. The data inside does not change during a run; freshness is controlled by when you call the matching ingestion or refresh tool.

/upstream/ is the pipeline bus. It exists only during a run_model(mode='sync') call and only when PIPELINE is defined in run.py. As each file in the pipeline completes, its runtime output dict (a module-level result dict or supported outputs dict) is serialised to /upstream/<file_name>/outputs.json so the next file in the list can read it. /upstream/ transport requires the file to define a runtime output dict — files that write directly to /outputs/ without defining result (or an outputs dict) execute normally but are not materialised to /upstream/ and cannot be consumed by downstream files via this path. When the run finishes, /upstream/ is discarded — it is not persisted to S3 and is not visible between runs or branches.

Google Sheets are not pipeline transport

Do not write intermediate file results to Google Sheets and then read them from the next file in the same run. That pattern couples pipeline execution to an external service, breaks reproducibility, and requires round-trip network I/O inside the sandbox (which has no network access anyway). Use /upstream/ for intra-run, intra-pipeline data exchange. Google Sheets are for importing external assumptions at the start of a pipeline and for exporting polished outputs at the end.

Enabling the pipeline

In run.py, define a top-level PIPELINE list naming your files in execution order:

run.py

PIPELINE = ["revenue", "expenses", "summary"]

When run_model(mode='sync') executes run.py, it detects PIPELINE and:

1. Runs model/revenue.py; writes its result dict (or outputs dict) to /upstream/revenue/outputs.json.
2. Runs model/expenses.py; writes its result dict (or outputs dict) to /upstream/expenses/outputs.json.
3. Runs model/summary.py.

Each /upstream/ write only occurs when the file defines a runtime output dict — either a module-level result dict (preferred, pairs with the outputs = [...] contract metadata convention) or a module-level outputs dict. A file that writes directly to /outputs/ without defining either runs and appears in run_model(mode='sync') results, but it is not materialised to /upstream/ for downstream files.

Files not in PIPELINE are skipped. If a file listed in PIPELINE does not exist, it is skipped with a warning on stderr and execution continues.

Without PIPELINE, the scaffold auto-discovers all model/*.py files and runs them in alphabetical order without upstream materialisation.

The /upstream first, /data fallback pattern

Downstream files should prefer /upstream/ when available and fall back to a /data/ snapshot when running outside a pipeline context (for example, during a single-file run_model(mode='sync', path='<name>.py') call or early development before all pipeline stages exist):

model/summary.py

"""Combine revenue and expense outputs into an executive summary."""
import json
import pathlib
import pandas as pd

MONTHS = 12

# --- revenue inputs ---
_upstream_rev = pathlib.Path("/upstream/revenue/outputs.json")
_data_rev = pathlib.Path("/data/revenue_actuals")  # Parquet snapshot fallback

if _upstream_rev.exists():
    rev_data = json.loads(_upstream_rev.read_text())
    total_revenue = sum(rev_data.get("monthly_revenue", [0] * MONTHS))
else:
    # Standalone run: read the Google Sheet snapshot instead.
    df = pd.read_parquet(str(_data_rev))
    total_revenue = float(df["revenue"].sum())

# --- expense inputs ---
_upstream_exp = pathlib.Path("/upstream/expenses/outputs.json")
_data_exp = pathlib.Path("/data/expense_actuals")

if _upstream_exp.exists():
    exp_data = json.loads(_upstream_exp.read_text())
    total_expenses = sum(exp_data.get("monthly_expenses", [0] * MONTHS))
else:
    df = pd.read_parquet(str(_data_exp))
    total_expenses = float(df["expenses"].sum())

# --- summary ---
inputs = ["monthly_revenue", "monthly_expenses"]
outputs = ["total_revenue", "total_expenses", "net_income"]
dependencies = ["revenue", "expenses"]

result = {
    "total_revenue": round(total_revenue, 2),
    "total_expenses": round(total_expenses, 2),
    "net_income": round(total_revenue - total_expenses, 2),
}

The fallback branch makes summary.py runnable as a standalone file (useful during development), while the primary branch composes the full pipeline result when run via run_model(mode='sync').

Complete three-file example

Here is a minimal but complete model showing the revenue → expenses → summary chain.

run.py (scaffold with PIPELINE defined):

"""Bridge Town model entry point — auto-discovery scaffold.

# BT-SCAFFOLD-SENTINEL
"""

PIPELINE = ["revenue", "expenses", "summary"]

# (The rest of this file is the standard Bridge Town scaffold.)

model/revenue.py:

"""12-month SaaS revenue projection."""
MONTHS = 12
BASE_ARR = 1_200_000
GROWTH_RATE = 0.07

monthly = []
arr = BASE_ARR
for _ in range(MONTHS):
    monthly.append(round(arr / 12, 2))
    arr *= 1 + GROWTH_RATE

inputs = ["base_arr", "growth_rate"]
outputs = ["monthly_revenue", "arr_eoy"]

dependencies = []

# Alternative runtime output pattern (dict).
result = {"monthly_revenue": monthly, "arr_eoy": round(arr, 2)}

model/expenses.py:

"""Monthly expense file — headcount + opex."""
import json, pathlib

_upstream = pathlib.Path("/upstream/revenue/outputs.json")
if _upstream.exists():
    rev = json.loads(_upstream.read_text())
    # Size opex as a percentage of revenue.
    monthly_revenue = rev.get("monthly_revenue", [100_000] * 12)
else:
    monthly_revenue = [100_000] * 12

HEADCOUNT_COST = 80_000          # per month
OPEX_PCT_OF_REVENUE = 0.18

monthly = [
    round(HEADCOUNT_COST + r * OPEX_PCT_OF_REVENUE, 2)
    for r in monthly_revenue
]

inputs = ["monthly_revenue"]
outputs = ["monthly_expenses"]
dependencies = ["revenue"]

result = {"monthly_expenses": monthly}

model/summary.py: (see /upstream first, /data fallback example above)

Running run_model(mode='sync') on this model produces combined output keyed by file name and triggers downstream scenario analysis naturally via compare_branches.

Scenario analysis with pipelines

For scenario analysis, use full pipeline runs — not Google Sheets — as the comparison mechanism:

1. Create a scenario branch: create_branch with branch_name="scenario/upside".
2. Edit the assumption in the relevant file on that branch: patch_file.
3. Run the full pipeline: run_model(mode='sync') with branch="scenario/upside".
4. Compare outputs: compare_branches with base_branch="main" and scenario_branch="scenario/upside".

Because /upstream/ materialises all intermediate file outputs during each run, compare_branches can diff the full pipeline — not just the file you edited. Changes propagate automatically through the dependency chain.

Do not export intermediate results to Google Sheets and import them back on a different branch to compare scenarios. This breaks the branch-scoped isolation that compare_branches depends on.

Authoring checklist

When writing a downstream file that reads from /upstream/:

• Use pathlib.Path("/upstream/<file_name>/outputs.json").exists() to guard the read — never assume the file is there.
• Provide a /data/ or constant fallback so the file is runnable standalone.
• Keep the upstream read near the top of the file, before any computation.
• Do not write back to /upstream/ — it is managed by run.py.

Related

• Models & Files — PIPELINE syntax reference and file lifecycle
• Sandbox Execution — mounted paths and security constraints
• Data Sources & Snapshots — /data/ mount and Google Sheets integration
• Google Sheets Integration — connecting sheets as external inputs and writing results back