Pulse-Check

Pulse-Check is the validation API for Pulse. It scores how closely simulated order book data matches real historical market data for the same symbol and date, using distributional distance metrics (L1 and Wasserstein), the Cont stylised facts, the MIND and FID inception distances on DeepLOB embeddings, and impact response analysis. It is a separate service from the native Pulse simulation API: validation jobs run under /validation and are accessed via client.validation in the Python SDK.

A Pro result never contains the real session. The distances, the per-fact verdicts and the two inception scores are aggregate scalars, so they are returned in full, and the impact response is returned for your simulated runs — that is your own generated data. What is withheld is anything derived from the historical series itself: the distribution histograms, the full stylised-fact series, and the historical half of the impact response.

How it works

1. Submit a validation job with run() — returns a job_id immediately
2. Historical data for the symbol and date is fetched automatically; simulation data comes from your sim_ids (or uploaded parquet files)
3. Poll get_job_status(job_id) until is_complete is true, then call get_job(job_id) once to fetch the result
4. Read results: per-metric distances, per-fact stylised-fact verdicts, and the MIND/FID scores

What a job computes

Six areas, one flag each. Every area defaults on, so a job submitted with no flags at all returns everything in the last column.

Validate simulation runs

The most common flow: run simulations with the native Pulse API, then pass the resulting sim_ids to Pulse-Check. Validation compares each run's sim_data.parquet against historical data for the same symbol and date.

The entry below documents the validation call itself. To poll the job and read its result, see get_job_status() and get_job().

# setup: the runs of a cached baseline
cached = client.simulation.list_cached(symbol="700.HK", scenario="normal")["simulations"][0]
base = cached["example_sim_id"].rsplit(":", 1)[0]
sim_ids = [f"{base}:{i:04d}" for i in range(min(cached["n_runs"], 3))]

job = client.validation.run(
    symbol=cached["symbol"],
    date=cached["date"],
    provider="omd",
    exchange="hkex_securities",
    sim_ids=sim_ids,
)
print(job["job_id"])

# two named populations, compared against each other and the real day
cached = client.simulation.list_cached(symbol="700.HK", scenario="normal")["simulations"][0]
base = cached["example_sim_id"].rsplit(":", 1)[0]
runs = [f"{base}:{i:04d}" for i in range(min(cached["n_runs"], 4))]

job = client.validation.run(
    symbol=cached["symbol"],
    date=cached["date"],
    provider="omd",
    exchange="hkex_securities",
    sim_ids={"first_half": runs[:2], "second_half": runs[2:]},
    plots=["statistical.radar", "stylised_facts.overall"],
)
print(job["job_id"])

# your own parquet beside platform runs, on one set of figures
from pathlib import Path

cached = client.simulation.list_cached(symbol="700.HK", scenario="normal")["simulations"][0]
base = cached["example_sim_id"].rsplit(":", 1)[0]

Path("my_sims").mkdir(exist_ok=True)
df = client.simulation.get_sim_data(cached["example_sim_id"])
df.write_parquet("my_sims/mine_0000.parquet")

job = client.validation.run(
    symbol=cached["symbol"],
    date=cached["date"],
    provider="omd",
    exchange="hkex_securities",
    sim_files={"mine": sorted(Path("my_sims").glob("mine_*.parquet"))},
    sim_ids={"platform": [f"{base}:0001"]},
)
print(job["job_id"])

Populations

Pass a mapping rather than a list and the job compares populations instead of averaging every run together. Results — distances, distributions, the verdicts and the FID/MIND scores — come back keyed by name, and the summary figures draw one series per population: one radar polygon each, one verdict-grid column each, one correlation heatmap each after the historical one. Runs within a population are pooled. The historical day is measured once, however many populations there are.

sim_ids and sim_files can be given together, so your own model and platform runs land on one set of figures — 25 runs in total across the two. With both present each source becomes a population of its own, so a bare list is given a name rather than merging into whatever else is there:

The uploaded files are measured first and the platform runs follow.

Check job status

Validation jobs move through: pending → running → completed (or failed). Only completed and failed are terminal. Poll the light status call, and fetch the full result once.

import time

# setup: your most recent validation job
job_id = client.validation.list_jobs(limit=1)["jobs"][0]["job_id"]

while not client.validation.get_job_status(job_id)["is_complete"]:
    time.sleep(5)

result = client.validation.get_job(job_id)
print(f"Status: {result['status']}")
if result["status"] == "failed":
    print(f"Error: {result['error']}")
elif result.get("errors"):
    print(f"Areas that failed: {result['errors']}")

# setup: your most recent validation job
job_id = client.validation.list_jobs(limit=1)["jobs"][0]["job_id"]

result = client.validation.get_job(job_id, plot_dir="figures")
print(f"Status: {result['status']}")
print(f"Window: {(result.get('metadata') or {}).get('historical_window')}")
print(f"Failed areas: {result.get('errors')}")

Plot ids

plots takes True for every figure the enabled areas can draw, or a list of the ids below for just those. An area switched off draws nothing either way. A figure's filename does not carry the id it was asked for, so every returned entry carries its id — match on that.

Validation metrics

With statistical=True, Pulse-Check computes stylised-fact distributions on both the historical and simulated order books, then scores the gap between them with L1 and Wasserstein distances. Lower is better. This distributional approach to scoring generative order book models follows the methodology of LOB-Bench (Nagy et al.). Metrics include:

The scores come back on a finished job as distances — see get_job() for the shape.

Stylised facts

With stylised_facts on (the default), Pulse-Check runs the eleven Cont stylised facts on the historical day and on every simulated run, and returns the verdicts. A real session is expected to exhibit them; the question a validation answers is whether your simulation does too.

{
  "heavy_tails": {
    "historical": true,
    "simulated": [true, true, false, true, true],
    "aggregate": {"n": 5, "passes": 4, "verdict": true, "significant": true},
    "policy": {
      "frequencies": ["50s", "100s", "150s", "200s", "..."],
      "judged": ["50s", "100s", "150s"],
      "mode": "any"
    }
  }
}

historical is the verdict on the real session and simulated holds one verdict per run, in sim_ids order; null means the fact could not be evaluated on that frame (usually too short). aggregate folds the per-run verdicts into one Monte Carlo signal with a one-sided binomial test against chance at the 5% level — significant says whether the run count was enough to call it. policy reports which sampling frequencies the methodology scores the fact over, which of them the data actually supported (judged), and how they combine: any, all, or slope across the ladder.

Impact response

With impact on (the default), Pulse-Check computes the Bouchaud response function for each simulated run: how far the mid-price moves, in ticks, in the events following a given kind of book event. It follows the same LOB-Bench methodology as the distance metrics, after Eisler, Bouchaud and Kockelkoren (2012).

Events that change a best price are classified into six types — market orders, limit orders and cancellations, each split by whether the event itself moved the mid (_1) or not (_0) — giving MO_0, MO_1, LO_0, LO_1, CA_0 and CA_1. For each, R(l) is the average signed mid-price move from just before the event to l events later, normalised by tick size, over a logarithmic ladder of lags from 1 to about 200. So R(1) is the event's own immediate move and later lags show whether that move persists or reverts. R(1) is zero by construction for the _0 types, so any growth in those curves reflects order-flow predictability rather than the event itself.

Inception distances (MIND and FID)

Both metrics compare 96-dimensional DeepLOB embeddings of 100-row L2 windows: FID is the Frechet distance between the two embedding clouds, MIND is a sliced-Wasserstein distance over the same embeddings, which makes it sensitive to differences beyond mean and covariance. Lower is closer to the historical day. Both need the full 10 book levels.

Each score has its own flag on run(), fid and mind. They share one embedding pass, which runs when either flag is on. To get only the scores, set statistical, stylised_facts, impact and volume_correlation to False. The result then carries the two score lists, in sim_ids order.

View past jobs

jobs = client.validation.list_jobs(limit=50)  # max 200

print(f"You have {jobs['total']} validation jobs")
for job in jobs["jobs"]:
    params = job["params"]
    print(f"{job['job_id']}  {job['status']}")
    print(f"  {params['symbol']} {params['date']} — {len(params.get('sim_ids') or [])} sims")
    print(f"  Created: {job['created_at']}")

REST endpoints

For non-Python integrations, Pulse-Check is plain HTTP. Authenticate with your Pro API key in the X-API-Key header.

The config object takes the area flags directly, one name per thing. There is no ticksize field on either route: it is read off the historical day.

An unset area flag runs that area. Any field not in this table is rejected with a 422 that names it — including the older spellings run_metrics, run_stylised_facts, run_fid, run_impact, plot_data and l2_only, which were removed. Whether the historical series is returned is set by your tier, not by the request.