Output format
The structure of tick-level simulation data returned by Pulse.
Pulse outputs data in Pulse Format — a normalised, exchange-agnostic representation of limit order book activity. The same schema works across any supported venue, so analysis code written for one exchange transfers directly to another without modification.
sim_data.parquet is a single frame whose columns fall into two groups: the book columns (bid_* / ask_*, ten levels a side) and the event columns (message_type, side, price, size, order_id). Each row is one event together with the book state it produced, so the two are aligned by construction — there is nothing to join.
The default file returned by get_sim_data(sim_id) is sim_data.parquet, which holds both column groups. See Download data for the full list of output files.
sim_id = client.simulation.list_cached()["simulations"][0]["example_sim_id"]
df = client.simulation.get_sim_data(sim_id)
book = [c for c in df.columns if c.startswith(("bid_", "ask_"))]
event = [c for c in df.columns if c not in book]
print(f"{df.height} rows, {df.width} columns")
print(f" {len(book)} book columns (10 levels x 3 fields x 2 sides)")
print(f" event columns: {event}")331848 rows, 70 columns
60 book columns (10 levels x 3 fields x 2 sides)
event columns: ['sequence', 'timestamp', 'timestamp_tick', 'message_type', 'side', 'price', 'size', 'order_id', '_to_drop', 'size_in_lots']The exact column count moves with the engine version, so derive it as above rather than hard-coding it.
Book depth: Level 1, 2 and 3
The two column groups sit at different depths, and that difference is why both are present. The book columns are aggregated by price level; the event columns are the individual orders behind them. Market data is sold at three such depths, and what you can ask of it changes with each.
Top of book only: the best bid and the best ask, each with its total size. Enough for a last price and a spread, and nothing about what sits behind them.
Full depth, aggregated. Every price level with its total size and order count — the shape of the book is visible, the orders inside each level are not.
Every resting order, in the queue position it holds. Time at price, fill probability, amendments and post-cancel behaviour become observable one order at a time.
Pulse simulates at Level 3 — agents place, amend and cancel orders one at a time — so the event columns carry every message at individual-order granularity, and the book columns are the aggregate view of those same events.
Book columns
Each row holds the state of the order book after that row's event has been applied. HKEX Securities and LSE both provide 10 price levels a side, so this is 60 columns: price, size and order count at each of 10 levels, bid and ask.
| Column | Type | Description |
|---|---|---|
| sequence | UInt32 | Monotonically increasing row index |
| timestamp | Datetime[ms] | Event time, timezone-naive exchange-local wall clock |
| bid_price_1 … bid_price_10 | Float64 | Bid prices from best (1) to 10th level |
| bid_size_1 … bid_size_10 | Int64 | Total volume resting at each bid level |
| bid_count_1 … bid_count_10 | Int64 | Number of individual orders at each bid level |
| ask_price_1 … ask_price_10 | Float64 | Ask prices from best (1) to 10th level |
| ask_size_1 … ask_size_10 | Int64 | Total volume resting at each ask level |
| ask_count_1 … ask_count_10 | Int64 | Number of individual orders at each ask level |
The pre-computed l2_by_second.parquet file gives you the same columns sampled once per second — useful for most analysis without processing tick-level data. See Download data for the schema.
Example: bid-ask spread
sim_id = client.simulation.list_cached()["simulations"][0]["example_sim_id"]
df = client.simulation.get_sim_data(sim_id)
spread = df["ask_price_1"] - df["bid_price_1"]
mid = (df["ask_price_1"] + df["bid_price_1"]) / 2
print(f"Mean spread: {spread.mean():.4f}")
print(f"Mean mid: {mid.mean():.4f}")Example: book depth at a snapshot
sim_id = client.simulation.list_cached()["simulations"][0]["example_sim_id"]
df = client.simulation.get_sim_data(sim_id)
# First snapshot
row = df.row(0, named=True)
print("Bid side:")
for i in range(1, 11):
print(f" L{i}: {row[f'bid_price_{i}']} x {row[f'bid_size_{i}']} ({row[f'bid_count_{i}']} orders)")
print("Ask side:")
for i in range(1, 11):
print(f" L{i}: {row[f'ask_price_{i}']} x {row[f'ask_size_{i}']} ({row[f'ask_count_{i}']} orders)")Event columns
Each row captures a single market event, alongside the book state it produced. Nothing needs joining — the alignment is positional.
size is in shares and size_in_lots is the same quantity in lots. Lots are the unit execution algorithms and the calibrated background agents work in, so size_in_lots is the column to compare an algo's order_size against.
| Column | Type | Description |
|---|---|---|
| sequence | UInt32 | Row index, shared with the book columns |
| timestamp | Datetime[ms] | Event time, timezone-naive exchange-local wall clock |
| timestamp_tick | Datetime[ms] | The event's own timestamp before book-state alignment |
| message_type | Int32 | Event classification — see message types below |
| side | Int32 | 1 = BID, −1 = ASK (passive/resting side — see side convention) |
| price | Float64 | Price level of the order |
| size | Int64 | Always positive, in shares — meaning depends on message type |
| size_in_lots | Float64 | The same quantity in lots, the unit exec_algos and the calibrated agents use |
| order_id | String | Exchange identifier linking related events for the same order. "NA" on generated foundation-model rows |
| _to_drop | Boolean | True where price or size could not be determined — filter these out |
Message types
| Value | Name | Description |
|---|---|---|
| 1 | PLACE | New limit order submitted to the book |
| 2 | TRADE | A resting order consumed by an incoming aggressive order |
| 3 | AMEND | Existing resting order modified |
| 4 | CANCEL | Resting order removed from the book |
Size semantics by message type
size is always positive but its meaning differs by event:
| Message type | size means… |
|---|---|
| PLACE | Original quantity submitted to the book |
| TRADE | Quantity removed from the resting order by this execution — one row per resting order consumed, never the aggressor's own quantity |
| AMEND | Remaining quantity of the order after the amendment — the order's new absolute size, not the change |
| CANCEL | Quantity removed from the book — the order's remaining quantity at the moment of cancellation, never 0 |
One aggressive order, several TRADE rows
A TRADE row is the passive side of an execution. An incoming order that crosses the spread produces one TRADE per resting order it consumes, and never a row of its own.
400 + 400 + 200 = 1,000 consumed
side and price. The incoming order never appears as a TRADE of its own, so no row here reads 1,000 even though 1,000 shares changed hands. Summing size over TRADE rows gives executed volume once; reading any single row as the aggressor's quantity does not.AMEND carries the new size, not the change
The most common misreading of the format. An amendment records the quantity the order is left with.
size is the quantity the order is left with, not the quantity taken off it. The engine replays an amendment by removing the resting order and re-adding it at the message's size, so reading it as a delta re-inserts the wrong quantity and corrupts the reconstructed book.Amendments that lose queue priority
This one is specific to the historical market data Pulse Format normalises — the data a simulation is calibrated and validated against — rather than to simulation output. It applies to the BMLL-sourced LSE and Cboe books.
An amendment is a single instruction, but under FIFO priority the venue can only honour some of them in place. Where the order loses its queue slot, Pulse Format records what actually happened to it rather than implying it held its position.
CANCEL withdrawing what was really resting followed by a PLACE at the new price and size — adjacent in sequence, CANCEL first, both keeping the original order_id. A single AMEND row would tell a queue model the order held its slot, which is the one thing it did not do.The split runs on XLON, BATE, CHIX and CEUX, where repricings are the majority of non-fill amendments. HKEX is unaffected — the venue itself requires a cancel and a new order, so it carries no price or upward amendments to split.
Because a split turns one raw message into two rows, sequence is renumbered to stay a contiguous index and row counts differ from the raw feed. Reconcile against a raw file on order_id + timestamp, not on row position. Fills are untouched: TRADE count and traded volume are identical either way.
Some exchanges do not include sizes in CANCEL and AMEND messages. Pulse back-fills these from the preceding PLACE or AMEND record. Rows where price or size cannot be determined are flagged with _to_drop = True and can be excluded via df.filter(~pl.col("_to_drop")).
Foundation-model output
Foundation-model runs use this same schema, with one difference: the file also contains the model's prompt context. The leading rows are real market data the model was conditioned on, not generated output, and the split is recorded in the parquet file-level metadata as pulse_fm.n_historical and pulse_fm.n_generated. Both keys are absent on agent-based output, where every row is generated.
See Foundation models for how to split the two sections and why it matters when comparing against historical data.
Side convention
side always reflects the passive (resting) order, not the aggressor. For a TRADE, this is the side of the resting order being consumed — not the side of the incoming order that triggered the fill.
| side value | Meaning |
|---|---|
| 1 | BID — the resting order was on the buy side |
| −1 | ASK — the resting order was on the sell side |
Example: filter by event type
import polars as pl
sim_id = client.simulation.list_cached()["simulations"][0]["example_sim_id"]
df = client.simulation.get_sim_data(sim_id)
MESSAGE_TYPES = {1: "PLACE", 2: "TRADE", 3: "AMEND", 4: "CANCEL"}
# All trades
trades = df.filter(pl.col("message_type") == 2)
# Cancels on the bid side
bid_cancels = df.filter(
(pl.col("message_type") == 4) & (pl.col("side") == 1)
)
print(f"Trades: {len(trades)}, Bid cancels: {len(bid_cancels)}")Example: reconstruct trade flow
import polars as pl
sim_id = client.simulation.list_cached()["simulations"][0]["example_sim_id"]
df = client.simulation.get_sim_data(sim_id)
trades = df.filter(pl.col("message_type") == 2)
# Volume on each side
buy_vol = trades.filter(pl.col("side") == -1)["size"].sum() # aggressor hit the ask
sell_vol = trades.filter(pl.col("side") == 1)["size"].sum() # aggressor hit the bid
print(f"Buy-initiated volume: {buy_vol:,}")
print(f"Sell-initiated volume: {sell_vol:,}")
print(f"Net order imbalance: {buy_vol - sell_vol:,}")