Datasets & Scaling
Collect causal examples in chronological order, normalize mixed-scale market features, and extract training and validation batches without contaminating the holdout.
Why scaling matters
Market inputs can differ by orders of magnitude: price may be 60,000 while a return is 0.002. Without normalization, large-scale columns dominate gradients even when they are not more informative.
var nl.WindowDataset dataset = nl.windowDataset(4, 1, 500, "returns_dataset")
if barstate.isfirst
dataset := dataset
.setInputScaler(nl.ScalerKind.zScore)
.setTargetScaler(nl.ScalerKind.zScore)Available scalers are none, zScore, minMax, and runningZScore.
Dataset types
| Dataset | Example shape | Use when |
|---|---|---|
nl.windowDataset(features, targets, rows, name) | One flat feature row | A bar snapshot is enough: RSI, momentum, volatility, distance from an MA. |
nl.rollingDataset(steps, features, targets, rows, name) | A chronological sequence | The order of recent rows matters for Conv1D, LSTM, GRU, or attention models. |
Pushing rows
dataset := dataset.pushRow(features, target)Rows are checked on insert. Wrong widths and any row containing na are rejected automatically.
pushNextReturnRow()builds a return target from current and future values.pushNextDirectionRow()builds a directional class target.nextReturnValue()andnextDirectionValue()expose target math for custom pipelines.
Extracting batches
if dataset.ready(64)
nl.Batch train = dataset.trainBatch(16)
nl.Batch validation = dataset.validationBatch(16)A Batch carries inputTensor and targetTensor, already scaled and ready for model calls.
| Method | Result |
|---|---|
lastBatch(size) | The newest requested rows. |
toBatch() | All stored rows in chronological order. |
unrollBatch(targetOffset) | Rolling windows paired with offset targets. |
trainBatch(validationRows) | Training side of a chronological split. |
validationBatch(validationRows) | Newest untouched holdout rows. |
FeatureBuilder
nl.featureBuilder(name) makes feature assembly self-documenting. Chain .push(value, "feature_name"), then use .toTensor() for inference or .toArray() for dataset insertion.
Ordering remains explicit
Builder labels help humans, but model semantics remain positional. Use the same builder order for historical examples and live inputs.
Leakage-safe scaling
NeuraLib fits validation transformations from the training-side profile. The holdout does not update the scaler it is evaluated with, preventing normalization from peeking at future distribution statistics.
- Scale live inputs with
dataset.scaleInput(). - Scale manual targets with
dataset.scaleTarget()when required. - Return predictions to original target units with
dataset.inverseScaleTarget(). - Use
clear()when a deliberate regime reset requires a fresh dataset.