Converting data

Overview

Here is the big picture of the different components that allow for data conversion (the data download components are grayed out):

        flowchart TB
    AbcSourceDataDirManager
    source_data@{ shape: cyl, label: "Source data" }
    converted_data@{ shape: cyl, label: "Converted data" }

    download@{ shape: lean-r, label: "download.py" }
    provider_infra@{ shape: cloud, label: "Provider infrastructure" }

    download -.->|call .start| AbcDownloader
    HttpResource -.->|fetch| provider_infra
    AbcDownloader -.->|create| HttpResource
    AbcDownloader -.-> AbcSourceDataDirManager
    HttpResource -.->|write| source_data

    convert@{ shape: lean-r, label: "convert.py" }

    convert -->|call .start| AbcConverter
    AbcConverter -->|create| MyDatasetConverter
    MyDatasetConverter --> AbcSourceDataDirManager
    AbcSourceDataDirManager -->|read| source_data
    MyDatasetConverter -->|write| converted_data

    classDef muted fill:#D3D3D31F,stroke:lightgray,stroke-dasharray: 5 5;
    class download,AbcDownloader,HttpResource,provider_infra muted;

Conversion script

This script can be called manually from the command line, and will be called by the DBnomics infrastructure in production.

Note the separation of concerns:

  • the ConvertCLI class handles the CLI arguments and options, then creates and runs the converter,

  • the AbcConverter class takes care of the conversion logic.

AbcConverter

The AbcConverter class, inherited from BaseConverter, is responsible for converting source data downloaded from the infrastructure of this specific provider into the DBnomics data model.

Dataset converters

A dataset converter is responsible for transforming raw source data into the DBnomics data model. Each dataset from the provider must have its own converter class that inherits from BaseDatasetConverter.

In the ABC fetcher example, we have one dataset (GAS_PRICE), so we create one dataset converter class:

Lifecycle methods

The BaseDatasetConverter class defines an abstract lifecycle that your converter must implement:

  1. start(*, dataset_writer): Called once at the beginning of the conversion process for this dataset. Use this method to initialize any resources needed for the conversion, such as loading and parsing source data files.

  2. _create_dataset_metadata(): Called to define the structure of the dataset. Returns a DatasetMetadata object describing dimensions, attributes, and metadata about the dataset.

  3. _iter_dataset_series(*, dataset_metadata): Called to iterate over all series in the dataset. Yields Series objects containing observations.

Modeling dimensions

Dimensions are the categorical axes of your data. In the ABC fetcher, the gas price data has one dimension: KIND (the type of gas).

Dimensions are defined upfront in the _create_dataset_metadata() method:

kind_dimension = Dimension.create(
    "KIND",
    label="Gas kind",
    values=[
        DimensionValue.create(
            slugify(label),
            label=label,
        )
        for label in self._df["kind"].unique()
    ],
)

Each dimension has:

  • A code (e.g., "KIND") used as an identifier

  • A label (e.g., "Gas kind") for display purposes

  • A list of dimension values representing each possible value of that dimension

Dimension values are also slugified to create unique, URL-safe codes from their labels.

Creating series

Series represent individual time series within the dataset. For each series, you must specify:

  • Dimension values: The specific combination of dimension values for this series (e.g., {"KIND": "natural-gas"})

  • Observations: A list of period-value pairs, with optional attributes for each observation

In the ABC fetcher, series are created by grouping the CSV data by the kind column:

def _iter_dataset_series(self, *, dataset_metadata: DatasetMetadata) -> Iterator[Series]:
    df = self._df
    for kind, group_df in df.groupby("kind"):
        observations = [
            Observation.create(
                period=row["year"],
                value=row["price"],
            )
            for _, row in group_df.sort_values("year").iterrows()
        ]
        yield Series.create(
            add_missing_dataset_dimensions=True,
            dataset_dimensions=dataset_metadata.dimensions,
            dimensions={"FREQ": "A", "KIND": kind},
            observations=observations,
        )

The add_missing_dataset_dimensions=True parameter ensures that any dimensions assigned to the series that was not defined upfront in the dataset metadata are added to it. This allows converting datasets with dynamic dimensions, when the set of dimension values is not known in advance.

Loading and parsing source data

Use helper methods like _load_csv_file() to load and parse source data files downloaded by the downloader. This keeps your conversion logic clean and testable:

def _load_csv_file(self) -> pd.DataFrame:
    return pd.read_csv(
        self._source_data_dir_manager.gas_price_file,
        na_values={"price": ["N/A"]},
    )

In this example, we use pandas to load a CSV file and handle missing values.

Note

The path of the CSV file is provided by the AbcSourceDataDirManager that was introduced in the Source data directory manager section.

Its purpose is to abstract away the details of where the source data files are located, so that the downloader and the converter can focus on their respective tasks.

Run the convert script

Note

TOOLBOX_NO_LOG_COLOR=1 disables colors in the logs, TOOLBOX_NO_LOG_TIMESTAMP=1 hides timestamps in the logs, and TOOLBOX_PLAIN_TRACEBACK=1 shows plain exception tracebacks, for the sake of readability in the documentation.

The convert script processes source data and generates the DBnomics data model output:

TOOLBOX_NO_LOG_COLOR=1 TOOLBOX_NO_LOG_TIMESTAMP=1 TOOLBOX_PLAIN_TRACEBACK=1 uv run convert.py source-data converted-data

This command:

  • Reads from the source-data directory (populated by the download script)

  • Writes to the converted-data directory

  • Creates datasets structured according to the DBnomics data model

The converted data directory structure looks like:

converted-data/
├── provider.json          # Provider metadata
└── GAS_PRICE/             # One directory per dataset
    ├── dataset.json       # Dataset metadata (dimensions, attributes)
    └── series.jsonl       # Series data in JSON Lines format

Each line of series.jsonl contains a complete series object with all its dimensions and observations.

Storage adapters

The second argument of the convert script (converted-data) is resolved by a storage adapter — an abstraction layer that determines how and where the converted data is written.

URI resolution

When you pass a plain directory path like converted-data, it is automatically resolved to a filesystem URI:

filesystem:converted-data?default_series_format=JSON_LINES&single_provider=true

Note

The single_provider=true query parameter indicates that the directory contains data from a single provider, as opposed to a multi-provider storage layout where a subdirectory per provider would be expected. Since the convert script targets one provider at a time, single_provider=true is automatically set when a plain directory path is provided.

The two forms are therefore equivalent:

# Short form (auto-resolved)
uv run convert.py source-data converted-data

# Explicit URI form
uv run convert.py source-data "filesystem:converted-data?single_provider=true"

The filesystem scheme is currently the only storage adapter available. It writes data to the local filesystem.

The argument parser is setup by the ConvertCLI class.

Output format

By default, series data is written in JSON Lines format (series.jsonl). You can switch to TSV (tab-separated values) by passing the default_series_format query parameter in the URI:

uv run convert.py source-data "filesystem:converted-data?single_provider=true&default_series_format=TSV"

This produces a TSV file per time series instead of a single series.jsonl file.