# Converting data

## Overview

Here is the big picture of the different components that allow for data conversion (the data download components are grayed out):

```{mermaid}
flowchart TB
    AbcSourceDataDirManager
    source_data@{ shape: cyl, label: "Source data" }
    converted_data@{ shape: cyl, label: "Converted data" }

    download@{ shape: lean-r, label: "download.py" }
    provider_infra@{ shape: cloud, label: "Provider infrastructure" }

    download -.->|call .start| AbcDownloader
    HttpResource -.->|fetch| provider_infra
    AbcDownloader -.->|create| HttpResource
    AbcDownloader -.-> AbcSourceDataDirManager
    HttpResource -.->|write| source_data

    convert@{ shape: lean-r, label: "convert.py" }

    convert -->|call .start| AbcConverter
    AbcConverter -->|create| MyDatasetConverter
    MyDatasetConverter --> AbcSourceDataDirManager
    AbcSourceDataDirManager -->|read| source_data
    MyDatasetConverter -->|write| converted_data

    classDef muted fill:#D3D3D31F,stroke:lightgray,stroke-dasharray: 5 5;
    class download,AbcDownloader,HttpResource,provider_infra muted;
```

## Conversion script

```{literalinclude} ../../_external/abc-fetcher/convert.py
:caption: convert.py
```

This script can be called manually from the command line, and will be called by the DBnomics infrastructure in production.

Note the separation of concerns:

- the [](#ConvertCLI) class handles the CLI arguments and options, then creates and runs the converter,
- the `AbcConverter` class takes care of the conversion logic.

## AbcConverter

The `AbcConverter` class, inherited from [](#BaseConverter), is responsible for converting source data downloaded from the infrastructure of this specific provider into the DBnomics data model.

```{literalinclude} ../../_external/abc-fetcher/src/abc_fetcher/converter.py
:caption: src/abc_fetcher/converter.py
```

## Dataset converters

A dataset converter is responsible for transforming raw source data into the DBnomics data model.
Each dataset from the provider must have its own converter class that inherits from [](#BaseDatasetConverter).

In the ABC fetcher example, we have one dataset (`GAS_PRICE`), so we create one dataset converter class:

```{literalinclude} ../../_external/abc-fetcher/src/abc_fetcher/dataset_converters.py
:caption: src/abc_fetcher/dataset_converters.py
```

### Lifecycle methods

The [](#BaseDatasetConverter) class defines an abstract lifecycle that your converter must implement:

1. `start(*, dataset_writer)`: Called once at the beginning of the conversion process for this dataset.
   Use this method to initialize any resources needed for the conversion, such as loading and parsing source data files.

2. `_create_dataset_metadata()`: Called to define the structure of the dataset.
   Returns a [](#DatasetMetadata) object describing dimensions, attributes, and metadata about the dataset.

3. `_iter_dataset_series(*, dataset_metadata)`: Called to iterate over all series in the dataset.
   Yields [Series](#dbnomics_toolbox.model.series.Series) objects containing observations.

### Modeling dimensions

Dimensions are the categorical axes of your data.
In the ABC fetcher, the gas price data has one dimension: `KIND` (the type of gas).

Dimensions are defined upfront in the `_create_dataset_metadata()` method:

```python
kind_dimension = Dimension.create(
    "KIND",
    label="Gas kind",
    values=[
        DimensionValue.create(
            slugify(label),
            label=label,
        )
        for label in self._df["kind"].unique()
    ],
)
```

Each dimension has:

- A **code** (e.g., `"KIND"`) used as an identifier
- A **label** (e.g., `"Gas kind"`) for display purposes
- A list of **dimension values** representing each possible value of that dimension

Dimension values are also slugified to create unique, URL-safe codes from their labels.

### Creating series

Series represent individual time series within the dataset.
For each series, you must specify:

- **Dimension values**: The specific combination of dimension values for this series (e.g., `{"KIND": "natural-gas"}`)
- **Observations**: A list of period-value pairs, with optional attributes for each observation

In the ABC fetcher, series are created by grouping the CSV data by the `kind` column:

```python
def _iter_dataset_series(self, *, dataset_metadata: DatasetMetadata) -> Iterator[Series]:
    df = self._df
    for kind, group_df in df.groupby("kind"):
        observations = [
            Observation.create(
                period=row["year"],
                value=row["price"],
            )
            for _, row in group_df.sort_values("year").iterrows()
        ]
        yield Series.create(
            add_missing_dataset_dimensions=True,
            dataset_dimensions=dataset_metadata.dimensions,
            dimensions={"FREQ": "A", "KIND": kind},
            observations=observations,
        )
```

The `add_missing_dataset_dimensions=True` parameter ensures that any dimensions assigned to the series that was not defined upfront in the dataset metadata are added to it.
This allows converting datasets with dynamic dimensions, when the set of dimension values is not known in advance.

### Loading and parsing source data

Use helper methods like `_load_csv_file()` to load and parse source data files downloaded by the downloader.
This keeps your conversion logic clean and testable:

```python
def _load_csv_file(self) -> pd.DataFrame:
    return pd.read_csv(
        self._source_data_dir_manager.gas_price_file,
        na_values={"price": ["N/A"]},
    )
```

In this example, we use [pandas](https://pandas.pydata.org/) to load a CSV file and handle missing values.

```{note}
The path of the CSV file is provided by the `AbcSourceDataDirManager` that was introduced in the [](./downloading-data.md#source-data-directory-manager) section.

Its purpose is to abstract away the details of where the source data files are located, so that the downloader and the converter can focus on their respective tasks.
```

## Run the convert script

```{note}
`TOOLBOX_NO_LOG_COLOR=1` disables colors in the logs, `TOOLBOX_NO_LOG_TIMESTAMP=1` hides timestamps in the logs, and `TOOLBOX_PLAIN_TRACEBACK=1` shows plain exception tracebacks, for the sake of readability in the documentation.
```

The convert script processes source data and generates the DBnomics data model output:

```bash
TOOLBOX_NO_LOG_COLOR=1 TOOLBOX_NO_LOG_TIMESTAMP=1 TOOLBOX_PLAIN_TRACEBACK=1 uv run convert.py source-data converted-data
```

This command:

- Reads from the `source-data` directory (populated by the download script)
- Writes to the `converted-data` directory
- Creates datasets structured according to the DBnomics data model

The converted data directory structure looks like:

```text
converted-data/
├── provider.json          # Provider metadata
└── GAS_PRICE/             # One directory per dataset
    ├── dataset.json       # Dataset metadata (dimensions, attributes)
    └── series.jsonl       # Series data in JSON Lines format
```

Each line of `series.jsonl` contains a complete series object with all its dimensions and observations.

## Storage adapters

The second argument of the convert script (`converted-data`) is resolved by a **storage adapter** — an abstraction layer that determines *how* and *where* the converted data is written.

### URI resolution

When you pass a plain directory path like `converted-data`, it is automatically resolved to a filesystem URI:

```text
filesystem:converted-data?default_series_format=JSON_LINES&single_provider=true
```

```{note}
The `single_provider=true` query parameter indicates that the directory contains data from a single provider, as opposed to a multi-provider storage layout where a subdirectory per provider would be expected. Since the convert script targets one provider at a time, `single_provider=true` is automatically set when a plain directory path is provided.
```

The two forms are therefore equivalent:

```bash
# Short form (auto-resolved)
uv run convert.py source-data converted-data

# Explicit URI form
uv run convert.py source-data "filesystem:converted-data?single_provider=true"
```

The `filesystem` scheme is currently the only storage adapter available. It writes data to the local filesystem.

The argument parser is setup by the [](#ConvertCLI) class.

### Output format

By default, series data is written in **JSON Lines** format (`series.jsonl`). You can switch to **TSV** (tab-separated values) by passing the `default_series_format` query parameter in the URI:

```bash
uv run convert.py source-data "filesystem:converted-data?single_provider=true&default_series_format=TSV"
```

This produces a TSV file per time series instead of a single `series.jsonl` file.