# Data model

The [data-model page](https://docs.db.nomics.world/data-model/) of the main documentation of DBnomics gives a high-level overview of the DBnomics data model, and the main design principles behind it.

This page explains how this data model is implemented in Python and how to use it in fetchers.

## Overview

The data model is implemented with classes defined in [](#dbnomics_toolbox.model).

Those classes represent data in memory and have no methods to read or write data.
Read the [](data-storage/index.md) page for that.

Here are the main entities of the data model:

```{mermaid}
classDiagram
    class ProviderMetadata {
        +code: ProviderCode
        +name: str
        +region: str
        +terms_of_use: Url
        +website: Url
    }

    class CategoryTree {
        +children: Sequence[CategoryTreeNode]
    }

    class CategoryTreeNode {
        <<Abstract>>
    }

    class Category {
        +code: CategoryCode
        +name: str
        +children: Sequence[CategoryTreeNode]
    }

    class DatasetReference {
        +code: DatasetCode
        +name: str
    }

    class DatasetMetadata {
        +code: DatasetCode
        +name: str
        +dimensions: DatasetDimensions
        +attributes: DatasetAttributes
    }

    class DatasetDimensions {
        +roles: Mapping[DimensionRole, DimensionCode]
    }

    class DatasetAttributes

    class Dimension {
        +code: DimensionCode
        +label: str
        +values: Sequence[DimensionValue]
    }

    class DimensionValue {
        +code: DimensionValueCode
        +label: str
    }

    class Attribute {
        +code: AttributeCode
        +label: str
        +values: Sequence[AttributeValue]
    }

    class AttributeValue {
        +code: AttributeValueCode
        +label: str
    }

    class Series {
        +metadata: SeriesMetadata
        +observations: Sequence[Observation]
    }

    class SeriesMetadata {
        +code: SeriesCode
        +dimensions: SeriesDimensions
        +attributes: Mapping[AttributeCode, str]
    }

    class SeriesDimensions {
        <<Mapping[DimensionCode, DimensionValueCode]>>
    }

    class Observation {
        +period: Period
        +value: ObservationValue
        +attributes: Mapping[AttributeCode, str]
    }

    CategoryTree "1" --> "*" CategoryTreeNode : children
    CategoryTreeNode <|-- Category
    CategoryTreeNode <|-- DatasetReference
    Category "1" --> "*" CategoryTreeNode : children

    DatasetMetadata "1" --> "1" DatasetDimensions : dimensions
    DatasetMetadata "1" --> "1" DatasetAttributes : attributes
    DatasetDimensions "1" --> "*" Dimension : dimensions
    Dimension "1" --> "*" DimensionValue : values
    DatasetAttributes "1" --> "*" Attribute : attributes
    Attribute "1" --> "*" AttributeValue : values

    Series "1" --> "1" SeriesMetadata : metadata
    Series "1" --> "*" Observation : observations
    SeriesMetadata "1" --> "1" SeriesDimensions : dimensions
```

## Provider metadata

The [](#ProviderMetadata) class represents metadata about a provider.

Example:

```python
from dbnomics_toolbox.model import ProviderMetadata

ProviderMetadata.create(
    code="INSEE",
    name="Institut national de la statistique et des études économiques",
    # Cf https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2
    region="FR",
    terms_of_use="https://www.insee.fr/fr/information/2381863",
    website="https://www.insee.fr/",
)
```

On DBnomics website that metadata is displayed on each provider page, in this case <https://db.nomics.world/INSEE>.

See also: [](#ProviderReader.read_provider_metadata) and [](#ProviderWriter.write_provider_metadata).

## Category tree

The [](#CategoryTree) class represents the category tree of datasets.

Each node can be either a reference to a dataset by using the [](#DatasetReference) class, or a category by using the [Category](#dbnomics_toolbox.model.category_tree.category.Category) class.

Example:

```python
from dbnomics_toolbox.model import Category, CategoryTree, DatasetReference

CategoryTree(
    children=[
        Category.create(
            children=[
                DatasetReference.create(
                    "POP",
                    name="Population",
                )
            ],
            code="KEI",
            name="Key economic indicators",
        )
    ]
)
```

See also: [](#ProviderReader.read_category_tree) and [](#ProviderWriter.write_category_tree).

## Dataset metadata

The [](#DatasetMetadata) class represents metadata about a dataset.

```python
from dbnomics_toolbox.model import DatasetMetadata, Dimension, DimensionValue

dataset_metadata = DatasetMetadata.create(
    "PRODUCT_PRICES",
    dimensions=[
        Dimension.create(
            "SKU",
            label="Stock keeping unit",
            values=[
                DimensionValue.create("111", label="Computer"),
                DimensionValue.create("222", label="Smartphone"),
                DimensionValue.create("333", label="Television"),
            ],
        ),
        Dimension.create(
            "COUNTRY",
            label="Country",
            values=[
                DimensionValue.create("DE", label="Germany"),
                DimensionValue.create("FR", label="France"),
            ],
        ),
    ],
    name="Product prices",
)
```

Representing the actual time series is detailed in the next section.

See also: [](#DatasetReader.read_dataset_metadata) and [](#DatasetWriter.write_dataset_metadata).

## Series

The [Series](#dbnomics_toolbox.model.series.Series) and [Observation](#dbnomics_toolbox.model.observations.observation.Observation) classes represent series metadata and observations.

See also: [](#DatasetReader.read_series), [](#DatasetReader.iter_series) and [](#DatasetWriter.write_series).

### Auto-generated series code

```python
from dbnomics_toolbox.model import Observation, Series

series = Series.create(
    dataset_dimensions=dataset_metadata.dimensions,
    dimensions={
        "COUNTRY": "FR",
        "SKU": "111",
    },
    observations=[
        Observation.create(period="2000", value=12),
        Observation.create(period="2001", value=13),
        Observation.create(period="2002", value=11),
    ],
)
print(series)
# Series(
#     metadata=SeriesMetadata(
#         code="111.FR",
#         dimensions={"COUNTRY": "FR", "SKU": "111"},
#     ),
#     observations=[...],
# )
```

The series code `111.FR` was auto-generated from the dimension values.
Dataset dimensions were used to determine the order of dimensions.
As a consequence this only works when series have dimensions.

### Arbitrary series code

An arbitrary series code can also be given:

```python
from dbnomics_toolbox.model import Series

series = Series.create(
    code="foo",
    dimensions={
        "COUNTRY": "FR",
        "SKU": "111",
    },
)
print(series)
# Series(
#     metadata=SeriesMetadata(
#         code="foo",
#         dimensions={"COUNTRY": "FR", "SKU": "111"},
#     ),
#     observations=[],
# )
```

### Periods

There are different period types, each one being represented by a model class.

Those classes can parse `str` periods and their instances can be serialized to  `str`.

| Period type                                                               | Text representation | Example      | Model class                        |
| ------------------------------------------------------------------------- | ------------------- | ------------ | ---------------------------------- |
| year                                                                      | `{YYYY}`            | `2025`       | [](#YearPeriod)                    |
| [semester](https://en.wikipedia.org/wiki/Academic_term) (6 months)        | `{YYYY}-S{S:1-2}`   | `2025-S1`    | [](#SemesterPeriod)                |
| [quarter](https://en.wikipedia.org/wiki/Academic_quarter_(year_division)) | `{YYYY}-Q{Q:1-4}`   | `2025-Q1`    | [](#QuarterPeriod)                 |
| [bimester](https://en.wikipedia.org/wiki/Academic_term) (2 months)        | `{YYYY}-B{N:1-6}`   | `2025-B1`    | [](#BimesterPeriod)                |
| month                                                                     | `{YYYY}-{MM}`       | `2025-01`    | [](#MonthPeriod)                   |
| week                                                                      | `{YYYY}-W{WW}`      | `2025-W01`   | [](#WeekPeriod)                    |
| day                                                                       | `{YYYY}-{MM}-{DD}`  | `2025-01-01` | [](#DayPeriod)                     |

### NA values

To represent a NA (non-available) value, create an [Observation](#dbnomics_toolbox.model.observations.observation.Observation) with a value of `None`:

```python
from dbnomics_toolbox.model import Observation, Series

series = Series.create(
    code="foo",
    observations=[
        Observation.create(period="2000", value=12),
        Observation.create(period="2001", value=None),
        Observation.create(period="2002", value=11),
    ],
)
```

## Creating model instances

There are several ways to create a model class instance, each one offering a certain degree of convenience and/or performance.

Each model class defines a standard `__init__` method which expects already parsed valid values.

In the following example using [](#ProviderMetadata), the `code` kwarg must be given a parsed [ProviderCode](#dbnomics_toolbox.model.identifiers.provider_code.ProviderCode), and the `terms_of_use` and `website` kwargs must be given a parsed [](#Url):

```python
from dbnomics_toolbox.model.identifiers import parse_provider_code
from dbnomics_toolbox.model.provider_metadata import ProviderMetadata
from dbnomics_toolbox.model.url import parse_url

ProviderMetadata(
    code=parse_provider_code("INSEE"),
    name="Institut national de la statistique et des études économiques",
    # Cf https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2
    region="FR",
    terms_of_use=parse_url("https://www.insee.fr/fr/information/2381863"),
    website=parse_url("https://www.insee.fr/"),
)
```

Some model classes define a `create` classmethod acting as an alternative constructor that offers more relaxed arguments.

In the following example using [](#ProviderMetadata.create), the `code`, `terms_of_use` and `website` kwargs can be given `str` values that will be parsed by the `create` classmethod before calling the corresponding `__init__` method:

```python
from dbnomics_toolbox.model import ProviderMetadata

ProviderMetadata.create(
    code="INSEE",
    name="Institut national de la statistique et des études économiques",
    # Cf https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2
    region="FR",
    terms_of_use="https://www.insee.fr/fr/information/2381863",
    website="https://www.insee.fr/",
)
```

The `create` classmethods make life easier for callers, but they may re-parse common values (dimension codes, periods) many times in large conversions.