Data model¶
The data-model page of the main documentation of DBnomics gives a high-level overview of the DBnomics data model, and the main design principles behind it.
This page explains how this data model is implemented in Python and how to use it in fetchers.
Overview¶
The data model is implemented with classes defined in dbnomics_toolbox.model.
Those classes represent data in memory and have no methods to read or write data. Read the Data storage page for that.
Here are the main entities of the data model:
classDiagram
class ProviderMetadata {
+code: ProviderCode
+name: str
+region: str
+terms_of_use: Url
+website: Url
}
class CategoryTree {
+children: Sequence[CategoryTreeNode]
}
class CategoryTreeNode {
<<Abstract>>
}
class Category {
+code: CategoryCode
+name: str
+children: Sequence[CategoryTreeNode]
}
class DatasetReference {
+code: DatasetCode
+name: str
}
class DatasetMetadata {
+code: DatasetCode
+name: str
+dimensions: DatasetDimensions
+attributes: DatasetAttributes
}
class DatasetDimensions {
+roles: Mapping[DimensionRole, DimensionCode]
}
class DatasetAttributes
class Dimension {
+code: DimensionCode
+label: str
+values: Sequence[DimensionValue]
}
class DimensionValue {
+code: DimensionValueCode
+label: str
}
class Attribute {
+code: AttributeCode
+label: str
+values: Sequence[AttributeValue]
}
class AttributeValue {
+code: AttributeValueCode
+label: str
}
class Series {
+metadata: SeriesMetadata
+observations: Sequence[Observation]
}
class SeriesMetadata {
+code: SeriesCode
+dimensions: SeriesDimensions
+attributes: Mapping[AttributeCode, str]
}
class SeriesDimensions {
<<Mapping[DimensionCode, DimensionValueCode]>>
}
class Observation {
+period: Period
+value: ObservationValue
+attributes: Mapping[AttributeCode, str]
}
CategoryTree "1" --> "*" CategoryTreeNode : children
CategoryTreeNode <|-- Category
CategoryTreeNode <|-- DatasetReference
Category "1" --> "*" CategoryTreeNode : children
DatasetMetadata "1" --> "1" DatasetDimensions : dimensions
DatasetMetadata "1" --> "1" DatasetAttributes : attributes
DatasetDimensions "1" --> "*" Dimension : dimensions
Dimension "1" --> "*" DimensionValue : values
DatasetAttributes "1" --> "*" Attribute : attributes
Attribute "1" --> "*" AttributeValue : values
Series "1" --> "1" SeriesMetadata : metadata
Series "1" --> "*" Observation : observations
SeriesMetadata "1" --> "1" SeriesDimensions : dimensions
Provider metadata¶
The ProviderMetadata class represents metadata about a provider.
Example:
from dbnomics_toolbox.model import ProviderMetadata
ProviderMetadata.create(
code="INSEE",
name="Institut national de la statistique et des études économiques",
# Cf https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2
region="FR",
terms_of_use="https://www.insee.fr/fr/information/2381863",
website="https://www.insee.fr/",
)
On DBnomics website that metadata is displayed on each provider page, in this case https://db.nomics.world/INSEE.
See also: ProviderReader.read_provider_metadata and ProviderWriter.write_provider_metadata.
Category tree¶
The CategoryTree class represents the category tree of datasets.
Each node can be either a reference to a dataset by using the DatasetReference class, or a category by using the Category class.
Example:
from dbnomics_toolbox.model import Category, CategoryTree, DatasetReference
CategoryTree(
children=[
Category.create(
children=[
DatasetReference.create(
"POP",
name="Population",
)
],
code="KEI",
name="Key economic indicators",
)
]
)
See also: ProviderReader.read_category_tree and ProviderWriter.write_category_tree.
Dataset metadata¶
The DatasetMetadata class represents metadata about a dataset.
from dbnomics_toolbox.model import DatasetMetadata, Dimension, DimensionValue
dataset_metadata = DatasetMetadata.create(
"PRODUCT_PRICES",
dimensions=[
Dimension.create(
"SKU",
label="Stock keeping unit",
values=[
DimensionValue.create("111", label="Computer"),
DimensionValue.create("222", label="Smartphone"),
DimensionValue.create("333", label="Television"),
],
),
Dimension.create(
"COUNTRY",
label="Country",
values=[
DimensionValue.create("DE", label="Germany"),
DimensionValue.create("FR", label="France"),
],
),
],
name="Product prices",
)
Representing the actual time series is detailed in the next section.
See also: DatasetReader.read_dataset_metadata and DatasetWriter.write_dataset_metadata.
Series¶
The Series and Observation classes represent series metadata and observations.
See also: DatasetReader.read_series, DatasetReader.iter_series and DatasetWriter.write_series.
Auto-generated series code¶
from dbnomics_toolbox.model import Observation, Series
series = Series.create(
dataset_dimensions=dataset_metadata.dimensions,
dimensions={
"COUNTRY": "FR",
"SKU": "111",
},
observations=[
Observation.create(period="2000", value=12),
Observation.create(period="2001", value=13),
Observation.create(period="2002", value=11),
],
)
print(series)
# Series(
# metadata=SeriesMetadata(
# code="111.FR",
# dimensions={"COUNTRY": "FR", "SKU": "111"},
# ),
# observations=[...],
# )
The series code 111.FR was auto-generated from the dimension values.
Dataset dimensions were used to determine the order of dimensions.
As a consequence this only works when series have dimensions.
Arbitrary series code¶
An arbitrary series code can also be given:
from dbnomics_toolbox.model import Series
series = Series.create(
code="foo",
dimensions={
"COUNTRY": "FR",
"SKU": "111",
},
)
print(series)
# Series(
# metadata=SeriesMetadata(
# code="foo",
# dimensions={"COUNTRY": "FR", "SKU": "111"},
# ),
# observations=[],
# )
Periods¶
There are different period types, each one being represented by a model class.
Those classes can parse str periods and their instances can be serialized to str.
NA values¶
To represent a NA (non-available) value, create an Observation with a value of None:
from dbnomics_toolbox.model import Observation, Series
series = Series.create(
code="foo",
observations=[
Observation.create(period="2000", value=12),
Observation.create(period="2001", value=None),
Observation.create(period="2002", value=11),
],
)
Creating model instances¶
There are several ways to create a model class instance, each one offering a certain degree of convenience and/or performance.
Each model class defines a standard __init__ method which expects already parsed valid values.
In the following example using ProviderMetadata, the code kwarg must be given a parsed ProviderCode, and the terms_of_use and website kwargs must be given a parsed Url:
from dbnomics_toolbox.model.identifiers import parse_provider_code
from dbnomics_toolbox.model.provider_metadata import ProviderMetadata
from dbnomics_toolbox.model.url import parse_url
ProviderMetadata(
code=parse_provider_code("INSEE"),
name="Institut national de la statistique et des études économiques",
# Cf https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2
region="FR",
terms_of_use=parse_url("https://www.insee.fr/fr/information/2381863"),
website=parse_url("https://www.insee.fr/"),
)
Some model classes define a create classmethod acting as an alternative constructor that offers more relaxed arguments.
In the following example using ProviderMetadata.create, the code, terms_of_use and website kwargs can be given str values that will be parsed by the create classmethod before calling the corresponding __init__ method:
from dbnomics_toolbox.model import ProviderMetadata
ProviderMetadata.create(
code="INSEE",
name="Institut national de la statistique et des études économiques",
# Cf https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2
region="FR",
terms_of_use="https://www.insee.fr/fr/information/2381863",
website="https://www.insee.fr/",
)
The create classmethods make life easier for callers, but they may re-parse common values (dimension codes, periods) many times in large conversions.