API Reference¶

This part of the documentation covers all the interfaces of the DBnomics toolbox.

`dbnomics_toolbox.fetcher_utils`¶

Constants¶

Errors¶

exception DBnomicsToolboxError(*, msg: str)¶

exception FetcherError(*, msg: str)¶

Source data dir managers¶

Import utilities¶

load_class_from_string(class_path: str) → type¶

Logging utilities¶

ZIP utilities¶

CLI utilities¶

Fetcher definition¶

File utilities¶

normalize_end_of_lines(input_file: Path, *, tmp_dir: Path | None = None) → None¶

reformat_file(input_file: Path, *, tmp_dir: Path | None = None) → None¶

write_chunks(chunks: Iterator[bytes], *, output_file: Path, tmp_dir: Path | None = None) → int¶

class CommandNotFoundMixin(file: Path, *, command_name: str, env_var_name: str, tmp_file: Path)¶

Mixin for errors caused by a missing external command.

Intended to be used with a FileProcessingError subclass via multiple inheritance. Adds command_name and env_var_name attributes, and builds the details message.

exception FileProcessingError(file: Path, *, msg: str, tmp_file: Path | None = None)¶

exception NormalizeEndOfLinesCommandNotFoundError(file: Path, *, command_name: str, env_var_name: str, tmp_file: Path)¶

exception NormalizeEndOfLinesError(file: Path, *, details: str, tmp_file: Path)¶

class JsonReformatStrategy(*values)¶

AUTO = 'auto'¶

JQ = 'jq'¶

PYTHON = 'python'¶

reformat_json_file(json_file: Path, *, codec: TypedCodec[Json] | None = None, strategy: JsonReformatStrategy = JsonReformatStrategy.AUTO, tmp_dir: Path | None = None) → None¶

reformat_json_file_with_jq(json_file: Path, *, indent_level: int = 2, jq_path: Path | None = None, tmp_dir: Path | None = None) → None¶

reformat_json_file_with_python(json_file: Path, *, codec: TypedCodec[Json] | None = None, tmp_dir: Path | None = None) → None¶

fast_iter(context: etree.iterparse[tuple[Literal['start', 'end', 'comment', 'pi'], _Element]]) → Iterator[tuple[str, _Element]]¶

Iterate the elements of context keeping memory usage low.

MIME type utilities¶

Partitions¶

class BasePartition(*, id: str)¶

abstract property file_discriminator: str¶

abstractmethod classmethod from_json(partition_id: str, *, data: JsonObject) → Self¶

id: str¶

property should_pre_split: bool¶

Return True if the partition should not be downloaded, but split and ignored.

Useful when the partition parameters clearly show that attempting downloading the partition would not work.

split() → Sequence[Self]¶

abstractmethod to_json() → JsonObject¶

class DimensionPartition(*, dimension_filter: DimensionFilter)¶

bisect() → tuple[Self, Self]¶

classmethod create(*, dimensions: list[Dimension]) → Self¶

property file_discriminator: str¶

split() → Sequence[Self]¶

to_json() → JsonObject¶

class PeriodRangePartition(min_value: TRangeValue, max_value: TRangeValue, *, max_length: int | None = None)¶

classmethod from_json(partition_id: str, *, data: JsonObject) → Self¶

class RangePartition(min_value: TRangeValue, max_value: TRangeValue, *, max_length: int | None = None)¶

bisect() → tuple[Self, Self]¶

property file_discriminator: str¶

property should_pre_split: bool¶

Return True if the partition should not be downloaded, but split and ignored.

Useful when the partition parameters clearly show that attempting downloading the partition would not work.

split() → Sequence[Self]¶

to_json() → JsonObject¶

class UrlPartition(url: yarl.URL | str)¶

property file_discriminator: str¶

classmethod from_json(partition_id: str, *, data: JsonObject) → Self¶

to_json() → JsonObject¶

url: URL¶

class PartitionStateRepo(*, codec: TypedCodec[Json] | None = None, file: Path, initial_splits: OrderedDict[str, list[TPartition]] | None = None)¶

find_sub_partitions(partition_id: str) → list[TPartition] | None¶

classmethod from_file(input_file: Path, *, codec: TypedCodec[Json] | None = None, initial_splits: OrderedDict[str, list[TPartition]] | None = None) → Self¶

register_partition_split(partition_id: str, *, sub_partitions: list[TPartition]) → None¶

Raises:: StateFileSaveError: If there is an error while saving the updated splits to the state file.

to_json() → PartitionsJson | None¶

exception PartitionError(*, msg: str)¶

exception StateFileError(*, msg: str, state_file: Path)¶

exception StateFileLoadError(state_file: Path)¶

exception StateFileSaveError(state_file: Path)¶

exception UnsplittablePartition(partition: BasePartition)¶

Processors¶

class BaseProcessor(*, downloader_state_repo: DownloaderStateRepo | None = None, encoder: Encoder | None = None, modes: ProcessorModes | None = None, report_file: Path | None = None, selection_options: SelectionOptions | None = None, source_data_dir: Path)¶

property downloader_state_repo: DownloaderStateRepo¶

abstractmethod start() → None¶

property stats: ProcessorStats¶

class ProcessorModes(*, dry_run: bool = False, fail_fast: bool = False, incremental: bool = True, resume: bool = True)¶

dry_run: bool¶

fail_fast: bool¶

incremental: bool¶

resume: bool¶

class SelectionOptions(*, excluded_dataset_ids: collections.abc.Sequence[str] = <factory>, selected_dataset_ids: collections.abc.Sequence[str] = <factory>, skip_datasets: bool = False)¶

excluded_dataset_ids: Sequence[str]¶

selected_dataset_ids: Sequence[str]¶

skip_datasets: bool¶

class BaseConverter(*, converted_data_writer: ProviderWriter, dataset_converters: Iterable[DatasetConverter] | None = None, downloader_state_repo: DownloaderStateRepo | None = None, encoder: Encoder | None = None, modes: ProcessorModes | None = None, previous_converted_data_reader: ProviderReader | None = None, report_file: Path | None = None, selection_options: ConverterSelectionOptions | None = None, source_data_dir: Path)¶

abstractmethod classmethod get_provider_metadata() → ProviderMetadata¶

property provider_code: ProviderCode¶

start() → None¶

class ConverterSelectionOptions(*, excluded_dataset_ids: collections.abc.Sequence[str] = <factory>, selected_dataset_ids: collections.abc.Sequence[str] = <factory>, skip_datasets: bool = False, skip_category_tree: bool = False)¶

skip_category_tree: bool¶

class BaseDatasetConverter(source_dataset_id: str, *, progress_reporter: DatasetConversionProgressReporter | None = None, provider_code: ProviderCode)¶

start(*, dataset_writer: DatasetWriter) → None¶

class CategoryTreeConverter(*args, **kwargs)¶

create_category_tree() → CategoryTree¶

class Converter(*args, **kwargs)¶

classmethod get_provider_metadata() → ProviderMetadata¶

property provider_code: ProviderCode¶

class DatasetConversionProgressReporter(*args, **kwargs)¶

report_converted_series(*, converted_series_count: int, provider_code: ProviderCode, source_dataset_id: str) → None¶

class DatasetConverter(*args, **kwargs)¶

dataset_code: DatasetCode¶

provider_code: ProviderCode¶

source_dataset_id: str¶

start(*, dataset_writer: DatasetWriter) → None¶

class Downloader(*args, **kwargs)¶

class Processor(*args, **kwargs)¶

property downloader_state_repo: DownloaderStateRepo¶

start() → None¶

property stats: ProcessorStats¶

class StartConversionSessionFunc(*args, **kwargs)¶

class DownloaderState(*, dataset_updates: dict[str, dbnomics_toolbox.types.tz_aware_datetime.TZAwareDatetime]=<factory>)¶

dataset_updates: dict[str, TZAwareDatetime]¶

class DownloaderStateRepo(*, codec: TypedCodec[Json] | None = None, initial_state: DownloaderState | None = None)¶

Store and retrieve downloader state.

Keep track of the last update time of datasets.

get_updated_at(dataset_id: str) → TZAwareDatetime | None¶

iter_dataset_updates() → Iterator[tuple[str, str]]¶

load(input_file: Path) → None¶

save(output_file: Path) → None¶

set_updated_at(dataset_id: str, updated_at: datetime) → None¶

class ProcessorReport(*, stats: dbnomics_toolbox.fetcher_utils.processors.processor_report.ProcessorStats)¶

stats: ProcessorStats¶

class ProcessorStats(*, fail_count: int = 0, skip_count: int = 0, success_count: int = 0)¶

fail_count: int = 0¶

format(*, item_singular: str) → str¶

skip_count: int = 0¶

success_count: int = 0¶

property total_count: int¶

class DatasetResumeModeSkipReason(dataset_code: 'DatasetCode', *, log_reason: bool = True)¶

dataset_code: DatasetCode¶

class ExcludedSkipReason(*, log_reason: bool = True, dataset_ids: frozenset[str], label: str)¶

dataset_ids: frozenset[str]¶

label: str¶

class NotSelectedSkipReason(*, log_reason: bool = False, dataset_ids: frozenset[str], label: str)¶

dataset_ids: frozenset[str]¶

label: str¶

log_reason: bool = False¶

class NotUpdatedAfterOptionSkipReason(*, log_reason: bool = True, label: str, updated_after: datetime.datetime, updated_at: datetime.datetime)¶

label: str¶

updated_after: datetime¶

updated_at: datetime¶

class NotUpdatedAfterPreviousUpdatedAtSkipReason(*, log_reason: bool = True, label: str, previous_updated_at: datetime.datetime, updated_at: datetime.datetime)¶

label: str¶

previous_updated_at: datetime¶

updated_at: datetime¶

class ResourceResumeModeSkipReason(resource: 'FileResource', *, log_reason: bool = True)¶

resource: FileResource¶

class SkipDatasetsSkipReason(*, log_reason: bool = True, dataset_ids: frozenset[str], label: str)¶

dataset_ids: frozenset[str]¶

label: str¶

class SkipReason(*, log_reason: bool = True)¶

log_reason: bool = True¶

log_multiple_datasets(dataset_converters: Sequence, *, item_kind_plural: str | None = None, item_kind: str | None = None, skipped_count: int | None = None) → Iterator[tuple[TDatasetConverter, tuple[int, int]]]¶

log_multiple_items(items: Sequence, *, action_infinitive: str, get_item_id: Callable[[T], str] | None = None, item_kind_plural: str | None = None, item_kind: str, skipped_count: int | None = None) → Iterator[tuple[T, tuple[int, int]]]¶

exception CategoryTreeCreateError(category_tree_converter: CategoryTreeConverter)¶

exception ControlFlowException¶

exception ConverterError(*, msg: str)¶

exception DatasetConvertError(dataset_converter: DatasetConverter)¶

exception DirectoryCreateError(directory: Path, *, kind: str)¶

exception DownloaderError(*, msg: str)¶

exception DownloaderStateFileLoadError(input_file: Path)¶

exception DownloaderStateFileSaveError(output_file: Path)¶

exception DownloaderStateRepoError(*, msg: str)¶

exception EnqueueSubPartitions(sub_partitions: list[BasePartition])¶

exception Skip(reason: SkipReason)¶

Resources¶

`dbnomics_toolbox.file_utils`¶

add_suffix(path: Path, suffix: str) → Path¶

Add a suffix to the path, keeping the existing ones.

The suffix must include the ‘.’ (e.g. suffix=”.gz”).

delete_empty_dirs(paths: Iterable[Path]) → None¶: Delete empty directories from a list of paths (files or directories).

gunzip_file(source_file: Path, target_file: Path, *, delete: bool = True) → None¶

is_empty_dir(path: Path, *, include_hidden: bool = True) → bool¶

Check if a directory is empty.

If include_hidden is True, hidden files are considered, otherwise they are ignored.

is_path_hidden(path: Path, *, base_dir: Path | None = None) → bool¶

iter_child_directories(base_dir: Path, *, include_hidden: bool = True) → Iterator[Path]¶

Iterate over direct child directories of base_dir.

Raises:: OSError: If iteration fails.

iter_child_files(base_dir: Path, *, include_hidden: bool = True) → Iterator[Path]¶

Iterate over direct child files of base_dir.

Raises:: OSError: If iteration fails.

iter_child_paths(base_dir: Path, *, include_hidden: bool = True) → Iterator[Path]¶

Iterate over direct child files or directories of base_dir.

Raises:: OSError: If iteration fails.

iter_files_recursive(base_dir: Path, *, include_hidden: bool = True) → Iterator[Path]¶

move_files_transactionally(file_mapping: Iterable[tuple[Path, Path]], *, delete_empty_dir: bool = True, missing_ok: bool = False) → None¶

Move files transactionally: all files will be moved or none.

If any file fails to move, rollback by moving the files at their original location.

If delete_empty_dir is True (the default), any source directory that is empty after moving the file will be deleted.

remove_suffix(path: Path, suffix: str) → Path¶

Remove the last suffix of the path, keeping the existing ones.

The suffix must include the ‘.’ (e.g. suffix=”.gz”).

temporary_file(file_name: str, *, base_dir: Path | str | None = None, delete: bool = True, dir_name_prefix: str | None = None) → Generator[Path]¶

Generate a temporary file path for the given file name.

Don’t open the file, which NamedTemporaryFile does, so don’t use it.

write_file_content(file: Path, *, content: AnyStr | bytearray, ensure_parent_dirs: bool = True) → None¶

write_gitignore_all(directory: Path, *, exist_ok: bool = False) → None¶: Write a .gitignore file ignoring the whole directory to avoid committing it accidentally.

`dbnomics_toolbox.http_utils`¶

HTTP constants¶

DEFAULT_USER_AGENT: Final = 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0'¶

str(object=’’) -> str str(bytes_or_buffer[, encoding[, errors]]) -> str

Create a new string object from the given object. If encoding or errors is specified, then the object must expose a data buffer that will be decoded using the given encoding and error handler. Otherwise, returns the result of object.__str__() (if defined) or repr(object). encoding defaults to ‘utf-8’. errors defaults to ‘strict’.

Requests helpers¶

class Timeouts(*, connect: float | datetime.timedelta = datetime.timedelta(seconds=60), read: float | datetime.timedelta = datetime.timedelta(seconds=60))¶

property as_requests_timeout: _Timeout¶

connect: float | timedelta = datetime.timedelta(seconds=60)¶

classmethod create(value: float | timedelta | Timeouts | None) → Timeouts¶

read: float | timedelta = datetime.timedelta(seconds=60)¶

`dbnomics_toolbox.io`¶

Parsing¶

Formats¶

Codecs¶

Protocols¶

class Codec(*args, **kwargs)¶: A codec that can encode (to bytes) and decode (from bytes) values.

class Decoder(*args, **kwargs)¶

decode(raw: bytes) → TData¶

Decode a value from bytes.

Raises:: DecodeError: If decoding fails.

class Dumper(*args, **kwargs)¶

dump(value: Any) → Any¶

Dump data.

Raises:: DumpError: If dumping fails.

class Encoder(*args, **kwargs)¶

encode(value: Any) → bytes¶

Encode a value to bytes.

Raises:: EncodeError: If encoding fails.

class Loader(*args, **kwargs)¶

load(data: Any, *, type_: type[TValue]) → TValue¶

Load data as the specified type.

Raises:: LoadError: If loading fails.

class TypedCodec(*args, **kwargs)¶

A codec that can encode (to bytes), decode (from bytes) and load (as type) values.

decode_and_load(raw: bytes, *, type_: type[TValue]) → TValue¶

Decode a value from bytes and load it as the specified type.

Raises:: DecodeError: If decoding fails. LoadError: If loading fails.

dump_and_encode(value: Any) → bytes¶

Dump data and encode it to bytes.

Raises:: DumpError: If dumping fails. EncodeError: If encoding fails.

`dbnomics_toolbox.model`¶

Identifiers¶

class ProviderCode¶

Identify a provider within DBnomics.

alias of str

is_provider_code(raw: str) → TypeGuard[ProviderCode]¶

parse_provider_code(raw: str) → ProviderCode¶

Parse a value as a provider code.

Raises:: IdentifierParseError: If the parsing failed.

class ProviderSnapshotId¶

Identify a snapshot of the data fetched from a provider by a data fetcher.

Currently implemented as a Git commit ID.

alias of str

is_provider_snapshot_id(raw: str) → TypeGuard[ProviderSnapshotId]¶

parse_provider_snapshot_id(raw: str) → ProviderSnapshotId¶

Parse a value as a provider snapshot ID.

Raises:: IdentifierParseError: If the parsing failed.

Entities¶

class Attribute(*, code: dbnomics_toolbox.model.identifiers.attribute_code.AttributeCode, label: str | None = None, values: collections.abc.Sequence[dbnomics_toolbox.model.attributes.attribute_value.AttributeValue] | None = None)¶

code: AttributeCode¶

create(code: str, *, label: str | None = None, values: Sequence[AttributeValue] | None = None) → Attribute¶

Create an attribute from raw values.

Raises:: AttributeCreateError: If the attribute instance cannot be created.

label: str | None¶

merge(other: Attribute) → Attribute¶

values: Sequence[AttributeValue] | None¶

class AttributeValue(*, code: dbnomics_toolbox.model.identifiers.attribute_value_code.AttributeValueCode, label: str | None = None)¶

code: AttributeValueCode¶

create(code: str, *, label: str | None = None) → AttributeValue¶

Create an attribute value from raw values.

Raises:: AttributeValueCreateError: If the attribute value instance cannot be created.

label: str | None¶

class DatasetAttributes(attributes: collections.abc.Sequence['Attribute'] = <factory>)¶

property attribute_codes: Sequence[AttributeCode]¶

attributes: Sequence[Attribute]¶

find_attribute_by_code(attribute_code: AttributeCode) → Attribute | None¶: Return the attribute having this code, or None if not found.

get_attribute_by_code(attribute_code: AttributeCode) → Attribute¶

Get an attribute by its code.

Raises:: DatasetAttributeNotFound: If the attribute is not found.

merge(other: DatasetAttributes) → DatasetAttributes¶

class CategoryTree(*, children: Sequence[CategoryTreeNode] = <factory>)¶

A category tree referencing datasets.

children: Sequence[CategoryTreeNode]¶

delete_empty_categories() → CategoryTree¶

iter_dataset_references() → Iterator[DatasetReference]¶: Yield datasets referenced by the category tree recursively.

merge(other: CategoryTree) → CategoryTree¶

sorted() → CategoryTree¶

class Category(*, children: Sequence[CategoryTreeNode] = <factory>, code: CategoryCode | None = None, doc_href: Url | None = None, name: str | None = None)¶

A category node of a category tree.

children: Sequence[CategoryTreeNode]¶

code: CategoryCode | None¶

Create a category.

Raises:: CategoryCreateError: If the category cannot be created.

doc_href: Url | None¶

iter_dataset_references() → Iterator[DatasetReference]¶: Yield datasets referenced by the nodes recursively.

merge(other: Category) → Category¶

name: str | None¶

sorted() → Category¶

class CategoryTreeNode¶

abstractmethod iter_dataset_references() → Iterator[DatasetReference]¶: Yield datasets referenced by the nodes recursively.

merge_category_tree_nodes(*, source: Iterable[CategoryTreeNode], target: Iterable[CategoryTreeNode]) → list[CategoryTreeNode]¶

sort_category_tree_nodes(nodes: Iterable[CategoryTreeNode]) → list[CategoryTreeNode]¶

class DatasetReference(*, code: DatasetCode, name: str | None = None)¶

A dataset node of a category tree.

code: DatasetCode¶

classmethod create(code: str, *, name: str | None = None) → Self¶

Create a dataset reference.

Raises:: DatasetReferenceCreateError: If the dataset reference cannot be created.

iter_dataset_references() → Iterator[DatasetReference]¶: Yield datasets referenced by the nodes recursively.

name: str | None¶

class DatasetFamily(*, code: dbnomics_toolbox.model.identifiers.dataset_family_code.DatasetFamilyCode, description: str | None = None, name: str | None = None, notes: str | None = None, vintages: collections.abc.Sequence[dbnomics_toolbox.model.dataset_vintage.DatasetVintage])¶

code: DatasetFamilyCode¶

classmethod create(*, code: str, description: str | None = None, name: str | None = None, notes: str | None = None, vintages: Sequence[DatasetVintage] | None = None) → Self¶

Create a dataset family from raw values.

Raises:: DatasetFamilyCreateError: If the dataset family cannot be created.

description: str | None¶

name: str | None¶

notes: str | None¶

vintages: Sequence[DatasetVintage]¶

class DatasetMetadata(*, attributes: dbnomics_toolbox.model.attributes.dataset_attributes.DatasetAttributes = <factory>, code: dbnomics_toolbox.model.identifiers.dataset_code.DatasetCode, description: str | None = None, dimensions: dbnomics_toolbox.model.dimensions.dataset_dimensions.DatasetDimensions = <factory>, discontinued: bool = False, doc_href: dbnomics_toolbox.model.url.Url | None = None, name: str | None = None, next_release_at: datetime.date | datetime.datetime | None = None, notes: collections.abc.Sequence[str] = <factory>, source_href: dbnomics_toolbox.model.url.Url | None = None, updated_at: datetime.date | datetime.datetime | None = None)¶

property attribute_codes: Sequence[AttributeCode]¶

attributes: DatasetAttributes¶

The dataset attributes.

Attributes are informative, contrary to dimensions.

code: DatasetCode¶

Create an instance of dataset metadata from string values.

Raises:: DatasetMetadataCreateError: If the dataset metadata could not be created.

description: str | None¶

dimensions: DatasetDimensions¶

The dataset dimensions.

Dimensions classify the dataset in groups of time series.

discontinued: bool¶

Has the provider stopped the publication of this dataset?

Set to True if the dataset is no more available on the provider website, or if the provider declares it as discontinued.

doc_href: Url | None¶: URL to the documentation of the dataset.

merge(other: DatasetMetadata) → DatasetMetadata¶

name: str | None¶

next_release_at: date | datetime | None¶: When a new release of the dataset will occur, given by the provider.

notes: Sequence[str]¶

source_href: Url | None¶: URL to the page of the dataset on the provider website.

updated_at: date | datetime | None¶: When the dataset was last updated, given by the provider.

class DatasetVintage(*, dataset_code: DatasetCode, id: DatasetVintageId)¶

Represent a specific vintage of a dataset within a dataset family.

Each vintage corresponds to a dataset that may have been published by the provider at a specific date or release (e.g. WEO@2020-10).

Attributes:: dataset_code: Identifier of the dataset (e.g. WEO@2020_10). dataset_vintage_id: Identifier of this specific vintage (e.g. 2020-10).

classmethod create(id: str, *, dataset_code: str) → Self¶

Create a dataset vintage from raw values.

Raises:: DatasetVintageCreateError: If the dataset vintage could not be created.

dataset_code: DatasetCode¶

id: DatasetVintageId¶

class Dimension(*, code: 'DimensionCode', label: str | None = None, values: collections.abc.Sequence['DimensionValue'] = <factory>)¶

add_value(value: DimensionValue) → None¶

code: DimensionCode¶

create(code: str, *, label: str | None = None, values: Sequence[DimensionValue] | None = None) → Dimension¶

Create a dimension from raw values.

Raises:: DimensionCreateError: If the dimension instance cannot be created.

find_value_by_code(dimension_value_code: DimensionValueCode) → DimensionValue | None¶

Find a dimension value by its code.

Returns:: The dimension value, or None if not found.

get_value_by_code(dimension_value_code: DimensionValueCode) → DimensionValue¶

Get a dimension value by its code.

Raises:: DimensionValueNotFound: If the dimension value is not found.

label: str | None¶

merge(other: Dimension) → Dimension¶

property value_codes: list[DimensionValueCode]¶

values: Sequence[DimensionValue]¶

with_values_sorted_by_code() → Self¶

class DimensionRole(*values)¶

The role of a dimension.

Some dimensions can have a specific role (e.g. the “frequency” dimension). As DBnomics keeps the codes of the dimensions from the data providers as-is, the dimension role is a way to indicate which dimension have a specific role without relying on a conventional name.

FREQUENCY = 'FREQUENCY'¶

class DimensionValue(*, code: dbnomics_toolbox.model.identifiers.dimension_value_code.DimensionValueCode, label: str | None = None)¶

code: DimensionValueCode¶

create(code: str, *, label: str | None = None) → DimensionValue¶

Create a dimension value from raw values.

Raises:: DimensionValueCreateError: If the dimension value instance cannot be created.

label: str | None¶

class DatasetDimensions(dimensions: collections.abc.Sequence['Dimension'] = <factory>, *, roles: collections.abc.Mapping['DimensionRole', 'DimensionCode']=<factory>)¶

property dimension_codes: Sequence[DimensionCode]¶

dimensions: Sequence[Dimension]¶

find_dimension_by_code(dimension_code: DimensionCode) → Dimension | None¶

Find a dimension by its code.

Returns:: The dimension, or None if not found.

property frequency_dimension_code: DimensionCode | None¶

get_dimension_by_code(dimension_code: DimensionCode) → Dimension¶

Get a dimension by its code.

Raises:: DatasetDimensionNotFound: If the dimension is not found.

merge(other: DatasetDimensions) → DatasetDimensions¶

roles: Mapping[DimensionRole, DimensionCode]¶

update_from_series_dimensions(series_dimensions: SeriesDimensions) → None¶

exception ModelError(*, msg: str)¶

class Frequency(*values)¶

ANNUAL = FrequencyItem(code='A', label='Annual')¶

BIMESTRIAL = FrequencyItem(code='B', label='Bimestrial')¶

DAILY = FrequencyItem(code='D', label='Daily')¶

MONTHLY = FrequencyItem(code='M', label='Monthly')¶

QUARTERLY = FrequencyItem(code='Q', label='Quarterly')¶

SEMESTRIAL = FrequencyItem(code='S', label='Semestrial')¶

WEEKLY = FrequencyItem(code='W', label='Weekly')¶

classmethod frequency_codes() → Sequence[Literal['A', 'B', 'D', 'M', 'Q', 'S', 'W']]¶

classmethod from_code(code: Literal['A', 'B', 'D', 'M', 'Q', 'S', 'W']) → Self¶

Return the frequency enum member matching a code.

Raises:: InvalidFrequencyCode: If code is not a valid frequency code.

classmethod parse_code(value: str) → Self¶

Parse a frequency code.

Raises:: InvalidFrequencyCode: If value is not a valid frequency code.

class FrequencyItem(*, code: Literal['A', 'B', 'D', 'M', 'Q', 'S', 'W'], label: str)¶

code: Literal['A', 'B', 'D', 'M', 'Q', 'S', 'W']¶

label: str¶

class Observation(*, period: dbnomics_toolbox.model.periods.period.Period, value: float | decimal.Decimal | None, attributes: collections.abc.Mapping[dbnomics_toolbox.model.identifiers.attribute_code.AttributeCode, str]=<factory>)¶

attributes: Mapping[AttributeCode, str]¶

create(*, attributes: Mapping[str, str | None] | None = None, period: Period | str | int, value: ObservationValue) → Observation¶

Create an Observation instance from raw values.

Args:: attributes: Optional observation attributes. period: Period value as a Period, a parsable string, or an integer year. value: Observation value. NaN values are normalized to None (which represents NA).
Raises:: ObservationCreateError: If an Observation instance cannot be created.

period: Period¶

value: float | Decimal | None¶

property value_as_float: float | None¶

class ObservationCollectionServices(observations: Sequence[Observation])¶

detect_frequency() → Frequency | None¶

get_observation_attribute_codes() → set[AttributeCode]¶

iter_periods() → Iterator[Period]¶

iter_values() → Iterator[float | Decimal | None]¶

property max_period: Period¶

property max_value: float | Decimal¶

property min_period: Period¶

property min_value: float | Decimal¶

property period_domain: PeriodDomain | None¶

sort_by_period() → None¶

property value_range: ObservationValueRange | None¶

class ObservationValueRange(lowest: T, highest: T)¶

highest: T¶

lowest: T¶

class PeriodDomain(lowest: T, highest: T)¶

highest: T¶

lowest: T¶

class ProviderMetadata(*, attribution: str | None = None, code: dbnomics_toolbox.model.identifiers.provider_code.ProviderCode, description: str | None = None, name: str | None = None, region: str | None = None, terms_of_use: dbnomics_toolbox.model.url.Url | None = None, website: dbnomics_toolbox.model.url.Url)¶

attribution: str | None¶

code: ProviderCode¶

Create an instance of provider metadata from string values.

Raises:: ProviderMetadataCreateError: If the provider metadata could not be created.

description: str | None¶

name: str | None¶

region: str | None¶

terms_of_use: Url | None¶

website: Url¶

ProviderMetadata.__init__(*, attribution: str | None = None, code: ProviderCode, description: str | None = None, name: str | None = None, region: str | None = None, terms_of_use: Url | None = None, website: Url) → None¶

class ProviderSnapshot(*, author_email: str | None = None, author_name: str | None = None, created_at: datetime.datetime, id: dbnomics_toolbox.model.identifiers.provider_snapshot_id.ProviderSnapshotId, message: str, provider_code: dbnomics_toolbox.model.identifiers.provider_code.ProviderCode)¶

author_email: str | None¶

author_name: str | None¶

created_at: datetime¶

id: ProviderSnapshotId¶

message: str¶

provider_code: ProviderCode¶

class Series(*, metadata: SeriesMetadata, observations: Sequence[Observation])¶

A complete series with its metadata and observations.

create(*, add_missing_dataset_dimensions: bool = False, attributes: Mapping[str, str | None] | None = None, code: str | None = None, dataset_dimensions: DatasetDimensions | None = None, description: str | None = None, dimensions: Mapping[str, str] | None = None, doc_href: str | None = None, name: str | None = None, next_release_at: date | datetime | None = None, notes: Sequence[str] | None = None, observations: Sequence[Observation] | None = None, updated_at: date | datetime | None = None) → Series¶

merge(other: Series) → Series¶

Merge other into self.

Subfield behavior: - metadata: delegated to SeriesMetadata.merge. - observations: merged by period, sorted by period, with other winning when both series define the same period.

This method is immutable: it returns a new series and does not mutate either input series.

metadata: SeriesMetadata¶

observations: Sequence[Observation]¶

type SeriesDimensions = Mapping[DimensionCode, DimensionValueCode]¶

class SeriesMetadata(*, attributes: collections.abc.Mapping['AttributeCode', str]=<factory>, code: 'SeriesCode', description: str | None = None, dimensions: 'SeriesDimensions' = <factory>, doc_href: 'Url | None' = None, name: str | None = None, next_release_at: datetime.date | datetime.datetime | None = None, notes: collections.abc.Sequence[str] = <factory>, updated_at: datetime.date | datetime.datetime | None = None)¶

attributes: Mapping[AttributeCode, str]¶

The series attributes.

An attribute value can be a code or free text.

code: SeriesCode¶

Create a SeriesMetadata instance from raw values.

The series code is generated from dataset dimensions if it is not provided. In this case, the series code is generated by concatenating the dimension value codes of the series dimensions in the order of the dataset dimensions.

Args:: add_missing_dataset_dimensions: Whether to add missing dataset dimensions to dataset_dimensions if they are present in the series dimensions but not in dataset_dimensions. This is useful when the dataset dimensions are not explicitly provided and need to be inferred from the series dimensions.
Raises:: SeriesMetadataCreateError: If a SeriesMetadata instance cannot be created. ValueError: If dataset_dimensions is not provided when code is not provided also.

description: str | None¶

dimensions: SeriesDimensions¶

doc_href: Url | None¶: URL to the documentation of the series.

merge(other: SeriesMetadata) → SeriesMetadata¶

Merge other into self and return a new metadata instance.

Subfield behavior: - attributes: keys from both are kept; for duplicate keys, the resulting value is taken from other. - dimensions: keys from both are kept; for duplicate keys, the resulting value is taken from other.

Neither input instance is mutated.

name: str | None¶

next_release_at: date | datetime | None¶: When a new release of the series will occur, given by the provider.

notes: Sequence[str]¶

parse_dimensions(dimensions: Mapping[str, str] | None) → SeriesDimensions¶

Parse dimensions from a mapping of dimension code to dimension value code.

Raises:: CachedParserParseError: If any of the dimension codes or dimension value codes cannot be parsed.

updated_at: date | datetime | None¶: When the series was last updated, given by the provider.

is_url(raw: str, *, public: bool) → TypeGuard[Url]¶

parse_url(raw: str, *, public: bool = True) → Url¶

Parse a value as an URL.

Raises:: UrlParseError: If the parsing failed.

Url = dbnomics_toolbox.model.url.Url¶

NewType creates simple unique types with almost zero runtime overhead.

NewType(name, tp) is considered a subtype of tp by static type checkers. At runtime, NewType(name, tp) returns a dummy callable that simply returns its argument.

Usage:

UserId = NewType('UserId', int)

def name_by_id(user_id: UserId) -> str:
    ...

UserId('user')          # Fails type check

name_by_id(42)          # Fails type check
name_by_id(UserId(42))  # OK

num = UserId(5) + 1     # type: int

Periods¶

class BimesterPeriod(year_num: int, bimester_num: int)¶

A period of 2 consecutive months.

bimester_num: int¶

property first_day: date¶

property first_month: MonthPeriod¶

max_bimester_num: ClassVar[int] = 6¶

min_bimester_num: ClassVar[int] = 1¶

property next: Self¶

classmethod parse(value: str) → Self¶

Parse a string as a period.

Raises:: PeriodParseError: If the parsing failed.

property previous: Self¶

property quarter: QuarterPeriod¶

property semester: SemesterPeriod¶

type_: ClassVar[PeriodType] = 'bimester'¶

class DayPeriod(year_num: int, month_num: int, day_num: int)¶

property bimester: BimesterPeriod¶

day_num: int¶

property first_day: date¶

classmethod from_date(value: date) → Self¶

property month: MonthPeriod¶

month_num: int¶

property next: Self¶

classmethod parse(value: str) → Self¶

Parse a string as a period.

Raises:: PeriodParseError: If the parsing failed.

property previous: Self¶

property quarter: QuarterPeriod¶

property semester: SemesterPeriod¶

type_: ClassVar[PeriodType] = 'day'¶

property week: WeekPeriod¶

class MonthPeriod(year_num: int, month_num: int)¶

property bimester: BimesterPeriod¶

property first_day: date¶

classmethod from_date(value: date) → Self¶

max_month_num: ClassVar[int] = 12¶

min_month_num: ClassVar[int] = 1¶

month_num: int¶

property next: Self¶

classmethod parse(value: str) → Self¶

Parse a string as a period.

Raises:: PeriodParseError: If the parsing failed.

property previous: Self¶

property quarter: QuarterPeriod¶

property semester: SemesterPeriod¶

type_: ClassVar[PeriodType] = 'month'¶

class Period(year_num: int)¶

property first_day: date¶

property next: Self¶

classmethod parse(value: str) → Self¶

Parse a string as a period.

Raises:: PeriodParseError: If the parsing failed.

property previous: Self¶

type_: ClassVar[PeriodType]¶

property year: YearPeriod¶

year_num: int¶

class QuarterPeriod(year_num: int, quarter_num: int)¶

A period of 3 consecutive months.

property first_day: date¶

property first_month: MonthPeriod¶

max_quarter_num: ClassVar[int] = 4¶

min_quarter_num: ClassVar[int] = 1¶

property next: Self¶

classmethod parse(value: str) → Self¶

Parse a string as a period.

Raises:: PeriodParseError: If the parsing failed.

property previous: Self¶

quarter_num: int¶

property semester: SemesterPeriod¶

type_: ClassVar[PeriodType] = 'quarter'¶

class SemesterPeriod(year_num: int, semester_num: int)¶

A period of 6 consecutive months.

property first_day: date¶

property first_month: MonthPeriod¶

max_semester_num: ClassVar[int] = 2¶

min_semester_num: ClassVar[int] = 1¶

property next: Self¶

classmethod parse(value: str) → Self¶

Parse a string as a period.

Raises:: PeriodParseError: If the parsing failed.

property previous: Self¶

semester_num: int¶

type_: ClassVar[PeriodType] = 'semester'¶

class WeekPeriod(year_num: int, week_num: int)¶

property bimester: BimesterPeriod¶

property first_day: date¶

classmethod from_date(value: date) → Self¶

classmethod from_week(value: Week) → Self¶

property month: MonthPeriod¶

property next: Self¶

classmethod parse(value: str) → Self¶

Parse a string as a period.

Raises:: PeriodParseError: If the parsing failed.

property previous: Self¶

property quarter: QuarterPeriod¶

property semester: SemesterPeriod¶

type_: ClassVar[PeriodType] = 'week'¶

week_num: int¶

class YearPeriod(year_num: int)¶

property first_day: date¶

property first_month: MonthPeriod¶

property next: Self¶

classmethod parse(value: str) → Self¶

Parse a string as a period.

Raises:: PeriodParseError: If the parsing failed.

property previous: Self¶

type_: ClassVar[PeriodType] = 'year'¶

`dbnomics_toolbox.retry_utils`¶

Run¶

Requests¶

`dbnomics_toolbox.sdmx_utils`¶

SDMX 2.1¶

SDMX 2.1 directory manager¶

class SdmxDirManager(*, base_dir: Path, dataset_data_file_name: str | None = None, dataset_structures_file_name: str | None = None, datasets_dir_name: str | None = None, sdmx_parser_factory: SdmxParserFactory | None = None)¶

Defines a conventional way to organize SDMX files in a directory and allows loading artefacts.

cache_global_structures(structures: set[StructureType]) → Generator[None]¶

get_dataset_data_file(dataflow_id: str) → Path¶

get_dataset_dir(dataflow_id: str) → Path¶

get_dataset_structure_file(dataflow_id: str) → Path¶

get_global_structure_file(structure_type: StructureType) → Path¶

iter_dataflow_ids_from_datasets_dir() → Iterator[str]¶

iter_dataset_data_files(dataflow_id: str) → Iterator[Path]¶

iter_dataset_directories() → Iterator[Path]¶

load_categorisations() → Categorisations¶

load_category_schemes() → CategorySchemes¶

load_codelists() → Codelists¶

load_concepts() → Concepts¶

load_constraints() → Constraints¶

load_data_structures(*, dataflow_id: str | None = None) → DataStructures¶

load_dataflows() → Dataflows¶

parse_dataset_structure_file(dataflow_id: str) → Generator[StructureParser]¶

parse_global_structure_file(structure_type: StructureType) → Generator[StructureParser]¶

SDMX 2.1 normalization utils¶

normalize_digits(text: bytes) → bytes¶

normalize_sdmx_file_header(input_file: Path, *, max_buffer_size: int = 2048, replacements: dict[str, Callable[[bytes], bytes]] | None = None) → None¶

Overwrite ID and timestamp in SDMX header by fixed values.

This is specifically useful to avoid creating false revisions of the file.

This implementation is based on the fact that the tags to modify are found in the first bytes of the file. It doesn’t load the entire file in memory.

SDMX 2.1 processors¶

SDMX 2.1 model¶

class ActualContentConstraint(*, descriptions: dict[str, str], names: dict[str, str], agency_id: str, version: str, id: str, type: str, valid_from: datetime.datetime)¶

valid_from: datetime¶

class Annotation(*, id: str | None, text: dict[str, str], title: str | None, type: str | None, url: str | None)¶

id: str | None¶

text: dict[str, str]¶

title: str | None¶

type: str | None¶

url: str | None¶

class Annotations(*, children: list['Annotation'])¶

children: list[Annotation]¶

find(type_: str) → Annotation | None¶

get(type_: str) → Annotation¶

get_title(type_: str) → str¶

get_url(type_: str) → str¶

class Attribute(*, id: str, assignment_status: Literal['Conditional', 'Mandatory'], codelist_ref: dbnomics_toolbox.sdmx_utils.v2_1.model.codelists.CodelistRef | None, concept_identity_ref: dbnomics_toolbox.sdmx_utils.v2_1.model.concepts.ConceptIdentityRef)¶

assignment_status: Literal['Conditional', 'Mandatory']¶

codelist_ref: CodelistRef | None¶

concept_identity_ref: ConceptIdentityRef¶

class AttributeList(*, children: list[TIdentifiableArtefact], id: str)¶

class Categorisation(*, descriptions: dict[str, str], names: dict[str, str], agency_id: str, version: str, id: str, annotations: dbnomics_toolbox.sdmx_utils.v2_1.model.annotations.Annotations, source_ref: dbnomics_toolbox.sdmx_utils.v2_1.model.dataflows.DataflowRef, target_ref: dbnomics_toolbox.sdmx_utils.v2_1.model.categories.CategoryRef)¶

property ref: CategorisationRef¶

source_ref: DataflowRef¶

target_ref: CategoryRef¶

class CategorisationRef(*, class_: dbnomics_toolbox.sdmx_utils.v2_1.types.ClassName | None = <ClassName.Dataflow: 'Dataflow'>, package: dbnomics_toolbox.sdmx_utils.v2_1.types.PackageName | None = <PackageName.datastructure: 'datastructure'>, agency_id: str, version: str, id: str)¶

class_: ClassName | None = 'Dataflow'¶

package: PackageName | None = 'datastructure'¶

class Categorisations(*, children: list[TIdentifiableAndMaintainableArtefact])¶

filter_by_ref(*, agency_id: str | None = None, all_versions: bool | None = None, source_ref_class: ClassName | None = None, source_ref_package: PackageName | None = None, target_ref_id: str | None = None, target_ref_maintainable_parent_id: str | None = None) → Iterator[Categorisation]¶

iter_categorisations(*, agency_id: str | None = None, all_versions: bool | None = None) → Iterator[Categorisation]¶

class Category(*, descriptions: dict[str, str], names: dict[str, str], children: list[TIdentifiableArtefact], id: str)¶

class CategoryRef(*, class_: dbnomics_toolbox.sdmx_utils.v2_1.types.ClassName | None, package: dbnomics_toolbox.sdmx_utils.v2_1.types.PackageName | None, agency_id: str, maintainable_parent_id: str, maintainable_parent_version: str, id: str)¶

class CategoryScheme(*, agency_id: str, version: str, descriptions: dict[str, str], names: dict[str, str], children: list[TIdentifiableArtefact], id: str, annotations: dbnomics_toolbox.sdmx_utils.v2_1.model.annotations.Annotations)¶

class CategorySchemes(*, children: list[TIdentifiableAndMaintainableArtefact])¶

class Code(*, descriptions: dict[str, str], names: dict[str, str], id: str)¶

class Codelist(*, descriptions: dict[str, str], names: dict[str, str], children: list[TIdentifiableArtefact], agency_id: str, version: str, id: str)¶

class CodelistRef(*, class_: dbnomics_toolbox.sdmx_utils.v2_1.types.ClassName | None, package: dbnomics_toolbox.sdmx_utils.v2_1.types.PackageName | None, agency_id: str, version: str, id: str)¶

class Codelists(*, children: list[TIdentifiableAndMaintainableArtefact])¶

get_from_ref(ref: CodelistRef, *, strict: bool | None = None) → Codelist¶

class Concept(*, descriptions: dict[str, str], names: dict[str, str], id: str)¶

class ConceptIdentityRef(*, class_: dbnomics_toolbox.sdmx_utils.v2_1.types.ClassName | None, package: dbnomics_toolbox.sdmx_utils.v2_1.types.PackageName | None, agency_id: str, maintainable_parent_id: str, maintainable_parent_version: str, id: str)¶

class ConceptScheme(*, descriptions: dict[str, str], names: dict[str, str], children: list[TIdentifiableArtefact], agency_id: str, version: str, id: str)¶

property ref: ConceptSchemeRef¶

class ConceptSchemeRef(*, class_: dbnomics_toolbox.sdmx_utils.v2_1.types.ClassName | None = None, package: dbnomics_toolbox.sdmx_utils.v2_1.types.PackageName | None = None, agency_id: str, version: str, id: str)¶

class_: ClassName | None = None¶

package: PackageName | None = None¶

class Concepts(*, children: list[TIdentifiableAndMaintainableArtefact])¶

get_from_ref(ref: ConceptIdentityRef, *, strict: bool | None = None) → ConceptScheme¶

class Constraints(*, children: list[TIdentifiableAndMaintainableArtefact])¶

class ContentConstraint(*, descriptions: dict[str, str], names: dict[str, str], agency_id: str, version: str, id: str, type: str)¶

type: str¶

class DataStructure(*, descriptions: dict[str, str], names: dict[str, str], agency_id: str, version: str, id: str, attribute_list: dbnomics_toolbox.sdmx_utils.v2_1.model.data_structures.AttributeList | None, dimension_list: dbnomics_toolbox.sdmx_utils.v2_1.model.data_structures.DimensionList, measure_list: dbnomics_toolbox.sdmx_utils.v2_1.model.data_structures.MeasureList)¶

attribute_list: AttributeList | None¶

dimension_list: DimensionList¶

measure_list: MeasureList¶

property ref: StructureRef¶

class DataStructures(*, children: list[TIdentifiableAndMaintainableArtefact])¶

get_from_ref(ref: StructureRef, *, strict: bool | None = None) → DataStructure¶

class Dataflow(*, descriptions: dict[str, str], names: dict[str, str], agency_id: str, version: str, id: str, annotations: dbnomics_toolbox.sdmx_utils.v2_1.model.annotations.Annotations, structure_ref: dbnomics_toolbox.sdmx_utils.v2_1.model.data_structures.StructureRef | None)¶

property ref: DataflowRef¶

structure_ref: StructureRef | None¶

class DataflowRef(*, class_: dbnomics_toolbox.sdmx_utils.v2_1.types.ClassName | None = <ClassName.Dataflow: 'Dataflow'>, package: dbnomics_toolbox.sdmx_utils.v2_1.types.PackageName | None = <PackageName.datastructure: 'datastructure'>, agency_id: str, version: str, id: str)¶

class_: ClassName | None = 'Dataflow'¶

package: PackageName | None = 'datastructure'¶

class Dataflows(*, children: list[TIdentifiableAndMaintainableArtefact])¶

get_from_ref(ref: DataflowRef, *, strict: bool | None = None) → Dataflow¶

class Dimension(*, id: str, codelist_ref: dbnomics_toolbox.sdmx_utils.v2_1.model.codelists.CodelistRef | None, concept_identity_ref: dbnomics_toolbox.sdmx_utils.v2_1.model.concepts.ConceptIdentityRef, position: int)¶

codelist_ref: CodelistRef | None¶

concept_identity_ref: ConceptIdentityRef¶

position: int¶

class DimensionList(*, children: list[TIdentifiableArtefact], id: str, time_dimension: dbnomics_toolbox.sdmx_utils.v2_1.model.data_structures.TimeDimension | None)¶

time_dimension: TimeDimension | None¶

class Error(*, message: dbnomics_toolbox.sdmx_utils.v2_1.model.error_models.ErrorMessage)¶

message: ErrorMessage¶

class ErrorCode(*values)¶

NO_RESULT_FOUND = '100'¶

RESPONSE_TOO_LARGE = '510'¶

class ErrorMessage(*, code: str, text_items: list[str])¶

code: str¶

text_items: list[str]¶

class IdValues(*, pairs: list[tuple[str, str]])¶

property as_dict: dict[str, str]¶

iter_values() → Iterator[str]¶

pairs: list[tuple[str, str]]¶

property values: list[str]¶

class MeasureList(*, id: str, primary_measure: dbnomics_toolbox.sdmx_utils.v2_1.model.data_structures.PrimaryMeasure)¶

primary_measure: PrimaryMeasure¶

class Observation(*, attributes: dict[str, str] | None = None, dimension: str | None, value: str | None)¶

An SDMX Observation.

In general the observation dimension is the time period.

attributes¶

dimension¶

value¶

class PrimaryMeasure(*, id: str, concept_identity_ref: dbnomics_toolbox.sdmx_utils.v2_1.model.concepts.ConceptIdentityRef)¶

concept_identity_ref: ConceptIdentityRef¶

class Series(*, attributes: dbnomics_toolbox.sdmx_utils.v2_1.model.series.IdValues = <factory>, key: dbnomics_toolbox.sdmx_utils.v2_1.model.series.SeriesKey, observations: list['Observation'])¶

attributes: IdValues¶

key: SeriesKey¶

classmethod merge_many(series_iter: Iterable[Self]) → Self¶

observations: list[Observation]¶

class SeriesKey(*, pairs: list[tuple[str, str]])¶

A series key is composed of (ID, value) pairs of dimensions.

matches(other: Self) → bool¶

class StructureRef(*, class_: dbnomics_toolbox.sdmx_utils.v2_1.types.ClassName | None = <ClassName.DataStructure: 'DataStructure'>, package: dbnomics_toolbox.sdmx_utils.v2_1.types.PackageName | None = <PackageName.datastructure: 'datastructure'>, agency_id: str, version: str, id: str)¶

class_: ClassName | None = 'DataStructure'¶

package: PackageName | None = 'datastructure'¶

`dbnomics_toolbox.storage`¶

Storage interface¶

class DatasetReader(*args, **kwargs)¶

dataset_id: DatasetId¶

iter_provider_snapshots(*, series_codes: Sequence[SeriesCode] | None = None) → Iterator[ProviderSnapshot]¶

Iterate over the provider snapshots touching this dataset.

Args:: series_codes: Yield the provider snapshots touching these series only.

The snapshots are iterated in a reverse chronological order, from newer to older.

iter_series(*, series_codes: Sequence[SeriesCode] | None = None) → Iterator[Series]¶

Iterate over metadata and observations of the series of this dataset.

The iteration order is implementation-dependent.

Args:

series_codes: The codes of the series to read.: The series that are not found are not yielded.

Raises:

SeriesIterError: If the iteration over the series fails before a specific series can be identified. SeriesItemIterError: If a specific series cannot be read while iterating.

iter_series_codes() → Iterator[SeriesCode]¶

Iterate over the codes of the series of this dataset.

The iteration order is implementation-dependent.

Raises:: SeriesCodesIterError: If the iteration over the series codes fails.

iter_series_metadata(*, series_codes: Sequence[SeriesCode] | None = None) → Iterator[SeriesMetadata]¶

Iterate over metadata of the series of this dataset.

The iteration order is implementation-dependent.

Args:

series_codes: The codes of the series to read.: The series that are not found are not yielded.

Raises:

SeriesMetadataIterError:: If the iteration over series metadata fails before a specific series can be identified.
SeriesMetadataItemIterError:: If a specific series metadata cannot be read while iterating.

read_dataset_metadata() → DatasetMetadata¶

Read metadata of the current dataset.

Raises:: DatasetMetadataNotFound: If the dataset metadata cannot be found. DatasetMetadataReadError: If the dataset metadata cannot be read.

read_series(series_code: SeriesCode) → Series¶

Read metadata and observations of a series.

Raises:: SeriesNotFound: If the series does not exist in this dataset. SeriesReadError: For any other failure preventing reading.

read_series_metadata(series_code: SeriesCode) → SeriesMetadata¶

Read metadata of a series.

Raises:: SeriesNotFound: If the series does not exist in this dataset. SeriesMetadataReadError: For any other failure preventing reading.

class DatasetWriter(*args, **kwargs)¶

dataset_id: DatasetId¶

delete_dataset_metadata() → None¶

Delete the metadata of this dataset, if it exists.

Raises:: DatasetMetadataDeleteError: If dataset metadata cannot be deleted.

write_dataset_metadata(dataset_metadata: DatasetMetadata) → None¶

Write the metadata of this dataset.

Raises:: DatasetMetadataWriteError: If dataset metadata cannot be written.

write_series(series_iter: Iterable[Series]) → None¶

Write series to this dataset.

The errors raised during the iteration of series_iter are not handled by this method.

Raises:: SeriesWriteError: If a specific series item cannot be written.

class ProviderReader(*args, **kwargs)¶

at_snapshot(provider_snapshot_id: ProviderSnapshotId) → ProviderReader¶: Return a reader for this provider that reads data at a specific snapshot.

compute_dataset_hash(dataset_code: DatasetCode) → str¶: Compute a stable hash for a dataset content.

count_datasets() → int¶: Return the number of datasets of this provider.

get_dataset_reader(dataset_code: DatasetCode) → DatasetReader¶

Return a reader for the given dataset.

Raises:: DatasetNotFound: If no dataset with the given code exists. DatasetReaderOpenError: If a reader for the given dataset cannot be opened.

has_category_tree() → bool¶: Return True if this provider has a category tree.

has_dataset(dataset_code: DatasetCode) → bool¶: Return True if this dataset exists.

iter_dataset_codes() → Iterator[DatasetCode]¶

Iterate over the codes of the datasets of this provider.

Yields:: The dataset codes in alphabetical order.

iter_dataset_families() → Iterator[DatasetFamily]¶

Iterate over the dataset families of this provider.

Raises:

DatasetFamilyIterError:: If the iteration over the dataset families fails before a specific dataset family can be identified.
DatasetFamilyItemIterError:: If a specific dataset family cannot be read while iterating.

iter_dataset_readers() → Iterator[DatasetReader]¶

Iterate over the readers of the datasets of this provider.

Yields:: The dataset readers ordered by dataset code.
Raises:: DatasetReaderOpenError: If a reader for a specific dataset cannot be opened during the iteration.

iter_provider_snapshots() → Iterator[ProviderSnapshot]¶

Iterate over the snapshots of this provider.

The snapshots are iterated in a reverse chronological order, from newer to older.

provider_code: ProviderCode¶

read_category_tree() → CategoryTree | None¶

Read the category tree of this provider, if it exists.

Returns:: If not found, return None.
Raises:: CategoryTreeReadError: If the category tree reading fails.

read_dataset_family(dataset_family_code: DatasetFamilyCode) → DatasetFamily | None¶

Read a dataset family of this provider.

Returns:: If not found, return None.
Raises:: DatasetFamilyReadError: If the dataset family reading fails.

read_provider_metadata() → ProviderMetadata¶

Read metadata of this provider.

Raises:: ProviderMetadataNotFound: If no metadata is available. ProviderMetadataReadError: For any failure preventing reading.

read_provider_snapshot(provider_snapshot_id: ProviderSnapshotId) → ProviderSnapshot¶

Read a snapshot of this provider.

Raises:: ProviderMetadataNotFound: If no metadata is available.

class ProviderWriter(*args, **kwargs)¶

create_session(session_id: str) → ProviderWriterSession¶: Create a provider writer session.

delete_category_tree() → None¶

Delete the category tree of this provider, if it exists.

Raises:: CategoryTreeDeleteError: If the category tree could not be deleted.

delete_dataset(dataset_code: DatasetCode) → None¶

Delete the dataset with the given code, if it exists.

Raises:: DatasetDeleteError: If the dataset could not be deleted.

delete_dataset_family(dataset_family_code: DatasetFamilyCode) → None¶

Delete the dataset family with the given code, if it exists.

Raises:: DatasetFamilyDeleteError: If the dataset family could not be deleted.

delete_provider_metadata() → None¶

Delete the metadata of this provider, if it exists.

Raises:: ProviderMetadataDeleteError: If the provider metadata could not be deleted.

get_dataset_writer(dataset_code: DatasetCode) → DatasetWriter¶: Return a writer for the given dataset.

get_reader() → ProviderReader¶: Return a reader for this provider.

provider_code: ProviderCode¶

write_category_tree(category_tree: CategoryTree) → None¶

Write the category tree of this provider.

Raises:: CategoryTreeWriteError: If the category tree could not be written.

write_provider_metadata(provider_metadata: ProviderMetadata) → None¶

Write the metadata of this provider.

Raises:: ProviderMetadataWriteError: If provider metadata could not be written.

class ProviderWriterSession(*args, **kwargs)¶

Apply write operations to a provider atomically.

This class is a context manager to ensure that all write operations are applied atomically. That is, either all the changes are applied, or none if any of them fails.

The session must be explicitly committed, otherwise all changes will be rolled back when exiting the context manager.

commit() → None¶

Commit the session, applying all changes atomically.

Raises:: ProviderWriterSessionCommitError: If the session could not be committed.

property provider_writer: ProviderWriter¶

class StorageAdapter(*args, **kwargs)¶

classmethod from_uri(uri: Uri) → Self¶

get_reader() → StorageReader¶: Return a reader for this storage.

get_writer() → StorageWriter¶: Return a writer for this storage.

class StorageReader(*args, **kwargs)¶

get_provider_reader(provider_code: ProviderCode) → ProviderReader¶

Return a reader for the given provider.

Raises:: ProviderNotFound: If no provider with the given code exists. ProviderReaderOpenError: If the provider reader could not be opened.

iter_provider_codes() → Iterator[ProviderCode]¶

Iterate over the codes of the providers.

The iteration order is implementation-dependent.

iter_provider_readers() → Iterator[ProviderReader | ProviderReaderCreateError]¶

Iterate over the providers.

The iteration order is implementation-dependent.

If the storage reader is in single-provider mode, yields a single ProviderReader.

class StorageWriter(*args, **kwargs)¶

delete_provider(provider_code: ProviderCode) → None¶

Delete the provider with the given code.

Raises:: ProviderNotFound: If no provider with the given code exists. ProviderDeleteError: If the provider could not be deleted.

get_provider_writer(provider_code: ProviderCode) → ProviderWriter¶

Return a writer for the given provider.

Raises:: ProviderWriterOpenError: If the provider writer could not be opened.

class UpdateStrategy(*values)¶

MERGE = 'merge'¶

REPLACE = 'replace'¶

API Reference¶

dbnomics_toolbox.fetcher_utils¶

Constants¶

Errors¶

Source data dir managers¶

Import utilities¶

Logging utilities¶

ZIP utilities¶

CLI utilities¶

Fetcher definition¶

File utilities¶

See Also:¶

MIME type utilities¶

Partitions¶

Processors¶

Resources¶

dbnomics_toolbox.file_utils¶

dbnomics_toolbox.http_utils¶

HTTP constants¶

Requests helpers¶

dbnomics_toolbox.io¶

Parsing¶

Formats¶

Codecs¶

Protocols¶

dbnomics_toolbox.model¶

Identifiers¶

Entities¶

Periods¶

dbnomics_toolbox.retry_utils¶

Run¶

Requests¶

dbnomics_toolbox.sdmx_utils¶

SDMX 2.1¶

SDMX 2.1 directory manager¶

SDMX 2.1 normalization utils¶

SDMX 2.1 processors¶

SDMX 2.1 model¶

dbnomics_toolbox.storage¶

Storage interface¶

File-system adapter¶

dbnomics_toolbox.validation¶

Types¶

Protocols¶

Validators¶

Walkers¶

`dbnomics_toolbox.fetcher_utils`¶

`dbnomics_toolbox.file_utils`¶

`dbnomics_toolbox.http_utils`¶

`dbnomics_toolbox.io`¶

`dbnomics_toolbox.model`¶

`dbnomics_toolbox.retry_utils`¶

`dbnomics_toolbox.sdmx_utils`¶

`dbnomics_toolbox.storage`¶

`dbnomics_toolbox.validation`¶