# File-system storage adapter The file-system adapter is a concrete implementation of the storage protocols described in the [](index.md) page, section **StorageAdapter interface**. The `FsStorageAdapter` class implements the [](#StorageAdapter) protocol, relying on `FsStorageReader` / `FsStorageWriter` for reading/writing and `FsProviderReader` / `FsProviderWriter` for provider-level operations. This adapter has options that are detailed below. ## The physical data model Although using human-readable file formats like JSON, JSON Lines and TSV, it can be considered as a black box, meaning that reading and writing data should be done through the storage adapter. The data format of the files is designed to optimize reads and writes and can differ from the [](../data-model.md) classes. To represent the content of those files in-memory, another set of model classes is defined in [](#dbnomics_toolbox.storage.adapters.filesystem.model). We can talk about those classes as the *physical model*, whereas the classes of [](#dbnomics_toolbox.model) can be called the *domain model*. When calling load and save methods on `FsProviderReader` or `FsProviderWriter`, they are responsible for both: - transforming objects of the domain model from and to objects of the physical model, - knowing the path of the files that are read and written. ### Provider metadata The file-system adapter stores provider metadata in a file named `provider.json` directly at the root of the provider directory. The [](#ProviderMetadata) domain model class is mapped to the [](#ProviderJson) physical model class. See also: [](#ProviderReader.read_provider_metadata) and [](#ProviderWriter.write_provider_metadata). ### Category tree The file-system adapter stores the category tree in a file named `category_tree.json` directly at the root of the provider directory. The [](#CategoryTree) domain model class is mapped to the [](#CategoryTreeJson) physical model class. See also: [](#ProviderReader.read_category_tree) and [](#ProviderWriter.write_category_tree). ### Datasets and series The file-system adapter stores each dataset in a dedicated directory named after the dataset code. For example, the dataset `INSEE/IPC-2015` is stored in the directory `IPC-2015` at the root level of the provider directory. The way datasets and series are stored depends on the chosen dataset layout, the different variants being described in the next section. #### Dataset layout The storage URI of the file-system adapter accepts a `variant` parameter that takes one of those values: `jsonl` or `tsv` (e.g. `filesystem:insee-converted-data?variant=jsonl`). When instantiating a `FsStorageAdapter`, the variant will be detected by looking at the already written files, if any. If detection could not be done, for example because no file has been written yet, then the `jsonl` variant will be used by default for future writes. This allows clients to open any directory with the file-system adapter without knowing by advance which variant is used. #### TSV variant The TSV variant mainly uses [tab-separated values files](https://en.wikipedia.org/wiki/Tab-separated_values) to store series. When using the TSV variant, the following files are created for each dataset, in a sub-directory named after the dataset code. The file `{dataset_code}/dataset.json` contains dataset metadata coming from the [](#DatasetMetadata) model class, and series metadata coming from the [Series](#dbnomics_toolbox.model.series.Series) model class under the `series` property. The [](#TsvDatasetJson) physical model class represents the contents of this file. For each series, observations and their attributes are stored in a TSV file named after the series code. For example, the series `INSEE/IPC-2015/A.IPC.SO.00.00.INDICE.ENSEMBLE.FE.SO.BRUT.2015.FALSE` is stored in the file `IPC-2015/A.IPC.SO.00.00.INDICE.ENSEMBLE.FE.SO.BRUT.2015.FALSE.tsv`. A simple TSV file looks like this: ```text PERIOD VALUE 2000 19 2001 NA 2002 22 ``` Observation attributes can be stored in additional columns: ```text PERIOD VALUE OBS_STATUS 2000 19 2001 NA 2002 22 E ``` Historically the TSV variant was the only existing variant, but since it uses one TSV file per series, this could lead to a huge number of files for some datasets. Given that in file-systems, any file takes a minimum of one block (e.g. 4kb), this is not optimal for datasets having a huge number of small series. Quickly the disk was full due to those millions of small files. Also the fact that the file is named after the series code can lead to file names that are too long for the runtime environment (i.e. file-system, operating system). #### JSON Lines variant The JSON Lines variant mainly uses [JSON Lines files](https://jsonlines.org/) to store series. This variant was introduced to circumvent the issues and limitations of the TSV variant. The file `{dataset_code}/dataset.json` contains dataset metadata coming from the [](#DatasetMetadata) model class. The [](#JsonLinesDatasetJson) physical model class represents the contents of this file. The file `{dataset_code}/series.jsonl` contains all the series of the dataset, including metadata and observations coming from the [Series](#dbnomics_toolbox.model.series.Series) model class. The `JsonLinesSeriesItem` physical model class represents each line of this file. Example of `series.jsonl` (only a minimal sample is shown here): ```text {"code":"M.LB.B.TTP.SA","dimensions":{"frequency":"M","seasonally_adjusted":"SA","sex":"B","subject":"LB","unit":"TTP"},"observations":[["PERIOD","VALUE"],["1953-01",4122],["1953-02",4001],["1953-03",4008]]} {"code":"M.NILF.F.TTP.NSA","dimensions":{"frequency":"M","seasonally_adjusted":"NSA","sex":"F","subject":"NILF","unit":"TTP"},"observations":[["PERIOD","VALUE"],["1972-01","NA"],["1972-02","NA"],["1972-03","NA"],["1972-04","NA"]]} ``` Note that in the above example the series are already sorted alphabetically by code (`M.LB.B.TTP.SA` < `M.NILF.F.TTP.NSA`). This is a **constraint of the JSON Lines format**: series must appear in `series.jsonl` sorted alphabetically by their code. Sorting series enables efficient lookups via `JsonLinesSeriesOffsetIndex` — the file-system adapter can jump directly to the byte offset of a given series without scanning the entire file. However, sorting a potentially huge number of series entirely in memory is impractical. To avoid loading everything into memory, the writer uses a **temporary SQLite database** (`series.sqlite`) as an intermediate store: each series is serialized to JSON and indexed by its code in SQLite, then retrieved in sorted order via `ORDER BY`. This is implemented by `BlobIndexer`. ## Provider Writer Sessions `FsProviderWriter` supports [](#ProviderWriterSession) for atomic writes. See [](index.md), section **Provider Writer Sessions**, for the general concept. When using the file-system adapter, a session writes all changes to a temporary `.sessions/{session_id}/` directory. Only when [](#ProviderWriterSession.commit) is called, the files are moved to the target storage directory atomically. ## Single provider or multiple providers By design, the storage protocols are able to load and save data belonging to several providers: ```python provider_reader = storage_reader.get_provider_reader(parse_provider_code("INSEE")) provider_reader2 = storage_reader.get_provider_reader(parse_provider_code("IMF")) ``` However, by design also, fetchers are dedicated to a single provider. For convenience during fetcher development, a storage directory can be configured to handle a single provider (rather than requiring a top-level directory per provider). This avoids the extra directory nesting when working on one fetcher at a time. The file-system adapter reads and writes data from/to a directory, which can be used in 2 different modes, according to the `single_provider` boolean parameter of the storage URI. With `single_provider=false` (the default), the adapter works in multiple provider mode. The path of the storage URI will be used as a base directory containing one top-level directory per provider. With `single_provider=true`, the adapter works in single-provider mode. The path of the storage URI will be used as a directory containing data of a single provider only (the same as the top-level directories of the multiple provider mode). Even when being used in the single-provider mode, the provider codes must be given to the methods of the storage protocols in order to respect the common interface. Example for multiple provider mode: ```python from dbnomics_toolbox.model.provider_metadata import ProviderMetadata from dbnomics_toolbox.model.identifiers import parse_provider_code from dbnomics_toolbox.storage.adapters.factories import create_storage_adapter_from_uri from dbnomics_toolbox.storage.uris import Uri multiple_provider_storage = create_storage_adapter_from_uri(Uri.parse("filesystem:///multiple_provider_data")) multiple_provider_writer = multiple_provider_storage.get_writer().get_provider_writer(parse_provider_code("EUROSTAT")) multiple_provider_writer.write_provider_metadata( ProviderMetadata.create( code="EUROSTAT", name="Eurostat", # Cf https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 region="EU", terms_of_use="https://ec.europa.eu/eurostat/web/main/help/copyright-notice", website="https://ec.europa.eu/eurostat", ) ) multiple_provider_storage.get_writer().get_provider_writer(parse_provider_code("INSEE")).write_provider_metadata( ProviderMetadata.create( code="INSEE", name="Institut national de la statistique et des études économiques", # Cf https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 region="FR", terms_of_use="https://www.insee.fr/fr/information/2381863", website="https://www.insee.fr/", ) ) ``` ```bash $ tree multiple_provider_data multiple_provider_data ├── eurostat-json-data │   └── provider.json └── insee-json-data └── provider.json ``` Example for single-provider mode: ```python from dbnomics_toolbox.model.provider_metadata import ProviderMetadata from dbnomics_toolbox.model.identifiers import parse_provider_code from dbnomics_toolbox.storage.adapters.factories import create_storage_adapter_from_uri from dbnomics_toolbox.storage.uris import Uri single_provider_storage = create_storage_adapter_from_uri( Uri.parse("filesystem:///single_provider_data?single_provider=true") ) single_provider_storage.get_writer().get_provider_writer(parse_provider_code("INSEE")).write_provider_metadata( ProviderMetadata.create( code="INSEE", name="Institut national de la statistique et des études économiques", # Cf https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 region="FR", terms_of_use="https://www.insee.fr/fr/information/2381863", website="https://www.insee.fr/", ) ) ``` ```bash $ tree single_provider_data single_provider_data └── provider.json ``` In a fetcher, the convert CLI expects a directory or a storage URI as its second command-line argument. If it's a directory, it is turned into a storage URI like `filesystem:{directory}?single_provider=true`.