ferc_xbrl_extractor.xbrl

XBRL extractor.

Module Contents

Functions

extract(→ ExtractOutput)

Extract fact tables from instance documents as Pandas dataframes.

table_data_from_instances(→ tuple[dict[str, ...)

Turn FactTables into Dataframes by ingesting facts from instances.

process_batch(→ tuple[dict[str, pandas.DataFrame], ...)

Extract data from one batch of instances.

process_instance(→ dict[str, pandas.DataFrame])

Extract data from a single XBRL filing.

get_fact_tables(→ dict[str, ...)

Parse taxonomy from URL.

Attributes

ExtractOutput

ferc_xbrl_extractor.xbrl.ExtractOutput[source]
ferc_xbrl_extractor.xbrl.extract(instance_path: pathlib.Path, taxonomy_source: pathlib.Path | io.BytesIO, form_number: int, db_uri: str, entry_point: pathlib.Path | None = None, datapackage_path: pathlib.Path | None = None, metadata_path: pathlib.Path | None = None, requested_tables: set[str] | None = None, instance_pattern: str = '', workers: int | None = None, batch_size: int | None = None) → ExtractOutput[source]

Extract fact tables from instance documents as Pandas dataframes.

Parameters:

  • instance_path – where to find instance documents.

  • taxonomy_source – either a URL/path to taxonomy or in memory archive of taxonomy.

  • form_number – the FERC form number (1, 2, 6, 60, 714).

  • db_uri – the location of the database we are writing this form out to.

  • entry_point – if taxonomy_path is a ZIP archive, this is a Path to the relevant taxonomy file within the archive. Otherwise, this should be None.

  • datapackage_path – where to write a Frictionless datapackage descriptor to, if at all. Defaults to None, i.e., do not write one.

  • metadata_path – where to write XBRL metadata to, if at all. Defaults to None, i.e., do not write one.

  • requested_tables – only attempt to ingest data for these tables.

  • instance_pattern – only ingest data for instances matching this regex. Defaults to empty string which matches all.

  • workers – max number of workers to use.

  • batch_size – max number of instances to parse for each worker.

ferc_xbrl_extractor.xbrl.table_data_from_instances(instance_builders: list[ferc_xbrl_extractor.instance.InstanceBuilder], table_defs: dict[str, ferc_xbrl_extractor.datapackage.FactTable], batch_size: int | None = None, workers: int | None = None) → tuple[dict[str, pandas.DataFrame], dict[str, list]][source]

Turn FactTables into Dataframes by ingesting facts from instances.

To handle lots of instances, we split the instances into batches.

Parameters:

  • instances – A list of Instance objects used for parsing XBRL filings.

  • table_defs – the tables defined in the taxonomy that we will match facts to.

  • archive_path – Path to taxonomy entry point within archive. If not None, then taxonomy should be a path to zipfile, not a URL.

  • batch_size – Number of filings to process before writing to DB.

  • workers – Number of threads to create for parsing filings.

ferc_xbrl_extractor.xbrl.process_batch(instance_builders: collections.abc.Iterable[ferc_xbrl_extractor.instance.InstanceBuilder], table_defs: dict[str, ferc_xbrl_extractor.datapackage.FactTable]) → tuple[dict[str, pandas.DataFrame], set[str]][source]

Extract data from one batch of instances.

Splitting instances into batches significantly improves multiprocessing performance. This is done explicitly rather than using ProcessPoolExecutor’s chunk_size option so dataframes within the batch can be concatenated prior to returning to the parent process.

Parameters:

  • instance_builders – Iterator of instance builders which can be parsed into instances.

  • table_defs – Dictionary mapping table names to FactTable objects describing table structure.

ferc_xbrl_extractor.xbrl.process_instance(instance: ferc_xbrl_extractor.instance.Instance, table_defs: dict[str, ferc_xbrl_extractor.datapackage.FactTable]) → dict[str, pandas.DataFrame][source]

Extract data from a single XBRL filing.

This function will use the Instance object to parse a single XBRL filing. It then iterates through requested tables and populates them with data extracted from the filing.

Parameters:

  • instance – A single Instance object that represents an XBRL filing.

  • table_defs – Dictionary mapping table names to FactTable objects describing table structure.

ferc_xbrl_extractor.xbrl.get_fact_tables(taxonomy_source: pathlib.Path | io.BytesIO, form_number: int, db_uri: str, entry_point: pathlib.Path | None = None, filter_tables: set[str] | None = None, datapackage_path: str | None = None, metadata_path: str | None = None) → dict[str, ferc_xbrl_extractor.datapackage.FactTable][source]

Parse taxonomy from URL.

XBRL defines ‘fact tables’ that groups related facts. These fact tables are used directly to create the tables that will make up the output SQLite database, and their structure. The output of this function is a dictionary that maps table names to ‘FactTable’ objects that specify their structure.

This function will also output a json file containing a Frictionless Data-Package descriptor describing the output database if requested.

Parameters:

  • taxonomy_path – URL of taxonomy.

  • form_number – FERC Form number (can be 1, 2, 6, 60, 714).

  • db_uri – URI of database used for constructing datapackage descriptor.

  • archive_path – Path to taxonomy entry point within archive. If not None, then taxonomy should be a path to zipfile, not a URL.

  • filter_tables – Optionally specify the set of tables to extract. If None, all possible tables will be extracted.

  • datapackage_path – Create frictionless datapackage and write to specified path as JSON file. If path is None no datapackage descriptor will be saved.

  • metadata_path – Path to metadata json file to output taxonomy metadata.

Returns:

Dictionary mapping to table names to structure.