Python API

The public Python API currently lives under scrape_smith.tools.

Table helpers

Extract tabular data from HTML documents.

class scrape_smith.tools.tables.HtmlTable(caption: str | None, headers: list[str], rows: list[list[str]])

Bases: object

A table extracted from an HTML document.

caption: str | None

headers: list[str]

rows: list[list[str]]

to_dict() → dict[str, object]

scrape_smith.tools.tables.extract_tables(target: str | Path) → list[HtmlTable]

Extract tables from a local HTML file path or HTTP(S) URL.

scrape_smith.tools.tables.extract_tables_from_html(html: str) → list[HtmlTable]

Extract tables from an HTML string.

scrape_smith.tools.tables.read_html(target: str | Path) → str

Read HTML from a local path or HTTP(S) URL.

List helpers

Extract list data from HTML documents.

class scrape_smith.tools.lists.HtmlDefinitionList(items: list[dict[str, str]])

Bases: object

A definition list extracted from an HTML document.

items: list[dict[str, str]]

to_dict() → dict[str, object]

class scrape_smith.tools.lists.HtmlList(tag: str, items: list[str])

Bases: object

An ordered or unordered list extracted from an HTML document.

items: list[str]

tag: str

to_dict() → dict[str, object]

scrape_smith.tools.lists.extract_lists(target: str | Path) → list[HtmlList | HtmlDefinitionList]

Extract lists from a local HTML file path or HTTP(S) URL.

scrape_smith.tools.lists.extract_lists_from_html(html: str) → list[HtmlList | HtmlDefinitionList]

Extract lists from an HTML string.

Content helpers

Extract body content from HTML documents as JSONL-ready records.

scrape_smith.tools.content.extract_content_records(target: str | Path) → list[dict[str, str]]

Extract body text records from a local HTML file path or HTTP(S) URL.

scrape_smith.tools.content.extract_content_records_from_html(html: str) → list[dict[str, str]]

Extract JSONL-ready body text records from an HTML string.

scrape_smith.tools.content.write_jsonl(records: list[dict[str, str]], output_file: TextIO) → None

Write content records as newline-delimited JSON.

Extract helpers (three-in-one)

Extract all content, tables, and lists from an HTML document as JSONL records.

scrape_smith.tools.extract.extract_all(target: str | Path) → list[dict[str, str]]

Extract all records from a local HTML file path or HTTP(S) URL.

scrape_smith.tools.extract.extract_all_from_html(html: str) → list[dict[str, str]]

Extract all records from an HTML string.

Returns a list of {"type": ..., "data": ...} dicts in document order. type is one of "content", "table", or "list". data is a JSON string with the record payload.

scrape_smith.tools.extract.write_jsonl(records: list[dict[str, str]], output_file: TextIO) → None

Write extract records as newline-delimited JSON.

Download helpers

Download file URLs into a local directory.

class scrape_smith.tools.downloads.DownloadResult(url: str, status: str, path: Path | None = None, reason: str | None = None)

Bases: object

Result for one URL from a download run.

path: Path | None = None

reason: str | None = None

status: str

url: str

class scrape_smith.tools.downloads.DownloadSummary(output_dir: Path, results: list[DownloadResult])

Bases: object

Summary for a file download run.

property downloaded_count: int

property failed_count: int

output_dir: Path

results: list[DownloadResult]

property skipped_count: int

scrape_smith.tools.downloads.available_directory(path: Path) → Path

Return path or a suffixed sibling that does not exist.

scrape_smith.tools.downloads.available_path(path: Path) → Path

Return path or a suffixed sibling that does not exist.

scrape_smith.tools.downloads.content_type_extension(content_type: str | None) → str | None

Return a supported target extension for a Content-Type value.

scrape_smith.tools.downloads.default_download_dir(url_list_path: str | Path) → Path

Return a collision-safe output directory for a URL list file.

scrape_smith.tools.downloads.download_files(url_list_path: str | Path, output_dir: str | Path | None = None, *, delay_seconds: float = 1.0, timeout_seconds: int = 30, reporter: Callable[[str], None] | None = None) → DownloadSummary

Download target file URLs listed in a text file.

Blank lines and lines starting with # are ignored. Downloads run sequentially with a delay between requests by default.

scrape_smith.tools.downloads.download_url(url: str, output_dir: str | Path, *, timeout_seconds: int = 30) → DownloadResult

Download one URL when it resolves to a supported target file.

scrape_smith.tools.downloads.fallback_filename(extension: str) → str

Return an epoch-based fallback filename with fixed-width milliseconds.

scrape_smith.tools.downloads.read_url_list(url_list_path: str | Path) → list[str]

Read non-empty, non-comment URL lines from a text file.

scrape_smith.tools.downloads.response_filename(content_disposition: str | None) → str | None

Extract a filename from a Content-Disposition header.

scrape_smith.tools.downloads.safe_filename(filename: str) → str

Return a filesystem-safe basename preserving the original extension.

scrape_smith.tools.downloads.target_extension(filename: str | None) → str | None

Return a supported target extension for a filename.

scrape_smith.tools.downloads.url_filename(url: str) → str | None

Extract a filename from a URL path.

CLI helpers

The CLI module is importable for tests and advanced integrations, but most callers should use the scrape command directly.

Command line interface for scrape-smith.

scrape_smith.cli.available_output_path(stem: str, suffix: str) → Path

scrape_smith.cli.build_parser() → ArgumentParser

scrape_smith.cli.default_content_output_path(target: str) → Path

scrape_smith.cli.default_lists_output_path(target: str, output_format: str) → Path

scrape_smith.cli.default_output_path(target: str, output_format: str) → Path

scrape_smith.cli.default_three_output_path(target: str) → Path

scrape_smith.cli.main(argv: Sequence[str] | None = None) → int

scrape_smith.cli.pluralize(word: str, count: int) → str

scrape_smith.cli.run_content(args: Namespace) → int

scrape_smith.cli.run_download(args: Namespace) → int

scrape_smith.cli.run_list(args: Namespace) → int

scrape_smith.cli.run_table(args: Namespace) → int

scrape_smith.cli.run_three(args: Namespace) → int

scrape_smith.cli.source_slug(target: str) → str

scrape_smith.cli.write_csv(tables: list[HtmlTable], output_file: TextIO) → None

scrape_smith.cli.write_json(tables: list[HtmlTable], output_file: TextIO) → None

scrape_smith.cli.write_lists(lists: list[HtmlList | HtmlDefinitionList], output_format: str, output_file: TextIO) → None

scrape_smith.cli.write_tables(tables: list[HtmlTable], output_format: str, output_file: TextIO) → None