File Downloads

The download tool saves known document and data files from a URL list. It is intentionally sequential and small: it does not crawl pages or open concurrent requests.

Supported sources

download_files and scrape download accept a text file containing one URL per line. Blank lines and lines starting with # are ignored.

Only HTTP and HTTPS URLs are downloaded. Other URL schemes are skipped.

Target files

The supported target extensions are:

  • .csv

  • .pdf

  • .docx

  • .xlsx

  • .pptx

scrape-smith first checks the URL path and Content-Disposition filename. If no filename is available, it can infer a target extension from common Content-Type values.

Output directory

When no output directory is supplied, scrape-smith creates a directory based on the URL list filename:

urls.txt -> urls-downloads/

If that directory already exists, a numeric suffix is added, such as urls-downloads-2.

Filenames

Downloaded files keep their original filename when the URL path or response headers provide one. Existing files are not overwritten; a numeric suffix is added before the extension.

Warning

Treat downloaded files as untrusted. Scan them before opening, and do not open files blindly; documents and spreadsheets can contain harmful content.

If no filename is available, scrape-smith uses a fixed-width epoch-millisecond filename with the detected extension:

1700000000123.pdf

Request behavior

Downloads run one at a time. By default, the CLI waits one second between requests. Use --delay to choose a longer or shorter delay.

The CLI prints important events to stdout: the start, output directory, safety warning, each downloaded/skipped/failed URL, and the final summary. If any download fails, scrape-smith finishes the remaining URLs and exits with status code 1.