Building a Python Workflow for Downloading Internet Archive Files

A download can finish without giving you a usable file. A connection drops halfway through. A script saves an error response under a PDF filename. A folder fills with several versions of a document, and you cannot tell which one you wanted. A reliable Python workflow handles these problems before they become cleanup work.

For Internet Archive files, the process starts with metadata. You inspect what an item contains, select the files you need, download them with sensible retries, and check the saved bytes against the published checksum. Each step gives the next one a clear job. The result is a workflow you can run again without guessing what happened last time.

Start With an Item Identifier and a Clear Scope

An Internet Archive item is a container for files and descriptive metadata. One item might contain a scanned book, OCR text, thumbnails, and several generated formats. Another might contain audio recordings and cover images. Downloading everything is rarely the best starting point.

The item identifier is the value that follows /details/ in an item address. Pass that identifier to the metadata endpoint, then inspect the returned files array. File records commonly contain a name, format, size, source classification, and checksums. Treat optional fields as optional. An absent checksum should change your handling of a file rather than trigger an unexplained crash.

This workflow addresses downloadable item files. Reconstructing a website from Wayback Machine captures requires a different process. Restricted files also remain restricted. A retry loop cannot supply access rights or resolve a missing file.

Before writing code, decide what success means. For this example, success means downloading public PDF files whose metadata includes a size and MD5 checksum. Files without those fields will be skipped with a message. That strict policy keeps the meaning of “verified” consistent.

Read Metadata Before Spending Bandwidth

Start with Python 3.9 or newer and create a virtual environment. The example uses requests for HTTP and standard library modules for the remaining work.

python -m venv .venv
source .venv/bin/activate
python -m pip install requests

The activation command above is for a Linux or macOS shell. On Windows, activate the environment through the appropriate script inside its Scripts directory.

A metadata request uses https://archive.org/metadata/{identifier}. File downloads use https://archive.org/download/{identifier}/{filename}. Build those addresses with URL encoding. Spaces, punctuation, and directory separators need deliberate handling.

Inspect the available names before choosing a filter. A PDF suffix is useful when your goal is simply “get PDFs.” A stricter archival task may need an exact filename or a condition involving the source field. Generated files can be useful reading copies, while original uploads may be preferable for preservation work.

File Selection Choices and Their Tradeoffs

For a large item, print the candidate names and add their reported sizes before transferring anything. This small inspection step can prevent an unexpectedly large download. Available disk space also needs room for the temporary file used during replacement.

Choose a Browser When the Task Is Occasional

A script earns its keep when you repeat a selection rule, process several items, or need a record of successful transfers. An occasional download may not justify maintaining that code. An archive.org downloader offers a browser-based alternative for readers who prefer to work through a visual interface.

Choose the approach around the job. A browser can make a small manual task more comfortable. Python gives you explicit selection rules, bounded retries, and local verification that you can repeat. For the rest of this workflow, those repeatable checks are the priority.

Build Retries Around Temporary Failures

Retry connection failures, timeouts, rate limiting, and selected server errors. Stop promptly for permanent HTTP failures. A forbidden response needs investigation. A missing file may require refreshing metadata. Repeating either request indefinitely adds traffic without fixing the underlying problem.

The script below allows four attempts per operation. It waits longer after each failure and adds a little random delay. When a retryable response supplies Retry-After, it accepts either a delay in seconds or an HTTP date. If the requested wait exceeds five minutes, the script stops so a later scheduled run can try again.

The timeout tuple sets connection and read timeouts. It does not impose a total download deadline. A large transfer can continue while data keeps arriving. Failed downloads restart from the beginning in this example, which keeps the verification logic straightforward.

Download Into a Temporary File and Verify It

Save the following as download_archive.py. It prints the available files, selects PDFs with verification metadata, and processes them sequentially. Local filenames use a hash of the remote filename. This avoids interpreting remote paths as local directories and avoids collisions caused by flattening different remote directories.

import hashlib
import random
import sys
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from pathlib import Path
from urllib.parse import quote

import requests

RETRYABLE = {429, 500, 502, 503, 504}
ATTEMPTS = 4
TIMEOUT = (10, 60)


def retry_delay(error, attempt):
    response = getattr(error, "response", None)
    value = response.headers.get("Retry-After") if response is not None else None
    if value:
        try:
            delay = max(0, int(value))
        except ValueError:
            try:
                when = parsedate_to_datetime(value)
                delay = max(
                    0, (when - datetime.now(timezone.utc)).total_seconds()
                )
            except (TypeError, ValueError, OverflowError):
                delay = None
        if delay is not None:
            if delay > 300:
                raise RuntimeError("Server requested a longer wait") from error
            return delay + random.uniform(0, 1)
    return 2 ** attempt + random.uniform(0, 1)


def retry(operation):
    for attempt in range(ATTEMPTS):
        try:
            return operation()
        except (requests.ConnectionError, requests.Timeout,
                requests.exceptions.ChunkedEncodingError,
                requests.HTTPError, ValueError) as error:
            if isinstance(error, requests.HTTPError):
                if error.response.status_code not in RETRYABLE:
                    raise
            if attempt == ATTEMPTS - 1:
                raise
            delay = retry_delay(error, attempt)
            print(f"Retrying in {delay:.1f}s: {error}")
            time.sleep(delay)


def matches(path, expected_size, expected_md5):
    if not path.is_file() or path.stat().st_size != expected_size:
        return False
    digest = hashlib.md5(usedforsecurity=False)
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest() == expected_md5.lower()


def download(session, identifier, record, folder):
    name = record["name"]
    expected_size = int(record["size"])
    expected_md5 = record["md5"]
    local_name = hashlib.sha256(name.encode("utf-8")).hexdigest() + ".pdf"
    target = folder / local_name
    partial = target.with_suffix(".pdf.part")

    if matches(target, expected_size, expected_md5):
        print(f"Already verified: {name}")
        return

    url = (
        "https://archive.org/download/"
        + quote(identifier, safe="")
        + "/"
        + quote(name, safe="/")
    )

    def transfer():
        try:
            with session.get(url, stream=True, timeout=TIMEOUT) as response:
                response.raise_for_status()
                with partial.open("wb") as handle:
                    for chunk in response.iter_content(1024 * 1024):
                        if chunk:
                            handle.write(chunk)
            if not matches(partial, expected_size, expected_md5):
                raise ValueError(f"Verification failed: {name}")
            partial.replace(target)
        finally:
            partial.unlink(missing_ok=True)

    retry(transfer)
    print(f"Verified: {name} -> {target.name}")


def main(identifier):
    item_key = hashlib.sha256(identifier.encode("utf-8")).hexdigest()
    folder = Path("downloads") / item_key
    folder.mkdir(parents=True, exist_ok=True)

    with requests.Session() as session:
        session.headers["User-Agent"] = "ArchiveFileWorkflow/1.0"

        def fetch_metadata():
            url = "https://archive.org/metadata/" + quote(identifier, safe="")
            with session.get(url, timeout=TIMEOUT) as response:
                response.raise_for_status()
                return response.json()

        metadata = retry(fetch_metadata)
        files = metadata.get("files") or []
        if not files:
            raise RuntimeError("No files returned; inspect item availability")

        selected = []
        for record in files:
            name = record.get("name", "")
            print(name, record.get("format", ""), record.get("size", "?"))
            if name.lower().endswith(".pdf"):
                if record.get("md5") and record.get("size") is not None:
                    selected.append(record)
                else:
                    print(f"Skipping without verification metadata: {name}")

        if not selected:
            raise RuntimeError("No verifiable PDFs matched")

        for record in selected:
            download(session, identifier, record, folder)


if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("Usage: python download_archive.py ITEM_IDENTIFIER")
    main(sys.argv[1])

Run it with an actual item identifier:

python download_archive.py ITEM_IDENTIFIER

The .part file stays beside its final destination. After size and checksum verification, replace() publishes the completed file at the target path. Ordinary transfer failures remove the temporary file. An abrupt process termination can leave one behind, but the next attempt opens it in write mode and starts over.

The script intentionally allows one process per item directory. Concurrent runs targeting the same files would compete for the same temporary paths. Add a process lock or separate working directories before introducing that kind of concurrency.

Understand What the Checksum Actually Proves

A matching MD5 digest helps confirm that the downloaded bytes match the digest supplied in the item metadata. The size check provides another useful signal. Neither check establishes that a document is trustworthy, safe to execute, or correctly described by its uploader.

MD5 also has known collision weaknesses. Python’s hashing documentation explains the available algorithms and their security limitations. Here, MD5 serves as a compatibility check against a published file digest. The usedforsecurity=False argument states that purpose explicitly, though algorithm availability can still depend on the Python build.

When verification fails repeatedly, keep the file out of your completed collection. Refresh the metadata and investigate whether the item changed during the transfer. Do not update the expected checksum merely to match the local file. That would remove the independent comparison that makes the check useful.

Turn a Single Run Into Repeatable Automation

Once one item works, automation becomes a matter of recording inputs and outcomes. Keep the identifier, remote filename, expected size, expected checksum, local destination, and completion time in a manifest. This also makes the hashed local filenames easier to trace back to their source.

A scheduled workflow can follow this sequence:

1. Fetch current metadata and record the selection rule.
2. Check existing files against the current expected values.
3. Download missing or invalid files using limited retries.
4. Record verified results and report failures for review.

Use an absolute interpreter path and working directory in a scheduler. Capture standard output and standard error. The example exits unsuccessfully when an exception reaches the command line, which gives a scheduler a useful failure signal. It stops at the first failed file; a larger batch runner can catch errors per file and produce a final failure report.

Keep transfers sequential until you have a measured reason to add concurrency. For large files, resuming partial downloads may eventually be worthwhile. That requires checking range support and validating the complete assembled file. Appending bytes blindly can produce a corrupt result that looks complete.

Make Every Completed File Earn Its Place

A dependable download folder should have a clear meaning: its accepted files passed the checks you defined. Metadata tells your script what to request. Selection rules keep the scope manageable. Retries handle temporary trouble. Temporary paths and checksum validation keep incomplete transfers from quietly becoming finished work.

Begin with one public item and inspect its file records. Run the script twice to confirm that verified files are skipped. Then adapt the selection rule and add a manifest for your own project. With those habits in place, retrieving Internet Archive files becomes a process you can inspect, repeat, and trust within its stated limits.