We maintain a small Python tool aimed at that job: page-image-downloader (MIT, standard library only, Python 3.9+). This post is about the problem shape and a pattern you can run locally. A hosted runner is optional later — start with the free repo.

Page images are not the same problem as a CSV

If someone already handed you a spreadsheet of direct image URLs, use a list downloader. We publish one of those too: bulk-image-downloader. A page is different: you must discover candidates from HTML first, resolve them against the page URL, then download.

What page-image-downloader collects from HTML (at commit 263a6f6):

• img via lazy-load attrs (data-src, data-lazy-src, data-original) before src
• largest srcset / data-srcset / data-lazy-srcset candidate (highest w, else highest x)
• each <picture><source srcset="..."> largest candidate
• og:image / twitter:image (and a few related meta keys)
• optional linked files with --links pdf (or other extensions)

Relative URLs are resolved against <base href> and the page URL. The same absolute URL is fetched once per page. Whatever you collect, write a manifest with one row per candidate. Status beats vibes.

Minimal local run

Verified CLI at commit 263a6f6 (python3 page_image_downloader.py --help): URLs are positional (there is no --url). Output directory is -o / --out (default images).

git clone https://github.com/fecklesspleb/page-image-downloader
cd page-image-downloader
python3 page_image_downloader.py \
  "https://example.com/" \
  -o page-images

Or pass a text file of URLs with -i / --input (one URL per line, # comments allowed).

What that run writes:

• Folder page-images/<page-host>/ with files named <stem>-<first 10 hex of sha256>.<ext> (extension from sniffed bytes, not the URL)
• page-images/manifest.csv (always)
• A JSON summary printed to stdout (not a separate .json file). Progress lines go to stderr (-q hides them).

manifest.csv columns (exact header):

page_url, image_url, final_url, found_in, alt, status, file, content_type, bytes, width, height, sha256, error

status values the tool writes: saved, duplicate, too_small, too_narrow, too_large, not_an_image, html_instead_of_file, robots_disallowed, error.

Exit code is 1 only when nothing was saved and at least one request failed.

Pattern: resolve, fetch, sniff, record

Even if you write your own script, keep this shape. The snippets below are illustrative and use only the standard library (Python 3.9+; Optional keeps the annotation 3.9-safe).

from urllib.parse import urljoin, urlparse
from urllib.request import Request, urlopen
import hashlib
from pathlib import Path
from typing import Optional

UA = "page-image-example/0.1 (+https://github.com/fecklesspleb/page-image-downloader)"

def fetch(url: str, timeout: float = 30) -> bytes:
    req = Request(url, headers={"User-Agent": UA})
    with urlopen(req, timeout=timeout) as resp:
        return resp.read()

def looks_like_image(data: bytes) -> Optional[str]:
    if data.startswith(b"\x89PNG\r\n\x1a\n"):
        return "png"
    if data[:3] == b"\xff\xd8\xff":
        return "jpg"
    if data[:4] == b"GIF8":
        return "gif"
    if data[:4] == b"RIFF" and data[8:12] == b"WEBP":
        return "webp"
    head = data[:200].lstrip().lower()
    if head.startswith(b"<svg") or b"<svg" in head:
        return "svg"
    return None

def save_unique(out_dir: Path, url: str, data: bytes) -> Path:
    kind = looks_like_image(data)
    if not kind:
        raise ValueError("not_an_image")
    digest = hashlib.sha256(data).hexdigest()[:10]
    host = urlparse(url).netloc or "unknown"
    path = out_dir / host
    path.mkdir(parents=True, exist_ok=True)
    dest = path / f"asset-{digest}.{kind}"
    if not dest.exists():
        dest.write_bytes(data)
    return dest

page = "https://example.com/articles/one"
candidate = "/static/hero.jpg"
absolute = urljoin(page, candidate)
print(absolute)

Running only the resolve lines prints:

https://example.com/static/hero.jpg

Sniffing the first bytes matters more than trusting Content-Type or a .jpg suffix. Soft 404 pages and interstitial HTML with Content-Type: text/html fail the sniff and should be recorded as not_an_image, not written as a corrupt JPEG that breaks the next pipeline step. (page-image-downloader does that when the bytes do not sniff as an image and the MIME type is not an image type either.)

Manifest columns worth keeping

At minimum — and these are the columns the tool actually writes:

With that table you can answer "how many assets did we actually get?" without opening the folder.

Politeness and rights

• Send a User-Agent that names the project and points at the repo.
• Cap concurrency; the tool defaults to a 0.5 s per-host delay (--delay).
• It checks robots.txt unless you pass --ignore-robots (only for sites you run).
• Only download pages and assets you have a right to use. "It was on a public page" is not automatically a license to redistribute.

Free repo first

The free GitHub project is the default. Use it in CI, on a laptop, or inside a container with no API keys. If you later want the same workflow on a hosted crawler platform, treat that as an optional convenience for people who already live there — not as the primary link in a how-to.

Try it: fecklesspleb/page-image-downloader. For CSV/text URL lists instead of HTML discovery, use bulk-image-downloader.

Disclosure

We built and maintain page-image-downloader (and the sibling bulk-image-downloader for CSV lists). This article and the tool/README were written with AI assistance and reviewed before publishing.