Skip to content
TzZek
grab
Public

About

Download media and docs, compress to fit size limits, transcribe, and summarize.

Topics

climediaobsidianpdfpythonsummarizationtranscriptionyt-dlp

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags

Latest commit

 

History

17 Commits

Folders and files

NameName
Last commit message
Last commit date
src/grab
src/grab
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
pyproject.toml
pyproject.toml
 
 

Repository files navigation

grab

Download, compress, transcribe, and summarize media. Modular Python CLI — each module is under 300 lines and independently runnable.

Install

git clone https://github.com/TzZek/grab.git
cd grab
uv venv && source .venv/bin/activate
uv pip install -e .

Optional dependencies

# Speech-to-text (local transcription)
uv pip install -e ".[transcribe]"    # installs faster-whisper

# LLM summarization (local via Ollama)
uv pip install -e ".[summarize]"     # installs ollama python client

# Instagram stories/highlights photo support
uv pip install -e ".[stories]"       # installs gallery-dl

# Everything
uv pip install -e ".[all]"

Model setup

Summarization model (Ollama)

grab uses Ollama for local LLM summarization. Install Ollama, then pull a model:

# Pick a model for your hardware:
ollama pull llama3.2          # ~2GB  — lightweight, works anywhere (default fallback)
ollama pull qwen3.5:9b       # ~7GB  — good balance for most machines
ollama pull qwen3.5:35b      # ~24GB — MoE, only ~3B active, beats models 7x its size
ollama pull qwen3.5:122b     # ~81GB — MoE, ~10B active, best quality (needs ~75GB RAM)

Set your default model:

grab config set summarize_model qwen3.5:9b    # or whichever you pulled

Why Qwen3.5? The MoE variants (35B, 122B) activate only a fraction of their parameters per token, giving high quality at lower compute cost. The default prompt uses /no_think to disable chain-of-thought reasoning, which is unnecessary for summarization and would only slow things down. If no model is configured, grab falls back to llama3.2.

Other LLM backends — if you don't want to run models locally:

# Claude API
uv pip install anthropic
grab config set summarize_backend anthropic
# Set ANTHROPIC_API_KEY in your environment

# OpenAI API
uv pip install openai
grab config set summarize_backend openai
# Set OPENAI_API_KEY in your environment

# Any OpenAI-compatible server (llama.cpp, vLLM, etc.)
uv pip install openai
grab config set summarize_backend openai-compatible
grab config set summarize_api_base http://localhost:8080/v1

Transcription model

Transcription tries free subtitle extraction via yt-dlp first. If no subtitles exist, it falls back to a local whisper model.

# Install faster-whisper (default backend)
uv pip install faster-whisper

# Set whisper model size (trades speed for accuracy)
grab config set transcribe_model base      # fast, good enough for clear audio
grab config set transcribe_model small     # better accuracy
grab config set transcribe_model medium    # good balance
grab config set transcribe_model large-v3  # best accuracy, slower

Transcription backends:

Backend Install Notes
ytdlp-subs (included) Extracts existing YouTube subtitles — free, fast, tried first
faster-whisper pip install faster-whisper Default model backend, fast on CPU
whisper pip install openai-whisper Original OpenAI whisper
whisper.cpp binary install Via subprocess, set WHISPER_CPP_BIN env var
mlx-whisper pip install mlx-whisper Apple Silicon optimized

To change the backend:

grab config set transcribe_backend faster-whisper   # default
grab config set transcribe_backend whisper          # original openai-whisper

Usage

Download & compress

# Download a video
grab "https://youtube.com/watch?v=xyz"

# Download and compress for Discord (10MB)
grab "https://instagram.com/reel/abc123" -p discord

# Custom size target in MB
grab "https://tiktok.com/@user/video/123" -p 8

# Audio only
grab "https://youtube.com/watch?v=xyz" -a

# Convert to GIF
grab "https://twitter.com/user/status/123" --gif

# GIF from local file
grab gif video.mp4 --fps 10 --width 320

# Batch download
grab --batch urls.txt -p discord

JSON output keeps path as the saved file and also includes folder_path, so you can choose whether to open the file itself or just its containing folder. Related file fields such as text_path, srt_path, and json_path get matching *_folder_path entries.

Supported platforms

URLs from these platforms route straight to the media pipeline (cobalt where it helps, yt-dlp otherwise):

YouTube, Instagram, Twitter/X, TikTok, Facebook (incl. fb.watch), Reddit (incl. redd.it), Pinterest, Snapchat, Tumblr, Bluesky, Twitch, Kick, Vimeo, Dailymotion, Rumble, Streamable, Loom, Bilibili, VK, OK.ru, Rutube, Newgrounds, Xiaohongshu, SoundCloud, Bandcamp.

Anything else is classified by file extension or Content-Type (image, PDF, podcast feed) and falls back to the article pipeline for HTML pages.

Twitch and Kick VODs

Finished broadcasts download like any other video: pass the VOD link (twitch.tv/videos/<id>, kick.com/<channel>/videos/<uuid>) or a clip link. Kick sits behind Cloudflare bot protection, so grab installs yt-dlp's curl-cffi extra and impersonates a browser for kick.com URLs. Bare channel URLs (twitch.tv/<channel>, kick.com/<channel>) point at the live stream, not a VOD, and are rejected with a hint. Sub-only VODs use the same cookie plumbing as other platforms: grab cobalt cookies <browser> picks up your Twitch and Kick sessions.

VODs are long; grab <vod> -a downloads the audio-only stream (Twitch exposes one directly) instead of pulling hours of video, which pairs well with --transcribe/--summarize.

Images vs videos vs GIFs

grab types every downloaded file and reports it as media_type in the JSON output:

  • image: static files (jpg, png, single-frame gif, webp, ...). Resized and converted with ImageMagick.
  • gif: animated images (multi-frame gif, animated webp/apng). Animated GIFs are compressed with ffmpeg palette generation and stay GIFs; presets never silently turn them into webp or mp4.
  • video: anything with a real video stream. Compressed with two-pass x264.
  • audio: audio-only files, including mp3s with embedded cover art.

Twitter/X serves its "GIFs" as looping mp4s; cobalt converts them back to real .gif files on download. Gallery and carousel posts can mix photos, videos, and gifs; each item is typed and processed independently.

Direct images

Direct image URLs (jpg/jpeg/png/webp/gif/heic/avif/bmp/tiff/svg) get their own pipeline. They are downloaded and optionally converted to a different format via ImageMagick.

# Save as-is
grab 'https://example.com/photo.webp'

# Convert format on download
grab 'https://example.com/photo.webp' --convert jpg
grab 'https://example.com/anim.gif'   --convert png

# Local files: same flag works
grab image photo.webp --convert jpg
grab image photo.png  --resize 1920x1080
grab image photo.jpg  --preset discord     # squeeze under 10 MB

Detection works either by file extension in the URL path or by the image/* Content-Type returned from a HEAD request. URLs that resolve to HTML (such as a Shopify product page) still classify as articles; you need the underlying image CDN URL for direct conversion.

Article preservation

When grab detects an article URL, it can also push the page to archive.today and capture a PDF snapshot via headless Chromium. Both are off by default and fail soft (a missing browser or rate-limit will warn but not stop the run).

# Per-run flags
grab "https://example.com/post" --archive --pdf

# Or turn them on globally via config
grab config set archive_articles true
grab config set pdf_articles true

# Standalone use
grab archive submit "https://example.com/post"
grab archive pdf "https://example.com/post" -o snapshot.pdf

The PDF backend uses any installed Chromium-family browser (chromium, google-chrome, brave-browser, etc.). If you also pass --vault, the resulting Obsidian note picks up archive and pdf fields in its YAML frontmatter and adds the links to the source callout.

Transcribe

Extract speech-to-text from videos. Tries free subtitle extraction first, falls back to local whisper models.

# Transcribe a local file
grab transcribe video.mp4

# Choose backend and model
grab transcribe video.mp4 --backend faster-whisper --model large-v3

# Transcribe with a specific language
grab transcribe lecture.mp4 --language ja

# Skip subtitle extraction, go straight to whisper
grab transcribe video.mp4 --no-subs

Outputs: .srt subtitle file + .transcript.json with full text and segments.

Summarize

Summarize a transcript using a local or remote LLM.

# Summarize an SRT file
grab summarize video.srt

# Summarize a transcript JSON
grab summarize video.transcript.json

# Use a different backend
grab summarize video.srt --backend anthropic

# Use a specific model
grab summarize video.srt --backend ollama --model qwen3.5:9b

# Override model for a single run
grab "https://youtube.com/watch?v=xyz" --summarize --summarize-model qwen3:32b

Outputs a .summary.md with structured notes: TL;DR, Key Ideas, Details, Resources.

Full pipeline

Download + transcribe + summarize in one command:

# Download and transcribe
grab "https://youtube.com/watch?v=xyz" --transcribe

# Download, transcribe, and summarize (--summarize implies --transcribe)
grab "https://youtube.com/watch?v=xyz" --summarize

# Full pipeline with Obsidian integration (--vault implies --summarize --transcribe)
grab "https://youtube.com/watch?v=xyz" --vault

Obsidian integration

Save summaries directly to your Obsidian vault as searchable notes with YAML frontmatter.

# One-time setup: point grab at your vault
grab config set obsidian_vault ~/obsidian-git-sync
grab config set obsidian_folder reference/videos   # default

# Now --vault does everything: download → transcribe → summarize → vault note
grab "https://youtube.com/watch?v=xyz" --vault

Each --vault run creates two notes in your vault:

  1. Summary note — structured summary with YAML frontmatter, info callout, and a [[backlink]] to the transcript
  2. Transcript note — full transcript text, tagged transcript, backlinked from the summary

Notes include:

  • YAML frontmatter: tags (auto-generated from category + channel), title, author, source URL, date, duration
  • Info callout: channel, URL, duration at a glance
  • Structured summary: TL;DR → Key Ideas → Details → Resources
  • Transcript backlink: [[Video Title — Transcript]] so you can always reference the source
  • Clickable terminal link: click the note title in your terminal to open it in Obsidian

Example summary note:

---
tags:
  - video-note
  - education
  - 3blue1brown
type: video-note
title: "But what is a neural network? | Deep learning chapter 1"
author: "3Blue1Brown"
source: "https://www.youtube.com/watch?v=aircAruvnKk"
date: 2017-10-05
duration: "18m40s"
---

# But what is a neural network? | Deep learning chapter 1

> [!info] Source
> **Channel:** 3Blue1Brown
> **URL:** https://www.youtube.com/watch?v=aircAruvnKk
> **Duration:** 18m40s

**Full transcript:** [[But what is a neural network  Deep learning chapter 1 — Transcript]]

**TL;DR** Neural networks recognize patterns by passing data through
layers of neurons with adjustable weights and biases...

## Key Ideas
- ...

## Details
- ...

## Resources
- ...

The companion transcript note (Video Title — Transcript.md) contains the full text with its own frontmatter:

---
tags:
  - transcript
type: transcript
title: "But what is a neural network? | Deep learning chapter 1"
author: "3Blue1Brown"
source: "https://www.youtube.com/watch?v=aircAruvnKk"
---

# Transcript — But what is a neural network? | Deep learning chapter 1

This is a 3. It's sloppily written and rendered at an extremely low
resolution of 28x28 pixels, but your brain has no trouble recognizing
it as a 3...

Config

All settings with their defaults:

grab config show                          # show all current settings

Download settings

grab config set output_dir ~/Pictures/content       # default save directory
grab config set organize_by_type true                # subfolder per content type (default)
grab config set cobalt_api https://your-instance     # cobalt API URL
grab config set default_preset discord               # always compress for a preset
grab config set default_quality 1080                 # video quality
grab config set filename_template "{source}_{title}" # naming pattern
grab config set cookies_from_browser ""              # browser for cookies

Output layout

With organize_by_type = true (the default), output_dir is split by content kind:

~/Pictures/content/
  videos/        yt, ig reels, tt, x clips
  images/        direct image URLs
  articles/      text + summary
  documents/     PDFs
  podcasts/      audio + transcripts
  stories/
    ig_someuser_20260531_120015/    one folder per stories/highlights download
      001_xxx.mp4
      002_xxx.jpg

Set organize_by_type = false to keep everything flat. Passing -d/--dir on a single run always overrides the typed routing.

Transcription settings

grab config set transcribe_backend faster-whisper    # STT backend
grab config set transcribe_model base                # whisper model (base/small/medium/large-v3)
grab config set transcribe_language ""               # auto-detect, or set "en", "ja", etc.

Summarization settings

grab config set summarize_backend ollama             # LLM backend
grab config set summarize_model qwen3.5:9b           # Ollama model name
grab config set summarize_prompt ""                  # custom prompt (use {text} placeholder)
grab config set summarize_api_base http://localhost:8080/v1  # for openai-compatible
grab config set summarize_api_key ""                 # for openai-compatible

Obsidian settings

grab config set obsidian_vault ~/obsidian-git-sync   # vault path
grab config set obsidian_folder reference/videos     # subfolder for notes

Article preservation settings

grab config set archive_articles true                # auto-submit articles to archive.today
grab config set pdf_articles true                    # auto-save PDF snapshot of articles

Settings file: ~/.config/grab/config.toml

Profiles

Bundle common flag combinations into named profiles in your config file, then apply with --profile NAME.

# ~/.config/grab/config.toml

[profiles.research]
vault = true
archive = true
pdf = true

[profiles.discord]
preset = "discord"

[profiles.podcast]
vault = true
summarize = true
language = "en"
grab <url> --profile research      # vault + archive + pdf
grab <url> --profile discord       # compress to 10 MB
grab <url> --profile podcast --no-archive   # explicit flags always win

Profile values are additive: they fill in for flags you did not pass on the command line. Anything you pass explicitly (including --no-archive, --no-pdf, etc.) overrides the profile.

Valid profile keys mirror the long flag names (with - or _): vault, archive, pdf, summarize, transcribe, preset, quality, convert, audio_only, no_compress, gif, gif_fps, gif_width, keep, json, language, transcribe_backend, transcribe_model, summarize_backend, summarize_model.

Shell completion

grab supports tab completion via argcomplete (optional dep). Install and activate:

uv pip install -e ".[completion]"

# Per-user global activation (one-time setup)
activate-global-python-argcomplete --user

# Or, eval into one shell session at a time
eval "$(register-python-argcomplete grab)"

After that, grab --<TAB> lists every flag, grab archive <TAB> shows subcommands, grab <url> --convert <TAB> offers jpg/jpeg/png/webp, grab <url> -p <TAB> completes preset names, and so on.

Colored output

grab emits ANSI color codes on stderr when it detects a terminal. Action lines (ending in ...) are cyan, warnings are yellow, errors are red, and successful results (saved/downloaded/archived/extracted/summary) are green. Streaming progress is dim grey.

Color is disabled automatically when stderr is piped to a file or another process. To override:

grab <url> --no-color           # disable for one run
NO_COLOR=1 grab <url>           # disable globally (https://no-color.org)
FORCE_COLOR=1 grab <url> | tee  # force-enable when piping

Presets

Compress to a site's upload limit with -p/--preset. Run grab presets to list them in the terminal.

Preset Size
discord 10 MB
discord-nitro 50 MB
telegram 50 MB
email 25 MB
twitter 512 MB

Custom sizes work anywhere a preset name does. A bare number means megabytes; KB/MB/GB units and decimals are accepted:

grab <url> -p discord      # 10 MB target
grab <url> -p 8            # 8 MB
grab <url> -p 25MB         # units are case-insensitive (25mb, 25M, ...)
grab <url> -p 500KB
grab <url> -p 1.5GB

Typos are caught before the download starts, with a did-you-mean suggestion (-p discrod suggests discord). If a saved file comes out over 10 MB and no preset was in play, grab prints a one-line tip about -p so the flag is discoverable when it matters.

Modules

Each module works standalone:

python -m grab.download <url>                    # download → JSON
python -m grab.probe video.mp4                   # file metadata → JSON
python -m grab.compress file.mp4 --preset discord  # compress video
python -m grab.image photo.jpg --max-bytes 1048576 # optimize image
python -m grab.image photo.webp --convert jpg     # convert format
python -m grab.image https://x.com/img.webp -d .  # download an image URL
python -m grab.gif video.mp4 --fps 15 --width 480  # video → GIF
python -m grab.transcribe video.mp4              # speech → text
python -m grab.summarize transcript.srt          # text → summary
python -m grab.obsidian summary.md --vault ~/vault  # summary → Obsidian note
python -m grab.naming <url>                      # generate clean filename
python -m grab.presets                           # list all presets
python -m grab.cobalt status                     # cobalt container status
python -m grab.archive submit <url>              # push to archive.today
python -m grab.archive pdf <url> -o file.pdf     # capture PDF via headless Chromium
python -m grab.config show                       # show config

Cobalt integration

grab automatically spins up a cobalt Docker container when downloading from platforms where yt-dlp needs authentication (Instagram, Twitter/X, TikTok, etc.). The container stops automatically when grab exits.

# First time: pull the cobalt image
grab cobalt pull

# That's it — grab handles the rest automatically.
# When you run:
grab "https://www.instagram.com/reel/abc123"
# grab will: start cobalt → download → stop cobalt

Instagram stories and highlights

Stories and highlights bypass cobalt (which has no reliable support for them) and use yt-dlp directly. Both require a logged-in Instagram session:

grab "https://www.instagram.com/stories/someuser/"            # all active stories
grab "https://www.instagram.com/stories/someuser/123456789/"  # a single story
grab "https://www.instagram.com/stories/highlights/12345/"    # all items in a highlight

Cookie source is auto-resolved in this order:

  1. ~/.config/grab/cookies.json if it has an instagram entry (preferred — same cookies cobalt uses, no keyring access required).
  2. cookies_from_browser as a fallback (grab config set cookies_from_browser chrome|brave|firefox|zen|...).

Each URL resolves to a gallery (multiple files), so output is a JSON array and standard compression/preset flags apply to every item.

Photos in stories require gallery-dl. yt-dlp downloads videos reliably but drops photo-only items (its format selector doesn't match images). Install the [stories] extra to enable a supplemental gallery-dl pass that fills them in:

uv pip install -e ".[stories]"

Without gallery-dl, story/highlight downloads will skip photos and log a hint.

Platform cookies

Instagram, Twitter, and Reddit require cookies even through cobalt. Create ~/.config/grab/cookies.json:

{
    "instagram": ["mid=...; ig_did=...; csrftoken=...; ds_user_id=...; sessionid=..."],
    "twitter": ["auth_token=...; ct0=..."],
    "reddit": ["client_id=...; client_secret=...; refresh_token=..."]
}

The cookies file is automatically mounted into the cobalt container when it starts.

Manual control

grab cobalt status    # check if container is running
grab cobalt start     # start manually (stays running)
grab cobalt stop      # stop and remove container
grab cobalt pull      # pull/update cobalt image

If you prefer a persistent cobalt instance, set the URL directly:

grab config set cobalt_api http://localhost:9000

When cobalt_api is set, grab uses that URL directly and skips auto container management.

Dependencies

  • Python 3.11+
  • ffmpeg/ffprobe — video compression and probing
  • ImageMagick — image processing
  • Docker — (optional) for auto cobalt container management

Installed automatically as Python dependencies:

  • yt-dlp — media downloading (bundled so it stays current; an outdated system yt-dlp gets blocked by site WAFs)
  • httpx — HTTP client (for cobalt API)

This project is not affiliated with cobalt.

License

MIT

About

Download media and docs, compress to fit size limits, transcribe, and summarize.

Topics

climediaobsidianpdfpythonsummarizationtranscriptionyt-dlp

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages