Skip to content

Accept prefilled Project Urls and add a local preprocessor - #36

Merged
HangYeung1 merged 1 commit into
mainfrom
accept-project-urls
Oct 4, 2026
Merged

HangYeung1 merged 1 commit into
mainfrom
accept-project-urls

Conversation

@pradhamk

@pradhamk pradhamk commented Oct 4, 2026

Copy link
Copy Markdown
Member

Summary

  • POST /datasets now accepts a Project Url column. Rows with a value keep it (it must be an https devpost.com link, or the row fails as INVALID_DEVPOST_URL), and only blank rows are resolved against Devpost. If every row is filled in, the server makes no Devpost requests.
  • New app/preprocess.py (uv run --group preprocess python -m app.preprocess export.csv -o resolved.csv) resolves submission URLs locally in Playwright Chromium. It keeps a Devpost login in a browser profile (--login, --profile), can show the browser so you can get past Cloudflare (--headed), and on a rerun of its own output retries only the blank rows. It needs no MDREDD_API_TOKEN.
  • devpost.resolve_all takes the cookie and concurrency as arguments, so the script runs without the server's settings.
  • Playwright is in a separate preprocess dependency group, so the Docker image (uv sync --no-dev) doesn't install it.
  • README documents the preprocessor and the new DEVPOST_NOT_PROJECT_PAGE code.

Notes

  • The server still resolves blank rows as a fallback. It doesn't yet require every row to be prefilled.
  • No automated tests cover the prefilled path.

Test plan

  • Run the preprocessor on a real export with --login and confirm it exits 0
  • Upload resolved.csv and confirm the server sends no Devpost requests
  • Upload the same file again and confirm it's accepted as a repeat with no changes

🤖 Generated with Claude Code

POST /datasets keeps any Project Url already filled in and only resolves
blank rows against Devpost. app/preprocess.py fills that column locally
in Playwright Chromium, so the upload can skip Devpost entirely.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@HangYeung1
HangYeung1 merged commit f50ec64 into main Oct 4, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants