Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Errors are JSON: `{"detail":{"code":"..."}}`. Rate limits add `retry_after_ms` a

## Organizer flow

1. Upload the Devpost projects export with `POST /datasets` as multipart form data, file field `entities_csv`. The CSV must be UTF-8 with unique non-empty headers, and must have the columns `Project Title`, `Submission Url`, `M Hacks Main Track`, and `Sponsor Opt In Prizes`; a missing one is `422` `INVALID_COLUMNS` with `detail.names`. Rows with an empty `Submission Url` (drafts) are dropped, and at least two must remain. Devpost headers only the first team member, so cells past the last header are dropped and short rows are padded with empty strings. Row ids are the zero-based index among the kept rows. Cell values are returned later as strings.
1. Upload the Devpost projects export with `POST /datasets` as multipart form data, file field `entities_csv`. The CSV must be UTF-8 with unique header names, and must have the columns `Project Title`, `Submission Url`, `M Hacks Main Track`, and `Sponsor Opt In Prizes`; a missing one is `422` `INVALID_COLUMNS` with `detail.names`. Rows with an empty `Submission Url` (drafts) are dropped, and at least two must remain. Columns with a blank header, which spreadsheet apps add when they re-save the export, are dropped. Devpost headers only the first team member, so cells past the last header are dropped and short rows are padded with empty strings. Row ids are the zero-based index among the kept rows. Cell values are returned later as strings.

Before storing anything, the upload follows each `Submission Url` to the public page it redirects to and stores it on that row as a `Project Url` column, which is appended to the headers. Every hop must stay on `https` `devpost.com`. If any row does not resolve, nothing is stored and the upload is `502` `DEVPOST_UNRESOLVED` with `detail.failures`, one `{ "title", "submission_url", "code" }` per failed row. `code` is `INVALID_DEVPOST_URL`, `DEVPOST_LOGIN_REQUIRED`, `DEVPOST_NOT_FOUND`, `DEVPOST_REDIRECTED_OFFSITE`, `DEVPOST_TOO_MANY_REDIRECTS`, or `DEVPOST_UNAVAILABLE`. While the hackathon's submissions are private, Devpost sends anonymous requests to its login page (`DEVPOST_LOGIN_REQUIRED`); set `MDREDD_DEVPOST_COOKIE` to the `Cookie` header of an organizer's logged-in Devpost session to resolve them anyway. Resolution makes one request per row, `MDREDD_DEVPOST_CONCURRENCY` at a time, so a large upload can take a while.
2. A successful upload is `201` and turns judging on:
Expand Down Expand Up @@ -139,7 +139,7 @@ If the judge worker is dead, stuck, or its queue is full, the call is `503` `WOR
| 409 | `ABSENT_NOT_IN_PAIR` | Absence ids are not the current pair |
| 409 | `POOL_EXHAUSTED` | Fewer than two projects are still active |
| 422 | `TOO_FEW_ENTITIES` | CSV has fewer than two rows |
| 422 | `INVALID_COLUMNS` | Headers are empty, duplicated, or unreadable. `detail.names` lists the bad headers when known |
| 422 | `INVALID_COLUMNS` | Headers are duplicated, all blank, or unreadable, or a required column is missing. `detail.names` lists the bad headers when known |
| 422 | `INCORRECT_PAIR_FORMAT` | Winner is not one of the two ids |
| 429 | `RATE_LIMITED` | Shared bucket is empty |
| 502 | `DEVPOST_UNRESOLVED` | Some submission URLs did not resolve. `detail.failures` lists them |
Expand Down
27 changes: 19 additions & 8 deletions app/entity.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,12 @@ def list_from_csv(raw_csv: bytes) -> tuple[list[str], list[Entity]]:
text = raw_csv.decode("utf-8-sig")
except UnicodeDecodeError as exc:
raise InvalidColumnsException([]) from exc
headers = _header_row(text)
header_row = _header_row(text)
# A spreadsheet that re-saves the export pads every row, header
# included, with unnamed columns. They have no name to store a value
# under, so they are dropped rather than rejected.
named = [index for index, name in enumerate(header_row) if name.strip()]
headers = [header_row[index] for index in named]
_require_headers(headers)
try:
rows = list(csv.reader(io.StringIO(text)))[1:]
Expand All @@ -24,9 +29,13 @@ def list_from_csv(raw_csv: bytes) -> tuple[list[str], list[Entity]]:
# Devpost headers only the first team member, so larger teams' rows run
# past the header. Blank lines are skipped, short rows padded, and
# cells past the last header dropped.
width = len(headers)
entities = [
Entity(attributes=dict(zip(headers, [*row, *[""] * (width - len(row))])))
Entity(
attributes={
headers[position]: row[index] if index < len(row) else ""
for position, index in enumerate(named)
}
)
for row in rows
if any(cell.strip() for cell in row)
]
Expand All @@ -43,12 +52,14 @@ def _header_row(text: str) -> list[str]:


def _require_headers(headers: list[str]) -> None:
if not headers:
raise InvalidColumnsException([])
seen: set[str] = set()
invalid: list[str] = []
duplicates: list[str] = []
for header in headers:
if header == "" or header in seen:
invalid.append(header)
if header in seen:
duplicates.append(header)
continue
seen.add(header)
if invalid:
raise InvalidColumnsException(invalid)
if duplicates:
raise InvalidColumnsException(duplicates)
Loading