Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Errors are JSON: `{"detail":{"code":"..."}}`. Rate limits add `retry_after_ms` a
4. `GET /pool` lists every project in upload order: `id`, `attributes`, `strikes`, and `removed`. `removed` is true once `strikes` reaches the strike limit. `POST /pool/{id}/restore` clears that project's strikes and returns it to the draw. Restoring a project that is still active succeeds and changes nothing.
5. `GET /projects` lists every project in upload order. The response is ids and attributes only. Before any dataset exists the list is empty. `GET /rankings` returns every row, strongest first, with the same shape. Rankings stay readable after stop. Before any dataset exists they are `409` `JUDGING_NEVER_STARTED`. `GET /columns` lists headers. `GET /rows/{id}` returns one row, or `404` `UNKNOWN_ROW`.
6. `POST /archive` moves the SQLite database and the log file into a new folder under `archive/` and starts empty. Each call keeps the earlier folders. The response `path` is that folder's name. `GET /archives` lists those names, newest first, and `GET /archives/{id}` downloads that folder as a zip. An unknown id is `404` `UNKNOWN_ARCHIVE`. Judging is off. If startup cannot read the file, it logs that and keeps serving. Other routes are `503` `DATABASE_UNREADABLE` until `POST /archive`.
7. `PUT /tables` with `{"tables": {"https://devpost.com/software/project-a": 12, ...}}` replaces the whole project URL to table number mapping. Build it from each team's saved Devpost link and reserved table. Table numbers must be positive. URLs match a project's `Project Url` ignoring case, `www.`, a trailing slash, the query, and the scheme. The response is `{ "stored": 2, "unknown_urls": [...] }`, where `unknown_urls` are the sent URLs that match no uploaded project. The mapping is kept in SQLite and survives a new upload, so send it again whenever a team changes tables. Send `{"tables": {}}` to clear it.
7. `PUT /tables` with `{"tables": {"https://devpost.com/software/project-a": 12, ...}}` replaces the whole project URL to table number mapping. Build it from each team's saved Devpost link and reserved table. Table numbers must be positive. URLs match a project's `Project Url` ignoring case, `www.`, a trailing slash, the query, and the scheme. The response is `{ "stored": 2, "unknown_urls": [...] }`, where `unknown_urls` are the sent URLs that match no uploaded project. The mapping is kept in SQLite and survives a new upload, so send it again whenever a team changes tables. Send `{"tables": {}}` to clear it. **Only projects with a table are drawn for judges**, so until the mapping is sent, `POST /pairs` is `409` `POOL_EXHAUSTED`. A pair a judge already holds is still returned if one of its projects loses its table.
8. `GET /export` downloads every project in upload order as `projects.csv`: `id`, every stored column (including `Project Url`), then `Table Number`, which is empty when no table is mapped to that project.

## Judge flow
Expand Down Expand Up @@ -85,12 +85,18 @@ POST /pairs → show the two projects → POST /comparisons → POST /pair
"tracks": ["Actually Intelligent (AI)", "Figma Best Design"]
},
{ "id": 11, "url": "https://devpost.com/software/project-b", "name": "Project B", "tracks": [] }
]
],
"assigned_at": 1790000000.0,
"server_time": 1790000042.5
}
```

`url` is the row's `Project Url`, `name` its `Project Title`, and `tracks` its `M Hacks Main Track` followed by each prize in `Sponsor Opt In Prizes`. No other CSV column is sent to judges. Match `url` against the Devpost links teams saved to find the team and its table.

`assigned_at` is the Unix time this pair was handed out, and stays the same each time the judge asks for the pair they hold, including across restarts. `server_time` is MDredd's clock when it answered, so a client can time the pair as `server_time - assigned_at` without trusting its own clock.

**Skip a pair.** Call `POST /pairs` with the open pair's two ids in `skip`, for example when the judge's time runs out. Nothing is recorded about either project: no comparison, no strike, and the appearance the draw counted for each is given back. The new pair avoids both skipped projects when enough others are drawable. If `skip` no longer matches the judge's open pair, because it was already replaced, MDredd returns the current pair instead of skipping again, so a retry is safe. `skip` and `absent` cannot be sent together (`422`).

**Record a winner.** `entity_ids` must be the two ids of that judge's open pair. Order does not matter. `winner_id` must be one of them.

```json
Expand Down Expand Up @@ -122,7 +128,8 @@ Drawing a pair is separate from strength. Each draw increments an appearance cou
|---|---|---|---|
| `POST /pairs` | each judge | 6 | about 1 every 5 seconds |
| `POST /comparisons` | each judge | 2 | about 1 per minute |
| Admin writes (upload, start, stop, restore, tables) | everyone | 4 | about 1 every 30 seconds |
| Admin writes (upload, start, stop, restore) | everyone | 4 | about 1 every 30 seconds |
| `PUT /tables` | everyone | 30 | about 1 per second |

`429` is `{"detail":{"code":"RATE_LIMITED","retry_after_ms":...}}` plus `Retry-After`. Retry the same body. `GET /judging`, `/projects`, `/rankings`, `/pool`, `/rows`, `/columns`, and `/export` are not limited.

Expand All @@ -137,7 +144,7 @@ If the judge worker is dead, stuck, or its queue is full, the call is `503` `WOR
| 409 | `JUDGING_NEVER_STARTED` | No dataset has been stored |
| 409 | `JUDGE_DOES_NOT_OWN_PAIR` | Comparison does not match this judge's open pair |
| 409 | `ABSENT_NOT_IN_PAIR` | Absence ids are not the current pair |
| 409 | `POOL_EXHAUSTED` | Fewer than two projects are still active |
| 409 | `POOL_EXHAUSTED` | Fewer than two projects are still active and have a table |
| 422 | `TOO_FEW_ENTITIES` | CSV has fewer than two rows |
| 422 | `INVALID_COLUMNS` | Headers are duplicated, all blank, or unreadable, or a required column is missing. `detail.names` lists the bad headers when known |
| 422 | `DEVPOST_UNRESOLVED` | Some submission URLs did not resolve. `detail.failures` lists them |
Expand Down
2 changes: 1 addition & 1 deletion app/api/admin_router.py
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ def restore_pool_entity(entity_id: int, session: SessionDep) -> PoolEntryModel:
"each project's Project Url, ignoring case, www, a trailing slash, and the query."
),
response_description="How many entries are stored, and the URLs that match no project.",
dependencies=[limited("admin")],
dependencies=[limited("tables")],
responses=error_responses(limited=True),
)
def put_tables(body: TablesInputModel, session: SessionDep) -> TablesModel:
Expand Down
11 changes: 8 additions & 3 deletions app/api/judge_router.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import time

from fastapi import APIRouter, Depends, Request, status

from app.api.deps import SessionDep, enforce_limit, require_session
Expand Down Expand Up @@ -35,7 +37,8 @@ def get_projects(session: SessionDep) -> list[RowModel]:
description=(
"Draw this judge's open pair, or return the pair they already hold. "
"One absent project from that pair forfeits, and the other project wins. "
"Both absent strikes each project and draws a new pair."
"Both absent strikes each project and draws a new pair. "
"skip gives up the open pair unjudged and draws a new one."
),
response_description="The two projects to compare: id, Devpost URL, name, and tracks.",
responses=error_responses(status.HTTP_409_CONFLICT, limited=True),
Expand All @@ -45,9 +48,11 @@ def create_pair(
) -> PairModel:
# Each judge has their own bucket, since every judge may come through one client.
enforce_limit(request, "pair", body.judge_id)
left, right = session.worker.request_pair(body)
left, right, assigned_at = session.worker.request_pair(body)
return PairModel(
pair=(ProjectModel.from_entity(left), ProjectModel.from_entity(right))
pair=(ProjectModel.from_entity(left), ProjectModel.from_entity(right)),
assigned_at=assigned_at,
server_time=time.time(),
)


Expand Down
75 changes: 73 additions & 2 deletions app/db.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import logging
import time
from dataclasses import dataclass
from datetime import UTC, datetime
from pathlib import Path
Expand All @@ -8,6 +9,7 @@
from peewee import (
BlobField,
BooleanField,
FloatField,
Case,
IntegerField,
JSONField,
Expand Down Expand Up @@ -59,6 +61,8 @@ class Assignment(Model):
judge_id = TextField(primary_key=True)
entity_id_1 = IntegerField()
entity_id_2 = IntegerField()
# Unix time the pair was handed out, for the judge's timer.
assigned_at = FloatField(null=True)

class Meta:
database = db
Expand Down Expand Up @@ -127,6 +131,8 @@ class JudgeRecord:
headers: list[str]
entities: list[Entity]
assignments: dict[str, tuple[int, int]]
# Judge id to the Unix time their open pair was handed out.
assigned_at: dict[str, float]
completed: dict[str, tuple[int, int, int]]
strikes: list[int]
last_skips: dict[str, AbsentSkip]
Expand All @@ -152,6 +158,7 @@ def open_db() -> None:
]
)
_migrate_strikes()
_migrate_assignment_time()
_migrate_legacy_state()


Expand Down Expand Up @@ -267,9 +274,17 @@ def load_state() -> JudgeRecord:
Entity(attributes=row.attributes)
for row in EntityRow.select().order_by(EntityRow.id)
]
assignment_rows = list(Assignment.select())
assignments = {
row.judge_id: (row.entity_id_1, row.entity_id_2)
for row in Assignment.select()
for row in assignment_rows
}
loaded_at = time.time()
assigned_at = {
row.judge_id: float(row.assigned_at)
if row.assigned_at is not None
else loaded_at
for row in assignment_rows
}
completed = {
row.judge_id: (row.entity_id_1, row.entity_id_2, row.winner_id)
Expand All @@ -285,6 +300,7 @@ def load_state() -> JudgeRecord:
headers=[],
entities=entities,
assignments=assignments,
assigned_at=assigned_at,
completed=completed,
strikes=strikes,
last_skips=last_skips,
Expand All @@ -297,6 +313,7 @@ def load_state() -> JudgeRecord:
headers=list(judge.headers),
entities=entities,
assignments=assignments,
assigned_at=assigned_at,
completed=completed,
strikes=strikes,
last_skips=last_skips,
Expand All @@ -313,6 +330,7 @@ def replace_state(
headers=list(headers),
entities=list(entities),
assignments={},
assigned_at={},
completed={},
strikes=[0] * len(entities),
last_skips={},
Expand Down Expand Up @@ -359,7 +377,11 @@ def replace_state(


def save_assignment(
frequency: Any, key: Any, judge_id: str, pair: tuple[int, int]
frequency: Any,
key: Any,
judge_id: str,
pair: tuple[int, int],
assigned_at: float,
) -> None:
left, right = pair
stored_frequency = np.asarray(frequency)
Expand Down Expand Up @@ -388,6 +410,7 @@ def save_assignment(
judge_id=judge_id,
entity_id_1=left,
entity_id_2=right,
assigned_at=assigned_at,
).execute()


Expand Down Expand Up @@ -443,6 +466,7 @@ def save_absence(
judge_id: str,
absent_ids: tuple[int, ...],
pair: tuple[int, int] | None,
assigned_at: float,
) -> None:
stored_frequency = np.asarray(frequency)
if stored_frequency.shape != (len(strikes),):
Expand Down Expand Up @@ -476,6 +500,7 @@ def save_absence(
judge_id=assigned_judge,
entity_id_1=left,
entity_id_2=right,
assigned_at=assigned_at,
).execute()
if completed is not None:
completed_judge, (entity_id_1, entity_id_2, winner_id) = completed
Expand Down Expand Up @@ -505,6 +530,42 @@ def save_tables(tables: dict[str, int]) -> None:
).execute()


def save_skip(
frequency: Any,
key: Any,
judge_id: str,
pair: tuple[int, int] | None,
assigned_at: float,
) -> None:
"""Store a skipped pair's refund and the judge's next pair, if any."""
stored_frequency = np.asarray(frequency)
with db.atomic():
for index, count in enumerate(stored_frequency):
updated = (
EntityState.update(frequency=int(count))
.where(EntityState.entity_id == index)
.execute()
)
if updated != 1:
raise RuntimeError("Entity state is missing")
updated = (
JudgeState.update(key=_array_bytes(key, np.dtype("<u4")))
.where(JudgeState.id == 1)
.execute()
)
if updated != 1:
raise RuntimeError("Judge state is missing")
if pair is None:
Assignment.delete().where(Assignment.judge_id == judge_id).execute()
return
Assignment.replace(
judge_id=judge_id,
entity_id_1=pair[0],
entity_id_2=pair[1],
assigned_at=assigned_at,
).execute()


def save_enabled(enabled: bool) -> None:
updated = Judge.update(enabled=enabled).where(Judge.id == 1).execute()
if updated != 1:
Expand Down Expand Up @@ -582,6 +643,16 @@ def _migrate_strikes() -> None:
)


def _migrate_assignment_time() -> None:
# Pairs handed out before this column existed load as handed out at startup.
if "assignments" not in db.get_tables():
return
names = {column.name for column in db.get_columns("assignments")}
if "assigned_at" in names:
return
db.execute_sql("ALTER TABLE assignments ADD COLUMN assigned_at REAL")


def _migrate_legacy_state() -> None:
judge = Judge.get_or_none(Judge.id == 1)
if judge is None or judge.bdp is None:
Expand Down
14 changes: 13 additions & 1 deletion app/models.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from typing import Annotated

from pydantic import BaseModel, Field
from pydantic import BaseModel, Field, model_validator

from app import project
from app.entity import Entity
Expand All @@ -19,6 +19,14 @@ class ComparisonInputModel(BaseModel):
class PairRequestModel(BaseModel):
judge_id: str = Field(min_length=1)
absent: list[int] = Field(default_factory=list, max_length=2)
# The open pair to give up unjudged, e.g. when the judge's timer runs out.
skip: tuple[int, int] | None = None

@model_validator(mode="after")
def one_action(self) -> PairRequestModel:
if self.skip is not None and self.absent:
raise ValueError("Send either absent or skip, not both")
return self


class PoolEntryModel(BaseModel):
Expand Down Expand Up @@ -87,6 +95,10 @@ def from_entity(cls, entity: EntityWithId) -> ProjectModel:

class PairModel(BaseModel):
pair: tuple[ProjectModel, ProjectModel]
# Unix time the pair was handed out, and MDredd's clock when it answered,
# so a client can time the pair without trusting its own clock.
assigned_at: float
server_time: float


class ComparisonResultModel(BaseModel):
Expand Down
4 changes: 3 additions & 1 deletion app/ratelimit.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from app.settings import settings

Bucket = tuple[float, float]
Operation = Literal["admin", "pair", "submit"]
Operation = Literal["admin", "pair", "submit", "tables"]


@dataclass(frozen=True)
Expand Down Expand Up @@ -65,6 +65,8 @@ def _limits(operation: Operation) -> tuple[int, float]:
return settings.SUBMIT_CAPACITY, settings.SUBMIT_REFILL_PER_SECOND
case "admin":
return settings.ADMIN_CAPACITY, settings.ADMIN_REFILL_PER_SECOND
case "tables":
return settings.TABLES_CAPACITY, settings.TABLES_REFILL_PER_SECOND


limiter = TokenBucket()
4 changes: 4 additions & 0 deletions app/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ class Settings(BaseSettings):
SUBMIT_REFILL_PER_SECOND: float = 1 / 60
ADMIN_CAPACITY: int = 4
ADMIN_REFILL_PER_SECOND: float = 1 / 30
# The dashboard re-sends tables whenever assignments change, so table
# syncs have their own, looser bucket.
TABLES_CAPACITY: int = 30
TABLES_REFILL_PER_SECOND: float = 1.0
WORKER_TIMEOUT_SECONDS: float = 10
WORKER_STUCK_SECONDS: float = 30
WORKER_QUEUE_SIZE: int = 8
Expand Down
Loading
Loading