diff --git a/README.md b/README.md index 1b4ad20..ac0cf3c 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ Errors are JSON: `{"detail":{"code":"..."}}`. Rate limits add `retry_after_ms` a 4. `GET /pool` lists every project in upload order: `id`, `attributes`, `strikes`, and `removed`. `removed` is true once `strikes` reaches the strike limit. `POST /pool/{id}/restore` clears that project's strikes and returns it to the draw. Restoring a project that is still active succeeds and changes nothing. 5. `GET /projects` lists every project in upload order. The response is ids and attributes only. Before any dataset exists the list is empty. `GET /rankings` returns every row, strongest first, with the same shape. Rankings stay readable after stop. Before any dataset exists they are `409` `JUDGING_NEVER_STARTED`. `GET /columns` lists headers. `GET /rows/{id}` returns one row, or `404` `UNKNOWN_ROW`. 6. `POST /archive` moves the SQLite database and the log file into a new folder under `archive/` and starts empty. Each call keeps the earlier folders. The response `path` is that folder's name. `GET /archives` lists those names, newest first, and `GET /archives/{id}` downloads that folder as a zip. An unknown id is `404` `UNKNOWN_ARCHIVE`. Judging is off. If startup cannot read the file, it logs that and keeps serving. Other routes are `503` `DATABASE_UNREADABLE` until `POST /archive`. -7. `PUT /tables` with `{"tables": {"https://devpost.com/software/project-a": 12, ...}}` replaces the whole project URL to table number mapping. Build it from each team's saved Devpost link and reserved table. Table numbers must be positive. URLs match a project's `Project Url` ignoring case, `www.`, a trailing slash, the query, and the scheme. The response is `{ "stored": 2, "unknown_urls": [...] }`, where `unknown_urls` are the sent URLs that match no uploaded project. The mapping is kept in SQLite and survives a new upload, so send it again whenever a team changes tables. Send `{"tables": {}}` to clear it. +7. `PUT /tables` with `{"tables": {"https://devpost.com/software/project-a": 12, ...}}` replaces the whole project URL to table number mapping. Build it from each team's saved Devpost link and reserved table. Table numbers must be positive. URLs match a project's `Project Url` ignoring case, `www.`, a trailing slash, the query, and the scheme. The response is `{ "stored": 2, "unknown_urls": [...] }`, where `unknown_urls` are the sent URLs that match no uploaded project. The mapping is kept in SQLite and survives a new upload, so send it again whenever a team changes tables. Send `{"tables": {}}` to clear it. **Only projects with a table are drawn for judges**, so until the mapping is sent, `POST /pairs` is `409` `POOL_EXHAUSTED`. A pair a judge already holds is still returned if one of its projects loses its table. 8. `GET /export` downloads every project in upload order as `projects.csv`: `id`, every stored column (including `Project Url`), then `Table Number`, which is empty when no table is mapped to that project. ## Judge flow @@ -85,12 +85,18 @@ POST /pairs → show the two projects → POST /comparisons → POST /pair "tracks": ["Actually Intelligent (AI)", "Figma Best Design"] }, { "id": 11, "url": "https://devpost.com/software/project-b", "name": "Project B", "tracks": [] } - ] + ], + "assigned_at": 1790000000.0, + "server_time": 1790000042.5 } ``` `url` is the row's `Project Url`, `name` its `Project Title`, and `tracks` its `M Hacks Main Track` followed by each prize in `Sponsor Opt In Prizes`. No other CSV column is sent to judges. Match `url` against the Devpost links teams saved to find the team and its table. +`assigned_at` is the Unix time this pair was handed out, and stays the same each time the judge asks for the pair they hold, including across restarts. `server_time` is MDredd's clock when it answered, so a client can time the pair as `server_time - assigned_at` without trusting its own clock. + +**Skip a pair.** Call `POST /pairs` with the open pair's two ids in `skip`, for example when the judge's time runs out. Nothing is recorded about either project: no comparison, no strike, and the appearance the draw counted for each is given back. The new pair avoids both skipped projects when enough others are drawable. If `skip` no longer matches the judge's open pair, because it was already replaced, MDredd returns the current pair instead of skipping again, so a retry is safe. `skip` and `absent` cannot be sent together (`422`). + **Record a winner.** `entity_ids` must be the two ids of that judge's open pair. Order does not matter. `winner_id` must be one of them. ```json @@ -122,7 +128,8 @@ Drawing a pair is separate from strength. Each draw increments an appearance cou |---|---|---|---| | `POST /pairs` | each judge | 6 | about 1 every 5 seconds | | `POST /comparisons` | each judge | 2 | about 1 per minute | -| Admin writes (upload, start, stop, restore, tables) | everyone | 4 | about 1 every 30 seconds | +| Admin writes (upload, start, stop, restore) | everyone | 4 | about 1 every 30 seconds | +| `PUT /tables` | everyone | 30 | about 1 per second | `429` is `{"detail":{"code":"RATE_LIMITED","retry_after_ms":...}}` plus `Retry-After`. Retry the same body. `GET /judging`, `/projects`, `/rankings`, `/pool`, `/rows`, `/columns`, and `/export` are not limited. @@ -137,7 +144,7 @@ If the judge worker is dead, stuck, or its queue is full, the call is `503` `WOR | 409 | `JUDGING_NEVER_STARTED` | No dataset has been stored | | 409 | `JUDGE_DOES_NOT_OWN_PAIR` | Comparison does not match this judge's open pair | | 409 | `ABSENT_NOT_IN_PAIR` | Absence ids are not the current pair | -| 409 | `POOL_EXHAUSTED` | Fewer than two projects are still active | +| 409 | `POOL_EXHAUSTED` | Fewer than two projects are still active and have a table | | 422 | `TOO_FEW_ENTITIES` | CSV has fewer than two rows | | 422 | `INVALID_COLUMNS` | Headers are duplicated, all blank, or unreadable, or a required column is missing. `detail.names` lists the bad headers when known | | 422 | `DEVPOST_UNRESOLVED` | Some submission URLs did not resolve. `detail.failures` lists them | diff --git a/app/api/admin_router.py b/app/api/admin_router.py index 00b1ae2..e93ea3d 100644 --- a/app/api/admin_router.py +++ b/app/api/admin_router.py @@ -208,7 +208,7 @@ def restore_pool_entity(entity_id: int, session: SessionDep) -> PoolEntryModel: "each project's Project Url, ignoring case, www, a trailing slash, and the query." ), response_description="How many entries are stored, and the URLs that match no project.", - dependencies=[limited("admin")], + dependencies=[limited("tables")], responses=error_responses(limited=True), ) def put_tables(body: TablesInputModel, session: SessionDep) -> TablesModel: diff --git a/app/api/judge_router.py b/app/api/judge_router.py index cfa7cf1..70f98bf 100644 --- a/app/api/judge_router.py +++ b/app/api/judge_router.py @@ -1,3 +1,5 @@ +import time + from fastapi import APIRouter, Depends, Request, status from app.api.deps import SessionDep, enforce_limit, require_session @@ -35,7 +37,8 @@ def get_projects(session: SessionDep) -> list[RowModel]: description=( "Draw this judge's open pair, or return the pair they already hold. " "One absent project from that pair forfeits, and the other project wins. " - "Both absent strikes each project and draws a new pair." + "Both absent strikes each project and draws a new pair. " + "skip gives up the open pair unjudged and draws a new one." ), response_description="The two projects to compare: id, Devpost URL, name, and tracks.", responses=error_responses(status.HTTP_409_CONFLICT, limited=True), @@ -45,9 +48,11 @@ def create_pair( ) -> PairModel: # Each judge has their own bucket, since every judge may come through one client. enforce_limit(request, "pair", body.judge_id) - left, right = session.worker.request_pair(body) + left, right, assigned_at = session.worker.request_pair(body) return PairModel( - pair=(ProjectModel.from_entity(left), ProjectModel.from_entity(right)) + pair=(ProjectModel.from_entity(left), ProjectModel.from_entity(right)), + assigned_at=assigned_at, + server_time=time.time(), ) diff --git a/app/db.py b/app/db.py index f7b223b..419f64e 100644 --- a/app/db.py +++ b/app/db.py @@ -1,4 +1,5 @@ import logging +import time from dataclasses import dataclass from datetime import UTC, datetime from pathlib import Path @@ -8,6 +9,7 @@ from peewee import ( BlobField, BooleanField, + FloatField, Case, IntegerField, JSONField, @@ -59,6 +61,8 @@ class Assignment(Model): judge_id = TextField(primary_key=True) entity_id_1 = IntegerField() entity_id_2 = IntegerField() + # Unix time the pair was handed out, for the judge's timer. + assigned_at = FloatField(null=True) class Meta: database = db @@ -127,6 +131,8 @@ class JudgeRecord: headers: list[str] entities: list[Entity] assignments: dict[str, tuple[int, int]] + # Judge id to the Unix time their open pair was handed out. + assigned_at: dict[str, float] completed: dict[str, tuple[int, int, int]] strikes: list[int] last_skips: dict[str, AbsentSkip] @@ -152,6 +158,7 @@ def open_db() -> None: ] ) _migrate_strikes() + _migrate_assignment_time() _migrate_legacy_state() @@ -267,9 +274,17 @@ def load_state() -> JudgeRecord: Entity(attributes=row.attributes) for row in EntityRow.select().order_by(EntityRow.id) ] + assignment_rows = list(Assignment.select()) assignments = { row.judge_id: (row.entity_id_1, row.entity_id_2) - for row in Assignment.select() + for row in assignment_rows + } + loaded_at = time.time() + assigned_at = { + row.judge_id: float(row.assigned_at) + if row.assigned_at is not None + else loaded_at + for row in assignment_rows } completed = { row.judge_id: (row.entity_id_1, row.entity_id_2, row.winner_id) @@ -285,6 +300,7 @@ def load_state() -> JudgeRecord: headers=[], entities=entities, assignments=assignments, + assigned_at=assigned_at, completed=completed, strikes=strikes, last_skips=last_skips, @@ -297,6 +313,7 @@ def load_state() -> JudgeRecord: headers=list(judge.headers), entities=entities, assignments=assignments, + assigned_at=assigned_at, completed=completed, strikes=strikes, last_skips=last_skips, @@ -313,6 +330,7 @@ def replace_state( headers=list(headers), entities=list(entities), assignments={}, + assigned_at={}, completed={}, strikes=[0] * len(entities), last_skips={}, @@ -359,7 +377,11 @@ def replace_state( def save_assignment( - frequency: Any, key: Any, judge_id: str, pair: tuple[int, int] + frequency: Any, + key: Any, + judge_id: str, + pair: tuple[int, int], + assigned_at: float, ) -> None: left, right = pair stored_frequency = np.asarray(frequency) @@ -388,6 +410,7 @@ def save_assignment( judge_id=judge_id, entity_id_1=left, entity_id_2=right, + assigned_at=assigned_at, ).execute() @@ -443,6 +466,7 @@ def save_absence( judge_id: str, absent_ids: tuple[int, ...], pair: tuple[int, int] | None, + assigned_at: float, ) -> None: stored_frequency = np.asarray(frequency) if stored_frequency.shape != (len(strikes),): @@ -476,6 +500,7 @@ def save_absence( judge_id=assigned_judge, entity_id_1=left, entity_id_2=right, + assigned_at=assigned_at, ).execute() if completed is not None: completed_judge, (entity_id_1, entity_id_2, winner_id) = completed @@ -505,6 +530,42 @@ def save_tables(tables: dict[str, int]) -> None: ).execute() +def save_skip( + frequency: Any, + key: Any, + judge_id: str, + pair: tuple[int, int] | None, + assigned_at: float, +) -> None: + """Store a skipped pair's refund and the judge's next pair, if any.""" + stored_frequency = np.asarray(frequency) + with db.atomic(): + for index, count in enumerate(stored_frequency): + updated = ( + EntityState.update(frequency=int(count)) + .where(EntityState.entity_id == index) + .execute() + ) + if updated != 1: + raise RuntimeError("Entity state is missing") + updated = ( + JudgeState.update(key=_array_bytes(key, np.dtype(" None: updated = Judge.update(enabled=enabled).where(Judge.id == 1).execute() if updated != 1: @@ -582,6 +643,16 @@ def _migrate_strikes() -> None: ) +def _migrate_assignment_time() -> None: + # Pairs handed out before this column existed load as handed out at startup. + if "assignments" not in db.get_tables(): + return + names = {column.name for column in db.get_columns("assignments")} + if "assigned_at" in names: + return + db.execute_sql("ALTER TABLE assignments ADD COLUMN assigned_at REAL") + + def _migrate_legacy_state() -> None: judge = Judge.get_or_none(Judge.id == 1) if judge is None or judge.bdp is None: diff --git a/app/models.py b/app/models.py index 1e79f36..8d96f6e 100644 --- a/app/models.py +++ b/app/models.py @@ -1,6 +1,6 @@ from typing import Annotated -from pydantic import BaseModel, Field +from pydantic import BaseModel, Field, model_validator from app import project from app.entity import Entity @@ -19,6 +19,14 @@ class ComparisonInputModel(BaseModel): class PairRequestModel(BaseModel): judge_id: str = Field(min_length=1) absent: list[int] = Field(default_factory=list, max_length=2) + # The open pair to give up unjudged, e.g. when the judge's timer runs out. + skip: tuple[int, int] | None = None + + @model_validator(mode="after") + def one_action(self) -> PairRequestModel: + if self.skip is not None and self.absent: + raise ValueError("Send either absent or skip, not both") + return self class PoolEntryModel(BaseModel): @@ -87,6 +95,10 @@ def from_entity(cls, entity: EntityWithId) -> ProjectModel: class PairModel(BaseModel): pair: tuple[ProjectModel, ProjectModel] + # Unix time the pair was handed out, and MDredd's clock when it answered, + # so a client can time the pair without trusting its own clock. + assigned_at: float + server_time: float class ComparisonResultModel(BaseModel): diff --git a/app/ratelimit.py b/app/ratelimit.py index 3f53e3a..9ad65fd 100644 --- a/app/ratelimit.py +++ b/app/ratelimit.py @@ -7,7 +7,7 @@ from app.settings import settings Bucket = tuple[float, float] -Operation = Literal["admin", "pair", "submit"] +Operation = Literal["admin", "pair", "submit", "tables"] @dataclass(frozen=True) @@ -65,6 +65,8 @@ def _limits(operation: Operation) -> tuple[int, float]: return settings.SUBMIT_CAPACITY, settings.SUBMIT_REFILL_PER_SECOND case "admin": return settings.ADMIN_CAPACITY, settings.ADMIN_REFILL_PER_SECOND + case "tables": + return settings.TABLES_CAPACITY, settings.TABLES_REFILL_PER_SECOND limiter = TokenBucket() diff --git a/app/settings.py b/app/settings.py index 6bfea00..2704c6a 100644 --- a/app/settings.py +++ b/app/settings.py @@ -27,6 +27,10 @@ class Settings(BaseSettings): SUBMIT_REFILL_PER_SECOND: float = 1 / 60 ADMIN_CAPACITY: int = 4 ADMIN_REFILL_PER_SECOND: float = 1 / 30 + # The dashboard re-sends tables whenever assignments change, so table + # syncs have their own, looser bucket. + TABLES_CAPACITY: int = 30 + TABLES_REFILL_PER_SECOND: float = 1.0 WORKER_TIMEOUT_SECONDS: float = 10 WORKER_STUCK_SECONDS: float = 30 WORKER_QUEUE_SIZE: int = 8 diff --git a/app/worker.py b/app/worker.py index 54a6266..919978d 100644 --- a/app/worker.py +++ b/app/worker.py @@ -22,6 +22,7 @@ save_absence, save_assignment, save_comparison, + save_skip, save_enabled, save_strikes, save_tables, @@ -104,6 +105,7 @@ def __init__( self.headers: list[str] = [] self._entities: list[Entity] = [] self._assignments: dict[str, tuple[int, int]] = {} + self._assigned_at: dict[str, float] = {} self._completed: dict[str, tuple[int, int, int]] = {} self._strikes: list[int] = [] self._last_skips: dict[str, AbsentSkip] = {} @@ -166,8 +168,15 @@ def stop(self) -> bool: def request_pair( self, pair_request: PairRequestModel - ) -> tuple[EntityWithId, EntityWithId]: - return self._call(lambda: self._get_pair(pair_request)) + ) -> tuple[EntityWithId, EntityWithId, float]: + """The judge's pair and the Unix time it was handed out.""" + + def job() -> tuple[EntityWithId, EntityWithId, float]: + left, right = self._get_pair(pair_request) + assigned_at = self._assigned_at.get(pair_request.judge_id, time.time()) + return left, right, assigned_at + + return self._call(job) def submit(self, comparison: ComparisonInputModel) -> None: self._call(lambda: self._submit(comparison)) @@ -318,6 +327,8 @@ def _get_pair( ) -> tuple[EntityWithId, EntityWithId]: if not self.enabled: raise JudgingNotStartedException() + if pair_request.skip is not None: + return self._skip(pair_request.judge_id, pair_request.skip) if pair_request.absent: return self._report_absence(pair_request.judge_id, pair_request.absent) assigned = self._assignments.get(pair_request.judge_id) @@ -371,6 +382,7 @@ def _submit(self, comparison: ComparisonInputModel) -> None: for entity_id, strike_count in strike_counts: self._strikes[entity_id] = strike_count del self._assignments[judge] + self._assigned_at.pop(judge, None) self._completed[judge] = completed logger.info( "Applied comparison", @@ -388,14 +400,15 @@ def _draw_for( exclude: tuple[int, ...], absent_key: tuple[int, ...] | None = None, ) -> tuple[EntityWithId, EntityWithId]: - mask = _active_mask(self._strikes, exclude) + mask = self._drawable_mask(self._strikes, exclude) if int(mask.sum()) < 2: raise PoolExhaustedException() bdp = self._require_bdp() left, right, frequency, key = bdp.propose_pair(mask) pair = (left, right) + now = time.time() if absent_key is None: - save_assignment(frequency, key, judge_id, pair) + save_assignment(frequency, key, judge_id, pair, now) else: save_absence( frequency, @@ -407,10 +420,12 @@ def _draw_for( judge_id, absent_key, pair, + now, ) bdp.frequency = frequency bdp.key = key self._assignments[judge_id] = pair + self._assigned_at[judge_id] = now if absent_key is not None: self._last_skips[judge_id] = (absent_key, pair) return self._pair(*pair) @@ -483,7 +498,7 @@ def _report_absence( ] assignment_updates: dict[str, tuple[int, int] | None] = {} exclude = absent_key if len(absent_key) == 2 else () - mask = _active_mask(strikes, exclude) + mask = self._drawable_mask(strikes, exclude) new_pair = None if int(mask.sum()) < 2: assignment_updates[judge_id] = None @@ -497,6 +512,7 @@ def _report_absence( completed = None if ordered is not None and winner_id is not None: completed = (judge_id, (*ordered, winner_id)) + now = time.time() save_absence( frequency, strikes, @@ -507,6 +523,7 @@ def _report_absence( judge_id, absent_key, new_pair, + now, ) bdp.frequency = frequency bdp.key = key @@ -516,8 +533,10 @@ def _report_absence( for other, pair in assignment_updates.items(): if pair is None: self._assignments.pop(other, None) + self._assigned_at.pop(other, None) else: self._assignments[other] = pair + self._assigned_at[other] = now if completed is not None: self._completed[judge_id] = completed[1] self._last_skips[judge_id] = (absent_key, new_pair) @@ -577,10 +596,84 @@ def _export(self) -> tuple[list[str], list[tuple[EntityWithId, int | None]]]: for index, entity in enumerate(self._entities) ] + def _drawable_mask( + self, strikes: list[int], exclude: tuple[int, ...] = () + ) -> np.ndarray: + """Projects that may be drawn: not struck out, and with a table. + + A project with no table in the mapping from PUT /tables has nowhere + for a judge to go, so it is left out until a table is mapped to it. + """ + mask = _active_mask(strikes, exclude) + seated = np.array( + [self._table_for(entity) is not None for entity in self._entities], + dtype=bool, + ) + return mask & seated + def _table_for(self, entity: Entity) -> int | None: key = project.normalize_url(entity.attributes.get(project.PROJECT_URL, "")) return self._tables.get(key) if key else None + def _skip( + self, judge_id: str, skip: tuple[int, int] + ) -> tuple[EntityWithId, EntityWithId]: + """Give up the judge's open pair unjudged and draw another. + + Nothing is recorded about either project: no comparison and no + strike, and each gets back the appearance the draw counted. `skip` + names the pair being given up, so a retry after it was already + replaced returns the replacement instead of skipping it too. + """ + skipped = (min(skip), max(skip)) + current = self._assignments.get(judge_id) + if current is None or (min(current), max(current)) != skipped: + logger.info( + "Ignored stale skip", + extra={ + "event": "skip_ignored", + "user_id": judge_id, + "entity_ids": list(skipped), + }, + ) + if current is not None: + return self._pair(*current) + return self._draw_for(judge_id, ()) + + bdp = self._require_bdp() + frequency = bdp.frequency + for entity_id in skipped: + frequency = _refund(frequency, entity_id) + # A different pair if the pool allows it, otherwise any pair. + mask = self._drawable_mask(self._strikes, skipped) + if int(mask.sum()) < 2: + mask = self._drawable_mask(self._strikes) + now = time.time() + if int(mask.sum()) < 2: + save_skip(frequency, bdp.key, judge_id, None, now) + bdp.frequency = frequency + self._assignments.pop(judge_id, None) + self._assigned_at.pop(judge_id, None) + raise PoolExhaustedException() + left, right, frequency, key = bdp.propose_pair( + mask, frequency=frequency, key=bdp.key + ) + pair = (left, right) + save_skip(frequency, key, judge_id, pair, now) + bdp.frequency = frequency + bdp.key = key + self._assignments[judge_id] = pair + self._assigned_at[judge_id] = now + logger.info( + "Skipped pair", + extra={ + "event": "pair_skipped", + "user_id": judge_id, + "entity_ids": list(skipped), + }, + ) + return self._pair(*pair) + def _pool(self) -> list[PoolEntryModel]: return [self._pool_entry(index) for index in range(len(self._entities))] @@ -615,6 +708,7 @@ def _install(self, record: JudgeRecord) -> None: self.headers = list(record.headers) self._entities = list(record.entities) self._assignments = dict(record.assignments) + self._assigned_at = dict(record.assigned_at) self._completed = dict(record.completed) self._strikes = list(record.strikes) self._last_skips = dict(record.last_skips)