Added .sql to create user_activity view.

[anna-stacey]

• I haven't tested the whole big select in RDS yet because I wasn't sure how billing works there and thus how pricey it would be, but I've tested some dummy/smaller subversions.
• This is the .sql for the canvas version of the view; we would also need to run this against catalog to make the second view.
• I've written a normal response.txt-style file with some notes, but I'm not actually sure how we should notify EASD. Probably not until the connection (what is it called? VPC to VPC or something?) is made anyways.

[jlongland]

I ran the full query (using DBeaver) directly against the database. Took 107 seconds and output 2,599,415 rows. Reasonably acceptable for a nightly job. I ran an explain plan with ANALYZE and BUFFERS. Claude's interpretation:

Where the time actually goes: it's almost entirely storage I/O, not CPU or spills. Total shared reads are ~53 GB (6.95M blocks) with 218 s of cumulative read wait, and two branches dominate — submissions (~28 GB read, 44 s of the runtime) and quiz_submissions (~18 GB, 27 s). Together they're two-thirds of the wall clock. The reason the volume is so high: those tables are fat (~850 bytes/row for quiz_submissions, thanks to columns like submission_data), and a seq scan reads every block to aggregate three narrow columns. The temp spills you can see everywhere (65-batch HashAggregate on pseudonyms, external-merge sorts) are real but cheap — under 3 s of temp I/O total.

Tips, in order of value:

If ~2 minutes nightly is acceptable, ship it as-is. Nothing here is pathological.
Cheap win — session settings for the job: SET work_mem = '256MB' (everything ran at the ~4 MB default; this collapses all the batching/spilling, worth maybe 10–20 s) and SET max_parallel_workers_per_gather = 4 (only 2 workers ran; the big scans are I/O-bound and Aurora storage handles parallel reads well). Session-scoped only.

The real lever, if you ever need it much faster: narrow partial covering indexes enabling index-only scans on the two fat tables, e.g. (user_id, submitted_at) WHERE workflow_state IN ('submitted','graded') AND submitted_at IS NOT NULL on submissions, and the analogue on quiz_submissions. Note the filters barely reduce rows (quiz_submissions keeps 90%) — the win is width: an index a fraction of the 28 GB/18 GB tables cuts that I/O 10–20×. Two caveats: it taxes every 3-hourly DAP upsert with index maintenance, and the view revisions we discussed (adding pending_review, updated_at for last_changed_at) change the needed columns — so settle the view shape first. I'd hold off unless the runtime or ACU footprint becomes a problem.

Fix the n_distinct underestimates anyway. The costs in this plan are byte-identical to Tuesday's, so table stats haven't changed in between. Each aggregate underestimates distinct user_id 2–10× (conversation_messages: est 17,552 vs actual 188,020), which is what compounds into the quintillion-row join estimates. Harmless for this query, but poisonous for anyone filtering the view (the bad-plan risk I flagged earlier). Fix: ALTER TABLE canvas.conversation_messages ALTER COLUMN author_id SET STATISTICS 1000; (likewise user_id on the other four event tables), then ANALYZE — larger sample, much better ndistinct. Confidence: medium-high that this substantially repairs the estimates; ndistinct sampling on skewed columns can stay stubborn, in which case ALTER COLUMN ... SET (n_distinct = ) pins it exactly.

Two operational notes: ~7M block reads per run is billable I/O if the cluster is on Aurora Standard rather than I/O-Optimized — trivial once nightly, another reason not to let interactive consumers hit the view. And this run pulled 53 GB cold, so expect the nightly job to spike Serverless ACUs while it runs; that's expected, just make sure min/max capacity accommodates it without starving a concurrent DAP sync.

Something we should consider is whether to implement a delta. Rather than having the consumer filter this view by timestamp, we compute the change set in the database and expose only the delta:

• A function (easd.refresh_user_activity()) takes one consistent snapshot of canvas.user_activity into a baseline table, diffs it against the previous night's baseline with EXCEPT (which treats NULLs as equal, so nullable key/timestamp columns diff correctly), appends the changes to an append-only easd.user_activity_delta table stamped with a run_id, and rotates the baseline — all in a single transaction.
• Delta rows are upsert (new or changed row, full new values) or delete (key gone), keyed on (user_id, sis_integration_id, sis_user_id, pseudonym_workflow_state). The nightly job calls the function, reads WHERE run_id > :last_applied, and stores its new watermark.

Why not updated_at-based filtering: CD2 sync applies hard deletes (a dropped enrollment or deleted pseudonym changes a row without advancing any timestamp), and a failed sync delivers rows later with older updated_at values than the watermark — both silently missed. Snapshot-diff catches every change uniformly, and the append-only feed makes failed or skipped consumer runs replayable for 30 days.

[jlongland]

I think what we should provide to them is a copy of the view DDL plus a detailed description. I asked Claude for a draft. I'd interleave your observations/experience into something like this.

Data dictionary — canvas.user_activity

Grain. One row per distinct (user_id, sis_integration_id, sis_user_id, pseudonym_workflow_state) combination. Activity metrics are computed per user_id and then repeated on every row that user has — if one user_id carries two integration_ids, both rows show identical activity. Never SUM metrics across rows; aggregate per user_id first. Neither direction is 1:1: one user_id can have multiple integration_ids, and one integration_id can appear under multiple user_ids (un-merged duplicate accounts).

Coverage. Only users with at least one pseudonym (login record) carrying a non-null integration_id appear. Users whose only pseudonyms lack an integration_id — and some users with no pseudonym row at all, which CD2 documents as possible — are absent entirely. No filtering on deleted/suspended states: deleted logins and deleted users are included.

Freshness. This is a replica: Instructure's CD2 feed (typically up to ~4 h behind live Canvas) synced into this database every 3 hours. Treat values as "as of a few hours ago," never real-time. Upstream deletions are hard deletes — rows can disappear between reads.

Scope of counts. All counts are lifetime totals across the whole Canvas history — no term or date scoping. All timestamps are UTC (timestamptz).

Identity columns

sis_integration_id — varchar(255)

pseudonyms.integration_id; the PUID. All non-null values are kept, including atypical formats (e.g. 12345ABCD_INC12345) — don't assume a fixed pattern.

sis_user_id — varchar(255), nullable

pseudonyms.sis_user_id; the student number. Can be NULL even when integration_id is present.

user_id — bigint

users.id, the Canvas internal ID; the key all activity is aggregated on. Oddity: IDs > 10,000,000,000,000 denote users from other Canvas instances (trust/consortium); they can appear here if they hold a local pseudonym.

pseudonym_workflow_state — enum: active, deleted, suspended

Lifecycle of the login record, not the user. Included in the row key, so a state change shows up as one row disappearing and another appearing.

merged_into_user_id — bigint, nullable

From users; set when this user account was merged into another. Activity does not follow the merge in this view — treat rows with this set as aliases pointing at the surviving user_id, whose own row carries the ongoing activity.

Login columns

Source: pseudonyms, aggregated over the row's matching login records.

last_login_at — timestamptz, nullable · [revision pending]

MAX of pseudonyms.last_login_at. Known oddity: Canvas stores the previous login here; the most recent login is current_login_at, which this view doesn't currently expose. Expect this to understate recency by one login.

last_request_at — timestamptz, nullable

MAX of pseudonyms.last_request_at; the best "last seen" signal — any authenticated page/API request, not just logins. Canvas throttles updates (roughly once per 10 min), so it's approximate. Activity via mobile apps or API tokens may not always register against the pseudonym.

login_count — bigint, 0 if none

SUM of pseudonyms.login_count across the row's matching login records. Counts session logins (including each SSO sign-in); not page views.

Enrolment columns

Source: enrollments.

enrolment_count_active — bigint

Count of enrollment records with workflow_state = 'active'. Three oddities: (1) these are enrollment records, not courses — enrollments are unique per course × section × role, so one student in a 3-section course contributes 3; (2) all role types count (Teacher, TA, Designer, Observer, Student); (3) "active" includes past terms — enrollments usually stay active after a term ends unless explicitly concluded (completed).

enrolment_count_total — bigint

All enrollment records in any state, including deleted, rejected, invited, inactive, completed, creation_pending.

Assignment submission columns

Source: submissions.

submission_count — bigint · [revision pending]

Submissions with a non-null submitted_at and state submitted or graded. Oddities: submissions holds one row per user × assignment, so this is "assignments with a submission," not submission acts (resubmissions don't increment it). Excludes pending_review (submitted, awaiting manual grading). Undercounts paper/external-tool assignments, where Canvas legitimately leaves submitted_at NULL even when graded. Includes New Quizzes (they flow through submissions, not quiz_submissions) and graded discussions.

last_submission_at — timestamptz, nullable

MAX submitted_at over those rows; resubmitting updates it to the latest attempt.

Quiz columns

Source: quiz_submissions — classic quizzes only.

quiz_submission_count — bigint · [revision pending]

Quiz submissions with state complete. Oddities: the table keeps one row per user × quiz (latest attempt), so this is "distinct classic quizzes completed," not attempts. Excludes pending_review (submitted, essay questions awaiting grading). New Quizzes are absent from this table entirely — they appear under submission_count instead, so don't compare these two columns as if disjoint or complete.

last_quiz_submission_at — timestamptz, nullable

MAX finished_at of completed classic quiz submissions.

Discussion columns

Source: discussion_entries.

discussion_entry_count — bigint

Non-deleted (active) discussion replies. The initial post of a topic lives in discussion_topics and is not counted. Replies on graded discussions also produce a submissions row, so that activity is visible in both columns.

last_discussion_entry_at — timestamptz, nullable

MAX created_at of those replies; later edits don't move it.

Inbox columns

Source: conversation_messages.

conversation_message_count — bigint · [revision pending]

Messages authored in the Canvas Inbox. Known oddity: includes system-generated rows ("X was added to the conversation") because the generated flag isn't filtered yet — expect modest inflation for some users. Messages persist here even if participants deleted them from their inboxes.

last_conversation_message_at — timestamptz, nullable

MAX created_at of authored messages, system-generated included.

[jlongland]

Just want to make sure that this is still on your radar for this PR.

[anna-stacey]

submissions (~28 GB read, 44 s of the runtime) and quiz_submissions (~18 GB, 27 s). Together they're two-thirds of the wall clock.

We could also just get rid of quiz_submissions. It was listed in the original issue as to-be-included (from Claude, I assume) but I was already kind of on the fence about it since, as I mentioned in the comments, it is just a subset of submissions. If we'd like to shorten the duration, this is a pretty harmless change.

[anna-stacey]

Fix the n_distinct underestimates anyway. The costs in this plan are byte-identical to Tuesday's, so table stats haven't changed in between. Each aggregate underestimates distinct user_id 2–10× (conversation_messages: est 17,552 vs actual 188,020), which is what compounds into the quintillion-row join estimates. Harmless for this query, but poisonous for anyone filtering the view (the bad-plan risk I flagged earlier). Fix: ALTER TABLE canvas.conversation_messages ALTER COLUMN author_id SET STATISTICS 1000; (likewise user_id on the other four event tables), then ANALYZE — larger sample, much better ndistinct. Confidence: medium-high that this substantially repairs the estimates; ndistinct sampling on skewed columns can stay stubborn, in which case ALTER COLUMN ... SET (n_distinct = ) pins it exactly.

I don't follow this part; was it in reference to a previous part of your conversation w Claude?

[anna-stacey]

Something we should consider is whether to implement a delta.

Hmm I think this should probably be your call since it's a matter of a) what better suits the EASD folks and b) what's better for our data processing costs, neither of which is my area of knowledge : ) But if you want me to check in with them about it, I can do that.

[jlongland]

submissions (~28 GB read, 44 s of the runtime) and quiz_submissions (~18 GB, 27 s). Together they're two-thirds of the wall clock.

We could also just get rid of quiz_submissions. It was listed in the original issue as to-be-included (from Claude, I assume) but I was already kind of on the fence about it since, as I mentioned in the comments, it is just a subset of submissions. If we'd like to shorten the duration, this is a pretty harmless change.

Yeah, I'm good with just using submissions. I don't think the distinction between assignments and quizzes makes a big difference for the person determining the account merge.

[jlongland]

Fix the n_distinct underestimates anyway. The costs in this plan are byte-identical to Tuesday's, so table stats haven't changed in between. Each aggregate underestimates distinct user_id 2–10× (conversation_messages: est 17,552 vs actual 188,020), which is what compounds into the quintillion-row join estimates. Harmless for this query, but poisonous for anyone filtering the view (the bad-plan risk I flagged earlier). Fix: ALTER TABLE canvas.conversation_messages ALTER COLUMN author_id SET STATISTICS 1000; (likewise user_id on the other four event tables), then ANALYZE — larger sample, much better ndistinct. Confidence: medium-high that this substantially repairs the estimates; ndistinct sampling on skewed columns can stay stubborn, in which case ALTER COLUMN ... SET (n_distinct = ) pins it exactly.

I don't follow this part; was it in reference to a previous part of your conversation w Claude?

Yeah, it was just concerned about query performance if the client is filtering to specific PUIDs. I don't think it's worth pursuing.