fix(hardware): use hardware_status window for listing and details trees

[alanpeixinho]

What it is

• Hardware listing no longer joins latest_checkout, so boards tested inside the selected time window still appear even when they were not on the newest tip.
• Hardware details tree heads no longer use absolute tree tips. get_hardware_trees_head_commits / get_hardware_trees_data resolve the latest checkout per tree from hardware_status in the window, so boards with tests on older checkouts still show data.
• Expands the hardcoded Hardware TEST_ORIGINS list so origins such as linaro_pull_labs (and other active pull-lab origins) show in the origin dropdown.

How to test

• Make sure origin dropdown now includes linaro_pull_labs, pullab_cloud_aws_arm64, and pull_labs_aws_ec2.
• Make sure hardware that are not tested on the latest checkout, but were tested in the specified time window (default is 5 days, but can be changed via ?i query parameter) are listed. (exynos on linaro_pull_labs is the example highlighted on issue Hardware missing from Hardware table #1983)
• Open hardware details for a board that has tests on an older checkout but not on the current tip (e.g. cases from Bug: Hardware details doesn't show tests from older checkouts #1264 / dashboard test result is not in sync with Maestro #1705). Trees and summary should show that older checkout’s data instead of empty / “no commits”.

Closes #1983
Closes #1264
Closes #1705

[mentonin]

I think we might need to iterate a little more on what should be counted as "current" for a given hardware platform or tree.

Deciding which builds and tests to show for a given platform should be the "easier" part, but deciding which hardwares, builds and tests to show for a given tree is complex

Example complications

Consider the following commit graph, where commits are marked according to the key:

• #: checkout with tests on platform X
• *: checkout with tests on platform Y
• $: checkout with tests on platform Z
• branch names in parentheses

  G  (dog)
 /
A#*$ ---------B#*  (cat)
 \
  C#*$ --- D* --- E  (cow)
            \
             F#$ --- H#  (elk)

Assuming every checkout has at least one test, latest_checkouts maps: cat -> B, elk -> F and cow -> D (no checkout on E yet). dog points to nothing (checkout on A declared cat for its branch)

Old implementation:

Only tests at latest_checkouts. Tests shown:

By platform:

• X: cat@B, elk@H
• Y: cat@B, cow@D
• Z: none

By tree:

• cat: X@B, Y@B
• cow: Y@D
• elk: X@H

New implementation:

Gets all tests in range. Tests shown:

By platform:

• X: cat@(A,B), cow@C, elk@(F,H)
• Y: cat@(A,B), cow@(C,D), `elk@F
• Z: cat@A, cow@C, elk@F

By tree:

• cat: X@(A,B), Y@(A,B), Z@A
• cow: X@C, Y@(C,D), Z@F
• elk: X@(F,H), Z@F

My expectation:

Gets latest checkout per (branch, platform) pair:

By platform:

• X in cat@B, cow@C, elk@H
• Y in cat@B, cow@D
• Z in cat@A, cow@C, elk@F

By tree:

• cat: (X,Y)@B, Z@A
• cow: Y@D, (X,Z)@C
• hawk: X@H, Y@F

Optionally, also fall back to checkouts at git topological (first) parents (would require major changes):

• dog would show (X,Y,Z)@A
• hawk would show Z@D

Some questions:

1. When looking at/aggregating a platform, which builds/tests should be included?
   a) in the latest checked out commit per tree (current)
   b) in any commit (this PR)
   d) in the latest checked out commit with at least one build in that platform
   c) in the latest checked out commit per {tree, identifier} (my suggestion)

2. When looking at/aggregating a tree, which builds/tests should be included?
   a) in the latest checked out commit (current)
   b) in the latest commit and same tree
   c) latest by identifier
   d) latest by platform by identifier

[mentonin]

I think this will overcount tests (getting all builds tests in the time range, including repeats/redos).

[alanpeixinho]

You have a good point here.
We might need to go for the latest checkout where that hardware is present. What you think?
I think this is another point in favor of #1947.

[alanpeixinho]

I have changed so we get the latest checkout on that hardware, without overcounting tests made in multiple checkouts within the time window.

[alanpeixinho]

I think we might need to iterate a little more on what should be counted as "current" for a given hardware platform or tree.

Deciding which builds and tests to show for a given platform should be the "easier" part, but deciding which hardwares, builds and tests to show for a given tree is complex

Example complications
Consider the following commit graph, where commits are marked according to the key:

• #: checkout with tests on platform X
• *: checkout with tests on platform Y
• $: checkout with tests on platform Z
• branch names in parentheses

  G  (dog)
 /
A#*$ ---------B#*  (cat)
 \
  C#*$ --- D* --- E  (cow)
            \
             F#$ --- H#  (elk)

Assuming every checkout has at least one test, latest_checkouts maps: cat -> B, elk -> F and cow -> D (no checkout on E yet). dog points to nothing (checkout on A declared cat for its branch)

Old implementation:

Only tests at latest_checkouts. Tests shown:

By platform:

• X: cat@B, elk@H
• Y: cat@B, cow@D
• Z: none

By tree:

• cat: X@B, Y@B
• cow: Y@D
• elk: X@H

New implementation:

Gets all tests in range. Tests shown:

By platform:

• X: cat@(A,B), cow@C, elk@(F,H)
• Y: cat@(A,B), cow@(C,D), `elk@F
• Z: cat@A, cow@C, elk@F

By tree:

• cat: X@(A,B), Y@(A,B), Z@A
• cow: X@C, Y@(C,D), Z@F
• elk: X@(F,H), Z@F

My expectation:

Gets latest checkout per (branch, platform) pair:

By platform:

• X in cat@B, cow@C, elk@H
• Y in cat@B, cow@D
• Z in cat@A, cow@C, elk@F

By tree:

• cat: (X,Y)@B, Z@A
• cow: Y@D, (X,Z)@C
• hawk: X@H, Y@F

Optionally, also fall back to checkouts at git topological (first) parents (would require major changes):

• dog would show (X,Y,Z)@A
• hawk would show Z@D

Some questions:

1. When looking at/aggregating a platform, which builds/tests should be included?
   a) in the latest checked out commit per tree (current)
   b) in any commit (this PR)
   d) in the latest checked out commit with at least one build in that platform
   c) in the latest checked out commit per {tree, identifier} (my suggestion)
2. When looking at/aggregating a tree, which builds/tests should be included?
   a) in the latest checked out commit (current)
   b) in the latest commit and same tree
   c) latest by identifier
   d) latest by platform by identifier

Good points.
Updated it here, so we are now using the (c) strategy for 1.
The point on 2 is a little bit more tricky.
Because would affect the tree page.
It might be safer to open a new ticket to discuss that. What you think?