Skip to content

Add filter startpoint option (height only) - #227

Closed
optout21 wants to merge 2 commits into
2140-dev:masterfrom
optout21:anchor2
Closed

optout21 wants to merge 2 commits into
2140-dev:masterfrom
optout21:anchor2

Conversation

@optout21

@optout21 optout21 commented Nov 28, 2024 •

Copy link
Copy Markdown

Motivations for this change:

  • to be able to specify a start point for filter download, just by height and not block hash
  • more fine-controlled start point.

Block headers and filter headers build on each other, and the chain is used to check consistency. It makes sense to require a block hash if one wants to start from a later point. This is not true for filters: they can be started processing from any point, without loss of confidence.

With this change, if one specifies an arbitrary height for filter startpoint:

  • the closest earlier known block hash is used for block header and filter header checkpoint
  • the provided height is used as starting point for filter download.

Anchor checkpoint can be set separately.

Background: This is useful in the 'balance checker' wallet use case, when the user may have the wallet birth height as an info (but no block hash); see Icepeek.

@rustaceanrob

Copy link
Copy Markdown
Member

Hi, apologies I am just getting back from a holiday. I will review this soon

@rustaceanrob

Copy link
Copy Markdown
Member

This is not true for filters: they can be started processing from any point, without loss of confidence.

Filters are keyed by the corresponding block hash. Without such a blockhash, there is no way to construct the filter. Also should be noted that the filter headers commit to the hash of the filter, which in turn commits to the block hash.

I am still unclear on the use case. Are you:

  1. Scanning filters after the anchor block header, ignoring some initial heights
  2. Attempting to check filters before the anchor block header

@optout21

optout21 commented Dec 6, 2024

Copy link
Copy Markdown
Author

The use case is this:
I, as a user, have the public info for my wallet: XPub, etc. I also have the hint for wallet birth height, i.e. the height of the first transaction in this wallet. Let's say it's 845,000. I don't have any block hashes, nor want I to look them up.
After entering this info in the kyoto-powered balance checker, we should:

  • automatically take a known earlier anchor point, in this case block 840,000.
  • download block headers starting from the anchor
  • download filter headers starting from the anchor
  • for filters we can start downloading later, so we can start with 845,000. Since the filter download is much slower than the header downloads, even a diff of 5000 blocks can make a noticeable difference.

@rustaceanrob

rustaceanrob commented Dec 6, 2024 •

Copy link
Copy Markdown
Member

I see, so the differential between the checkpoint and when the relevant birthday is. May I ask why we must resync from the birthday every time? If you have recovered all your scripts, you may safely sync from the last chain tip you saw.

As a side note, I've tested your app and I concur with you that syncing feels slow. I may fork your app and see if I can improve the sync time in other ways.

@rustaceanrob

Copy link
Copy Markdown
Member

Please rebase as there is a new CI script and the testing suite has changed.

@optout21

optout21 commented Dec 9, 2024

Copy link
Copy Markdown
Author

Please rebase

Rebased

@optout21

optout21 commented Dec 9, 2024

Copy link
Copy Markdown
Author

May I ask why we must resync from the birthday every time? If you have recovered all your scripts, you may safely sync from the last chain tip you saw.

If state has already synced, it's OK to continue from there. Only at the first sync it starts from the hint height, to speed up the process.

The typical use case is that you don't have a state, but always start from a new one. This may not be the best suited for kyoto (or compact block filters in general).

@rustaceanrob

Copy link
Copy Markdown
Member

With the recent performance improvements I think using the checkpoints should be sufficient enough, and the extra handful of filter messages is the cost of having simple logic internally.

@rustaceanrob

Copy link
Copy Markdown
Member

In the end I think we will add something like this via #605, except it specifies only give me the last N filters once you do headers sync.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants