Skip to content

rfc(indexer): Multiple IPNI chains #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alanshaw wants to merge 1 commit into
base: main
Choose a base branch
from
Open

rfc(indexer): Multiple IPNI chains #59

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions rfc/indexer-multi-ipni-chains.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# RFC(Indexer): Multiple IPNI chains

## Background

The indexer publishes an IPNI advert chain with adverts for index and equality claims. For a short period yesterday the indexer was in production and receiving ALL equality claims for all uploads. It was observed to be behind the current head of the advert chain by multiple hours. It also reported a [`Lag`](https://github.com/ipni/go-libipni/blob/d8f9f451cc1d5d123f9ed270304211e9bc3678e6/find/model/provider_info.go#L19-L22) value of over 2,000 adverts.

The IPNI chain is a linked list and there are only so many opportunities to paralleise that work to increase ingest speed. Being behind the current head by such a large amount suggests that an IPNI node outside of the same datacenter/region as the source chain is unlikely to be able to keep up. That is to say, the IPNI node that was being observed was in the same region as the indexer - cid.contact is not.

## Proposal

The best way to increase ingest rate is to shard into multiple chains. At the very least we should shard into 2 chains - one for equality claims and one for index data.

Unfortunately aggregating multiple hashes into a single advert is not really possible since we use advert metadata - so all hashes in the advert have to carry the same metadata in order to be aggregateable. Note: we already do this for index data, but it is not possible for equality claims.