RecyclerView is scrolling to top, while user is scrolling down (bad anchorPosition?)

[damiancioo]

After continuously scrolling down, after a while RecyclerView is switching his position to some element which was before, resulting in not smooth displaying of elements in it.
I only change NETWORK_PAGE_SIZE to 20, and switch to latest versions of:
roomVersion = '2.4.3'
pagingVersion = '3.1.1'

I added logs to GithubRemoteMediator, when scrolling down, there is:

15:21:54.884 RemoteMediator           D  load, type: REFRESH, state.pages.size: 0, anchorPosition: null
15:21:55.935 RemoteMediator           D  load, type: PREPEND, state.pages.size: 1, anchorPosition: null
15:21:55.999 RemoteMediator           D  load, type: APPEND, state.pages.size: 1, anchorPosition: 3
15:21:56.918 RemoteMediator           D  load, type: APPEND, state.pages.size: 1, anchorPosition: 0
15:21:57.748 RemoteMediator           D  load, type: APPEND, state.pages.size: 1, anchorPosition: 0
15:22:02.064 RemoteMediator           D  load, type: APPEND, state.pages.size: 2, anchorPosition: 40
15:22:04.697 RemoteMediator           D  load, type: APPEND, state.pages.size: 2, anchorPosition: 30
15:22:08.322 RemoteMediator           D  load, type: APPEND, state.pages.size: 4, anchorPosition: 81
15:22:10.000 RemoteMediator           D  load, type: APPEND, state.pages.size: 2, anchorPosition: 41

As You can see, anchorPosition after element 81 is going to be 41, which result in RecyclerView most likely going to that element.