Despite all the caching that goes on, after page 5000 the performance of the post listing starts to degrade, getting as bad as 7 seconds per page on page 25000+. This is just a side effect of how OFFSET works in most databases.
I'm planning the following change: after page 1000, the default paginator will switch to a next/previous system that doesn't show the page numbers. This will remove the need for using OFFSET and the resulting sequential scan.
There's no easy way to update the post API to use this, so instead there will be a hard limit of 1000 pages. This is easy enough to get around though, just use the id metatag.
Like having your pilgrimage to the end of the /touhou tag abruptly ended on page 2134, for example. Essentially 'lost', but the posts are still there if you use the manual way instead of page-by-page...
The endless danbooru scripts will probably need patched to work with the new system. (though I'm not sure how many people scroll all the way to page 1000)
Shinjidude said: The endless danbooru scripts will probably need patched to work with the new system. (though I'm not sure how many people scroll all the way to page 1000)
Eh. Probably won't be too bad for UPW (not so "Unlimited" after all, huh.). It feels less elegant having to make a few exceptions for Danbooru, but I'm always looking for an excuse to patch things anyway. I'll take this chance to corner the market!
albert said: There's no easy way to update the post API to use this, so instead there will be a hard limit of 1000 pages. This is easy enough to get around though, just use the id metatag.
Is deleted posts are no longer visible using status:deleted a side result of this change? Is that intentional behavior? Some times I like to go through deleted posts to see if there's anything worth while for undelete, but now that seems to be impossible.
If you have a page bookmarked, simply substitute the page with the appropriate before_id fragment and the next/previous links will start working for you. As an added bonus, you don't have to worry about new posts messing up the order of the pages.
RaisingK said: How does the "limit" parameter affect things?
Limit works as it always has. It basically translates to posts per page. But the hard limit on 1000 posts per page has always been there and hasn't changed.
Looks like you'll have to update the Endless Danbooru Comments script, too. The alternate page select now applies to that section, too, from the very beginning, so your script repeats the first page and stops. (And if you're going to fix this, might as well fix that comment search bug I mentioned on the discussion.)
Mine still works, though, thanks to the aforementioned change.
Are the paginators going to be replaced for any other pages, Albert?
I'm not sure where I should ask this, but anyway: I favorite pretty much everything I like for no actual reason, and I know I'm probably an idiot for that, but when this system starts past page 1000 of my favorites, it gets all messed up. As far as I can tell it works fine when I'm seraching things normally, but when I go through my favorites it seems to reset to page 2 after this 'Previous' 'Next' thing starts up. Is this some kind of bug?
It isn't exactly repeating pages, but yeah, favorites are screwed up after page 1000. Using any order besides order:id_desc breaks the Prev/Next system--clicking Next effectively adds "id:<NNNNN" to your search, where NNNNN is the id of the last post on your current page. You'll have to settle for fav:Sideswipe order:id_desc.
So if I am geting this right you will have to change the way you serch dirasticly after you reach page 1000, ether in the post pages or your favorits. and the way you search after 1000 is either with a seperit script or with a updated URL ?
