Paginator changes

Posted under General

Despite all the caching that goes on, after page 5000 the performance of the post listing starts to degrade, getting as bad as 7 seconds per page on page 25000+. This is just a side effect of how OFFSET works in most databases.

I'm planning the following change: after page 1000, the default paginator will switch to a next/previous system that doesn't show the page numbers. This will remove the need for using OFFSET and the resulting sequential scan.

There's no easy way to update the post API to use this, so instead there will be a hard limit of 1000 pages. This is easy enough to get around though, just use the id metatag.

Updated by blaze1514

Like having your pilgrimage to the end of the /touhou tag abruptly ended on page 2134, for example. Essentially 'lost', but the posts are still there if you use the manual way instead of page-by-page...

Shinjidude said:
The endless danbooru scripts will probably need patched to work with the new system. (though I'm not sure how many people scroll all the way to page 1000)

Eh. Probably won't be too bad for UPW (not so "Unlimited" after all, huh.). It feels less elegant having to make a few exceptions for Danbooru, but I'm always looking for an excuse to patch things anyway. I'll take this chance to corner the market!

albert said:
There's no easy way to update the post API to use this, so instead there will be a hard limit of 1000 pages. This is easy enough to get around though, just use the id metatag.

How does the "limit" parameter affect things?


Is deleted posts are no longer visible using status:deleted a side result of this change? Is that intentional behavior? Some times I like to go through deleted posts to see if there's anything worth while for undelete, but now that seems to be impossible.

BCI_Temp said:
Interesting. What will this do to the URLs if you're working your way through a single popular tag for several weeks?

It's not elegant but there's a before_id query parameter now. You can see the transition take place here: http://danbooru.donmai.us/post/index?page=1000

If you have a page bookmarked, simply substitute the page with the appropriate before_id fragment and the next/previous links will start working for you. As an added bonus, you don't have to worry about new posts messing up the order of the pages.

RaisingK said:
How does the "limit" parameter affect things?

Limit works as it always has. It basically translates to posts per page. But the hard limit on 1000 posts per page has always been there and hasn't changed.

ttfn said:
I've updated Endless Danbooru Pages as well. Get it while it's fresh at http://userscripts.org/scripts/show/14360

Looks like you'll have to update the Endless Danbooru Comments script, too. The alternate page select now applies to that section, too, from the very beginning, so your script repeats the first page and stops. (And if you're going to fix this, might as well fix that comment search bug I mentioned on the discussion.)

Mine still works, though, thanks to the aforementioned change.

Are the paginators going to be replaced for any other pages, Albert?


That's what I thought. IMO it's so short now that it detracts from the usability of the comments page.

Oh well, I hope that endless comments thing gets fixed, I I'll have to look into installing if that happens.

I'm not sure where I should ask this, but anyway:
I favorite pretty much everything I like for no actual reason, and I know I'm probably an idiot for that, but when this system starts past page 1000 of my favorites, it gets all messed up. As far as I can tell it works fine when I'm seraching things normally, but when I go through my favorites it seems to reset to page 2 after this 'Previous' 'Next' thing starts up.
Is this some kind of bug?

So if I am geting this right you will have to change the way you serch dirasticly after you reach page 1000, ether in the post pages or your favorits. and the way you search after 1000 is either with a seperit script or with a updated URL ?