Danbooru

Danbooru 2

Posted under General

This might be way off in wonderland but is there a way of setting up a template for certain tags with regards to an image?

What I mean is have a base image of say a character with the basic tags that should be used. I can't decide sometimes what tags should be used and since tagging is not entirely uniform digging around to other images doesn't always help.

Also a template format for pools would be nifty. I use RaisingK's and MrGT's style for them and I assume that is more or less correct but a plug in template for that would be super nifty.

Lastly on the topic of tags, when you have say Hong Meiling and are tagging, do you tag braids and twin braids or is that redundant?

Siegmund said:
This might be way off in wonderland but is there a way of setting up a template for certain tags with regards to an image?

You can't guarantee that a character will always be drawn with any given tag. "Related Tags" gives you the basics, albeit with a bit of noise.

Lastly on the topic of tags, when you have say Hong Meiling and are tagging, do you tag braids and twin braids or is that redundant?

Twin braids already implicates braid. You tag the former and end up with both.

RaisingK said:
You can't guarantee that a character will always be drawn with any given tag. "Related Tags" gives you the basics, albeit with a bit of noise.

I guess I was thinking of more of a tutorial deal (unless I missed it), or make the related tags more relevant to the character.

I swear a few times the related tags are actually related to characters related to the one you are tagging but not the ones you might need. As an example not finding yellow eyes under the related tags for Marissa, but finding blue, brow, etc.

RaisingK said:
You can't guarantee that a character will always be drawn with any given tag. "Related Tags" gives you the basics, albeit with a bit of noise.

I personally wouldn't mind "Related Tags" just giving general tags as results since all of the other "Related" links cover the other three tag-types.

Algasir said:
I personally wouldn't mind "Related Tags" just giving general tags as results since all of the other "Related" links cover the other three tag-types.

I would. Maybe if it ranked them by frequency but this would be hell to do, I'm sure. Related characters, when applied to a copyright, always gives a ton of false positives. Related tags gives you the most popular characters and a bunch of general tags.

I'm reading a lot of good ideas here. Thanks for responding. And don't be afraid to post old Trac tickets. Everything is fair game.

Regarding nested tags:

There's a technical aspect and a usability aspect. Both are equally important and, in my opinion, both are equally problematic.

It's feasible to do nested tags right now, and maybe the way forward is adding some syntactic sugar.

You have two tags: hat and blonde_hair. When you search for both tags currently you might get someone with blonde hair who isn't wearing a hat. But what if someone added a blonde_hair+hat tag, to represent a single character who had both attributes? Then you could search for that specific tag. The sugar might be that whenever you tag something xxx+yyy it would also tag it with xxx and yyy.

But what happens when you tag something xxx+yyy+zzz? You would need xxx, yyy, zzz, xxx+yyy, yyy+zzz, and xxx+zzz. You can see how it starts spiraling out of control.

But even if we could come up with a clever solution, tag searches have a major constraint. They are by far the most heavily used part of the site. Any sort of implementation for searching has to be extremely fast or it simply will not work. That's why I think anything that doesn't flatten out these nested tags is going to be slow. Any sort of fast solution will probably involve either a custom index (either in PostgreSQL or in some secondary data store like Lucene or Sphinx), or a clever way of overriding existing datatypes and functions. I'll think about it but I don't have high hopes for this.

The other aspect is usability. Will nested tags make the site easier to use or harder to use? It makes a few tags searches easier obviously, but at what cost to taggers?

Say I have an image of three girls, all of them wearing hats, two wearing skirts, one wearing a jacket, and another wearing a scarf. These types of images are ubiquitous on Danbooru. How would you tag this image using nested tags?

(hat skirt jacket girl) (hat skirt scarf girl) (hat girl)

That's a lot of duplication. Maybe we could refactor...

(girl (hat (skirt (jacket scarf) ())))

I'll be honest. I don't want to type in Lisp code every time I want to tag an image. You would have to maintain this structure whenever you edit tags, also. Compare it to what you would tag it now:

girl hat skirt jacket scarf

No you can't find images of a girl wearing a hat, a skirt, a jacket, and a scarf at the exclusion of everything else, but people are also more likely to tag posts with the things you're interested in.

The other issue is that while nested tags are certainly a nice-to-have feature, how many people would use it? Most users of the site don't even bother with two tag searches, let alone three or more. Is it worthwhile implementing a costly solution for nested tags that only benefits 3% of users?

Nested tags are the type of thing where if you and a couple friends are the only users, I think it would be worth pursuing. But for a site that gets thousands of visitors a day I don't think it's tenable.

Regarding notes:

I agree that the UI for creating and editing notes could be improved a lot. Part of what makes jQuery appealing is that I would have access to a lot of concise syntax for manipulating the DOM and storing data in attributes. If I'm going to rework notes it may make sense to just rewrite the entire thing.

Regarding profiles:

I will probably never implement avatars, but if people are interested in adding more custom data to their profile page I'm not averse to it. I think Metafilter strikes a good balance.

Regarding security:

I agree it's a problem. I'm not sure if people care enough about security to want an SSL cert for the login page but I could try and take steps to eliminate the password_hash cookie. Perhaps a separate account type for API users.

Regarding DText:

I previously used Textile, which I hated, because it had a lot of behavior I didn't want and there was no easy way to only enable the features I want. For example, _text_ would emphasize anything in between the underscores. This type of thing was problematic when rendering wiki pages, and my impression of Markdown was that it had the same sort of excessive syntax and lack of customization.

There's also the issue of adding additional custom syntax. Markdown doesn't support post #nnnn or tag search link, so I'd be adding a layer on top of it anyway.

Other ideas:

  • Many image sites will actually offer multiple sizes of an image you upload. Flickr does this. Currently Danbooru offers three sizes: thumbnail, sample, and original. Maybe adding additional sizes makes sense to fit common monitor resolutions.
  • A mobile version of the site. This fits well with the above idea.
  • Instead of related tags, for popular tags maybe a user defined list of commonly associated tags makes more sense.

Read everything, and don't have much to say right now since the focus is backend stuff rather than interface and features.

Regarding avatars and signatures, I am adamantly opposed. Absolutely not. As "mean" as it sounds, I do not want you to socialize on the forums, I want you to treat it almost like a mixture of a customer service hotline and a trac system. That is, you ask for help, and propose changes. That's about it. It's work, not play.

Regarding nested tagging and semantic relationships, I have to agree with albert. It's a great idea, I've wished for it many times and being obsessed with information organization as I am, I would likely waste many, many hours working with it. But it's just a usability nightmare. It's too complicated for most people to use (we still have people confused that _ = space) and even for more advanced users it's a real time drain. And that's even if we don't retag any old posts. It would also need a ludicrously well designed tagging interface for everything to go smoothly.

Some of my "blue sky" feature requests have already been mentioned, but I'll go ahead and show my support and arguments for them along with some others.

Hierarchical tagging: I think this feature would greatly enhance the semantics encoded and the power we have to describe and query posts. It also provides for a way to generate more relevantly related tags as it would directly link tags to their truly related tags rather than depending on mere coincidence.

Hierarchical tagging will also provide a means to clean up the somewhat kludgey "combo tags", that lead to dilution of concepts and potential exponential specification. By "combo tags" I mean things like white_thighhighs, striped_panties, or short_twintails. In fact we could even use this to reify things like hair color, length, and style.

We'd probably need to experiment to find an efficient and usable way of implementing this, but I picture something like the following taglist:
"(ayanami_rei (hair short blue)(eyes red)(plugsuit white))"

Using that you could query characters with short blue hair with (hair blue short) without the need of a short_blue_hair tag.

Tag ontologies: I think by explicitly defining ontologies based on tag generality or specificity could be very useful, would provide semantics to a common category of implication, and would allow us to have shorter, less cluttered tag lists. For example we could define (miniskirt < skirt < bottom < clothing). That way anything tagged "miniskirt" would imply all of the above without having to explicitly state it in the taglist either via implication or otherwise. We could then use those semantics to say query bottom to get all instances of miniskirts, pants, bike_shorts, etc. Combined with the above we could query "(clothing blue)" to get any instance of blue clothing of any type.

Expanded tag types: Even if for usability sake we only color a handful of them, I think categorizing tags might be useful. It could be used to link things in the wiki. Also being able to query by count or existence of these might be useful in various contexts. For example automatically tagging exact duplicates, you often would like to exclude tags of a "meta" type, such as resolution size, quality, etc.

Spatial tagging: The ability to associate spatial properties (specifically coordinates) to tags. There are a couple things I see as being potentially useful with this. One, we could point out the characters tagged (a la Facebook's tagging) This would eliminate the necessity of using notes for this purpose, which is pretty ugly, and could be very helpful for people identifying characters of uncommon series. It could also be used to automatically tag colors, which would be an easy way to automatically increase our semantic information.

This could be done either by a point sample at the coordinates a tag is identified with, or more sophisticated analysis could be done. I don't know if it's something that could be incorporated as-is, but there is an interesting face detecting script I came across a while ago, that automatically detects anime faces and returns their position, hair color, eye color, and skin color. The code is available, and it's something I've been thinking of playing with for a while now.

Guided semi-automatic tagging: To aide in quickly and easily providing accurate and rich descriptive taglists, the site could automatically provide suggested tags. This could be done by a number of means.

  • Cross referencing synsets of tags scraped or queried from other sites. This could be done with Pixiv uploads (as has been suggested in the past, and the Pixiv Translation Plus script already provides a jumping off point for this), other *boorus that use similar tagging schemes as us could easily be queried by IQDB or MD5 hash, and the existing tags there suggested.
  • Automatically suggesting related tags, especially if the accuracy of this can be improved via tag hierarchies.
  • Things such as character could be inferred by hair-color, eye-color, etc, as determined by the above facial detection script or something similar.
  • The above methods used in synergy and/or iteratively based on existing input. For example if a facial detection script detects two characters one with blue hair & red eyes, and another with red hair and blue eyes, and pixiv provides the tag "エヴァ", and the system knows rei_ayanami is almost always tagged (hair short blue), the system could automatically provide "neon_genesis_evangelion (ayanami_rei (hair short blue)(eyes red))(soryuu_asuka_langley (hair long red)(eyes blue))" as a list of suggested tags.

Of course as has been mentioned with any sort of automatic tagging in the past these should be merely suggestions and need to be manually selected by a human user. But by doing this we could vastly shorten the time and greaten the ease it takes to accurately and thoroughly tag posts. This would make it easier to go back through and enrich old posts as well as encourage new and casual users to better tag what they post.

Semi-automatic duplicate detection: Use IQDB or a clone thereof to automatically search for visual duplicates above a given threshold, optionally comparing resolution, format, filesize, or filesize per area to gauge quality.

This could warn users against uploading potential duplicates, as well as automatically tag the existing duplicate, and provide suggested tags based on the duplicate (maybe excluding things like "Meta" categorized tags).

If a file is detected as matching above a threshold, the user would need to verify this fact and examine any suggestions, if the pic is automatically assessed to be of higher quality they can then upload without a warning. If it is assessed to be of lower quality, the user would need to override to upload, and it could potentially be flagged for review.

Clustering based on tags / features / metafeatures: This could be used for sorting, drilling-down, and suggesting similar tags.

Suggested posts: Based on a user's uploads and favorites, and coupled with the above clustering, suggest pics the user might like.

Automatic thumbnails for flash / video

Built-in auto-suggestion / color coding of tags a la Danbooruup: This would mitigate the need to update the extension. It's pretty much the only feature of it I use anymore.

Checking new tags for spelling errors / alternative variants before creating them (and verifying the correction with the user before continuing): I think this is a large contributor to the huge number of largely useless 1-use tags we have.

Embedding meta-information (taglists) into images: I'm not sure if I'm fully with this one, but it was mentioned above, and it's something I've thought about in depth before.

I think it would be useful to the end-user who downloads the pics, and if it catches on elsewhere could be another means to help auto-tag here or be used personally as things drift across the Internet, but I'm wary of changing the hash of the file.

Perhaps this could be done optionally on-demand, with the hash being computed only after meta-data has been stripped?

Normalizing use of DTEXT / HTML: We should use the same syntax everywhere, including translation notes (which currently use only HTML). I'd argue against disallowing style / formatting tags in notes though.

Better documentation throughout: with the ability for privileged users the ability to edit. Perhaps rankings set such that only mods can change some pages (like rules) whereas privileged can edit "how-to" lists, etc.

Richer API such that almost any feature can be used by a 3rd party application. Updated documentation on the API.

Better documented & streamlined installation of Danbooru on multiple platforms The ability to host in a subdirectory or shared server.

A polling system to automatically count yays/nays in the forum. This seems to be the most common form a thread takes here and could be useful both for whoever ends up deciding / enforcing the issue as well as for people who have an opinion, but no real commentary.

----

Ok, this sort of exploded as I got started. It's already too big, and I could probably keep going with "blue sky" ideas. This post probably needs a page of it's own.

Obviously many of these are sort of out-there, and many are probably infeasible for efficiency's sake, but I think they are all ideas that I think could be very useful, and I think are worth exploring or at least consideration.

Most of them are ideas I've been thinking about for some time, though I've never really done any sort of work on them or looked into particular implementations.

Updated

Ok, I took forever writing my post so I missed Albert and
jxh2154's responses to nested/hierarchical tagging. As for the format, I'd probably use Albert's "(hat skirt jacket girl) (hat skirt scarf girl) (hat girl)" (as alluded in my examples", it's a lot of duplication, but it's the easiest to understand and parse in my opinion.

As for usability, I would suggest perhaps using something other than a simple text box. Perhaps using something like a filetree with newly inputted tags placed in the hierarchy depending on what is currently selected. I agree that forcing users to type LISP is a bad idea.

Also if we had better related tags and semi-automatic suggestions, it could make input faster and easier than one might think. The more the system can automatically suggest, the faster and easier tagging of any form gets.

As for the technical issues, I have no real good solutions, and it's probably the idea's Achilles heel. If flattening the levels of the hierarchy fixes things, it wouldn't be hard to do internally, but I'm afraid the number of combinations that would need to be added as internal combo tags would be too excessive to be worthwhile. Things such as counting tags and the like might also become complicated.

I still think that it would be an immensely useful feature, and I'm sure a large proportion of the power users on the site would use it. If the majority of users don't, perhaps using two taglists (one hierarchical, one flat), and two query methods (one shallow and fast, one deep and slow) might help mitigate the additional work load? Perhaps hierarchical search could be an additional feature to entice base members to upgrade? Either way would limit the impact of more costly searches.

In any case, I don't think we should dismiss it out of hand, and at least think of ways in which it could be implemented successfully.

Updated

Mmm, the more I see and think of it, the more I'm on albert's side regarding nested tagging. I just can't think of a way it wouldn't be so complicated in actual practice as to be unfeasible.

edit: We have an indication in the tag wiki entries for what tags are aliased to and from any given tag. I'd like to see a similar thing for implications, as those can sometimes be just as (if not more) confusing.

Updated

Shinjidude said:
Semi-automatic duplicate detection

I can't believe I forgot this. I use the bookmarklet to upload, and as things stand, I am basically forced to click the "Find Similar" button every time I upload something, because a fair percentage of the time, the results will include a 95-99% similar image that has already been uploaded, but happens to have a few different pixels and a different checksum as a result. The only thing keeping me from switching to the Firefox plugin for uploading is the fact that I can't do this "Find Similar" search, which is a pain, since the plugin is better in just about every other aspect. An automatic IQDB check for similar images and a warning if you try to upload an image greater than, say, 90% similar to one already in Danbooru, would be a huge help.

albert said:
Regarding nested tags:

As I know virtually nothing about the inner workings of Danbooru, I'll have to take your word for it if you say there's no good, fast way of implementing this. If you do think of a way, though, don't hold back for the sake of simplicity for users; I, for one, could stomach having to enter LISP code for every upload if it meant gaining all that additional search power, and even if hierarchical tagging didn't catch on among the majority of uploaders, the userbase is large enough that I'm sure new posts would end up being retagged hierarchically.

Make it easier to grab the resize widget in the translation notes. I can't tell how often I get frustrated by accidentally a moving note that I just put in the right spot by not carefully edging my mouse into the tiny little space where the resize widget is.

I also second the request for a "translation mode."

UI stuff:

  • Translation and notes don't work on iPhone, since you can't click on things. This might apply to other mobile devices too. Quick fix: a link on the side that expands every note, but I'd like to correct them too.
  • If two people add new notes at the same time, neither of them will notice until they reload it some random time later and see two sets of translations. Admittedly I can't think of anything to do about this.
  • To the extent that people upload multiple-page comics as pools, a mode that can switch between images with AJAX loading to flip pages would be nice.
  • Loading related tags doesn't work half the time in Safari. (guess I should put that in trac…)
  • The default view in Tags isn't very useful; it could show post_tag_history instead.
  • A warning if you define a new tag by spelling something wrong or using spaces instead of _.
  • Some automatic tag-type setting; for instance, adding a new artist could set the tag type for that.
  • An upload progress bar (if there's a way to do that without custom Flash).
  • I can't tell what the pool index is sorted by. Also, maybe it should show the most relevant tags to the pool, if there's a good way to calculate those.
  • The first page of order:score for some larger tags is always filled with everyone images and flash videos. This comes back to finding the relevance of a tag for some image.
  • On the other hand, order:views (counting views by privileged members only) might be interesting.

evazion said:
Switch to Imagemagick or something for resizing images. The current resizer has a long standing bug with generating corrupt thumbnails/samples sometimes. It doesn't handle transparent GIFs/PNGs well either. Also, turn on progressive JPEG encoding. It's a loseless option and last time I tested it generated on average ~10% smaller files.

It's already switched away from imagemagick because the resizer was very slow.
I have a design for one that's much faster again than the current one, but the FFmpeg JPEG decoder needs some new features first - it should be much faster and better about memory use than libjpeg (which I'd want to get rid of), but it only supports decoding entire pictures instead of one-row-at-a-time. That's kind of hard, so I don't have the time...

Updated

memento_mori said:
If two people add new notes at the same time, neither of them will notice until they reload it some random time later and see two sets of translations. Admittedly I can't think of anything to do about this.

One way of avoiding this might be to have users "check out" posts to translate and allow no one to edit the notes until they "check" them back "in". An automatic check-in can be employed after a certain time limit to avoid posts getting stuck in limbo. Just a thought.

Shinjidude said:

Semi-automatic duplicate detection

Most of this is way over my head so I apologize if I am being redundant here but is there a way to check for duplicates when they are using transparent backgrounds?

I bumped into this just recently and unless you happen to check the artist pictures already uploaded you end up adding a duplicate image.

--Image Tutorial--

Another thing I think would help new posters like myself is a somewhat robust tutorial involving what is Danbooru quality images. I have more or less just been using trial and error so far. While I have a much better grasp on what is Danbooru quality, I am far from confident.

Get the mods together and maybe pick a few images that fall into the "no thanks it needs to be better" category and list why. A sort of visual tutorial.

memento_mori said:
I can't tell what the pool index is sorted by.

I'm pretty sure it's sorted by most recently updated pool, sort of like the comment index.

Siegmund said:
Most of this is way over my head so I apologize if I am being redundant here but is there a way to check for duplicates when they are using transparent backgrounds?

I've never tried, but the "Ignore colors" checkbox might be be helpful for that.

Siegmund said:
Get the mods together and maybe pick a few images that fall into the "no thanks it needs to be better" category and list why. A sort of visual tutorial.

This is actually an somewhat interesting idea. Maybe select a few images that just barely meet Danbooru standards due to bad art/TOS skirting and say "nothing worse than this"?

Rule based vandalism detection

It should be feasible to detect any kind of vandalism automatically, because vandals usually tend to show a specific behavior. Examples for these behavior are:

  • Removing a huge amount (>70%) of tags from an image.
  • Continuous removing of a few tags from a huge amount of images.
  • Adding of not used or rarely used tags.
  • The speed of operation.
  • ...

The implementation should be relatively unproblematic. Store the last n modifying actions of a user and weight them based on rules, that are formal descriptions of the above mentioned behaviors. If the overall weight exceeds a certain threshold the user could be mentioned in a query for further manual moderation, moderated automatically, slowed down by reducing the allowed number of modifying actions per hour, ...

Of course there is the risk of false positives, so I would go with manual moderation and an automatic correction of the rules.

DschingisKhan said:
You should read up on RDF. I think you'd like it.

Thanks for pointing out that one. :) RDF goes straight into the direction that I had in mind.

albert said:
Many image sites will actually offer multiple sizes of an image you upload. Flickr does this. Currently Danbooru offers three sizes: thumbnail, sample, and original. Maybe adding additional sizes makes sense to fit common monitor resolutions.

An option for jumbo-sized thumbnails would be nice. Bigger thumbnails would make tag scripting a lot easier.

Shinjidude said:
Normalizing use of DTEXT / HTML: We should use the same syntax everywhere, including translation notes (which currently use only HTML). I'd argue against disallowing style / formatting tags in notes though.

I don't think translators should have to deal directly with formatting. Ideally, they would just describe the semantics of the text using some XML-ish syntax and let the site handle the actual formatting. For example, sound effects would be marked with a <sfx> tag and the site would format them appropriately, rather than relying on translators to format them manually.

This way translators don't have to mess with things like font sizes directly, so translations will be formatted more consistently. Also, it would make it easier to format notes in different ways if need be (for mobile devices, for instance).

1 2 3 4 5 6 7 13