10 | 20 | 40 | 60 | 80 | 100 per page
Oldest First | Newest First

Review Process and Transparency

Bonjour everyone! First, thanks for all your feedback and excitement so far, it is really encouraging.

One thing that came up multiple times in my last topic was the review process, and having it open for everyone to see. This was something we thought about very heavily when we started the site. We looked at all the other sites that have review processes, and what could be done to make it better for all involved.

Currently, the application system is 'confidential' because if someone gets declined, that may not be a very good feeling, no. Also, as Amara Fox pointed out, there can be a lot of jibing and stuff said that is no-so-good in nature based on what an artist applies with.

The other interesting tidbit is that reviews are even confidential between reviewers! The way the system is set up, a reviewer cannot see others comments or their decision whether to accept or decline. This is so that every decision is impartial and not based upon what another reviwer has said. Comments can only be seen when the application is done, and even then there is no name associated with them.

We will have sample portfolios soon which will show what a passing one is, and what a not passing one is. The easy way to find out what a passing one is, is to look at the art on the site! :)

Also, there is the critique forum which the reviewers will parouse. If you want to know if a piece of artwork has the muster to pass, post it there, and a reviewer might stop by and let you know if it will pass, and if not, show you why, using the exact same criteria we use to review the portfolios.

As part of transparency though, well, translucency perhaps, I will show you a screenshot from the application review system. Also because it is a very nifty thing, I think! A couple, even. The first is of the app review screen.

Okay. Now you have seen what a reviewer sees when they review, and all that they see when they have to make a decision. (Except for the whole pixelating thing. It would be very hard to review with that in effect!)

This is the list screen. Note that when a reviewer submits a decision, it is marked as "Submitted" and they cannot view it anymore until it gets 3 reviews total.

Pretty nifty, no?

For those curious about the criteria we go by, there is a link to it when you are applying, but for your convenience (provided you are logged in), here is a link to our submission criteria.

As with any suggestion, we will think about it. And we will think about transparency, perhaps an option the applicant can set. But for now, our system is set up to be as impartial as it can be, and opening the in-progress reviews up to the public would also mean opening them up to the other reviewers. Perhaps finished ones, maybe. Again, stuff we will think about.

I will be happy to answer questions in the meantime!

Personally I like that you keep applications confidential. Public applications just create drama. As long as you plan to add sample portfolios(a fantastic idea), there should be no issues.

Interested applicants can just take a look the the uploads to get an idea of the skill required. :)

What do we have to do if we would like to volunteer to help review? I had offered to help review for yerf right before they went down the most recent time, but seeing as that's going nowhere fast, and this seems like a much more efficient system, I'd like to do what I can to help, when I have the time.

Now that you mention the fact that currently, the reviewers can't see each others' reviews (it'd seriously be too easy to bump a friend's app in the right direction if they wouldn't otherwise get in . . . or vise-versa for someone the reviewer doesn't like), I like the idea that it's hidden. Makes a good drama llama repellent. Do the applicants know who said what in the review or not? I don't remember what anonymity ended up being kept or not.

I really like the automated, efficent systems on this site.

I still like public applications more, or at least a few samples so new applicants know what to aim for. I really felt like I was shooting in the dark when I submitted my app.

I think a sort of "status bar" would be nice, also. I really would like to know what direction my application is going in. I'd also like to see commentary (shown to me as anonymous) as it happens.

I think the way you have it now is perfectly fine. :3 In making it so that the other reviewers don't know what they've rated the application.

Perhaps if an applicant is declined, they can select whether to make their 'failed' application public or not? I would allow it, anyway. I'm not sure how 'common' it would be for someone to show off that they were turned down, but that allows a certain amount of extra translucency to the ap process.

A status bar or real-time commentary on my ap would be interesting, but not necessary.

Perhaps a notice somewhere (anywhere) saying what the current ap turn around is? For EX: "The average application is reviewed in X days, Y hours. The oldest application in the queue was submitted on [date]."

But I'm a statfiend. :3

Turn around time is fairly quick, looks like anywhere from about a day to three or four at most. =)

Hee - well, it's not something to work on right away while the gallery is still getting on its feet and turn-around is quick. But should it ever become "The" hotspot to upload, things might slow down a bit. ;)

I'm a stat fiend too, I admit. When we have more data in the database, I'll probably put up something like, "Application acceptance rate" for total and past month, etc. Stuff like that. A status bar sounds good, although right now it seems pretty minimal. "Your application is being reviewed, has finished being reviewed, or has been sent." Those are pretty much the categories. If there was a status bar, what information exactly would people want to see in it?

Our system is pretty streamlined. In the first rush after opening applications, the reviewers completed over 30 applications in 10 days. Not bad. :)

Since opening up applications on October 16, 15 days ago, ArtSpots has completed 48 applications. 28 of those were accepted. This is a 58.3% acceptance rate. However, I think this is mildly skewed because a lot of the initial rush were Yerf artists. :) I'm told (and expect) that after more data comes in, the acceptance rate will be more around 30% or so.

Oldest application in the queue right now is October 29, 2006 at 03:30 PM EST. Admittedly, of the ones waiting to be sent out, the oldest is from October 21st. It's done, just needs to be finalized and sent.

Sadly, there is no field currently to mark the time when an application has been sent. Technically, I could check submitted at vs. when the last review was made, but that does not take into account redlining or waiting to be sent out. Seeing as how we aren't a company trying to measure metrics, I don't see much point to add one right now. ;) Turn-around does seem to be a few days, a week at most.

*rolls on stats* :D Hee! Thanks for those. ;)

What's sad is that I do that sort of thing for fun. >_> A friend of mine got a new gallery on another site and I tracked her hits, comments, favorites recieved, and watches for two days. And if she uploads a third piece to the gallery, I'll do it then too, probably. :B (Maybe... I can stop anytime I want to! I swear!)

I'm not in any lather to have any stats-capable things implimented since it's really only for my personal squee than anything else. I'm a PR Major; I think it's encoded somewhere. :B

I can tell a lot of thought went into administrating this site and it's really done well.

Although I most likely won't apply for artspots, if a site had to be administrated for high quality art it would be in this manner.

I really like that everything with the application process is so anonymous. Not that I'm saying I wouldn't give in to the option to have my application seen, but it does peevent alot of unessacary drama, which no one needs.

I, too, really like this review system. It gives viewers of the site a lot less drama to wade through and more art to look at quicker!

Like Nevar, I also would like to volunteer to be a reviewer if there's a call for them =)

Klawzie said:

Perhaps if an applicant is declined, they can select whether to make their 'failed' application public or not? I would allow it, anyway. I'm not sure how 'common' it would be for someone to show off that they were turned down, but that allows a certain amount of extra translucency to the ap process.

I like this idea! That way, there's still anonymity between reviewers during the time it's important to have that, and yet people can still see what they're getting into.

Maybe accepted artists can make their apps public, too, just so people can see exactly what the reviewers liked about their art and such.

I was turned down, so I stand by my offer if anyone coding the place wants to take me up on it.

EDIT:

Actually, that sounded sort of bad. what I meant to say was - I think before Yerf had a forum, it sort of had a goofus and gallant-type thing where, basically, six images with a description on each, and what about the piece would be cause for rejection or admission.

The only thing I actually liked about Yerf's new review process was you could see how someone at your own skill level did.

Vekke said:

Maybe accepted artists can make their apps public, too, just so people can see exactly what the reviewers liked about their art and such.

We are working on examples, yes, and pieces that show what reviewers found and did not find in a piece of artwork. I do apologize for the delay, but we cannot haphazardly put someone's art onto a page using it as an example without their permission. However.. hmm! I think later I will look through a few pieces and link to them, at least. If folks do not want their pieces linked to, I will remove or change the links.

M. "billysheep" Farmer said:

Actually, that sounded sort of bad. what I meant to say was - I think before Yerf had a forum, it sort of had a goofus and gallant-type thing where, basically, six images with a description on each, and what about the piece would be cause for rejection or admission.

The only thing I actually liked about Yerf's new review process was you could see how someone at your own skill level did.

We are working on something like that, see above! We do look at different criteria than Yerf did, focusing on measurable qualities like anatomy, perspective, and lighting. I have seen some beautiful renderings and pieces with such style, but the applicants were still rejected because we do not look at how well a piece is rendered nor the style it is rendered in. Those are artistic decisions and cannot be measured. Once the fundamentals underneath are grasped, everything falls into place above them.

Oh, there I go babbling again. Needless to say, we will have examples up soon. I have been thinking about, and it is only thinking at this point, after we get through our current influx and post such examples, of resetting all the reapply times so that people can determine whether they want to try again right away, or wait and practice.

I was wondering if the link to the app guidelines could be in the initial application page (that is, before you hit the Apply button) just so people can see what they get into? Because I was trying to decide whether to apply and was searching everywhere for the criteria, but I didn't find the criteria page link until I decided to apply. If that...makes sense.

Also I don't know if I'm being dense but it is possible for accepted members to see his/her own redlines/crits as well?

Thanks for the quick turn-around time (2 days :O)!

Msh said:

I was wondering if the link to the app guidelines could be in the initial application page (that is, before you hit the Apply button) just so people can see what they get into?

Ah, yes! That would be what we call, um, a minor oversight. Others have mentioned that very exact thing as well. It will be fixed come tomorrow.

Also I don't know if I'm being dense but it is possible for accepted members to see his/her own redlines/crits as well?

If there is a redline, the link goes out in the email. If there were any common crits those are also included in the email. Not all accepted applicants get redlines, mainly because that is something a reviewer might do before the letter is sent out. They are not required for accepted apps, and since the weekend the reviewers have been focusing most of their effort on trying to get through all the declined ones which, by definition, need them more. Even if one had an inclination to point something out in an accepted one, there just is not much time right now! :)

Others have been bringing this up, and we will be creating one of those FAQ pages to help explain some of this. We had a.. pretty big growth spurt we did not necessarily anticipate. We will get to these things soon! I do appreciate folks patience, and am glad you are enjoying the site so far.

Thanks for the quick turn-around time (2 days :O)!

We try to be efficient!