New Black List Thread - Franklin Leonard answers your questions

[FranklinLeonard]

Re: New Black List Thread - Franklin Leonard answers your questions

Franklin:

Why is there no tag for "high concept" available on The Black List?

FA4

Because it's highly likely that people would abuse it, rendering it valueless within the system. We'll return it to the list of tags under consideration.

[finalact4]

Re: New Black List Thread - Franklin Leonard answers your questions

Franklin:

Why is there no tag for "high concept" available on The Black List?

FA4

[FranklinLeonard]

Re: New Black List Thread - Franklin Leonard answers your questions

Franklin-

Here's a different question. How do you decide in what order you put "top uploaded pilots" and "top unrepresented pilots" on the page ? I would think this would be in order of scoring but it doesn't appear to be that way. In fact there's a pilot in the "top unrepresented pilots" section where the scores aren't even made public. Thanks

It's not based on the average score. We try to incorporate all the ratings a script has received along with the # of ratings it has received. We currently use Bayesian modifiers to implement this, just like IMDb and Amazon do for their top lists.

One does not have to make their scores public for their script to be in the top list. They just need to have the scores to merit inclusion.

[FranklinLeonard]

Re: New Black List Thread - Franklin Leonard answers your questions

Franklin,
Just wondering, and forgive me if this has been asked and answered already (its a damned long thread here), but do you collect and analyze data on the reader's scoring patterns? To perhaps detect if certain of your readers are giving out an inordinate amount of high or low scores to scripts. And perhaps a step further, see if a particular reader is at odds with other readers in their scoring on a consistent basis, by seeing what other scores scripts they've read and rated have gotten.
Not that I'm saying you should make this known to writers who have scripts on the page, just for an internal QC check.

Yeah we do! Quite a bit actually.

If you've been following any of these threads (or my personal Twitter account), it's probably obvious just how big a data geek I am. If there's something we can track numerically, we're probably already tracking it and using it to make the site better. No where is that more true than with our readers.

[FranklinLeonard]

Re: New Black List Thread - Franklin Leonard answers your questions

First, I want to thank Franklin again for his tireless attention to our questions and issues.

Now that the interminable "secret shopper" thread has closed, I wanted to pick up on your mention of the recommendation algorithm. Could you tell us a little bit more about the factors that go into it and how it works? I assume it is what is used to generate the recommended scripts that appear at the bottom of the page when you view another script. When I check those out, however, they seem to vary widely (in everything from genre to subject matter) from the script that is being viewed. Is it different for the industry pros?

Also, another somewhat related question. In a separate thread, Wise asked about the number, and specificity of tags to include. I imagine that part of the answer to that question might hinge on how and if those tags are used to generate recommendations, so it would be helpful if you could touch on that as well.

Re: the recommendation algorithm, it's not different for pros, and you've astute to note that the recommendations often vary widely by genre and subject matter. This is because for script based recommendations we're looking for scripts that that got high ratings from other users who rated the original script highly. It really is as simple as "people who liked this script also like these scripts."

The recommendations to individuals are a bit more complicated. We're looking at all of the scripts you've rated, and all of the people who have rated those scripts too, and then all of the scripts they've rated that you haven't rated, and algorithmically making an assessment of our best guess as to what you would rate the script, based on comparing your ratings to other people's ratings of scripts you share in comment.

As for tags, you can include whatever tags you want, but the caution is that if someone goes and reads your script because of a tag, and it's not in evidence, you can reasonably assume that the person who sought it out isn't going to be very pleased. We generally advise people to tag your scripts only with tags that someone looking for those tags would be happy to read based on their search.

[CrissCross]

Re: New Black List Thread - Franklin Leonard answers your questions

Franklin-

Here's a different question. How do you decide in what order you put "top uploaded pilots" and "top unrepresented pilots" on the page ? I would think this would be in order of scoring but it doesn't appear to be that way. In fact there's a pilot in the "top unrepresented pilots" section where the scores aren't even made public. Thanks

[Exponent5]

Re: New Black List Thread - Franklin Leonard answers your questions

In a separate thread, Wise asked about the number, and specificity of tags to include. I imagine that part of the answer to that question might hinge on how and if those tags are used to generate recommendations, so it would be helpful if you could touch on that as well.

^^THIS^^

More info on tags, please!

[tinlizzie]

Re: New Black List Thread - Franklin Leonard answers your questions

First, I want to thank Franklin again for his tireless attention to our questions and issues.

Now that the interminable "secret shopper" thread has closed, I wanted to pick up on your mention of the recommendation algorithm. Could you tell us a little bit more about the factors that go into it and how it works? I assume it is what is used to generate the recommended scripts that appear at the bottom of the page when you view another script. When I check those out, however, they seem to vary widely (in everything from genre to subject matter) from the script that is being viewed. Is it different for the industry pros?

Also, another somewhat related question. In a separate thread, Wise asked about the number, and specificity of tags to include. I imagine that part of the answer to that question might hinge on how and if those tags are used to generate recommendations, so it would be helpful if you could touch on that as well.

[ATSayre]

Re: New Black List Thread - Franklin Leonard answers your questions

Franklin,
Just wondering, and forgive me if this has been asked and answered already (its a damned long thread here), but do you collect and analyze data on the reader's scoring patterns? To perhaps detect if certain of your readers are giving out an inordinate amount of high or low scores to scripts. And perhaps a step further, see if a particular reader is at odds with other readers in their scoring on a consistent basis, by seeing what other scores scripts they've read and rated have gotten.
Not that I'm saying you should make this known to writers who have scripts on the page, just for an internal QC check.

[UpandComing]

Re: New Black List Thread - Franklin Leonard answers your questions

Perhaps the Black List could add a top list for scripts on which reviewers have disagreed. If I was a producer that would be a list I'd keep an eye on.

I guess it would be a bit awkward to define, but I'd suggest something like any script for which two of the first three reader reviews disagreed by three or more points.

Cool idea! A "Scripts That Incite Passion" List

[Howie428]

Re: New Black List Thread - Franklin Leonard answers your questions

There are other websites where writers anonymously rate each other's work and the result spread is published. Overall average scores vary, but that's not the only thing that varies.

The range of opinion also varies. Some scripts produce tight agreement, while others get opinions spread across the range. There are even some scripts that get an inverted curve, with a group of people that love them and a group that really don't.

Having read a lot of scripts on one of these sites, it is my experience that the scripts on which opinion spreads are often the most memorable and in my view the ones with the most potential. Often they include material that challenges the reader or has an original aspect in either the story or the story telling premise.

Perhaps the Black List could add a top list for scripts on which reviewers have disagreed. If I was a producer that would be a list I'd keep an eye on.

I guess it would be a bit awkward to define, but I'd suggest something like any script for which two of the first three reader reviews disagreed by three or more points.

[CJ Walley]

Re: New Black List Thread - Franklin Leonard answers your questions

Duly noted. See my previous answer.

No worries dude

I beg to differ on this one. I think that in cases where there are two polarizing reviews and there is a chance that a brilliant script wasn't given a fair chance, it is completely reasonable to offer a half-price review.

I agree to disagree. I appreciate the intent but think it sends out the wrong message.

As for pulled/free evaluations, I think that quite a few customers have experiences where the reader clearly did a sloppy job (me being one of them). So I'm glad BL offers that policy, and commend them for it.

I agree. I wasn't covering sloppy reviews with my point.

[UpandComing]

Re: New Black List Thread - Franklin Leonard answers your questions

From what I can gather (certain posters are on my ignore list so I can't be sure) people are actually denouncing the fact that there is no rigid scoring system. That personal taste and judgement in a creative field is now wrong! Am I reading that right? And the Black List is unfair and unjust to use the exact same reliance on personal judgement that the industry itself uses??? Really???

I won't take the time to respond to this post (TL; DR); I will just say that I'm honored that a forum member who has me on his ignore list (namely because the poster wasn't able to come up with a response to a past argument of mine) would take so much time to respond to one of my arguments

I'll also say that if I pulled up some of said forum member's past rants against the BL, it would be interesting to see the almost 180 degree contrast in tone with what he currently writes. It's fascinating how getting repped can make you get amnesia. But then again, when I get repped, I'll probably begin suffering from that condition, too

[UpandComing]

Re: New Black List Thread - Franklin Leonard answers your questions

Here's the crux of this: you are calling for a decrease in subjectivity in a process that has been proven to be remarkably objective. The BL is hitting .960. What percentage should they aim for?

I'll just say one last time that I think that average seems strangely low. I can see the typical deviation being 3 points or less, but only 1.5 for a system with no defined scoring criteria for the overall score? I've asked FL to provide some more detail around it, but that question has strangely (and unfortunately) gone unanswered a number of times

To be fair, as impressive as the correlation is, it doesn't do the writer who gets a 7 instead of an 8 any good. I can't imagine anything - nor have I heard anything proposed - that would eliminate that.

True.

I aggressively disagree with the Pass/Consider/Recommend suggestion. I think it's really helpful to both the writer and potential readers just how "hard" a Pass the reader intends.

Kay, I see your point here. I'll revoke that argument. See, I'm reasonable

I did say something to that effect dude, but I don't want BL changing and I don't want numbers replacing with words.

Duly noted. See my previous answer.

The half price evaluation/free evaluation/pulled evaluation policies are not a remedy to reader subjectivity, they are a blatant and understandable panacea to our complaints. Offering them is wrong, and I say that as someone who's received a fair few. They are wrong because it causes BL to almost appear apologetic for subjectivity. They do have a side benefit of helping us gauge the general appeal of their script but that's not what BL is there to do.

I beg to differ on this one. I think that in cases where there are two polarizing reviews and there is a chance that a brilliant script wasn't given a fair chance, it is completely reasonable to offer a half-price review. Besides, it's a wise business decision on BL's part, as it'll only serve to generate repeat customers and more money, no?

As for pulled/free evaluations, I think that quite a few customers have experiences where the reader clearly did a sloppy job (me being one of them). So I'm glad BL offers that policy, and commend them for it.

[SundownInRetreat]

Re: New Black List Thread - Franklin Leonard answers your questions

I think it's really helpful to both the writer and potential readers just how "hard" a Pass the reader intends. A score of 1 pretty much indicates that the reader can't find ANY merit in the material at all. A score of 6 or 7 allows for the possibility that the script is a fixer-upper that could be "flipped" with a reasonable amount of effort. If PASS is assumed to be a 1 or a 0 on the scale, than that rating is going to hurt you a lot more than a 6 or 7 will.

On the American grading scale, a grade below 60% is considered failling. So if you have a 59% average going into your final exam, it's not as dire for your advancement as if you have a 10%.

+1 for more common sense. And IMO calling for a more vague scoring method is not borne out of improving the service but of ego once again. It sounds better for them to tell people (or even themselves) "it was a pass" as opposed to "it was a 3". A pass is just one step away from a consider - ooh, so close! (That's the illusion anyway) when in reality it could have scored 1 when the cut off for a consider is 8. Not so close now, huh?

The sad thing about all this is the current scoring system, ego-denting as it can be to many, actually, puts things into perspective and is less likely to mislead writers and kid them into continuing to try again, to throw more money away and to keep chasing a career they are just not skilled for. If a writer writes 4 scripts and they all score no higher than a 5 then that is the truth hitting him in the face and if he dismisses this then he's in denial. 4 passes on the other hand - well, you can forgive the guy for thinking he's still in with a shout.