Click here for Done Deal Pro home page

Done Deal Pro Home Page

Loading

Go Back   Done Deal Pro Forums > General > Sites, Services, Software, & Supplies
FAQ Members List Calendar Search Today's Posts Mark Forums Read

Closed Thread
 
Thread Tools Display Modes
Old 01-26-2014, 12:37 AM   #321
JeffLowell
Member
 
Join Date: May 2005
Posts: 3,051
Default Re: New Black List Thread - Franklin Leonard answers your questions

Quote:
Originally Posted by UpandComing View Post
The difference is, students don't pay their teachers to grade their essays.
Whether through tax dollars or tuition, I'm pretty sure that teachers are being paid to grade those essays.

Quote:
As it stands right now, the system leans heavily towards "gut feelings", which doesn't sit well.
I understand the thought, but when those gut feelings are producing such remarkable correlation, doesn't it argue that there's a functioning, organizing system behind the "grades?" I know a little bit about math, and if something is random, it doesn't produce results like that.
JeffLowell is offline  
Old 01-26-2014, 12:51 AM   #322
UpandComing
Member
 
UpandComing's Avatar
 
Join Date: Dec 2012
Location: NYC
Posts: 1,429
Default Re: New Black List Thread - Franklin Leonard answers your questions

Quote:
Originally Posted by JeffLowell View Post
Whether through tax dollars or tuition, I'm pretty sure that teachers are being paid to grade those essays.
Yeah, but it's not coming directly out of students' pockets. For BL customers, the hit is more direct

Quote:
Originally Posted by JeffLowell View Post
I understand the thought, but when those gut feelings are producing such remarkable correlation, doesn't it argue that there's a functioning, organizing system behind the "grades?" I know a little bit about math, and if something is random, it doesn't produce results like that.
I thought this would come up. My response would be: Just because the system's results are positive, doesn't mean there aren't flaws in its design, or that it couldn't be made better.

Readers for a contest reading the same script probably often give similar scores as well. However, they are still required to grade using defined criteria. Why? To maximize fairness and decrease subjectivity in the process. It just increases faith in the process, especially when there's money being spent.

To an extent, the BL operates like a year-round contest, but with less accountability with regard to scoring.
__________________
I love being a writer. What I can't stand is the paperwork.- Peter De Vries
UpandComing is offline  
Old 01-26-2014, 01:06 AM   #323
Bitter Script Reader
User
 
Join Date: Jun 2009
Posts: 138
Default Re: New Black List Thread - Franklin Leonard answers your questions

Quote:
Originally Posted by UpandComing View Post
Readers for a contest reading the same script probably often give similar scores as well. However, they are still required to grade using defined criteria. Why? To maximize fairness and decrease subjectivity in the process. It just increases faith in the process, especially when there's money being spent.

To an extent, the BL operates like a year-round contest, but with less accountability with regard to scoring.
I see aspiring writers get hung up on this sort of thing all the time and I never quite understand why. In a creative industry, decisions about a script at any level - assistant, executive, studio head - are going to come down to a subjective "Do I like this? Is something I am passionate enough about that I'm going to commit a year's resources to making it?"

That's not a conclusion one arrives at by reading a script and and only arriving at PASS or CONSIDER after tallying up all the plus-and-minuses. PASS or CONSIDER (and especially RECOMMEND) is something that you know the instant you put down the script. At that point, the coverage or analysis is merely academic.

It's always amusing to me that aspiring writers will rail against the idea that their scripts should fit into some kind of rigid formula, then demand a rigid, Vulcan-like assessment of their work.

What I fear is people essentially wanting a system like a 10-point scale to rate each of the following: CHARACTER, CONCEPT, PLOT, STRUCTURE, and then to have some kind of accountable grade that determines PASS or CONSIDER based on the accrued points. So let's say 30/40 is the threshold for CONSIDER. So if you got a 10 on Plot, 10 on Structure, but only 5 each on Character and Concept, should the script automatically get a consider? What if the concept is weak and hackneyed? What if the characters are so bland no actor could be attached?

Worry less about improving the system for how reactions are delivered. Focus more on writing material that gets superlative reactions.
Bitter Script Reader is offline  
Old 01-26-2014, 01:26 AM   #324
UpandComing
Member
 
UpandComing's Avatar
 
Join Date: Dec 2012
Location: NYC
Posts: 1,429
Default Re: New Black List Thread - Franklin Leonard answers your questions

Quote:
Originally Posted by Bitter Script Reader View Post
That's not a conclusion one arrives at by reading a script and and only arriving at PASS or CONSIDER after tallying up all the plus-and-minuses. PASS or CONSIDER (and especially RECOMMEND) is something that you know the instant you put down the script. At that point, the coverage or analysis is merely academic.
I'm actually glad you brought this up. Some people on here have suggested using the actual labels "Pass/Consider/Recommend" for the BL (I think CJ Walley was one), which would actually be more clearcut than a range of numbers from 1 to 10. I could see this as an alternative to the system I mentioned previously, if only because the numbers from 1 to 7 now seem pretty meaningless.

Or maybe the numbers could be made more meaningful if Franklin attached explicit labels to them, as you hinted at earlier:

Quote:
Originally Posted by Bitter Script Reader View Post
They're more like 8 and above - tell your boss to read it immediately. 6-7 - close, but wouldn't recommend. 4-5 - average. 3 and under - not close.
Part of the problem with this whole thing is communication, which is important in any business. When you have a range as wide as 1 - 10 it is understandable for people to wonder what exactly every number means.

Quote:
Originally Posted by Bitter Script Reader View Post
What I fear is people essentially wanting a system like a 10-point scale to rate each of the following: CHARACTER, CONCEPT, PLOT, STRUCTURE, and then to have some kind of accountable grade that determines PASS or CONSIDER based on the accrued points. So let's say 30/40 is the threshold for CONSIDER. So if you got a 10 on Plot, 10 on Structure, but only 5 each on Character and Concept, should the script automatically get a consider? What if the concept is weak and hackneyed? What if the characters are so bland no actor could be attached?
You make a valid point here. See my suggestion above about a P/C/R system or explicit labels for scores 1 - 7.

Quote:
Originally Posted by Bitter Script Reader View Post
Worry less about improving the system for how reactions are delivered. Focus more on writing material that gets superlative reactions.
I assure you, the vast majority of my time is spent on improving my material. Really, I don't sit here all day twiddling my thumbs over BL However, Alfred brought up what I thought was a compelling argument, and I wanted to lend my support. And as a paying customer, I will never hesitate to express my opinion about a service, especially one that I care so much about.

Okay, I really do need to go get my beauty rest now. Goodnight fellas, was a great discussion!
__________________
I love being a writer. What I can't stand is the paperwork.- Peter De Vries
UpandComing is offline  
Old 01-26-2014, 01:26 AM   #325
JeffLowell
Member
 
Join Date: May 2005
Posts: 3,051
Default Re: New Black List Thread - Franklin Leonard answers your questions

Quote:
Originally Posted by UpandComing View Post
Yeah, but it's not coming directly out of students' pockets. For BL customers, the hit is more direct
There are a few students who pay for their own education. (I believe loans are available for this purpose? ) And whether it's students or their parents, the principle is the same: these teachers are being paid to grade essays.

Quote:
Readers for a contest reading the same script probably often give similar scores as well. However, they are still required to grade using defined criteria. Why? To maximize fairness and decrease subjectivity in the process.
Here's the crux of this: you are calling for a decrease in subjectivity in a process that has been proven to be remarkably objective. The BL is hitting .960. What percentage should they aim for?

Quote:
To an extent, the BL operates like a year-round contest, but with less accountability with regard to scoring.
Franklin publishes reams of statistics, and when the rare script falls outside of his high standards, he is totally accountable and fixes the problem. I don't know of a contest that comes close to his level of accountability.

To be fair, as impressive as the correlation is, it doesn't do the writer who gets a 7 instead of an 8 any good. I can't imagine anything - nor have I heard anything proposed - that would eliminate that.

Last edited by JeffLowell : 02-06-2014 at 10:13 PM.
JeffLowell is offline  
Old 01-26-2014, 02:24 AM   #326
Bitter Script Reader
User
 
Join Date: Jun 2009
Posts: 138
Default Re: New Black List Thread - Franklin Leonard answers your questions

Quote:
Originally Posted by UpandComing View Post
I'm actually glad you brought this up. Some people on here have suggested using the actual labels "Pass/Consider/Recommend" for the BL (I think CJ Walley was one), which would actually be more clearcut than a range of numbers from 1 to 10. I could see this as an alternative to the system I mentioned previously, if only because the numbers from 1 to 7 now seem pretty meaningless.
I aggressively disagree with the Pass/Consider/Recommend suggestion. I think it's really helpful to both the writer and potential readers just how "hard" a Pass the reader intends. A score of 1 pretty much indicates that the reader can't find ANY merit in the material at all - that getting this script up to Consider is going to take more than a mere polish.

A score of 6 or 7 allows for the possibility that the script is a fixer-upper that could be "flipped" with a reasonable amount of effort. As someone who uses the site, if I saw a logline that intrigued me and the rating was 6 or 7, I might be inclined to give it a read. If I see a 1 or a 2 - not a chance in hell.

There's also the fact that the script's average score plays a role in how it shows up in the top lists. If PASS is assumed to be a 1 or a 0 on the scale, than that rating is going to hurt you a lot more than a 6 or 7 will.

On the American grading scale, a grade below 60% is considered failling. So if you have a 59% average going into your final exam, it's not as dire for your advancement as if you have a 10%.
Bitter Script Reader is offline  
Old 01-26-2014, 02:31 AM   #327
Bitter Script Reader
User
 
Join Date: Jun 2009
Posts: 138
Default Re: New Black List Thread - Franklin Leonard answers your questions

Quote:
Originally Posted by JeffLowell View Post
Here's the crux of this: you are calling for a decrease in subjectivity in a process that has been proven to be remarkably objective. The BL is hitting .960. What percentage should they aim for?

Franklin publishes reams of statistics, and when the rare script falls outside of his high standards, he is totally accountable and fixes the problem. I don't of a contest that comes close to his level of accountability.
Two really good points made here. It's easy to forget that this debate about scores is largely using "outliers" as evidence of flaws in the system. Most of the time, the Black List readers are remarkably consistent. And in the instances when their feedback is contradictory, Franklin already has a really fair remedy. Confused what to do about a script that's gotten a 4 and an 8? Here's a free read to break the tie.

I can see the argument coming that this does you little good if you pay for just one read and get the 4. While that may be true, the possibility of a second opinion is one not offered at any production company or agency that you would submit to, and I'm not sure many contests have multiple readers on the same screening stage.
Bitter Script Reader is offline  
Old 01-26-2014, 03:26 AM   #328
CJ Walley
Member
 
CJ Walley's Avatar
 
Join Date: Dec 2012
Posts: 458
Default Re: New Black List Thread - Franklin Leonard answers your questions

Quote:
Originally Posted by UpandComing View Post
I'm actually glad you brought this up. Some people on here have suggested using the actual labels "Pass/Consider/Recommend" for the BL (I think CJ Walley was one), which would actually be more clearcut than a range of numbers from 1 to 10. I could see this as an alternative to the system I mentioned previously, if only because the numbers from 1 to 7 now seem pretty meaningless.
I did say something to that effect dude, but I don't want BL changing and I don't want numbers replacing with words.

Franklin has previously detailed some very valid reasoning for the 1-10 system. I respect the decision to use it and I believe both options were considered. Personally I found it very flattering when my first two screenplays got overall 7's. At such an early stage in my writing it was highly motivating to know there were professional readers out there feeling I was on the cusp of their recommendation. No, it was incredibly motivating.

Yes I would like to see an alternative to BL which uses a slightly more finite p/c/r system that only allows scores to be adjusted positively and doesn't work to the law of subjective averages. But I'm not suggesting it would be better. I'd just like to see how that system performs at identifying valuable scripts, complaints over subjectivity and in terms of conflict of interest. It could be significantly worse and I could see how.

I also think it's wrong that Franklin feels the need to, or is expected to defend, the differentiation in overall reader scores. There's nothing wrong with a wild variation in scoring. There's nothing wrong with polarised ratings. In my opinion it's an irrelevant debate, partly born out of defending concerns over a scripts average rating, and partly to stop writers throwing their money at the system like a lottery.

The fact is a script that isn't very generically appealing isn't going to perform will on a site that gives equal weight to positive and negative reactions. Just as a movie won't on the MetaScore system. But there is the chance of being spotlighted on one readers opinion alone. Failing that, the site scores and reader opinions do seem to carry some weight in querying.
CJ Walley is offline  
Old 01-26-2014, 03:56 AM   #329
CJ Walley
Member
 
CJ Walley's Avatar
 
Join Date: Dec 2012
Posts: 458
Default Re: New Black List Thread - Franklin Leonard answers your questions

The half price evaluation/free evaluation/pulled evaluation policies are not a remedy to reader subjectivity, they are a blatant and understandable panacea to our complaints. Offering them is wrong, and I say that as someone who's received a fair few. They are wrong because it causes BL to almost appear apologetic for subjectivity. They do have a side benefit of helping us gauge the general appeal of their script but that's not what BL is there to do.

BL should not be striving toward objectivity, and certainly not gauging its readers on that level, not that I think Franklin, Dino & Terry would ever do so. This only further makes subjectivity appear like a bad thing and compound the confusion we feel when we face it. And more often than not, our real issue isn't with the subjectivity it's our precious attitude to our scripts overall scores.
CJ Walley is offline  
Old 01-26-2014, 07:40 AM   #330
SundownInRetreat
Member
 
SundownInRetreat's Avatar
 
Join Date: Aug 2011
Posts: 1,892
Default Re: New Black List Thread - Franklin Leonard answers your questions

Quote:
Originally Posted by JeffLowell View Post
The stat you cite was my biggest surprise that emerged from the data… but upon thinking about it, I shouldn't have been surprised.

The scores aren't "pick a number between one and ten." They're more like 8 and above - tell your boss to read it immediately. 6-7 - close, but wouldn't recommend. 4-5 - average. 3 and under - not close.

If you take a reader who's read hundreds or thousands of scripts professionally and tell them they have to put a script in one of those four groups, it doesn't surprise me that most readers would pick the same groups as other readers. The numbers back that up - 24 out of 25 times, readers do exactly that. There's obviously a little subjectivity even within those groups, but 96% of the time, they don't vary by more than two points.

This isn't a pure science. It's writing. So people waiting for someone to come up with a formula that is perfectly accurate will keep waiting.

To try an analogy, readers aren't grading math tests, they're grading essays. If you handed a hundred English teachers an essay, I bet the grades would be as close as the BL reader results. Would one teacher give an essay an A while another gave it a B? Of course. Pretty rare that one would give an A while another gave a C.

No one is saying that English grades are completely random by their nature, because you can't come up with a defined methodology.
Common Sense quote of the month. Yet BL detractors, or those affronted by their personal scores, will ignore it as they've ignored pretty much everything that doesn't give credence to the notion they've suffered unfairly.
__________________
Gotta Let You Go

I gotta let you go...

'Cause now you've got to fly
(Fly high)
Fly to the angels
Heaven awaits your heart
And flowers bloom in your name

Last edited by SundownInRetreat : 01-26-2014 at 10:38 AM.
SundownInRetreat is offline  
Closed Thread


Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump


All times are GMT -6. The time now is 04:23 AM.


Powered by vBulletin Version 3.6.2
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Done Deal Pro

eXTReMe Tracker