New Black List Thread - Franklin Leonard answers your questions

Collapse

Announcement

Collapse
No announcement yet.
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Re: New Black List Thread - Franklin Leonard answers your questions

    Originally posted by Alfred Parker View Post
    By your own description of the process, there is no methodology for the readers to follow. Absent methodology, each reaction is by definition random. Every evaluation simply boils down to that individual reader's subjective taste, no? And that can easily differ between two readers. Which is exactly what happens at any prod co, contest, or anywhere you send a script.

    Because that's what I'm talking about -- not any specific situation -- but how in general, no one can know how the reader is going to react to a script. Thus, random. Just pointing it out as a reality of the business.
    Have to admit, there is validity to this statement. When the only criteria for scoring a script is a "gut feeling" about whether one would recommend it to a peer/superior in the industry, that inherently creates randomness.

    Yes, there is subjectivity in any scoring system (as is frequently argued in these forums); I won't dispute that. But some systems invite more subjectivity than others. A lot more. You can't directly compare the BL to contests, because their overall assessments are usually tied directly to specific scores (whereas, as is noted in BL's policy, it's overall score is not a direct reflection of component scores).

    It's why I find the statement that "fewer than 4% of scripts with more than one paid rating (that's one in 25) had a standard error of mean >= 1.5 (e.g. 5,8 or 4,7, etc.)" a bit questionable. You'd think a system like this that relies so heavily on gut feelings would exhibit more disparity in scores. But -- even if these numbers are spot-on accurate -- it doesn't change the fact that a lack of defined methodology by definition makes a scoring system random.

    The thing is, when you're paying for a service, you hope for a little bit more structure around it than that.

    I can understand why there would be limited disparity among scripts scoring the highest (8s, 9s, 10s). But for everything scoring a 7 or below, it does very much seem like a crapshoot as to where you place along that scale.

    Despite my belief in the power of and necessity for the BL, I continue to think that this is one of its biggest, most obvious flaws. I understand the desire to give readers flexibility -- but I kinda wish the BL would try to understand more that the way it currently works makes many users feel like they could easily be throwing away their money (and is a source of much of the frustration often thrown around here).
    Last edited by UpandComing; 01-25-2014, 10:12 PM.
    "I love being a writer. What I can't stand is the paperwork.-- Peter De Vries

    Comment


    • Re: New Black List Thread - Franklin Leonard answers your questions

      Originally posted by UpandComing View Post
      It's why I find the statement that "fewer than 4% of scripts with more than one paid rating (that's one in 25) had a standard error of mean >= 1.5 (e.g. 5,8 or 4,7, etc.)" a bit questionable. You'd think a system like this that relies so heavily on gut feelings would exhibit more disparity in scores. But -- even if these numbers are spot-on accurate -- it doesn't change the fact that a lack of defined methodology by definition makes a scoring system random.
      The stat you cite was my biggest surprise that emerged from the data... but upon thinking about it, I shouldn't have been surprised.

      The scores aren't "pick a number between one and ten." They're more like 8 and above - tell your boss to read it immediately. 6-7 - close, but wouldn't recommend. 4-5 - average. 3 and under - not close.

      If you take a reader who's read hundreds or thousands of scripts professionally and tell them they have to put a script in one of those four groups, it doesn't surprise me that most readers would pick the same groups as other readers. The numbers back that up - 24 out of 25 times, readers do exactly that. There's obviously a little subjectivity even within those groups, but 96% of the time, they don't vary by more than two points.

      This isn't a pure science. It's writing. So people waiting for someone to come up with a formula that is perfectly accurate will keep waiting.

      To try an analogy, readers aren't grading math tests, they're grading essays. If you handed a hundred English teachers an essay, I bet the grades would be as close as the BL reader results. Would one teacher give an essay an A while another gave it a B? Of course. Pretty rare that one would give an A while another gave a C.

      No one is saying that English grades are completely random by their nature, because you can't come up with a defined methodology.

      Comment


      • Re: New Black List Thread - Franklin Leonard answers your questions

        I stopped by my manager's office yesterday. I talked to a younger manager who said that he signed many people off the Blacklist based on his preferences and the newsletters.

        And he was saying how great this is for younger generation of managers like him because it allows him to read and seek out hidden gems.

        Comment


        • Re: New Black List Thread - Franklin Leonard answers your questions

          Originally posted by JeffLowell View Post
          This isn't a pure science. It's writing. So people waiting for someone to come up with a formula that is perfectly accurate will keep waiting.

          To try an analogy, readers aren't grading math tests, they're grading essays. If you handed a hundred English teachers an essay, I bet the grades would be as close as the BL reader results. Would one teacher give an essay an A while another gave it a B? Of course. Pretty rare that one would give an A while another gave a C.

          No one is saying that English grades are completely random by their nature, because you can't come up with a defined methodology.
          Possibly the most eloquent deconstruction of this issue I've seen on DDP. Spot-on. Puts my baseball analogy to shame.

          Comment


          • Re: New Black List Thread - Franklin Leonard answers your questions

            Originally posted by JeffLowell View Post
            This isn't a pure science. It's writing. So people waiting for someone to come up with a formula that is perfectly accurate will keep waiting.

            To try an analogy, readers aren't grading math tests, they're grading essays. If you handed a hundred English teachers an essay, I bet the grades would be as close as the BL reader results. Would one teacher give an essay an A while another gave it a B? Of course. Pretty rare that one would give an A while another gave a C.

            No one is saying that English grades are completely random by their nature, because you can't come up with a defined methodology.
            That's a great analogy, Jeff - the readers are essentially scoring essays, not math tests.

            The difference is, students don't pay their teachers to grade their essays. When you pay for something, you expect slightly higher standards and a bit more structure built into the system.

            I don't think BL users expect a 100% accurate formula. However, I do think they would appreciate that as strong an effort as possible is being made to reduce subjectivity in the system. As it stands right now, the system leans heavily towards "gut feelings", which doesn't sit well. That could easily be solved by tying the overall score to component scores. I think it's a simple fix that would stanch a lot of the criticism thrown BL's way.

            Yes, some scripts are more than the "sum of their parts" (BL's standby argument) - maybe there can be a special additional component to reflect that, or it can be given an extra point.
            "I love being a writer. What I can't stand is the paperwork.-- Peter De Vries

            Comment


            • Re: New Black List Thread - Franklin Leonard answers your questions

              Originally posted by UpandComing View Post
              The difference is, students don't pay their teachers to grade their essays.
              Whether through tax dollars or tuition, I'm pretty sure that teachers are being paid to grade those essays.

              As it stands right now, the system leans heavily towards "gut feelings", which doesn't sit well.
              I understand the thought, but when those gut feelings are producing such remarkable correlation, doesn't it argue that there's a functioning, organizing system behind the "grades?" I know a little bit about math, and if something is random, it doesn't produce results like that.

              Comment


              • Re: New Black List Thread - Franklin Leonard answers your questions

                Originally posted by JeffLowell View Post
                Whether through tax dollars or tuition, I'm pretty sure that teachers are being paid to grade those essays.
                Yeah, but it's not coming directly out of students' pockets. For BL customers, the hit is more direct

                Originally posted by JeffLowell View Post
                I understand the thought, but when those gut feelings are producing such remarkable correlation, doesn't it argue that there's a functioning, organizing system behind the "grades?" I know a little bit about math, and if something is random, it doesn't produce results like that.
                I thought this would come up. My response would be: Just because the system's results are positive, doesn't mean there aren't flaws in its design, or that it couldn't be made better.

                Readers for a contest reading the same script probably often give similar scores as well. However, they are still required to grade using defined criteria. Why? To maximize fairness and decrease subjectivity in the process. It just increases faith in the process, especially when there's money being spent.

                To an extent, the BL operates like a year-round contest, but with less accountability with regard to scoring.
                "I love being a writer. What I can't stand is the paperwork.-- Peter De Vries

                Comment


                • Re: New Black List Thread - Franklin Leonard answers your questions

                  Originally posted by UpandComing View Post
                  Readers for a contest reading the same script probably often give similar scores as well. However, they are still required to grade using defined criteria. Why? To maximize fairness and decrease subjectivity in the process. It just increases faith in the process, especially when there's money being spent.

                  To an extent, the BL operates like a year-round contest, but with less accountability with regard to scoring.
                  I see aspiring writers get hung up on this sort of thing all the time and I never quite understand why. In a creative industry, decisions about a script at any level - assistant, executive, studio head - are going to come down to a subjective "Do I like this? Is something I am passionate enough about that I'm going to commit a year's resources to making it?"

                  That's not a conclusion one arrives at by reading a script and and only arriving at PASS or CONSIDER after tallying up all the plus-and-minuses. PASS or CONSIDER (and especially RECOMMEND) is something that you know the instant you put down the script. At that point, the coverage or analysis is merely academic.

                  It's always amusing to me that aspiring writers will rail against the idea that their scripts should fit into some kind of rigid formula, then demand a rigid, Vulcan-like assessment of their work.

                  What I fear is people essentially wanting a system like a 10-point scale to rate each of the following: CHARACTER, CONCEPT, PLOT, STRUCTURE, and then to have some kind of accountable grade that determines PASS or CONSIDER based on the accrued points. So let's say 30/40 is the threshold for CONSIDER. So if you got a 10 on Plot, 10 on Structure, but only 5 each on Character and Concept, should the script automatically get a consider? What if the concept is weak and hackneyed? What if the characters are so bland no actor could be attached?

                  Worry less about improving the system for how reactions are delivered. Focus more on writing material that gets superlative reactions.

                  Comment


                  • Re: New Black List Thread - Franklin Leonard answers your questions

                    Originally posted by Bitter Script Reader View Post
                    That's not a conclusion one arrives at by reading a script and and only arriving at PASS or CONSIDER after tallying up all the plus-and-minuses. PASS or CONSIDER (and especially RECOMMEND) is something that you know the instant you put down the script. At that point, the coverage or analysis is merely academic.
                    I'm actually glad you brought this up. Some people on here have suggested using the actual labels "Pass/Consider/Recommend" for the BL (I think CJ Walley was one), which would actually be more clearcut than a range of numbers from 1 to 10. I could see this as an alternative to the system I mentioned previously, if only because the numbers from 1 to 7 now seem pretty meaningless.

                    Or maybe the numbers could be made more meaningful if Franklin attached explicit labels to them, as you hinted at earlier:

                    Originally posted by Bitter Script Reader View Post
                    They're more like 8 and above - tell your boss to read it immediately. 6-7 - close, but wouldn't recommend. 4-5 - average. 3 and under - not close.
                    Part of the problem with this whole thing is communication, which is important in any business. When you have a range as wide as 1 - 10 it is understandable for people to wonder what exactly every number means.

                    Originally posted by Bitter Script Reader View Post
                    What I fear is people essentially wanting a system like a 10-point scale to rate each of the following: CHARACTER, CONCEPT, PLOT, STRUCTURE, and then to have some kind of accountable grade that determines PASS or CONSIDER based on the accrued points. So let's say 30/40 is the threshold for CONSIDER. So if you got a 10 on Plot, 10 on Structure, but only 5 each on Character and Concept, should the script automatically get a consider? What if the concept is weak and hackneyed? What if the characters are so bland no actor could be attached?
                    You make a valid point here. See my suggestion above about a P/C/R system or explicit labels for scores 1 - 7.

                    Originally posted by Bitter Script Reader View Post
                    Worry less about improving the system for how reactions are delivered. Focus more on writing material that gets superlative reactions.
                    I assure you, the vast majority of my time is spent on improving my material. Really, I don't sit here all day twiddling my thumbs over BL However, Alfred brought up what I thought was a compelling argument, and I wanted to lend my support. And as a paying customer, I will never hesitate to express my opinion about a service, especially one that I care so much about.

                    Okay, I really do need to go get my beauty rest now. Goodnight fellas, was a great discussion!
                    "I love being a writer. What I can't stand is the paperwork.-- Peter De Vries

                    Comment


                    • Re: New Black List Thread - Franklin Leonard answers your questions

                      Originally posted by UpandComing View Post
                      Yeah, but it's not coming directly out of students' pockets. For BL customers, the hit is more direct
                      There are a few students who pay for their own education. (I believe loans are available for this purpose? ) And whether it's students or their parents, the principle is the same: these teachers are being paid to grade essays.

                      Readers for a contest reading the same script probably often give similar scores as well. However, they are still required to grade using defined criteria. Why? To maximize fairness and decrease subjectivity in the process.
                      Here's the crux of this: you are calling for a decrease in subjectivity in a process that has been proven to be remarkably objective. The BL is hitting .960. What percentage should they aim for?

                      To an extent, the BL operates like a year-round contest, but with less accountability with regard to scoring.
                      Franklin publishes reams of statistics, and when the rare script falls outside of his high standards, he is totally accountable and fixes the problem. I don't know of a contest that comes close to his level of accountability.

                      To be fair, as impressive as the correlation is, it doesn't do the writer who gets a 7 instead of an 8 any good. I can't imagine anything - nor have I heard anything proposed - that would eliminate that.
                      Last edited by JeffLowell; 02-06-2014, 08:13 PM.

                      Comment


                      • Re: New Black List Thread - Franklin Leonard answers your questions

                        Originally posted by UpandComing View Post
                        I'm actually glad you brought this up. Some people on here have suggested using the actual labels "Pass/Consider/Recommend" for the BL (I think CJ Walley was one), which would actually be more clearcut than a range of numbers from 1 to 10. I could see this as an alternative to the system I mentioned previously, if only because the numbers from 1 to 7 now seem pretty meaningless.
                        I aggressively disagree with the Pass/Consider/Recommend suggestion. I think it's really helpful to both the writer and potential readers just how "hard" a Pass the reader intends. A score of 1 pretty much indicates that the reader can't find ANY merit in the material at all - that getting this script up to Consider is going to take more than a mere polish.

                        A score of 6 or 7 allows for the possibility that the script is a fixer-upper that could be "flipped" with a reasonable amount of effort. As someone who uses the site, if I saw a logline that intrigued me and the rating was 6 or 7, I might be inclined to give it a read. If I see a 1 or a 2 - not a chance in hell.

                        There's also the fact that the script's average score plays a role in how it shows up in the top lists. If PASS is assumed to be a 1 or a 0 on the scale, than that rating is going to hurt you a lot more than a 6 or 7 will.

                        On the American grading scale, a grade below 60% is considered failling. So if you have a 59% average going into your final exam, it's not as dire for your advancement as if you have a 10%.

                        Comment


                        • Re: New Black List Thread - Franklin Leonard answers your questions

                          Originally posted by JeffLowell View Post
                          Here's the crux of this: you are calling for a decrease in subjectivity in a process that has been proven to be remarkably objective. The BL is hitting .960. What percentage should they aim for?

                          Franklin publishes reams of statistics, and when the rare script falls outside of his high standards, he is totally accountable and fixes the problem. I don't of a contest that comes close to his level of accountability.
                          Two really good points made here. It's easy to forget that this debate about scores is largely using "outliers" as evidence of flaws in the system. Most of the time, the Black List readers are remarkably consistent. And in the instances when their feedback is contradictory, Franklin already has a really fair remedy. Confused what to do about a script that's gotten a 4 and an 8? Here's a free read to break the tie.

                          I can see the argument coming that this does you little good if you pay for just one read and get the 4. While that may be true, the possibility of a second opinion is one not offered at any production company or agency that you would submit to, and I'm not sure many contests have multiple readers on the same screening stage.

                          Comment


                          • Re: New Black List Thread - Franklin Leonard answers your questions

                            Originally posted by UpandComing View Post
                            I'm actually glad you brought this up. Some people on here have suggested using the actual labels "Pass/Consider/Recommend" for the BL (I think CJ Walley was one), which would actually be more clearcut than a range of numbers from 1 to 10. I could see this as an alternative to the system I mentioned previously, if only because the numbers from 1 to 7 now seem pretty meaningless.
                            I did say something to that effect dude, but I don't want BL changing and I don't want numbers replacing with words.

                            Franklin has previously detailed some very valid reasoning for the 1-10 system. I respect the decision to use it and I believe both options were considered. Personally I found it very flattering when my first two screenplays got overall 7's. At such an early stage in my writing it was highly motivating to know there were professional readers out there feeling I was on the cusp of their recommendation. No, it was incredibly motivating.

                            Yes I would like to see an alternative to BL which uses a slightly more finite p/c/r system that only allows scores to be adjusted positively and doesn't work to the law of subjective averages. But I'm not suggesting it would be better. I'd just like to see how that system performs at identifying valuable scripts, complaints over subjectivity and in terms of conflict of interest. It could be significantly worse and I could see how.

                            I also think it's wrong that Franklin feels the need to, or is expected to defend, the differentiation in overall reader scores. There's nothing wrong with a wild variation in scoring. There's nothing wrong with polarised ratings. In my opinion it's an irrelevant debate, partly born out of defending concerns over a scripts average rating, and partly to stop writers throwing their money at the system like a lottery.

                            The fact is a script that isn't very generically appealing isn't going to perform will on a site that gives equal weight to positive and negative reactions. Just as a movie won't on the MetaScore system. But there is the chance of being spotlighted on one readers opinion alone. Failing that, the site scores and reader opinions do seem to carry some weight in querying.
                            Script Revolution - A free to use script hosting website that offers screenwriters a platform to promote their scripts and a way for filmmakers to search through them.

                            Comment


                            • Re: New Black List Thread - Franklin Leonard answers your questions

                              The half price evaluation/free evaluation/pulled evaluation policies are not a remedy to reader subjectivity, they are a blatant and understandable panacea to our complaints. Offering them is wrong, and I say that as someone who's received a fair few. They are wrong because it causes BL to almost appear apologetic for subjectivity. They do have a side benefit of helping us gauge the general appeal of their script but that's not what BL is there to do.

                              BL should not be striving toward objectivity, and certainly not gauging its readers on that level, not that I think Franklin, Dino & Terry would ever do so. This only further makes subjectivity appear like a bad thing and compound the confusion we feel when we face it. And more often than not, our real issue isn't with the subjectivity it's our precious attitude to our scripts overall scores.
                              Script Revolution - A free to use script hosting website that offers screenwriters a platform to promote their scripts and a way for filmmakers to search through them.

                              Comment


                              • Re: New Black List Thread - Franklin Leonard answers your questions

                                Originally posted by JeffLowell View Post
                                The stat you cite was my biggest surprise that emerged from the data… but upon thinking about it, I shouldn't have been surprised.

                                The scores aren't "pick a number between one and ten." They're more like 8 and above - tell your boss to read it immediately. 6-7 - close, but wouldn't recommend. 4-5 - average. 3 and under - not close.

                                If you take a reader who's read hundreds or thousands of scripts professionally and tell them they have to put a script in one of those four groups, it doesn't surprise me that most readers would pick the same groups as other readers. The numbers back that up - 24 out of 25 times, readers do exactly that. There's obviously a little subjectivity even within those groups, but 96% of the time, they don't vary by more than two points.

                                This isn't a pure science. It's writing. So people waiting for someone to come up with a formula that is perfectly accurate will keep waiting.

                                To try an analogy, readers aren't grading math tests, they're grading essays. If you handed a hundred English teachers an essay, I bet the grades would be as close as the BL reader results. Would one teacher give an essay an A while another gave it a B? Of course. Pretty rare that one would give an A while another gave a C.

                                No one is saying that English grades are completely random by their nature, because you can't come up with a defined methodology.
                                Common Sense quote of the month. Yet BL detractors, or those affronted by their personal scores, will ignore it as they've ignored pretty much everything that doesn't give credence to the notion they've suffered unfairly.
                                Last edited by SundownInRetreat; 01-26-2014, 08:38 AM.

                                Comment

                                Working...
                                X