How A Ratings Service Can Deal With Subjectivity

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #46
    Re: How A Ratings Service Can Deal With Subjectivity

    Originally posted by DavidK View Post
    Because 42 percent of people misuse statistics, perhaps a different approach is needed. I've always advocated a zero-three rating system; it works like this:

    0 - I don't like this script, period.
    1 - I like this script/writer but not enough to recommend them to anyone.
    2 - I would recommend this script/writer to my boss.
    3 - I would stake my career on this work.

    It relies on all the same input data as any other rating system, is sublimely simple and less fallible, but most importantly the practical result is no different from any vastly more complex approach. As you can imagine, its simplicity has a lot of appeal for those of us whose intellectual skills are ... well, let's just say less than staggering.
    Perfect. I'd be more than okay with this rating system, and no "strengths" or "weaknesses" needed.

    Comment


    • #47
      Re: How A Ratings Service Can Deal With Subjectivity

      Originally posted by DavidK View Post
      Because 42 percent of people misuse statistics, perhaps a different approach is needed. I've always advocated a zero-three rating system; it works like this:

      0 - I don't like this script, period.
      1 - I like this script/writer but not enough to recommend them to anyone.
      2 - I would recommend this script/writer to my boss.
      3 - I would stake my career on this work.

      It relies on all the same input data as any other rating system, is sublimely simple and less fallible, but most importantly the practical result is no different from any vastly more complex approach. As you can imagine, its simplicity has a lot of appeal for those of us whose intellectual skills are ... well, let's just say less than staggering.
      This is simple and brilliant.

      Comment


      • #48
        Re: How A Ratings Service Can Deal With Subjectivity

        I wish everyone would stop being so flip about this. It's important. We need to figure this out otherwise nobody will buy any scripts or make any movies due to lack of empirical evidence that a script is good or not. This is the reason there has never been any movies made ever. Let's crack it.

        Comment


        • #49
          Re: How A Ratings Service Can Deal With Subjectivity

          Originally posted by DavidK View Post
          0 - I don't like this script, period.
          1 - I like this script/writer but not enough to recommend them to anyone.
          2 - I would recommend this script/writer to my boss.
          3 - I would stake my career on this work.
          I like the idea, except for #1. The writer would just whine: "He liked the script, but wouldn't recommend it!? What a douche!" I think the following, very familiar, system might be better:

          [ ] what the hell did I just read? = Very bad = pass
          [ ] wasn't for me = Bland = pass
          [ ] worth the read = OK = pass on script, rec writer
          [ ] impressive = Good = recommend script + writer
          [ ] genius = AMAZING! = recommend script + writer
          I'm never wrong. Reality is just stubborn.

          Comment


          • #50
            Re: How A Ratings Service Can Deal With Subjectivity

            Originally posted by Levenger View Post
            I wish everyone would stop being so flip about this. It's important. We need to figure this out otherwise nobody will buy any scripts or make any movies due to lack of empirical evidence that a script is good or not. This is the reason there has never been any movies made ever. Let's crack it.

            Comment


            • #51
              Re: How A Ratings Service Can Deal With Subjectivity

              Come on, you guys really need to think outside the box. How about this:


              1. PASS

              2. PASS WITH RESERVATIONS

              3. CONSIDER WITH RESERVATIONS

              4. CONSIDER

              5. RECOMMEND

              Comment


              • #52
                Re: How A Ratings Service Can Deal With Subjectivity

                Originally posted by Gary Dragan Milin View Post
                Come on, you guys really need to think outside the box. How about this:

                1. PASS
                2. PASS WITH RESERVATIONS
                3. CONSIDER WITH RESERVATIONS
                4. CONSIDER
                5. RECOMMEND
                First, this isn't exactly thinking outside the box. Second, it doesn't work for the same reasons most other score/rating systems don't work. The reason I'm serious about the zero-three rating system is that it conforms to the only four thresholds that actually matter in a practical sense. Whatever else a reader/director/agent/producer says or thinks about a script, it still lands in one of four categories: bad, okay but not okay enough, good enough to recommend to a superior, and (once every few years) so out of this world awesome they'd stake their job or reputation on it.

                Trying to refine a scoring system beyond that has always struck me as futile and fraught with all the issues and shortcomings that have been discussed recently. In every other system there is too much room for ambiguity and lack of precision and finding a reliable rating is onerous. I'm not knocking what BL or others are trying to achieve and I can see how the reliability of their ratings arguably grows with an increase in the number of assessments, but in the final analysis I still see only four thresholds that matter.
                "Friends make the worst enemies." Frank Underwood

                Comment


                • #53
                  Re: How A Ratings Service Can Deal With Subjectivity

                  Originally posted by DavidK View Post
                  First, this isn't exactly thinking outside the box. Second, it doesn't work for the same reasons most other score/rating systems don't work. The reason I'm serious about the zero-three rating system is that it conforms to the only four thresholds that actually matter in a practical sense. Whatever else a reader/director/agent/producer says or thinks about a script, it still lands in one of four categories: bad, okay but not okay enough, good enough to recommend to a superior, and (once every few years) so out of this world awesome they'd stake their job or reputation on it.

                  Trying to refine a scoring system beyond that has always struck me as futile and fraught with all the issues and shortcomings that have been discussed recently. In every other system there is too much room for ambiguity and lack of precision and finding a reliable rating is onerous. I'm not knocking what BL or others are trying to achieve and I can see how the reliability of their ratings arguably grows with an increase in the number of assessments, but in the final analysis I still see only four thresholds that matter.
                  Either you use a simple numerical system.

                  Or use the p/c/r system that the Industry already employs.

                  Using anything else is pointless.

                  And I would definitely not recommend using any system that has the word "like" in it. That sounds too subjective.

                  Remember, in this case the reader is not providing their personal opinion so much as their opinion on how the rest of the Industry will view the material.

                  Comment


                  • #54
                    Re: How A Ratings Service Can Deal With Subjectivity

                    0 - I want to meet the writer.
                    1 - I don't want to meet the writer.

                    Comment


                    • #55
                      Re: How A Ratings Service Can Deal With Subjectivity

                      We can try to "Figure this out" all we want, but ultimately, it's Franklin's business model to execute as HE sees fit!

                      Comment


                      • #56
                        Re: How A Ratings Service Can Deal With Subjectivity

                        Maybe someone's scoring should be something like this:

                        0 - 2 I don't like this script, period.
                        3 - 6 I like this script/writer but not enough to recommend them to anyone.
                        7 - 9 I would recommend this script/writer to my boss.
                        9+ I would stake my career on this work.

                        I'm freaking brilliant!
                        On Twitter @DeadManSkipping

                        Comment


                        • #57
                          Re: How A Ratings Service Can Deal With Subjectivity

                          Having an even number of categories makes sense, that way you can never be in the middle - everything has to either be above or below the middle figure. Otherwise a lot gravitates to the middle, because it's the safe number

                          Comment


                          • #58
                            Re: How A Ratings Service Can Deal With Subjectivity

                            I tend to use the American grading system in conversation to describe quality, as people (yanks) have a good instinctual feel for it.

                            "The first draft was a C- but he polished it into a B+."
                            "This director takes A scripts and makes B- movies from them."
                            "This project could be polished from a D- to a D+ but why bother."
                            etc

                            Comment


                            • #59
                              Re: How A Ratings Service Can Deal With Subjectivity

                              I know I've said my piece but seriously you guys are SO calculating more right now.

                              Comment


                              • #60
                                Re: How A Ratings Service Can Deal With Subjectivity

                                Originally posted by Gary Dragan Milin View Post
                                Either you use a simple numerical system.

                                Or use the p/c/r system that the Industry already employs.

                                Using anything else is pointless.

                                And I would definitely not recommend using any system that has the word "like" in it. That sounds too subjective.

                                Remember, in this case the reader is not providing their personal opinion so much as their opinion on how the rest of the Industry will view the material.
                                ...or you combine a numerical system with the familiar P/C/R spectrum, which is the approach we take with the Spec Scout Score.

                                At Spec Scout, each script gets covered by three readers, and each reader provides an overall rating as well as numerical scores for ten individual attributes of the script on a scale of 1 to 5. The overall rating and the numerical scale are basically equivalent, along the following lines:

                                5 / Strong Recommend = exceptional, amazing, loved it
                                4 / Recommend = excellent, very little room/need for improvement
                                3 / Consider = very strong/competent and has promise, but needs a polish
                                2 / Pass = average to weak
                                1 / Strong Pass = terrible

                                Here are the individual categories our readers assess, in alphabetical order:
                                - Character
                                - Conflict
                                - Craft
                                - Dialogue
                                - Internal Logic
                                - Originality
                                - Pacing
                                - Premise
                                - Structure
                                - Tone

                                Our algorithm weights some categories more heavily than others. Originality, for example, matters far less than Character, which in turn is subordinate to the Overall rating. Each reader's ratings are converted into a score on a scale of 1 to 100, and then the three readers' scores are averaged to determine the Spec Scout Score.

                                We've found that three readers is the magic number where scores start to stabilize. The fourth and fifth readers' scores really don't affect the Spec Scout Score all that much -- on the order of +/- 4% for the fourth reader and even less for the fifth.

                                At the risk of encouraging more calculating, I'd be happy to answer questions about this either here or off DDP.
                                Jason Scoggins

                                http://www.specscout.com

                                or for that matter

                                http://www.scogginsreport.com

                                Comment

                                Working...
                                X