Re: Spec Scout
Hi, all. My apologies for going quiet after the invigorating exchange during the holidays. I'll be more attentive to this thread going forward. In the meantime, I thought I'd re-start the conversation by giving an update on the changes we made to Spec Scout in the wake of that conversation. I'll stay high level for this post and get as granular as you want in my replies. And please forgive me in advance for the lengthy post.
As a reminder, there are three basic facets to SpecScout:
- We're tracking the spec market on a daily basis, and basic project info is free to anyone who comes to the site on a day to day basis;
- We're doing coverage on scripts that hit the spec market, which includes our Spec Scout Score, and making that coverage available in one place (i.e., in our coverage library, available to our pro subscribers for a monthly fee); and
- We're providing a paid coverage service to writers using the same rubric and scoring system we apply to spec market scripts. Writer clients whose scripts score above a 68 on our 1 to 100 point scale are invited to be included in the coverage library for no additional fee. (The cutoff was 70 for a while, but we changed it to 68 for reasons I explain below.)
Each script we cover is read and reviewed by three separate readers who, in addition to having significant experience doing coverage before they start working with us, a) have been trained by us by reading and covering literally dozens of scripts before their work is available on our site or through our paid service, and b) use a very lengthy (45+ page) and very granular rubric we developed in order to make sure they're all reviewing the material the same way. In addition to providing an overall Pass/Consider/Recommend-style rating, they evaluate 10 individual attributes of the script and provide ratings for each on a 1-to-5 scale. Our algorithm calculates those scores into a single number on a 1-to-100 scale, and the Spec Scout Score is the average of the three readers' scores.
To clear up a misconception I noticed above, subscribers cannot download scripts from our site. We're not charging for access to scripts. We're not a script library in any sense of the term. Our subscribers are paying for access to a library of coverage. When subscribers are interested in material they discover on our site, we put them in touch with the agent or manager or writer directly.
Another misconception I noticed is the idea that we're covering random drafts of scripts that are already in development around town, the way ScriptShadow reviews material in development, for example. We're not. We're only covering the drafts of scripts that were originally sent into the marketplace, and we're doing this because we think its a useful service in and of itself but also so we can compare apples to apples, if you will, between those scripts and the material submitted for our paid coverage service. This way, we can credibly recommend high-scoring scripts from our paid coverage service.
Here's the single biggest change we made to the site in the wake of the earlier conversation in this thread: We're suppressing scores and coverage comments for all scripts that score below a 68 on our 1 to 100 scale. This goes not just for the public site, which anyone can see, but also behind our paywall, which is only accessible to agents, managers, producers, executives and their ilk.
Practically speaking, any script that scores above 60 is actually quite good -- it means that each of our three readers have given it an overall "Consider" rating along with solid scores in the individual categories. That said, a score of 68 turns out to be the point at which negative comments in our coverage disappear, replaced by constructive comments about ways to make an already good script better. And that, to quote myself, is in line with our mission to highlight the best stuff out there.
We've also implemented (but not yet widely publicized) an opt-out policy in response to earlier suggestions. Any writer (or rep) whose script's scores and comments are visible on the site (because it's scored above a 68) may request that we suppress those scores and comments. We won't delete the project from our database, since that would undermine our tracking activities, but we're happy to suppress comments and scores on a case-by-case basis.
I was initially worried that the lack of a score would signal that a given project is by definition a bad script and thus cause harm. So far, that doesn't seem to be a problem in practice, since a) our coverage of spec market scripts currently lags the market by several weeks, thanks to the load on our readers from paid script submissions, b) we don't have access to every single script that hits the market, and c) we don't indicate which scripts have been opted-out. We'll be keeping an eye on this going forward to make sure the dynamic doesn't shift.
...and I think that's enough for this post. I'll do a better job of lurking on the threads in the days to come to respond to questions, slings and arrows.