This is something I’ve been spending a lot of time working on with the completion of my Reader Top 30, and I feel like I have things where I want them, so I wanted to make another detailed post talking about my new creation, the SONAR score. If you’ve been around for a while, you read my introduction pieces here and here back in November. Like any good social scientist, I set out with a goal; to try and figure out a way to evaluate minor league prospects across all leagues, factoring in all of the aspects of the game that can be accurately measured, to try and evaluate what prospects have done, and which players might succeed/struggle going forward based on their peripherals. Along the way, I encountered many difficulties, including the simple amount of work it entails to code over 5,000 players into a system, and then trying to figure out if the formulas I used were accurate, helpful, or off the mark. My test run was published in November, and then I started to do position by position breakdowns. It was during this process, when I was looking at the numbers in depth (and based on reader feedback) that I discovered some of the flaws in the system, and I set about fixing the errors, making adjustments, and trying to make the system the best it can be. As any person who tries to create something new, the first run (or first 10 in this case) rarely is ever perfect. But a good scholar always tries to improve, to figure out what is missing, and to try and make it the best it can be. So that is where we are now. Check below for the details.
I’m not going to re-hash all of the details from the introduction pieces linked above. Most of the tenets of the study are still the same. The goal is to try and place all prospects on an even line, that is, adjusting their raw batting lines/pitching lines for the league they play in, factoring in their age related to level, the size of the sample, factoring the impact of their home park, and then drilling right down to their core peripherals that indicate true levels of skill, removing as much of the luck as possible. As I’ve openly stated from the beginning of this project, the end product, the SONAR score, is not meant to replace a scouting report, or be the sole measure by which you judge a prospect. Far from it. The SONAR score is meant to be another piece of data, either a confirmation point or a jumping off point. If you read about Prospect X and scouts saying he’s one of the best prospects in baseball, I want to be able to look at his score and see if his performance has backed up the claims from scouts. On the other side, if I see a player with a very high score, I want to investigate him more, try to figure out why he’s been successful, and compare his numbers against his scouting reports. If you look at the prospect world, different outlets assign different weights to their evaluations of prospects. Some evaluators will rank a guy with a terrible batting line in their Top 50, based on the player’s raw tools and abilities, even if he hasn’t hit a lick or retired many batters as a pro. On the other side, some people weigh performance more heavily, and will rank players who’ve reached the upper minors higher than players further away from the majors but with possibly more upside. The best evaluators take all of the information they have, they blend it together, and they come up with a ranking/evaluation of a player. But even the best guys miss on prospects every year, sometimes they miss badly. Its the nature of the prospect game, where a guy can go from a .220 hitter with seemingly no skills to a bona-fide stud prospect in a matter of a month. Players develop at different speeds, some guys break out at a very young age and never look back. Some break out much later in their careers, for whatever reason. Some guys develop new skills, they radically change their swing/delivery, or add a new pitch, or some kind of twist that alters their development. And really, some guys just never break out, or never perform up to expectations. The game of baseball is really hard, some guys with seemingly endless potential simply never make it. Because of that fact, you can look back at years past and see guys ranked in the Top 10 of prospect lists and say “who?”…such is the nature of the beast. SONAR is simply meant to be one more piece of the pie, one more fly in the proverbial ointment, one more piece of data to use in evaluating prospects.
So, a quick summary of what SONAR tries to do
* Adjust for league/level – How does a line in the California League compare to a line in the Florida State League?
* Adjust for home park – Hitters playing half their games in pitchers parks deserve to have their numbers adjusted
* Adjust for age – A .300/.400/.500 line for a 22 year old in A ball is different than a .300/.400/.500 line for a 19 year old in A ball
* Measure the core components – Getting on base, hitting for power, and using your speed. For pitchers, its the three true outcomes; strike outs, walks, and home runs.
And now, the changes I’ve made from the initial release
* Changes to core components – I’ve figured out a way to better evaluate the core components for a hitter, getting on base and raw power. I don’t want to get into specifics on the formula, but I was discovering quite a few guys with noisy numbers based on flukish BABIP results in small sample sizes, so I found a way to weed most of these out. I was also able to improve the way power is treated by the system. The result should be a much cleaner approach to the core skills for hitters.
* The addition of speed – One of the biggest things I wanted to try and address was how to include the stolen base into the score. As I highlighted in my original piece, stolen base numbers in the minors are often times a bit misleading. Pitchers have less refined pickoff moves, many minor league catchers do not have major league catching skills, and base runners are still learning how to actually steal bases with a solid technique. Nevertheless, based on feedback received early on, and from looking more at the numbers, I figured out a way to incorporate speed in a way I felt comfortable, and it has drastic changes to some players numbers, but I feel the changes are merited. As you’ll see with a player like Anthony Gose, the change was massive, but his speed represents his best raw tool, and is something that had to be addressed. If you look at a player like Gose and discount his speed, you are taking away most of his present value and future potential, because his game is always going to be predicated on speed. The stolen base itself is a much debated item in the statistically astute community, but players that can steal tons of bases at a high percentage have value, especially if the speed accompanies an otherwise strong skill set.
* Rewarding contact – This kind of ties into the first point about core skills. Before, there was no real adjustment for guys who showed excellent contact skills, and conversely, guys who struck out a ton. I felt this was an important adjustment to make. Guys who strike out at alarmingly high rates in the minors are greater risks, because the pitching doesn’t get any easier as you climb the ladder. On the other side, guys who make tons of contact will always have a chance to hit for average, based on the luck they have on balls in play. Contact rate isn’t weighted as heavily as some of the other core skills, but its an important factor to consider.
* Re-weighted pitching components – I decided to tweak the formula slightly for pitchers, giving a bit more weight to the strikeout. For a hitter, a strikeout is not as harmful as it is helpful for a pitcher. When the ball is put in play, anything can happen (good or bad), but when the batter can’t hit it, there is only one outcome. Guys who show swing and miss stuff in the minors have a better chance of achieving a higher ceiling in the majors. At the same time, control/command is important, as guys who have no idea where its going face a tougher climb, but the ability to make a batter swing and miss is arguably the most important skill a pitcher can have. I also found that guys who showed ridiculously good control in the low minors, but didn’t have the swing and miss stuff, often times were found out at higher levels, and this kind of goes along with the consensus opinion of scouts.
* Altering of the scale to evaluate all prospects – With all of the tweaks to the system, the scale where prospects generally fall has changed, but it applies to both pitchers and hitters. There are 5,598 players ranked in the system. A few of these guys may be eliminated as I go through individual teams and remove players who no longer have rookie eligibility, but you get the idea in terms of the number of guys in the system. This number includes both hitters and pitchers. There were 2,684 position players and 2,914 pitchers. The raw average score, that is, the average of the 5,598 players, is -6.57. That basically means that the “average” player in the SONAR universe has a score below zero. This wasn’t anything I planned, but intuitively it makes sense. Of the 5,600 players I looked at, obviously only a small fraction of them will be stars, a slightly larger subset will go on to become above average big leaguers, a slightly larger subset will be average big leaguers, a larger group will be 4A/up and down guys, and the large majority of them will never make it as even bench players in the majors. Its a numbers game, there are only 750 big league spots available (25 man roster x 30 teams) and then a few more on the 40 man who will shuttle up and down before either becoming a fixture at the big league level or falling off the map. The standard deviation for the entire sample was about 20, which also makes some sense. The difference between the highest score in the system (Jason Heyward 138.20) and the lowest score (Jerry Gil -193.5) was 331.7. The difference between the average prospect (-6.57) and the lowest score was greater than the difference between the average prospect and the highest score. 2,374 players had a score of at least 0.01 or higher, but only 447 players had a score of 14.00 or higher, which is essentially one standard deviation (20.67) away from the average. Going further up, there are a total of 76 prospects with a score of 40.00 or higher, 42 with a score of 50 or higher, and 19 with a score of 70 or higher. So the distribution is not a standard bell curve, it is skewed more toward the below average, but I think its intuitive.
The 20-80 scouting scale is considered the gold standard when discussing the tools of a prospect, so to keep that applicable to this score, I’ve developed a chart for easy reference.
Pretty self-explanatory stuff. Because of the exact nature of the score, you can compare two prospects within the same range and rank them however you choose, but a rough guideline is often times helpful.
As I mentioned a while back, the current version of SONAR will only have 1 year of data, 2009. After the 2010 season, I will again compute a 1 year score, but I’ll also produce a score which blends together the 2009 and 2010 data to come up with a composite type score. Because of the nature of prospects and how rapidly they can change, I think its important to place a strong weighting on the most recent data, but a historical track record is also important and something to definitely consider.
So now that you know about the changes I’ve made, as well as the system in general, I wanted to present a comparison chart for Phillies prospects, and how their scores have changed from the old version to the updated, improved current version. I’ll split the graphic up into a few pieces and break it down part by part.
* As I mentioned above, Anthony Gose was one of the biggest benefactors of the changes made, and its intuitive. His approach at the plate remains raw, but he has a number of things working solidly in his favor. First, he was one of the youngest players in the SAL, which boosts his score. More importantly, his speed ranks at the top of the list among all minor league players. He stole 76 bases at a 79% success rate, which is elite level. Domonic Brown also sees a bump up as a result of his work on the base paths.
* Tyson Gillies actually sees a drop in his score, and part of this has to do with his speed working against him. He managed an impressive 44 stolen bases, but at a 69% success rate, which doesn’t help his score to the level that Gose’s number helps him. Gillies also derived some of his on base average through 18 HBP in 2009, and while part of that is skill, part of it is also somewhat unpredictable, and an adjustment to the formula helps to account for this. His score still puts him comfortably in the above average category.
* The biggest drop on the list is Domingo Santana, and this again goes back to the feedback given by a number of people when discussing the scores. The big area where Santana was impacted was the contact aspect of his skill set, as he posted a very high K rate in his 139 PA. His score still places him in the above average tier, and we all know his upside is substantial, but it also adds in more of a hedge because of the swings and misses, lowering his score a bit. Still, a score of more than 25.00 in a very limited sample is very impressive, its just more in line with what you would expect to see. In the same light, Sebastian Valle sees his score dip significantly, as the emphasis on drawing walks as a core skill hurts his overall profile. I highlighted this in my writeup of him in the Top 30.
* Travis Mattair and Derrick Mitchell receive big bumps here. With Mattair, the emphasis on drawing walks helped boost his score. Mitchell’s surface numbers were bad in 2009, but underneath he showed very nice power in a tough hitter’s league, and he wasn’t obscenely old for Clearwater.
* Quintin Berry gets a bump up, largely on the basis of his excellent speed and stolen base potential, as well as his ability to draw a walk. His score is held down by his age, and his propensity for the strikeout, as well as the lack of power. Aaron Altherr becomes a positive value prospect, as he drew a few walks and also swiped a few bases in his very limited sample.
* Tyler Cloyd gains some value here thanks to the tweak which placed a bit more emphasis on home run suppression. Guys who are home run prone in the minors are often times even more home run prone at the major league level, so it is definitely a peripheral to pay close attention to.
* Steve Susdorf sees his stock take a turn down, as he possesses almost no speed, and his walk rate and power numbers were just average for his age compared to his level.
* A bunch of big movers here, including Zach Collier and Winder Torres. Collier, especially, still showed a hint of secondary ability even with his massive struggles in 2009. He stole 20 bags and drew 32 walks, even though his contact rate is really holding his score down. It wasn’t a great year by any means, but not everything was negative.
* Juan Ramirez still shows up on the wrong side of the ledger, but the altered formula likes his performance a bit more. We really won’t have any idea about him until seeing him pitch in a somewhat neutral environment. Julio Rodriguez sees his score drop significantly, as his HR rate is a significant red flag going forward. Still a lot to like there, and a full season (or at least 70+ innings at Williamsport) should give us a better idea.
* Freddy Galvis and Joe Savery see a bump up, but both guys still have a long way to go.
Over the next few days/weeks, I hope to have the SONAR page at the top of the site updated with new spreadsheets for all 30 teams, and then the updated position by position breakdowns. For me, the initial output a few months ago was very important, because it allowed me to have something to look at, and when I went through the results on a more individual basis, it helped me to uncover some of the flaws in the system, and some of the things that needed to be improved. The feedback given originally was also very helpful, and I’m hoping that as the season progresses and we look back at this data, we can figure out ways to even further improve the system. I’m somewhat limited in what I can do, because I only have access to basic data, but I think for what I want SONAR to be (another data point), it can be very useful.