Evaluating the performance of MLB analytical Staff
Background
Now that the baseball season is over, I can finally take a break from spending too much time following the 2022 games. However, as a passionate member of the baseball community, I wanted to contribute in some way. Although I currently work as a data scientist consultant at Accenture and my professional activities and projects are not related to sports, I decided to use some of my free time to pursue a project in this area. Despite not being a reporter or directly involved in the media, I am excited to explore how my data analysis skills can be applied to baseball and potentially benefit the community.
As I sat down to write about baseball, I was unsure of where to begin. There are thousands of excellent articles, blogs, and data analyses already out there, and I didn’t want to simply replicate what had already been done. At the same time, the amount of available data was overwhelming. So many ideas came flooding into my head that it was difficult to know where to start. I spent hours poring over articles, examining data sets, and exploring different approaches in the hopes of finding a unique angle that would allow me to contribute to the baseball community in a meaningful way. Despite the initial challenges, I remained committed to the task at hand, determined to find a fresh perspective that would add value to the discussion around this beloved sport.
Public studies and articles out there, primarily focus on providing insights with predictions or classification methods (e.g., statistical and Machine Learning models), these methods are usually categorized into two groups:
1. Player performance impact: Creating insights from Machine Learning (ML) models to know when to substitute a player based on similar and previous situations.
2. Winning predictions: providing daily predictions with in-depth analysis for betting purposes.
Player performance impact methods are also applied inside every MLB team’s baseball operation. According to Max Wittenberg from Databricks who has worked with many MLB teams and discussed their challenges related to data-driven decisions, states in an article that teams mainly focus on three cores:
1. Speed: Ingesting and processing data for coaches more quickly than their rivals to employ data-driven decisions that may influence the result of the game.
2. Real-time analytics: The ability to produce insights from their Machine Learning models in real-time pitch by pitch also provides a competitive advantage. As Max said, knowing when to replace a pitcher due to exhaustion is one example of this, where a model evaluates pitcher movement and data points collected from the pitch itself and can estimate performance degradation pitch by pitch.
3. Ease of Use: When running data pipelines, analytics teams have trouble ingesting on their computers huge amounts of data from Statcast. When they try to scale their pipelines to collect data from the minor leagues and integrate them with other technologies, the situation becomes much more challenging. Teams desire a cooperative, scalable analytics platform that automates data input with performance and gives them the power to influence in-game judgment.
During my research, I found a couple of blogs that focused on the value that analytics staff has to the game. In the first post, Chad Rain evaluated the value of a quantitative analyst and investigated how analytics staff relates to winning. How do they contribute to WAR? Chad runs and visualizes the following correlations:
1. Payroll rank vs. R&D department size
2. R&D department size vs. the winning percentage
3. R&D department size vs. pitcher WAR
4. Payroll vs. pitcher WAR
5. Payroll vs. winning percentage
He also reports that According to Craig Edwards in his FranGraphs piece, the relationship between winning and payroll is cyclical, and this Collective Bargaining Agreement permits clubs that spend to have an advantage. Therefore, teams with significant payrolls coupled with superior analytical capabilities may be the difference between the haves and have-nots in MLB as teams become more comparable in their analytical capacities. The number of high-caliber data analysts is perhaps a contributing factor in staying ahead of the curve analytically.
Chad also reports that a study from FiveThirtyEight estimates that the addition of 5 data analysts would result in 2 more wins every season at a cost of $350,000. The market for R&D analysts is 52 times more efficient than the free agent market if we consider that the price of 1 WAR in the free agent market is around $9.1 million.
In another post called “What Are Teams Paying Per WAR in Free Agency?” by Ben Clemens on FanGraphs. He used a methodology by Craig Edwards and concludes that teams had offered record-breaking contracts. But as of right now (2021), they haven’t, and they’ve kept on paying top dollar for stars while giving the middle class of free agency a break; to support this, Ben updated Edwards’ other investigation: is the cost of WAR still linear? by removing relievers' contracts, because they receive different compensation from starters and position players. The comparison is fairer when relievers are excluded because they are rarely included in the 2+ WAR cohort.
This is where my idea started…
Seeing that some teams significantly spent less money and achieve better deals or obtained valuable players. Since MLB data roles have a huge impact on the game, from the players, managers, and fans. Their decisions are final and if they make poor choices, it can cost their teams wins, championships such as game 6 of a world series, and even their jobs.
I wanted to take a deep look into the literature and search if analytical departments are a factor (i.e., contribute) to tradeline acquisitions or signing Free Agents during the off-season.
For the past four weeks during my free time, I’ve been working on analyzing the performance of pitchers who were traded during the 2022 season or signed with a new team during the 2021 off-season. I wanted to look if the analytic staff has something to do with acquiring pitchers based on their performances since their new team.
Study 1 — Does R&D staff size still plays a role in starters and relievers WAR?
Teams with higher payrolls have an advantage in winning more games, hence they also have an advantage in analytical capabilities. For this reason, I wanted to explore if R&D staff size impacted pitcher WAR, based on this we can see if low-budget teams are taking advantage of investing in analytical resources.
For this, I had to select Research & Development roles to consider the staff size per team. In order to do this, I included the following positions from their media guides:
- Research, Research & Development, Research & Analytics, Research Scientist, Quantitative Analysis.
- Data Engineer, Data Scientist, Data Architect, Data Quality Assurance, DevOps Engineer, Software Engineer/Developer, Baseball System Developer, Database Engineer/Developer, and Data Operations.
- Business Intelligence, Business Insights, Business Analytics, Baseball Analytics, Business Strategy, Strategy.
- Baseball Operations Analyst, Baseball Operations Fellow, Player Development & Performance, Pitching Strategist, Pitching Analytics.
Some roles such as “software developer”, “Baseball Operations Fellow”, and “Player Development” are not exactly part of an analytical domain (i.e., data), but doing a deep look into their careers, they had a background in analytics activities, and others used their skills to develop applications to other staffs to generate reports and information, for example:
“Developed and maintained the Rangers internal scouting web application. Interfaced with scouts, coaches, and front office members to gather requirements, design, and build scouting reports, game reports, and player information. Interfaced with the Commissioner’s Office to gather, store, and integrate player contracts and tracking data” -Kim Eskew former Senior Software Developer for the Rangers from his LinkedIn profile.
Afterward, I excluded roles that were assistants and part of Minor League Baseball. For this study, I considered 504 positions out of 1,495 baseball staff from all teams, I ended up with the following results per team:
Before continuing to explain the evaluation, a summary of the tools and processes I used is shown in the figure below. To extract the information from FanGraphs I used a Python library called beautifulsoup4 that makes it easy to scrape information from web pages, i.e., it is a library that allows you to efficiently and easily pull-out information from HTML pages. Afterward, I used Python to aggregate and export tables into CSV files to be later used in PowerBI to visualize them.
To evaluate if R&D staff size still influenced pitchers' WAR, three Pearson correlation coefficients were computed to assess the linear relationship between Analytical staff size and Pitchers' WAR.
- Analytical staff size and starters WAR: there was a positive correlation between the two variables, r(28) = -.397, p = .030.
- Analytical staff size and relievers WAR: there was a positive correlation between the two variables, r(28) = -.321, p = .082.
- Analytical staff size and all pitchers WAR: there was a positive correlation between the two variables, r(28) = -.421, p = .020.
The correlations confirm there is still a positive trend that analytical staff size improves pitchers’ WAR, based on this, teams with low WAR such as DET, KCR, PIT, CHC, CIN, OAK, and WSN (lower than the quartile Q1 — IQR) could invest on hiring more analysts since they are 52 times more efficient than the free agent market. A 2022 team payroll vs. all pitchers WAR plot visualizes the distribution between all teams (payroll source):
2022 team payroll and all pitchers WAR: there was a positive correlation between the two variables, r(28) = -.357, p = .053.
Study 2 — Is there an effect between analytical staff size vs. change in the performance of pitchers?
I was interested in seeing if analytical staff had anything to do with signing and trading pitchers depending on their recent results for a new team. For this I considered the following groups and stats for each type of pitcher:
For a quick recap of the stats here is a table describing each of them:
2.1 Change of percentage for pitchers signed during the 2021 off-season
To have a list of the pitchers signed during the 2021 off-season, I compared all pitchers who played for a different team in the 2022 season, 183 pitchers played for a new team, and out of the 183 I excluded the following players because of particular reasons, Raynel Espinal: only pitched two innings during the 2021 season, Marwin Gonzalez: position player, Cesar Valdez, only pitched one inning during the 2022 season, and Willians Astudillo, position player. Here is an example of the change in the CSW% rate per player:
In the figure above we can see pitchers such as Zach Davies (2021: 26%, 2022: 26.7%) and Andrew Heaney (2021: 28.5%, 2022: 32.5%), improved their stuff and were able to command it well against hitters. A simple, change such as switching a pitch or getting a different grip could change everything. For example, Andrew Heaney switched his curveball to a slider in 2022 and his command consisted only of fastball (62,5%), slider (32,4%), and changeup (5.1%). Jomboy does a breakdown of how Heaney drastically improved (source).
Summarizing the change of percentage for pitchers signed during the 2021 off-season, here is a table showing the change of percentage per team:
Furthermore, I compared the analytical staff size against all stats considered in this study, being the change of percentage of CSW% vs. Analytical size the most positive correlation with r(28) = .354, although is not a strong correlation, it does tell us some meaning that there could be an effect between analytical size and other advanced stat. The following table shows a summary of the correlations between the analytical size vs. stats considered:
2.2 Change of percentage for pitchers traded during the 2022 season
Moving on to analyzing the performance of pitchers who were traded during the 2022 season I used the same procedure as before:
- Gathered information on every pitcher from the 2022 season FanGraphs.
- Processed starters and relievers stats and exported comparison tables.
- Conducted comparison analysis and visualized data.
I ended up with the change of percentage of CSW% vs. Analytical size being the most positive correlation with r(28) = .442, the same as the study of pitchers singed during 2021.
Summarizing the change of percentage for pitchers signed during the 2021 off-season, here is a table showing the change of percentage per team:
Change of percentage of stats including all pitchers traded during the 2022 season
These results, shows there is a good correlation between the CSW% and the analytical size of pitchers who played for a new team. Teams that have a lot of data roles not only have an advantage in analytical capabilities, but also in analyzing the value of pitchers who are not playing to their full potential.
Since CSW% it is a valuable metric to see if a pitcher has good stuff and is able to command it well, this means that the analytical staff is able to help to pitch coaches improve pitchers' stuff by analyzing the value of their pitches.
Additional examination
For further exploration, I looked into the correlation between the analytical size and the CSW% 2022 season, confirming that analytical staff has an effect on the CSW% of the pitchers.
Supporting this effect, the following table shows the distribution of each CSW% stats:
Moreover, exploring the linear relationship between Analytical staff size and Pitchers’ F-Strike, it also shows a good correlation r(28) = .514, and comparing them to the two previous studies we have:
- change of percentage of F-Strike for pitchers signed during the 2021 off-season: r(28) = .178
- change of percentage of F-Strike for pitchers traded during the 2022 season: r(28) = .306
Although there is no similarity between the F-Strike correlation (0.514, 0.178, 0.306) and the CSW% correlation (0.449, 0.354, 0.442), it does tell us that analytical staff has an effect on pitchers' performance, and it reflects in some stats.
Despite the fact I only considered 5 stats, some of them are constructed from other stats, for example CSW% of called strikes and whiffs; and WAR of FIP and pitcher Specific Runs Per Win, which tells us that there is a great possibility that a good correlation also exists.
Final thoughts…
As I analyzed the findings of two studies — one focused on pitchers traded during the 2022 tradeline and the other on free-agent pitchers signed during the 2021 offseason — I was struck by the lack of clear relationships between analytical staff size and statistical performance (except for the CSW%). While this was expected for a variety of reasons:
- The number of players per team was insufficient; for example, BOS, CHW, and DET only received one pitcher during the tradeline; a sample of only one individual per team is insufficient for concrete results, which biases the results and does not enrich them at the time of analysis.
- Moreover, even when teams had only one person dedicated to data analysis, the study did not consider removing outliers — a critical step in any analytical methodology. Although eliminating outliers isn’t always a rule of thumb, it’s important to mention the methodology behind any analysis. In this case, it was clear that more information was required before progressing with any outlier removal method. By addressing these issues and taking a more nuanced approach to data analysis, we can gain a deeper understanding of how analytical staff size affects statistical performance in the world of baseball.
Although these reasons do not violate the correlation assumptions, they expose the analyses to risk due to the absence of information. Still, the analytics provide some insight into how we can interpret analytics teams to assist teams in acquiring pitchers with promising potential.
Two additional correlation analyses were conducted to support this, looking at all pitchers in the 2022 season and their effect on analytics team size. The results supported the insights given by the two studies, with a strong correlation in WAR (0.421), CSW% (0.478), and F-Strike% (0.514).
What does this tell us? MLB teams with poor performance could invest in hiring more analytical staff, but they must decide how many front-office baseball analytics roles they will require soon; and ensure that the analytical staff has the knowledge, experience, and collaborative spirit necessary to achieve shared goals. Ultimately, the insights gained from these studies could prove invaluable in helping teams build successful rosters and achieve their championship aspirations.
Feel free to send me a message for any feedback or if you want to chat, you can reach me on Twitter, e-mail (andres.mitre@outlook.com), LinkedIn.
You can also find more about other analytical projects in my portfolio. The GitHub repository is shown below, you can find all exported files, scripts, and input files used for this study.