Data Exploration
So, I have a crapton of data. I didn’t really take too much of a scientific approach last time when analyzing team stats, I had some ideas in my head that I wanted to execute against, and essentially one thing just lead to another. Eventually I came to a point where I was analyzing offenses and defenses, and then I started watching film. It never really had a structure, and I got further and further away from the numbers.
My ADD mind wants to try to come at this from all angles. In one case, the film was explaining the numbers to me. In this case, I want to start purely from the numbers, essentially without any context of what basketball is or how basketball works (okay, I still need to know what rebounds and steals are… and the fact that certain points are worth 3 points… and that basketball is played with a ball… and that hoops are involved…).
Again, I’m writing these posts to learn about basketball as a primary reason, but I’m also an Engineer working in “analytics”, so the inner nerd wants to jump into advanced analytics and machine learning right off the bat. After all, I am trying to predict basketball outcomes so I can get rich and quit my day job (sorry, employer).
I will have to preface this whole post by saying I’m not a data scientist, but I’m a wannabe data scientist.
A data scientist knows what he wants and how he wants to get there. I’m not saying things don’t change along the way, or that they don’t use exploratory methods to inquire about data sets they have no context of, but I am saying that a data scientist probably has an understanding of what types of unknown information he will now know after specific methods. Me? I have none of that. I don’t know what I want, I don’t know how to get what I want because I don’t know what I want, and I’m more so going off shiny new objects than a methodical approach.
We come back to the idea of failing extremely fast and extremely frequently.
- I don’t have the analytical experience to really know what to look for and how to look for it
- I don’t have the understanding of the methods to understand how to use them in the right situations
The following, my friends, is called the “shotgun” approach. Let’s throw a bunch of stuff at some arbitrary target to see what we get!
What is the objective, you ask? It’s simply to know more information coming out of this post than I knew coming in. That’s it.
Scatterplot Matrices
There’s so many ways to explore data. Man, that was a dumb statement, but it’s true lol. We saw that I started with a bunch of stats. There’s per game, there’s per 36, there’s per 100, there’s advanced… Each category came with a good 10+ features. I narrowed it down by just taking some per 100 and advanced stats, but even then, I have… let’s check.
%load_ext rpy2.ipython
%%R
# Load libraries & initial config
library(ggplot2)
library(gridExtra)
library(scales)
# Load libraries & initial config
%matplotlib nbagg
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import boto3
from StringIO import StringIO
import warnings
warnings.filterwarnings('ignore')
# Retrieve team stats from S3
playerAggDfToAnalyze = pd.read_csv('https://s3.ca-central-1.amazonaws.com/2017edmfasatb/fas_boto/data/playerAggDfToAnalyze.csv', index_col = 0)
pd.set_option('display.max_rows', len(playerAggDfToAnalyze.dtypes))
print playerAggDfToAnalyze.dtypes
pd.reset_option('display.max_rows')
print 'There are {} columns in the data set'.format(len(playerAggDfToAnalyze.dtypes))
How in the world do we even make sense of 51 columns simultaneously? I’ve been using the pandas plot function with bar charts and histograms this entire time. Those are great for more targeted approaches.
Like in the last post, I wanted to check Kobe’s numbers per year. Easy, two variables. Year, points. There are a multitude of graphs that I could use to communicate this information. A bar chart is as good and as effective as any out there.
I also wanted to see the distribution of points scored by each player / 100 possessions per season. A single variable! Even easier! That’s what a histogram was built for.
Now I want to see 51 variables at a time, maybe get a sense of where I should start, perhaps see some interesting things that I wouldn’t see in two dimensions. As humans, we’re straight up dumb. Any more than basically 3 dimensions we’re useless. We live in a 3 dimensional world. It’s not fair for for me to try to think in 8 dimensions when I can’t even live in a world where 8 dimensions exist. 51 dimensions? Cool, I’d imagine it’s something like this:
And that I’d react something like this:
But NOT because 51 dimensions is so beautiful, but because I’d be so successful in my career and I’d be known as the guy that can think in 51 dimensions. Scatterplot what? Principal components what? Until then, though, let’s try out some methods.
While a scatterplot doesn’t allow us to necessarily think in 51 dimensions, it allows us to think in 2 dimensions over and over again very quickly. Let’s build a quick scatterplot matrix of some “key” metrics?
from pandas.tools.plotting import scatter_matrix
# Build scatterplot matrix
ax = scatter_matrix(playerAggDfToAnalyze[[
'per100Stats_FGPerc',
'per100Stats_FTPerc',
'per100Stats_TRB',
'per100Stats_AST',
'per100Stats_STL',
'per100Stats_BLK',
'per100Stats_TOV'
]])
# We have to set axis labels manaully with Pandas' scatter_matrix function. Maybe there's a better function out there for
# scatterplot matrices, but for now, this is fairly simple.
[plt.setp(item.xaxis.get_label(), 'size', 10) for item in ax.ravel()]
[plt.setp(item.yaxis.get_label(), 'size', 10) for item in ax.ravel()]
Nothing really interesting here. Not sure if I was really expecting something, but nice to see that basically nothing correlates. If there’s any semblance of any correlation, it would be AST and FT%. The more assists you get, the more likely you’re a guard? If you’re a guard, you probably shoot the ball pretty well too? Sure, it’s something, but it’s not really anything new or groundbreaking.
Maybe what I’m getting the most out of this though is all the outliers in the data. You’re gonna tell me that, on average, there are folks averaging 50 turnovers / 100 possessions? So they must first have the ball for at least half of their teams possessions, and then they must turn it over all 50 times? Not correct, but this is probably just guys who played only a few possessions in the season during garbage time, and they maybe got like 2 turnovers in 4 possessions and it’s just been extrapolated out.
Let’s take a look at anyone with over 40 turnovers / 100 possessions.
playerAggDfToAnalyze[playerAggDfToAnalyze['per100Stats_TOV'] > 40]
I know literally none of these players, and none averaged more than 2 minutes per game that season. Pretty much what I thought. I probably makes sense to remove these guys as they really are causing false outliers in the data.
I don’t think anyone should be average over even like… 10 turnovers per 100 possessions. That’s a lot! Let’s see how many minutes players have to play to shield them from this extrapolation error. Well not error… “phenomenon”.
ax = scatter_matrix(playerAggDfToAnalyze[[
'perGameStats_MP',
'per100Stats_TOV'
]])
This scatterplot matrix is actually quite helpful to figure out how much data I’d be cutting out if I only took players over a certain MP.
If we just take an arbitrary cut off of players who play more than, let’s say 10 minutes, we see that the turnovers / 100 poss comes right down to about 10. Let’s take a closer look at folks with turnovers between 10 and 20 (look at the TOV histogram… it’s a very small amount of data which makes me think this is fine).
playerAggDfToAnalyze[(playerAggDfToAnalyze['per100Stats_TOV'] > 10) & (playerAggDfToAnalyze['per100Stats_TOV'] < 20)][[
'perGameStats_MP'
]].plot(kind = 'hist')
playerAggDfToAnalyzeMin10Min = playerAggDfToAnalyze[(playerAggDfToAnalyze['perGameStats_MP'] > 10)]
# Build scatterplot matrix
ax2 = scatter_matrix(playerAggDfToAnalyzeMin10Min[[
'per100Stats_FGPerc',
'per100Stats_FTPerc',
'per100Stats_TRB',
'per100Stats_AST',
'per100Stats_STL',
'per100Stats_BLK',
'per100Stats_TOV'
]])
# We have to set axis labels manaully with Pandas' scatter_matrix function. Maybe there's a better function out there for
# scatterplot matrices, but for now, this is fairly simple.
[plt.setp(item.xaxis.get_label(), 'size', 10) for item in ax2.ravel()]
[plt.setp(item.yaxis.get_label(), 'size', 10) for item in ax2.ravel()]
Some of these other ones also look kinda odd… I feel like 30 rebounds / 100 possessions is also kinda crazy, no? Although you know what, there’s a decent amount of folks even in that 20-25 rebound range… crazy.
Let’s look at these culprits over 25 rebounds / 100 possessions and see if they make sense.
playerAggDfToAnalyzeMin10Min[(playerAggDfToAnalyzeMin10Min['per100Stats_TRB'] > 25)][[
'perGameStats_MP'
]].plot(kind = 'hist')
playerAggDfToAnalyzeMin10Min[(playerAggDfToAnalyzeMin10Min['per100Stats_TRB'] > 25)][[
'season_start_year',
'perGameStats_Player',
'perGameStats_Tm',
'perGameStats_G',
'perGameStats_MP',
'per100Stats_TRB'
]]
First of all, holy F#&@. Dennis Rodman. What the hell mate, the guy grabbed 25% of the boards that ever happened while he was on the court in this season… out of 10 guys, 1 guy grabbed 25% of the rebounds. SURE, WHY NOT.
But you can see a pattern with these other guys. It comes back down to the fact that each row represents one player per one season per one team. I don’t even remember Andrew Bynum being on Indiana, but I guess it happened.
Here comes another filter. My gut feel is that for a player to be truly represented in their stats that year, they should have played at least like a third of the season. I get injuries and whatnot wiping out half the season, but if an injury or trade hinders more than 66% of the season, is it even a good representation of what happened? Let’s try to throw GP into the scatterplot, that’s probably what I should have done with MP to be honest, but let’s see.
# Build scatterplot matrix
ax3 = scatter_matrix(playerAggDfToAnalyzeMin10Min[[
'perGameStats_G',
'per100Stats_FGPerc',
'per100Stats_FTPerc',
'per100Stats_TRB',
'per100Stats_AST',
'per100Stats_STL',
'per100Stats_BLK',
'per100Stats_TOV'
]])
# We have to set axis labels manaully with Pandas' scatter_matrix function. Maybe there's a better function out there for
# scatterplot matrices, but for now, this is fairly simple.
[plt.setp(item.xaxis.get_label(), 'size', 10) for item in ax3.ravel()]
[plt.setp(item.yaxis.get_label(), 'size', 10) for item in ax3.ravel()]
From these charts, it looks like 20 is a pretty good cutoff, or maybe even 15 actually. If we look at TOV, TRB, and FG%, at 20 games it pretty much equalizes.
Let’s go with 20 and try the scatterplot again lol.
playerAggDfToAnalyzeMin10Min20Games = playerAggDfToAnalyze[(playerAggDfToAnalyze['perGameStats_MP'] > 10) & (playerAggDfToAnalyze['perGameStats_G'] > 20)]
# Build scatterplot matrix
ax4 = scatter_matrix(playerAggDfToAnalyzeMin10Min20Games[[
'per100Stats_FGPerc',
'per100Stats_FTPerc',
'per100Stats_PTS',
'per100Stats_TRB',
'per100Stats_AST',
'per100Stats_STL',
'per100Stats_BLK',
'per100Stats_TOV'
]])
# We have to set axis labels manaully with Pandas' scatter_matrix function. Maybe there's a better function out there for
# scatterplot matrices, but for now, this is fairly simple.
[plt.setp(item.xaxis.get_label(), 'size', 10) for item in ax4.ravel()]
[plt.setp(item.yaxis.get_label(), 'size', 10) for item in ax4.ravel()]
Look at how nicely the data fills out the boxes now. Nice histograms throughout as well.
Your average player, per 100 possessions, has the following statline:
playerAggDfToAnalyzeMin10Min20Games[[
'per100Stats_FGPerc',
'per100Stats_FTPerc',
'per100Stats_PTS',
'per100Stats_TRB',
'per100Stats_AST',
'per100Stats_STL',
'per100Stats_BLK',
'per100Stats_TOV'
]].describe()
- 20 PTS
- 9 TRB
- 4.5 AST
- 1.5 STL
- 1 BLK
- 3 TOV
Good to know! Do I know more now than I did before? I think so.
In the next post, I’ll try to throw Principal Components Analysis at the problem because… well basically more or less because I want to learn it regardless of whether it applies here or not!
One thought on “All-NBA Predict #10 – Exploration of Historical NBA Players (Part II, Scatterplot Matrices)”