What A Journey It basically took me 20 posts to get come to some conclusions and then kind of contradict them. In terms of the data set itself, I definitely found some interesting insights, but my understanding of the topic kinda followed this trajectory: In [1]: # I’m starting to think the single graph this setup […]
Read More NYPD Crime #20 – ConclusionReview I ran into a major dead end in the last post. The problem? Data pre-processing… You don’t often think that data processing would be the activity that prevents you from moving forward eh? As a novice data scientist, you’re so infatuated with the high level objective, the meat of the analysis, the sexy chart […]
Read More NYPD Crime #19 – Clustering To Explore Neighbourhoods (Part IV – Continued Because Spark Hates Me)Review To sum up the last post, our driver’s RAM was essentially the bottleneck and what was causing our Spark application and the underlying JVM to crash. Before, we were using 3 AWS m4.large (8GB RAM) boxes for our master + 2 workers. In this notebook, I’ve spawned a new cluster keeping my workers the […]
Read More NYPD Crime #18 – Clustering To Explore Neighbourhoods (Part III – Continued Because Spark Hates Me)Review I’m so confused I don’t even know what to review. In the last post, I used Spark to load a file, tried to cluster, and ran into a memory issue. Made some changes, clustered, did some SQL, tried to transform the data with a one-hot encoder, and ran into more memory issues… Again, we’re […]
Read More NYPD Crime #17 – Clustering To Explore Neighbourhoods (Part II – Continued)Review In the last post, we reviewed some of the visualizations we made via datashader and learned a few things about the neighbourhoods in NYC. As someone who is not from NYC, has never lived in NYC, and has only ever visited Manhattan and Brooklyn (and even then, my exposure is quite limited), I was […]
Read More NYPD Crime #16 – Clustering To Explore Neighbourhoods (Part I)Themes, yes. What we’ve all been waiting for in this post after 14 posts of mostly troubleshooting haha. To review some of the plots we created with datashader: Figure 1: NYC Crime Density Figure 2: NYC Crime Density (Top 3% Most Dense Regions Highlighted) Figure 3: NYC Crime Density by Offense Level Figure 4: NYC […]
Read More NYPD Crime #15 – TakeawaysReview This is a direct continuation of last post, so I won’t review too much. Last time, we mapped NYPD complaint / offense density geographically using datashader. We explored the ideas of color gradients, color gradient floors, histogram equalization, and looking at top x% of densities. Here, we will look at categorical mapping, which allows […]
Read More NYPD Crime #14 – Data Exploration (Part VIIII – Lat Long Visualization By Category)Review Last post, we looked at Loading a parquet Installing datashader Using datashader We ran into enough issues to make a post out of those items. Parquet On the parquet side, I became familiar with the parquet file format and the partitioned nature of data storage within distributed computing platforms (in this case, I used […]
Read More NYPD Crime #13 – Data Exploration (Part VIII – Datashader Deep Dive)Review In the last 2 posts, we reviewed (largely using Spark and Spark SQL (very handy)) all of the interesting fields. All of them except latitude and longitude. I ended the last post puzzled about how to actually plot this many points (5 million points!). Spark didn’t have anything to do this, so I had […]
Read More NYPD Crime #12 – Data Exploration (Part VII – Lat Long Visualization)Intro In our last post, we looked at some time based fields. Summer months are big for incidents, so are the 1st, 10th, 15th, and 20th of the month. Why? We’re not sure yet, maybe with some more exploration, we’ll get to the bottom of that. Let’s load up our data from S3 again. In [1]: […]
Read More NYPD Crime #11 – Data Exploration (Part VI – Complaint Type & Location Exploration)
