NYPD Crime #13 – Data Exploration (Part VIII – Datashader Deep Dive)

Review Last post, we looked at Loading a parquet Installing datashader Using datashader We ran into enough issues to make a post out of those items. Parquet On the parquet side, I became familiar with the parquet file format and the partitioned nature of data storage within distributed computing platforms (in this case, I used […]

NYPD Crime #12 – Data Exploration (Part VII – Lat Long Visualization)

Review In the last 2 posts, we reviewed (largely using Spark and Spark SQL (very handy)) all of the interesting fields. All of them except latitude and longitude. I ended the last post puzzled about how to actually plot this many points (5 million points!). Spark didn’t have anything to do this, so I had […]

NYPD Crime #11 – Data Exploration (Part VI – Complaint Type & Location Exploration)

Intro In our last post, we looked at some time based fields. Summer months are big for incidents, so are the 1st, 10th, 15th, and 20th of the month. Why? We’re not sure yet, maybe with some more exploration, we’ll get to the bottom of that. Let’s load up our data from S3 again. In [1]: […]

NYPD Crime #10 – Data Exploration (Part V – Date-Time Exploration)

Intro At this point, we’ve gotten a feel for the data, but we really still don’t know what’s in it. After 9 posts, we’re finally ready to actually explore haha… Oh, the miraculous world of data science… Similar to the All-NBA prediction project I was doing, I really have no objective here. I came in […]

NYPD Crime #9 – Data Exploration (Part III – Feature Building)

Let’s get right into it. If you’ve just landed on this post, please read the past few “Data Exploration” posts on this project to understand the context of what I’m trying to do here in this post. Note that this post is almost identical to the last with the exception of replacing all 24:00:00 time […]

NYPD Crime #8 – Data Exploration (Part III – Feature Building)

Intro Okay, I’m starting to get the hang of Spark. I’ve totally abused the Spark SQL capabilities so far, and it’s been extremely user friendly doing all this in Jupyter. Honestly, it’s not that large of a data set so I haven’t had too much complexity navigating the fields so far. There are not many […]

NYPD Crime #7 – Data Exploration (Part II – Data Cleansing)

Intro In the last post, we wanted to dive into data exploration, but got caught up looking at the basic workflow of simple Spark processes. In fact, we really did no exploration whatsoever other than look at the top 10 rows of the data set haha. It’s okay though, that deeper dive was necessary and […]