NYPD Crime #10 – Data Exploration (Part V – Date-Time Exploration)

Intro At this point, we’ve gotten a feel for the data, but we really still don’t know what’s in it. After 9 posts, we’re finally ready to actually explore haha… Oh, the miraculous world of data science… Similar to the All-NBA prediction project I was doing, I really have no objective here. I came in […]

NYPD Crime #9 – Data Exploration (Part III – Feature Building)

Let’s get right into it. If you’ve just landed on this post, please read the past few “Data Exploration” posts on this project to understand the context of what I’m trying to do here in this post. Note that this post is almost identical to the last with the exception of replacing all 24:00:00 time […]

NYPD Crime #8 – Data Exploration (Part III – Feature Building)

Intro Okay, I’m starting to get the hang of Spark. I’ve totally abused the Spark SQL capabilities so far, and it’s been extremely user friendly doing all this in Jupyter. Honestly, it’s not that large of a data set so I haven’t had too much complexity navigating the fields so far. There are not many […]

NYPD Crime #7 – Data Exploration (Part II – Data Cleansing)

Intro In the last post, we wanted to dive into data exploration, but got caught up looking at the basic workflow of simple Spark processes. In fact, we really did no exploration whatsoever other than look at the top 10 rows of the data set haha. It’s okay though, that deeper dive was necessary and […]

NYPD Crime #6 – Data Exploration (Part I – Spark Basics)

Intro Alright, now we know the ins and outs of Spark! Spark be like: And, yes Spark, you would be absolutely correct. I won’t pretend to be a Spark expert after scouring the internet for random tutorial videos, but you know what? I feel like I have enough knowledge to get started. So, let’s do […]

NYPD Crime #5 – Apache Spark

Intro Oh man… SPARK… Where to start. First of all, pretty much everything in this post will be based on this seminar by Sameer Farooqi from Databricks. It’s a full-day seminar, and yes, I watched all 6 hours. Me after watching the video: Me realizing where my day has gone: Hey, this is the life […]

NYPD Crime #4 – Spinning Up EMR Cluster (Continued)

This is a direct continuation from the last post. The last post had already gotten way too long, so I’m starting another for organization sake. Wow, that sounded professional for a second. Spinning It Up Okay, so I clicked on “Create Cluster” after we finished step 4 of the AWS EMR console. It looks like […]

NYPD Crime #3 – Spinning Up EMR Cluster

In the last post, we went over briefly some examples of distributed computing and narrowly surveyed the landscape of EMR and Hadoop. In this post, I want to try to actually spin up a cluster. Don’t really know what else to say at this point because I’m so uneducated in the area and have read […]

NYPD Crime #2 – Distributed Computing & Amazon EMR

Distributed Computing Before we jump straight into EMR (Elastic Map Reduce (The “Map Reduce” refers to essentially a distributed computing computational engine that is more or less obsolete these days)) or Hadoop, let’s just spend a few minutes (well, paragraphs) talking about distributed computing in general. Distributed computing is the idea of distributing your computing […]

NYPD Crime #1 – Intro

Yello Hello! I’m back for another random data science project to try, fail, and sometimes succeed at! Key word… probably “fail”. All good, though, I’ve failed so many times now that it’s basically become second nature! In the last post, I tried to build a Convolutional Neural Network to identify my girlfriend’s face from mine. […]