Scalable Data Science - 1.x from Middle Earth

Scalable data science is a technical course in the area of Big Data, aimed at the needs of the
data industry. This course uses
Apache Spark, a fast and general engine for large-scale data processing via databricks to compute
with datasets that won't fit in a single computer. The course will introduce Spark’s core concepts
via hands-on coding, including resilient distributed datasets and map-reduce algorithms, DataFrame
and Spark SQL on Catalyst, scalable machine-learning pipelines in MlLib and vertex programs using
the distributed graph processing framework of GraphX. We will solve instances of real-world big data
decision problems from various scientific domains.

This is being prepared by Raazesh Sainudiin and Sivanand Sivaram
with assistance from Paul Brouwers, Dillon George and Ivan Sadikov.

All course projects by seven enrolled and four observing students for Semester 1 of 2016 at UC, Ilam are part of this content.

How to self-learn this content?

The 2016 instance of this scalable-data-science course finished on June 30 2016.

To learn Apache Spark for free try databricks Community edition by starting from https://databricks.com/try-databricks.

All course content can be uploaded for self-paced learning by copying the following URL for 2016/Spark1_6_to_1_3/scalable-data-science.dbc archive
and importing it from the URL to your free Databricks Community Edition.

The Gitbook version of this content is https://www.gitbook.com/book/raazesh-sainudiin/scalable-data-science/details.

The browsable git-pages version of the content is http://lamastex.github.io/scalable-data-science/.

How to cite this work?

Scalable Data Science - 1.x from Middle Earth, Raazesh Sainudiin and Sivanand Sivaram, Published by GitBook https://www.gitbook.com/book/lamastex/scalable-data-science/details, 791 pages, 24th July 2017.

Supported By

Databricks Academic Partners Program and Amazon Web Services Educate.

Summary of Contents

• Prelude of 2016 Version

• Some Basics and Essentials

• Week 1: Introduction to Scalable Data Science

  • Scalable Data Science
  • Why Spark?
  • Login to databricks
  • Scala Crash Course

• Week 2: Introduction to Spark RDDs, Transformations and Actions and Word Count of the US State of the Union Addresses

  • RDDs, Transformations and Actions
  • HOMEWORK: RDDs, Transformations and Actions
  • Word Count: US State of Union Addesses
  • EXTRA_Word Count: ETL of US State of Union Addesses

• Week 3: Introduction to Spark SQL, ETL and EDA of Diamonds, Power Plant and Wiki CLick Streams Data

  • Spark SQL Introduction
    • HOMEWORK: overview
    • HOMEWORK: getting started
    • HOMEWORK: data sources
    • HOMEWORK: performance tuning
    • HOMEWORK: distributed sql engine
  • ETL and EDA of Diamonds Data
  • ETL and EDA of Power Plant Data
  • ETL and EDA of Wiki Click Stream Data

• Week 4: Introduction to Machine Learning - Unsupervised Clustering and Supervised Classification

  • Introduction to Machine Learning
  • Unsupervised Clustering of 1 Million Songs via K-Means in 3 Stages
    • Stage 1: Extract-Transform-Load
    • Stage 2: Explore
    • Stage 3: Model
  • Supervised Classification of Hand-written Digits via Decision Trees

• Week 5: Introduction to Non-distributed and Distributed Linear Algebra and Applied Linear Regression

  • Linear Algebra Introduction
    • HOMEWORK: breeze linear algebra cheat sheet
  • Linear Regression Introduction
  • Distributed Linear Algebra for Linear Regression Introduction
    • HOMEWORK: Spark Data Types for Distributed Linear Algebra
      • Local Vector
      • Labeled Point
      • Local Matrix
      • Distributed Matrix
      • Row Matrix
      • Indexed Row Matrix
      • Coordinate Matrix
      • Block Matrix
  • Power Plant Pipeline: Model, Tune, Evaluate

• Week 6: Introduction to Spark Streaming, Twitter Collector, Top Hashtag Counter and Streaming Model-Prediction Server

  • Introduction to Spark Streaming
  • Tweet Collector - broken down
  • Tweet Collector - Generic
  • Tweet Hashtag Counter
  • Streaming Model-Prediction Server, the Full Powerplant Pipeline

• Week 7: Probabilistic Topic Modelling via Latent Dirichlet Allocation and Intro to XML-parsing of Old Bailey Online

  • Probabilistic Topic Modelling
  • HOMEWORK: Introduction to XML-parsing of Old Bailey Online

• Week 8: Graph Querying in GraphFrames and Distributed Vertex Programming in GraphX

  • Introduction to GraphFrames
  • HOMEWORK: On-Time Flight Performance with GraphFrames

• Week 9: Deep Learning, Convolutional Neural Nets, Sparkling Water and Tensor Flow

  • Deep Learning, A Crash Introduction
  • H2O Sparkling Water
  • H2O Sparkling Water: Ham or Spam Example
  • Setting up TensorFlow Spark Cluster
  • Scalable Object Identification with Sparkling TensorFlow

• Week 10: Scalable Geospatial Analytics with Magellan

  • What is Scalable Geospatial Analytics
  • Introduction to Magellan for Scalable Geospatial Analytics

• Week 11 and 12: Student Projects

  • Student Projects
  • Dillon George, Scalable Geospatial Algorithms
    • Scalable Spatio-temporal Constraint Satisfaction
    • Map-matching
    • OpenStreetMap to GraphX

  • Akinwande Atanda, Twitter Analytics

    • Chapter_Outline_and_Objectives
    • Unfiltered_Tweets_Collector_Set-up
    • Filtered_Tweets_Collector_Set-up_by_Keywords_and_Hashtags
    • Filtered_Tweets_Collector_Set-up_by_Class
    • ETL_Tweets
    • binary_classification
    • binary_classification_with_Loop
    • binary_classification_with_Loop_TweetDataSet

  • Yinnon Dolev, Deciphering Spider Vision

  • Xin Zhao, Higher Order Spectral CLustering

    • Case-study

  • Shanshan Zhou, Exploring EEG

  • Shakira Suwan, Change Detection in Random Graph Series
  • Matthew Hendtlass, The ATP graph
    • Yuki_Katoh_GSW_Passing_Analysis
  • Andrey Konstantinov, Keystroke Biometric
  • Dominic Lee, Random Matrices
    • References
  • Harry Wallace, Movie Recommender
  • Ivan Sadikov, Reading NetFlow Logs

• Extra Resources

  • AWS Educate
  • Databricksified Spark SQL Programming Guide 1.6
    • overview
    • getting started
    • data sources
    • performance tuning
    • distributed sql engine
  • Linear Algebra Cheat Sheet
  • Databricksified Data Types in MLLib Programming Guide 1.6
    • Local Vector
    • Labeled Point
    • Local Matrix
    • Distributed Matrix
    • Row Matrix
    • Indexed Row Matrix
    • Coordinate Matrix
    • Block Matrix
  • Introduction to XML-parsing of Old Bailey Online

Contribute

All course content is currently being pushed by Raazesh Sainudiin after it has been tested in
Databricks cloud (mostly under Spark 1.6 and some involving Magellan under Spark 1.5.1).

The markdown version for gitbook is generated from the Databricks .scala, .py and other source codes.
The gitbook is not a substitute for the Databricks notebooks available in the Databricks cloud. The following issues need to be resolved:

• need to find a stable solution for the output of various databricks cells to be shown in gitbook, including those from display_HTML and frameIt with their in-place embeds of web content.

Please feel free to fork the github repository:

• https://github.com/raazesh-sainudiin/scalable-data-science.

Furthermore, due to the anticipation of Spark 2.0 this mostly Spark 1.6 version could be enhanced with a 2.0 version-specific upgrade.

Please send any typos or suggestions to [email protected]

Please read a note on babel to understand how the gitbook is generated from the .scala source of the databricks notebook.

Raazesh Sainudiin,
Laboratory for Mathematical Statistical Experiments, Christchurch Centre
and School of Mathematics and Statistics,
University of Canterbury,
Private Bag 4800,
Christchurch 8041,
Aotearoa New Zealand

Sun Jun 19 21:59:19 NZST 2016