Apache Spark & Scala Certification Training Course in Brisbane, Australia

Apache Spark & Scala

• Lesson 00 - Course Overview

  04:11Preview

  • 0.001 Introduction

    00:12

  • 0.002 Course Objectives

    00:28

  • 0.003 Course Overview

    00:38

  • 0.004 Target Audience

    00:31

  • 0.005 Course Prerequisites

    00:21

  • 0.006 Value to the Professionals

    00:48

  • 0.007 Value to the Professionals (contd.)

    00:20

  • 0.008 Value to the Professionals (contd.)

    00:21

  • 0.009 Lessons Covered

    00:24

  • 0.010 Conclusion

    00:08

• Lesson 01 - Introduction to Spark

  25:17Preview

  • 1.001 Introduction

    00:15

  • 1.002 Objectives

    00:26

  • 1.3 Evolution of Distributed Systems

  • 1.004 Need of New Generation Distributed Systems

    01:15

  • 1.005 Limitations of MapReduce in Hadoop

    01:05

  • 1.006 Limitations of MapReduce in Hadoop (contd.)

    01:06

  • 1.007 Batch vs. Real-Time Processing

    01:09

  • 3.040 PairRDD Methods-Others

    00:06

  • 1.009 Application of In-Memory Processing

    01:47

  • 1.010 Introduction to Apache Spark

    00:44

  • 1.11 Components of a Spark Project

  • 1.012 History of Spark

    00:50

  • 1.013 Language Flexibility in Spark

    00:54

  • 1.014 Spark Execution Architecture

    01:13

  • 1.015 Automatic Parallelization of Complex Flows

    00:58

  • 1.016 Automatic Parallelization of Complex Flows-Important Points

    01:13

  • 1.017 APIs That Match User Goals

    01:05

  • 1.018 Apache Spark-A Unified Platform of Big Data Apps

    01:37

  • 1.019 More Benefits of Apache Spark

    01:05

  • 1.020 Running Spark in Different Modes

    00:40

  • 1.21 Installing Spark as a Standalone Cluster-Configurations

  • 1.022 Installing Spark as a Standalone Cluster-Configurations

    00:07

  • 1.023 Demo-Install Apache Spark

    00:07

  • 1.024 Demo-Install Apache Spark

    02:40

  • 1.025 Overview of Spark on a Cluster

    00:47

  • 1.026 Tasks of Spark on a Cluster

    00:36

  • 1.027 Companies Using Spark-Use Cases

    00:46

  • 1.028 Hadoop Ecosystem vs. Apache Spark

    00:31

  • 1.029 Hadoop Ecosystem vs. Apache Spark (contd.)

    00:42

  • 1.30 Quiz

  • 1.031 Summary

    00:39

  • 1.032 Summary (contd.)

    00:41

  • 1.033 Conclusion

    00:13

• Lesson 02 - Introduction to Programming in Scala

  37:15Preview

  • 2.001 Introduction

    00:11

  • 2.002 Objectives

    00:16

  • 2.003 Introduction to Scala

    01:32

  • 2.4 Features of Scala

  • 2.005 Basic Data Types

    00:24

  • 2.006 Basic Literals

    00:34

  • 2.007 Basic Literals (contd.)

    00:24

  • 2.008 Basic Literals (contd.)

    00:21

  • 2.009 Introduction to Operators

    00:31

  • 2.10 Types of Operators

  • 2.011 Use Basic Literals and the Arithmetic Operator

    00:07

  • 2.012 Demo Use Basic Literals and the Arithmetic Operator

    03:17

  • 2.013 Use the Logical Operator

    00:07

  • 2.014 Demo Use the Logical Operator

    01:40

  • 2.015 Introduction to Type Inference

    00:33

  • 2.016 Type Inference for Recursive Methods

    00:09

  • 2.017 Type Inference for Polymorphic Methods and Generic Classes

    00:30

  • 2.018 Unreliability on Type Inference Mechanism

    00:22

  • 2.019 Mutable Collection vs. Immutable Collection

    01:13

  • 2.020 Functions

    00:21

  • 2.021 Anonymous Functions

    00:21

  • 2.022 Objects

    01:07

  • 2.023 Classes

    00:36

  • 2.024 Use Type Inference, Functions, Anonymous Function, and Class

    00:09

  • 2.025 Demo Use Type Inference, Functions, Anonymous Function and Class

    07:39

  • 2.026 Traits as Interfaces

    00:57

  • 2.027 Traits-Example

    00:08

  • 2.028 Collections

    00:41

  • 2.029 Types of Collections

    00:25

  • 2.030 Types of Collections (contd.)

    00:26

  • 2.031 Lists

    00:28

  • 2.032 Perform Operations on Lists

    00:07

  • 2.033 Demo Use Data Structures

    04:09

  • 2.034 Maps

    00:45

  • 2.35 Maps-Operations

  • 2.036 Pattern Matching

    00:33

  • 2.037 Implicits

    00:36

  • 2.038 Implicits (contd.)

    00:17

  • 2.039 Streams

    00:21

  • 2.040 Use Data Structures

    00:07

  • 2.041 Demo Perform Operations on Lists

    03:24

  • 2.42 Quiz

  • 2.043 Summary

    00:37

  • 2.044 Summary (contd.)

    00:36

  • 2.045 Conclusion

    00:14

• Lesson 03 - Using RDD for Creating Applications in Spark

  50:27Preview

  • 3.001 Introduction

    00:11

  • 3.002 Objectives

    00:22

  • 3.003 RDDs API

    01:39

  • 3.4 Features of RDDs

  • 3.005 Creating RDDs

    00:36

  • 3.006 Creating RDDs-Referencing an External Dataset

    00:18

  • 3.007 Referencing an External Dataset-Text Files

    00:51

  • 3.008 Referencing an External Dataset-Text Files (contd.)

    00:49

  • 3.009 Referencing an External Dataset-Sequence Files

    00:32

  • 3.010 Referencing an External Dataset-Other Hadoop Input Formats

    00:46

  • 3.011 Creating RDDs-Important Points

    01:08

  • 3.012 RDD Operations

    00:37

  • 3.013 RDD Operations-Transformations

    00:47

  • 3.014 Features of RDD Persistence

    00:56

  • 3.015 Storage Levels Of RDD Persistence

    00:19

  • 3.16 Choosing The Correct RDD Persistence Storage Level

  • 3.017 Invoking the Spark Shell

    00:22

  • 3.018 Importing Spark Classes

    00:14

  • 3.019 Creating the SparkContext

    00:25

  • 3.020 Loading a File in Shell

    00:10

  • 3.021 Performing Some Basic Operations on Files in Spark Shell RDDs

    00:20

  • 3.022 Packaging a Spark Project with SBT

    00:50

  • 3.023 Running a Spark Project With SBT

    00:31

  • 3.024 Demo-Build a Scala Project

    00:06

  • 3.025 Build a Scala Project

    06:50

  • 3.026 Demo-Build a Spark Java Project

    00:07

  • 3.027 Build a Spark Java Project

    04:31

  • 3.028 Shared Variables-Broadcast

    01:20

  • 3.029 Shared Variables-Accumulators

    00:51

  • 3.030 Writing a Scala Application

    00:20

  • 3.031 Demo-Run a Scala Application

    00:07

  • 3.032 Run a Scala Application

    01:43

  • 3.033 Demo-Write a Scala Application Reading the Hadoop Data

    00:07

  • 3.034 Write a Scala Application Reading the Hadoop Data

    01:22

  • 3.035 Demo-Run a Scala Application Reading the Hadoop Data

    00:07

  • 3.036 Run a Scala Application Reading the Hadoop Data

    02:21

  • 3.37 Scala RDD Extensions

  • 3.038 DoubleRDD Methods

    00:08

  • 3.039 PairRDD Methods-Join

    00:46

  • 3.040 PairRDD Methods-Others

    00:06

  • 3.041 Java PairRDD Methods

    00:09

  • 3.042 Java PairRDD Methods (contd.)

    00:05

  • 3.043 General RDD Methods

    00:05

  • 3.044 General RDD Methods (contd.)

    00:05

  • 3.045 Java RDD Methods

    00:07

  • 3.046 Java RDD Methods (contd.)

    00:06

  • 3.047 Common Java RDD Methods

    00:09

  • 3.048 Spark Java Function Classes

    00:12

  • 3.049 Method for Combining JavaPairRDD Functions

    00:41

  • 3.050 Transformations in RDD

    00:33

  • 3.051 Other Methods

    00:07

  • 3.052 Actions in RDD

    00:08

  • 3.053 Key-Value Pair RDD in Scala

    00:31

  • 3.054 Key-Value Pair RDD in Java

    00:43

  • 3.055 Using MapReduce and Pair RDD Operations

    00:24

  • 3.056 Reading Text File from HDFS

    00:16

  • 3.057 Reading Sequence File from HDFS

    00:21

  • 3.058 Writing Text Data to HDFS

    00:18

  • 3.059 Writing Sequence File to HDFS

    00:12

  • 3.060 Using GroupBy

    00:07

  • 3.061 Using GroupBy (contd.)

    00:05

  • 3.062 Demo-Run a Scala Application Performing GroupBy Operation

    00:07

  • 3.063 Run a Scala Application Performing GroupBy Operation

    03:12

  • 3.064 Demo-Run a Scala Application Using the Scala Shell

    00:06

  • 3.065 Run a Scala Application Using the Scala Shell

    04:02

  • 3.066 Demo-Write and Run a Java Application

    00:06

  • 3.067 Write and Run a Java Application

    01:48

  • 3.68 Quiz

  • 3.069 Summary

    00:53

  • 3.070 Summary (contd.)

    00:59

  • 3.071 Conclusion

    00:15

• Lesson 04 - Running SQL Queries Using Spark SQL

  39:26Preview

  • 4.001 Introduction

    00:11

  • 4.002 Objectives

    00:17

  • 4.003 Importance of Spark SQL

    01:01

  • 4.004 Benefits of Spark SQL

    00:47

  • 4.005 DataFrames

    00:50

  • 4.006 SQLContext

    00:50

  • 4.007 SQLContext (contd.)

    01:12

  • 4.008 Creating a DataFrame

    00:10

  • 4.009 Using DataFrame Operations

    00:21

  • 4.010 Using DataFrame Operations (contd.)

    00:05

  • 4.011 Demo-Run SparkSQL with a Dataframe

    00:06

  • 4.012 Run SparkSQL with a Dataframe

    08:52

  • 4.13 Interoperating with RDDs

  • 4.014 Using the Reflection-Based Approach

    00:38

  • 4.015 Using the Reflection-Based Approach (contd.)

    00:08

  • 4.016 Using the Programmatic Approach

    00:44

  • 4.017 Using the Programmatic Approach (contd.)

    00:06

  • 4.018 Demo-Run Spark SQL Programmatically

    00:08

  • 4.019 Run Spark SQL Programmatically

    09:20

  • 4.20 Data Sources

  • 4.021 Save Modes

    00:31

  • 4.022 Saving to Persistent Tables

    00:45

  • 4.023 Parquet Files

    00:18

  • 4.024 Partition Discovery

    00:37

  • 4.025 Schema Merging

    00:28

  • 4.026 JSON Data

    00:34

  • 4.027 Hive Table

    00:45

  • 4.028 DML Operation-Hive Queries

    00:27

  • 4.029 Demo-Run Hive Queries Using Spark SQL

    00:06

  • 4.030 Run Hive Queries Using Spark SQL

    04:58

  • 4.031 JDBC to Other Databases

    00:49

  • 4.032 Supported Hive Features

    00:38

  • 4.033 Supported Hive Features (contd.)

    00:22

  • 4.034 Supported Hive Data Types

    00:13

  • 4.035 Case Classes

    00:14

  • 4.036 Case Classes (contd.)

    00:07

  • 4.37 Quiz

  • 4.038 Summary

    00:48

  • 4.039 Summary (contd.)

    00:48

  • 4.040 Conclusion

    00:12

• Lesson 05 - Spark Streaming

  34:48Preview

  • 5.001 Introduction

    00:11

  • 5.002 Objectives

    00:14

  • 5.003 Introduction to Spark Streaming

    00:49

  • 5.004 Working of Spark Streaming

    00:19

  • 5.5 Features of Spark Streaming

  • 5.006 Streaming Word Count

    01:34

  • 5.007 Micro Batch

    00:19

  • 5.008 DStreams

    00:34

  • 5.009 DStreams (contd.)

    00:38

  • 5.010 Input DStreams and Receivers

    01:19

  • 5.011 Input DStreams and Receivers (contd.)

    00:54

  • 5.012 Basic Sources

    01:14

  • 5.013 Advanced Sources

    00:49

  • 5.14 Advanced Sources-Twitter

  • 5.015 Transformations on DStreams

    00:15

  • 5.016 Transformations on Dstreams (contd.)

    00:06

  • 5.017 Output Operations on DStreams

    00:29

  • 5.018 Design Patterns for Using ForeachRDD

    01:14

  • 5.019 DataFrame and SQL Operations

    00:25

  • 5.020 DataFrame and SQL Operations (contd.)

    00:20

  • 5.021 Checkpointing

    01:25

  • 5.022 Enabling Checkpointing

    00:37

  • 5.023 Socket Stream

    00:59

  • 5.024 File Stream

    00:11

  • 5.025 Stateful Operations

    00:28

  • 5.026 Window Operations

    01:22

  • 5.027 Types of Window Operations

    00:11

  • 5.028 Types of Window Operations Types (contd.)

    00:05

  • 5.029 Join Operations-Stream-Dataset Joins

    00:20

  • 5.030 Join Operations-Stream-Stream Joins

    00:33

  • 5.031 Monitoring Spark Streaming Application

    01:18

  • 5.032 Performance Tuning-High Level

    00:20

  • 5.33 Performance Tuning-Detail Level

  • 5.034 Demo-Capture and Process the Netcat Data

    00:07

  • 5.035 Capture and Process the Netcat Data

    05:00

  • 5.036 Demo-Capture and Process the Flume Data

    00:07

  • 5.037 Capture and Process the Flume Data

    05:08

  • 5.038 Demo-Capture the Twitter Data

    00:06

  • 5.039 Capture the Twitter Data

    02:33

  • 5.40 Quiz

  • 5.041 Summary

    01:00

  • 5.042 Summary (contd.)

    01:04

  • 5.043 Conclusion

    00:11

• Lesson 06 - Spark ML Programming

  39:50Preview

  • 6.001 Introduction

    00:11

  • 6.002 Objectives

    00:19

  • 6.003 Introduction to Machine Learning

    01:35

  • 6.4 Common Terminologies in Machine Learning

  • 6.005 Applications of Machine Learning

    00:21

  • 6.006 Machine Learning in Spark

    00:33

  • 6.7 Spark ML API

  • 6.008 DataFrames

    00:32

  • 6.009 Transformers and Estimators

    00:59

  • 6.010 Pipeline

    00:48

  • 6.011 Working of a Pipeline

    01:41

  • 6.012 Working of a Pipeline (contd.)

    00:44

  • 6.013 DAG Pipelines

    00:33

  • 6.014 Runtime Checking

    00:20

  • 6.015 Parameter Passing

    00:59

  • 6.016 General Machine Learning Pipeline-Example

    00:06

  • 6.17 General Machine Learning Pipeline-Example (contd.)

  • 6.018 Model Selection via Cross-Validation

    01:15

  • 6.019 Supported Types, Algorithms, and Utilities

    00:30

  • 6.020 Data Types

    01:25

  • 6.021 Feature Extraction and Basic Statistics

    00:42

  • 6.022 Clustering

    00:37

  • 6.023 K-Means

    00:55

  • 6.024 K-Means (contd.)

    00:05

  • 6.025 Demo-Perform Clustering Using K-Means

    00:07

  • 6.026 Perform Clustering Using K-Means

    04:41

  • 6.027 Gaussian Mixture

    00:57

  • 6.028 Power Iteration Clustering (PIC)

    01:16

  • 6.029 Latent Dirichlet Allocation (LDA)

    00:34

  • 6.030 Latent Dirichlet Allocation (LDA) (contd.)

    01:45

  • 6.031 Collaborative Filtering

    01:13

  • 6.032 Classification

    00:16

  • 6.033 Classification (contd.)

    00:06

  • 6.034 Regression

    00:41

  • 6.035 Example of Regression

    00:56

  • 6.036 Demo-Perform Classification Using Linear Regression

    00:07

  • 6.037 Perform Classification Using Linear Regression

    02:00

  • 6.038 Demo-Run Linear Regression

    00:06

  • 6.039 Run Linear Regression

    02:14

  • 6.040 Demo-Perform Recommendation Using Collaborative Filtering

    00:05

  • 6.041 Perform Recommendation Using Collaborative Filtering

    02:23

  • 6.042 Demo-Run Recommendation System

    00:06

  • 6.043 Run Recommendation System

    02:45

  • 6.44 Quiz

  • 6.045 Summary

    01:14

  • 6.046 Summary (contd.)

    00:57

  • 6.047 Conclusion

    00:11

• Lesson 07 - Spark GraphX Programming

  46:16Preview

  • 7.001 Introduction

    00:13

  • 7.002 Objectives

    00:17

  • 7.003 Introduction to Graph-Parallel System

    01:13

  • 7.004 Limitations of Graph-Parallel System

    00:49

  • 7.005 Introduction to GraphX

    01:21

  • 7.006 Introduction to GraphX (contd.)

    00:06

  • 7.007 Importing GraphX

    00:10

  • 7.008 The Property Graph

    01:25

  • 7.009 The Property Graph (contd.)

    00:06

  • 7.010 Features of the Property Graph

  • 7.011 Creating a Graph

    00:13

  • 7.012 Demo-Create a Graph Using GraphX

    00:07

  • 7.013 Create a Graph Using GraphX

    10:08

  • 7.014 Triplet View

    00:30

  • 7.015 Graph Operators

    00:50

  • 7.016 List of Operators

    00:23

  • 7.017 List of Operators (contd.)

    00:05

  • 7.018 Property Operators

    00:18

  • 7.019 Structural Operators

    01:02

  • 7.020 Subgraphs

    00:21

  • 7.021 Join Operators

    01:09

  • 7.022 Demo-Perform Graph Operations Using GraphX

    00:07

  • 7.023 Perform Graph Operations Using GraphX

    05:46

  • 7.024 Demo-Perform Subgraph Operations

    00:07

  • 7.025 Perform Subgraph Operations

    01:36

  • 7.026 Neighborhood Aggregation

    00:43

  • 7.027 mapReduceTriplets

    00:41

  • 7.028 Demo-Perform MapReduce Operations

    00:08

  • 7.029 Perform MapReduce Operations

    09:17

  • 7.030 Counting Degree of Vertex

    00:32

  • 7.031 Collecting Neighbors

    00:28

  • 7.032 Caching and Uncaching

    01:09

  • 7.033 Graph Builders

  • 7.034 Vertex and Edge RDDs

    01:17

  • 7.035 Graph System Optimizations

    01:21

  • 7.036 Built-in Algorithms

  • 7.037 Quiz

  • 7.038 Summary

    01:12

  • 7.039 Summary (contd.)

    00:55

  • 7.040 Conclusion

    00:11