# What data scientist should know?

In this blog post, I will try to give you the first 10 things to become a Data Scientist .

For sure, depending of your background, you should learn many others things needed to become a great Data Scientist.

This is my personal list of the things that as data scientist should know:

I should remark that I am missing many other import programming languages that can be used in Data Science such as R, Spark, Scala, Ruby , JavaScript, Go and Swift and tools of ingestion of Data such as Apache Kafka and creation of the Cloud Infrastructure such as Terraform with Terragrunt and for the automatization of the ETL , Airflow and Jenkins and for sure the CLI in Linux and the use of Git and Unit test to check your programs. In addition I am skipping all the part of Deep Learning in details and Machine Learning in the Cloud that will be subject of future posts.

I know there are a toons of things that is needed to know to become a great Data Scientist but I will introduce only the essential topics based on Python and a little of SQL to manage the data from Cloud Databases.

You have also to know basis of Data Engineering such as in the previous post here and a little of Mathematics to understand how to solve the problems first by creating your algorithms which solves what you want to produce and analyze.

I have collected the information from different sources among them: Google , Udemy , Coursera, DataCamp , Pluralsight and EdX.

# Python for Data Science

Python Operator Precedence

From Python documentation on operator precedence (Section 5.15)

Highest precedence at top, lowest at bottom. Operators in the same box evaluate left to right.

Operator Description
() Parentheses (grouping)
f(args…) Function call
x[index:index] Slicing
x[index] Subscription
x.attribute Attribute reference
** Exponentiation
~x Bitwise not
+x, -x Positive, negative
*, /, % Multiplication, division, remainder
«, » Bitwise shifts
& Bitwise AND
^ Bitwise XOR
| Bitwise OR
in, not in, is, is not, <, <=, >, >=, <>, !=, == Comparisons, membership, identity
not x Boolean NOT
and Boolean AND
or Boolean OR
lambda Lambda expression

## Types and Type Conversion

str()
'5', '3.45', 'True' #Variables to strings

int()
5, 3, 1 #Variables to integers

float()
5.0, 1.0 #Variables to floats

bool()
True True True , , #Variables to boolean


### Libraries

Data analysis -> pandas

Scientific computing -> numpy

2D plotting -> matplotlib

Machine learning -> Scikit-Learn

### Import Libraries

import numpy
import numpy as np


### Selective import

from math import pi


### Strings

>>> my_string = 'thisStringisAwesome'
>>> my_string
'thisStringisAwesome'



String Operation

>>> my_string * 2
'thisStringisAwesomethisStringisAwesome'
>>> my_string +'Innit'
'thisStringisAwesomeinnit'
>>> 'm' in  my_string
True


String Indexing

Index starts at 0

>>> my_string[3]
>>> my_string[4:9]


String Methods

>>> my_string.upper() #String to uppercase
>>> my_string.lower() #String to lowercase
>>> my_string.count('w') #Count String elements
>>> my_string.replace('e','i') #Replace String elements
>>> my_string.strip() #Strip whitespoces


### Lists

>>> my_list = [1, 2, 3, 4]
>>> my_array = np.array(my_list)
>>> my_2darray = np.array([[1,2,3],[4,5,6]])



Selecting Numpy Array Elements

Index starts at 0

Subset
>>> my_array[1] #Select item at index 1
2
Slice
>>> my_array[ 0:2]#Select items at index 0 and 1
array([1, 2])
Subset 2D Numpy arrays
>>> my_2darray[:,0]#my_2darroy[rows, columns]
array([1, 4])


Numpy Array Operations

>>> my_array > 3
array([False, False, False, True], dtype=bool)
>>> my_array *  2
array([2, 4, 6, 8])
>>> my_array + np.array([5, 6, 7, 8])
array([6, 8, 10, 12])


Numpy Array Functions

>>> my_array.shape #Get the dimensions of the array
>>> np.append(other_array) #Append items to on array
>>> np.insert( my_array, 1, 5) #Insert items in on array
>>> np.delete( my_array,[1]) #Delete items in on array
>>> np.mean(my_array) #Mean of the array
>>> np.median(my_array) #Median of the array
>>> my_array.corrcoef() #Correlation coefficient
>>> np.std( my_array) #Standard deviation


### Lists

>>> a  =  'is'
>>> b  =  'nice'
>>> my_list = ['my','list', a, b]
>>> my_list2=[[4,5,6,7],[3,4,5,6]]


Selecting List Elements

Index starts at 0

Subset
>>> my_list[1] #Select item at index 1
>>> my_list[-3] #Select 3rd last item
Slice
>>> my_list[1:3] #Select items at index 1 and 2
>>> my_list[1:] #Select items after index 0
>>> my_list[:3] #Select items before index 3
>>> my_list[:] #Copy my_list
Subset Lists of Lists
>>> my_list2[1][0] #my_list[list][itemOfList]
>>> my_list2[1][:2]



List Operations

>>> my_list + my_list
['my','list','is','nice','my','list','is','nice']
>>> 2*my_list
['my','list','is','nice','my','list','is','nice']


List Methods

>>> my_list.index(a) #Get the index of an item
2
>>> my_list.count(a) #Count on item
1
>>> my_list.append( '!') #Append on item ot a time
['my','list','is','nice','!']
>>> my_list.remove( '!' ) #Remove  on  item
>>> del(my_list[0:1]) #Remove an item
['list','is','nice']
>>> my_list.reverse() #Reverse the list
['nice','is','list','my']
>>> my_list.extend('!') #Append on item
>>> my_list.pop(-1) #Remove on item
>>> my_list.insert(0, '!') #Insert on item
>>> my_list.sort() #Sort the list


>>> help(str)


# Section 2

### Importing Data

Most of the time, you’ll use either NumPy or pandas to import your data:

>>> import numpy as np
>>> import pandas as pd


Help

>>> np.info(np.ndarray.dtype)


### Text Files

Plain Text Files

>>>filename= 'huck_finn.txt'
>>>file= open(filename, mode= 'r' ) #Open the file for reading
>>> print(file.closed) #Check whether file is closed
>>> file.close() #Close file
>>> print(text)

>>> with open('huck_finn.txt', 'r' ) as file:


## Table Data: Flat Files

Importing Flat Files with NumPy

>>>filename= 'huck_finn.txt'
>>>file= open(filename, mode= 'r' ) #Open the file for reading
>>> print(file.closed) #Check whether file is closed
>>> file.close() #Close file
>>> print(text)


Files with one data type

>>>filename= 'mnist.txt'
delimiter=',' , #String used to separate values
skiprows=2, #Skip the first 2 lines
usecols=[0,2], #Read the 1st and 3rd column
dtype=str) #The type of the resulting array


Files with mixed data type

>>>filename= 'titanic.csv'
>>>data= np.genfromtxt(	filename,
delimiter = ',',
dtype=None)
>>> data_array = np.recfromcsv(filename)
#The default dtype of the np.recfromcsv() function is None


### Importing Flat Files with Pandas

>>>filename= 'winequality-red.csv'
nrows=5, #Number of rows of file to read
header=None, #Row number to use as col names
sep='\t', #Delimiter to use
na _values=[""]) #String to recognize as NA/NaN



NumPy Arrays

>> data_array.dtype #Data type of array elements
>>> data_array.shape #Array dimensions
>>> len(data_array) #Length of array


Pandas DataFrames

>>> df.head() #Return first DataFrame rows
>>> df.tail() #Return last OataFrame rows
>>> df.index #Describe index
>>> df.columns #Describe OataFrame columns
>>> df.info() #Info an DataFrame
>>> data_array = data.values #Convert a DataFrame to an a NumPy array



SAS File

>>> from sas7bdat import SAS7BDAT
>>> with SAS7BDAT( 'urbanpop .sas7bdat') as file:
df_sas = file.to_da ta_frame()


Stata File

>>>data= pd.read_stata('urbanpop .dta')


>>>file= 'urbanpop.xlsx'
>>>data= pd.ExcelFile(file)
>>> df_sheet2 = data.parse('1960-1966',
skiprows=[0], names=['Country',
'AAM: War(2002)'])
>>> df_sheetl = data.parse(0,
parse_cols=[0], skiprows=[0],
name s=['Country'])


To access the sheet names, use the sheet_names attribute:

>>> data.sheet_names


### Relational Databases

>>> from sqlalchemy import create _engine
>>>engine= create_engine('sq lite://Northwind.sqlite')


Use the table_name s() method to fetch a list of table names:

table_names = engine.table_names()


Querying Relational Databases

>>>con= engine.connect()
>>> rs= con.execute('SELECT* FROM Order s')
>>> df = pd.DataFrame(rs.fetchall())
>>> df.columns = rs.keys()
>>> con.close()


Using the context manager with

>>> with engine.connect() as con:
rs= con.execute('SELEC T OrderID FROM Order s')
df = pd.DataFrame(rs.fetchmany(size=5))
df.columns = rs.keys()


Querying relational databases with pandas

>>> df = pd.read_sql_query( ''SELECT*  FROM Orders'', engine)


Pickled File

>>> import pickle
>>> with open('pickled_fruit.pkl', 'rb' ) as file: pickled_data = pickle.load(file)


Matlab File

>>> import scipy.io
>>>filename= 1 workspace.m at 1


HDF5 Files

>>> import h5py
>>>filename= 'file.hdf5'
>>>data= h5py.File(filename, 'r' )


## Exploring Dictionaries

Querying relational databases with pandas

>>> print(mat.keys()) #Print dictionary keys
>>> for key in data.keys(): #Print dictionary keys
print(key)
meta quality strain
>>> pickled_data.values() #Return dictionary values
>>> print(mat.items()) #Returns items in list format of (key, value) tuple pairs


Accessing Data Items with Keys

>>> for key in data['meta'].keys() #Explore the HOF5
structure
print(key) Description DescriptionURL Detector
Duration GPSstart Observatory Type UTCstart
#Retrieve the value for a key
>>> print(data['meta']['Description'].value)


Magic Commands

!ls #List directory contents of files and directories
%cd .. #Change current working directory
%pwd #Return the current working directory path


OS Library

>>> import os
>>> path = "/usr/tmp"
>>> wd = os.getcwd() #Store the name of current directory in a string
>>> os.listdir(wd) #Output contents of the directory in a list
>>> os.chdir(path) #Change current working directory
>>> os.rename( "testl.txt", #Rename a file
"test2.txt" )
>>> os.remove( "test1. txt") #Oelete an existing file
>>> os.mkdir( "newdir") #Create a new directory


Pivot

>>> df3= df2.pivot(inde x='Date', #Spread rows into columns
col umns= 'Type' ,
values='Value' )


Pivot Table

>>> df4 = pd.pivot_table(df2, #Spread rows into
columns values='Value', index='Date', columns='Type'])


Stack / Unstack

>>>stacked= df5.stack() #Pivot o level of column labels
>>> stacked.unstack() #Pivot o level of index labels


Melt

>>> pd.melt(df2, #Gather columns into rows
id _vars=[°Date°],
value_var s=['Type','Value'],
value name=''Observations'')


### Iteration

>>> df.iteritems() #{Column-index, Series) pairs
>>> df.iterrows() #{Row-index, Series) pairs



### Missing Data

>>> df.dropna() #Drop NaN values
>>> df3.fillna(df3.mean()) #Fill NaN values with o predetermined value
>>> df2.replace("a" , "f") #Replace values with others



Selecting

>>> df3.loc[:,(df3>1).any()] #Select cols with any vols >1
>>> df3.loc[:,(df3>1).all()] #Select cols with vols> 1
>>> df3.loc[:,df3.isnull().any()] #Select cols with NaN
>>> df3.loc[:,df3.notnull().all()] #Select cols without NaN


Indexing With isin ()

>>> df[(df.Country.isin(df2.Type))] #Find some elements
>>> df3.filter(iterns="a","b"]) #Filter on values
>>> df.select(lambda x: not x%5) #Select specific elements


Where

>>> s.where(s > 0) #Subset the data


Query

>>> df6.query('second > first') #Query DataFrame


Setting/Resetting Index

>>> df.set_index('Country' ) #Set the index
>>> df4 = df.reset_index() #Reset the index
>>> df = df.rename(index=str, #Rename
DataFrame columns={	"Country":"cntry",
"Capital':"cptl",
"Population":pplt})


Reindexing

>>>s2=s.reindex(['a','c','d','e','b'])


Forward Filling

>>> df.reindex(	range(4),
method= 'ffill')

	Country	Capital		Population
0	Belgium	Brussels	11190846
1	India	New Delhi	1303171035
2	Brazil	Brasilia	207847528
3	Brazil	Brasilia	207847528


Backward Filling

>>> s3 = s.reindex(	range(5),
method= 'bfill')

0	3
1	3
2	3
3	3
4	3


Multilndexing

>>>arrays= [np.array([l,2,3]),
np.array([5,4,3])]
>>> df5 = pd.DataFrame(np.random.rand(3, 2), index=arrays)
>>>tuples= list(zip(*arrays))
>>>index= pd.Multilndex.from_tuples(tuples,
names= [ 'first' , 'second' ])
>>> df6 = pd.DataFrame(np.random.rand(3, 2), index=index)
>>> df2.set_index([ "Date", "Type"])


### Duplicate Data

>>> s3.unique() #Return unique values
>>> df2.duplicated('Type') #Check duplicates
>>> df2.drop_dup licates( 'Type', keep='last') #Drop duplicates
>>> df.index.duplicated() #Check index duplicates


### Grouping Data

Aggregation

>>> df2.groupby(by=['Date','Type']).mean()
>>> df4.groupby(level=0).sum()
>>> df4.groupby(level=0).agg({ 'a':lambda x:sum(x)/len (x), 'b': np.sum})


Transformation

>>> customSum = lambda x: (x+x%2)
>>> df4.groupby(level=0).transform(customSum)


## Combining Data

Merge

>>> pd.merge(data1,
data2,
how = 'left' ,
on='X1')


>>> pd.merge(data1,
data2,
how = 'right' ,
on='X1')


>>> pd.merge(data1,
data2,
how='inner',
on='X1')


>>> pd.merge(data1,
data2,
how='outer',
on='X1')


Join

>>> datal.join(data2, ho w='righ t')


Concatenate

Vertical

>>> s.append(s2)


Horizontal/Vertical

>>> pd.concat([s,s2],axis=l, keys=['One' ,'Two'])
>>> pd.concat([data1, data2], axis=l, join='inner')


### Dates

>>> df2['Date']= pd.to_da tetime(d f2['Date'])
>>> df2['Date']= pd.da te_range( '2000-1-1',
periods=6, freq='M' )
>>>dates= [datetime(2012,5,1), datetime(2012,5,2)]
>>>index= pd.Datetimelndex(dates)
>>>index= pd.date_range(datetime(2012,2,1), end, freq='BM' )



### Visualization

>>> import matplotlib.pyplot as plt
>>> s.plot()
>>> plt.show()


>> df2.plot()
>>> plt.show()


# Data Wrangling with Pandas

The Pandas library is built on NumPy and provides easy-to-use data structures and data analysis tools for the Python programming language.

Use the following import convention:

>>> import pandas as pd


### Pandas Data Structure

Series

A one-dimensional labeled array

capable of holding any data type

>>>  s = pd.Series([3,-5,7, s], index=['a','b','c','d'])


Dataframe

A two-dimensional labeled data structure with columns of potentially different types

>>> data = { 'Country' : ['Belgium' ,'India ',' Brazil'  ] ,
'Capital': ['Brussels','New Delhi','Brasilia'] ,
'Population': [111908s6, 1303171035, 207847528]}
>>> df = pd.DataFrame(data,columns=[ 'Country' , 'Capital' , 'Population' ])



Dropping

>>> s.drop(['a', 'c']) #Drop values from rows (axis=B)
>>> df.drop( 'Country', axis=l) #Drop values from columns(axis=l)


Sort & Rank

>>> df.sort_index() #Sort by labels along an axis
>>> df.sort_values( by='Country') #Sort by the values along on axis
>>> df.rank() #Assign ranks to entries


### I/O

>>> pd.read_csv('file.csv', header=None, nrows=5)
>>> df.to_csv('myDataFrame.csv')


>>> pd.read_excel( 'file.xlsx')
>>> df.to_excel('dir/myDataFrame.x lsx', sheet_name= 'Sheet1')


Read multiple sheets from the same file

>>> xlsx = pd.ExcelFile('file.xls')


Read and Write to SQL Query or Database Table

>>> from sqlalchemy import create_engine
>>> engine = create_eng ine('sqlite:///:memory:' )
>>> pd.read_sql( "SELECT* FROM my_tabl e;", engine)
>>> pd.read_sql_ tabl e('my_ tabl e', engine)
>>> pd.read_sql_query( "SELECT * FROM my_table;", engine)



>>> df.to_sql('myDf',engine)


### Selection

Getting

>>> s['b'] #Get one element
-5

>>> df[l:] #Get subset of a DataFrome
Country Capital Population 1 India New Delhi 1303171035
2 Brazil Brasilia 207847528


Selecting, Boolean Indexing & Setting

By Position

>>> df.iloc[[0],[0]] #Select single value by row & column
'Belgium'
>>> df.iat([0],[0])
'Belgium'


By Label

>>> df.loc[[0], [ 'Country']] #Select single value by row & column labels 'Belgium'
>>> df.at([0], [ 'Country ']) 'Belgium'


By Label/Position

>>> df.ix[2] #Select single row of subset of rows
Country Brazil Capital Brasilia Population 207847528
>>> df.ix[:,'Capital'] #Select a single column of subset of columns
0	Brussels
1	New Delhi
2	Brasilia
>>> df.ix[1,'Capital'] #Select rows and columns 'New Delhi'


Boolean Indexing

>>> s[N(s > 1)] #Series s where value is not >l
>>> s[(s < -1) I (s > 2)] #s where value is f-1 or >2
>>> df[df['Population']>1200000000] #Use filter to adjust DataFrame


Setting

>>> s['a'] = 6 #Set index a of Series s to 6


### Retrieving Series/DataFrame Information

Basic Information

>>> df.shape #(rows,columns)
>>> df.index #Describe index
>>> df.columns #Describe DataFrame columns
>>> df.info() #Info on DataFrame
>>> df.count() #Number of non-NA values


Summary

>>> df.sum() #Sum of values
>>> df.cumsum() #Cummulative sum of values
>>> df.min() /df.max() #11inimum/maximum values
>>> df.idxmin()/df.idxmax() #Minimum/Maximum index value
>>> df.describe() #Summary statistics
>>> df.mean() #11ean of values
>>> df.median() #Median of values


### Applying Functions

>>> f = lambda x: X*2
>>> df.apply(f) #Apply function
>>> df.applymap(f) #Apply function element-wise


Data Alignment

Internal Data Alignment

>>> s3 = pd.Series([7, -2, 31, index= ['a','c' ,'d'])
>>> s + s3
a 10.0
b NaN
C 5.0
d 7.0


Arithmetic Operations with Fill Methods

You can also do the internal data alignment yourself with the help of the fill methods:

>>> s.add(s3, fill_values=0) a 10.0
b -5. 0
C 5.0
d 7.0
>>> s.sub(s3, fill_value=2)
>>> s.div(s3, fill_value=4)
>>> s.mul(s3, fill_value=3)


# Section 4

## Data Analysis with Numpy

The NumPy library is the core library for scientific computing in Python. It provides a high performance multidimensional array object , and tools for working with these arrays

Use the following import convention

>> import numpy as np


Creating Array

>>> a = np.array([l,2,3])
>>> b = np.array([(l.5,2,3), (4,5,6)], dtype = float)
>>> c = np.array([[(l.5,2,3), (4,5,6)],[(3,2,1), (4,5,6)]], dtype = float)


Initial Placeholders

>>> np.zeros((3,4)) #Create an array af zeros
>>> np.ones((2,3,4),dtype=np.int16) #Create an array of ones
>>> d = np.arange(10,25,5) #Create an array of evenly spaced values (step value)
>>> np.linspace(0,2,9) #Create an array of evenly spaced values (number of samples)
>>> e = np.full((2,2),7) #Create a constant array
>>> f = np.eye(2) #Create a 2X2 identity matrix
>>> np.random.random((2,2)) #Create an array with random values
>>> np.empty((3,2)) #Create an empty array



### I/O

>>> np.save('my_array',a)
>>> np.save('array.npz',a, b)


>> np.loadtxt("myfile.txt")
>>> np.genfromtxt("my_file.csv", delimiter=',' )
>>> np.savetxt("myarray.txt" , a, delimiter=" " )


>>> a.shape #Array dimensions
>>> len(a) #Length of array
>>> b.ndim #Number of array dimensions
>>> e.size #Number of array elements
>>> b.dtype #Data type of array elements
>>> b.dtype.name #Name of data type
>>> b.astype(int) #Convert an array to a different type


Data Type

>>> np.int64 #Signed 64-bit integer types
>>> np.flaat32 #Standard double-precision floating paint
>>> np.complex #Complex numbers represented by 128 floats
>>> np.baol #Boolean type storing TRUE and FALSE values
>>> np.object #Python object type
>>> np.string _ #Fixed-length string type
>>> np.unicode_ #Fixed-length unicode type


## Array Mathematics

Arithmetic Operations

>>> g = a - b #Subtraction
array([[-0.5,0. , 0. ],
[-3. , -3. , -3. ]])
>>> np.subtract(a,b) #Subtraction
array([[ 2.5, 4. , 6. ],
[5.,7.,9.]])
>>> a / b #Division
array([[ 0.66666667, 1., 1.],
[ 0.25 , 0.4 , 0.5 ]])
>>> np.divide(a,b) #Division
>>> a * b #Multiplication
array([[ 1.5, 4. , 9. ],
[ 4. , 10. , 18. ]])
>>> np.multiply(a,b) #Multiplication
>>> np.exp(b) #Exponentiation
>>> np.sqrt(b) #Square root
>>> np.sin(a) #Print sines of an array
>>> np.cos(b) #Element-wise cosine
>>> np.log(a) #Element-wise natural logarithm
>>> e.dot(f) #Dot product
array([[ 7., 7.],


Comparison

>>>a == b #Element-wise comparison
array([[False, True , True],
[ False, False, False]], dtype=bool)
>>> a < 2 #Element-wise comparison
array([True , False, False], dtype=bool)
>>> np.array_equal(a, b) #Array-wise comparison


Aggregate Functions

>>> a.sum() #Array-wise sum
>>> a.min() #Array-wise minimum value
>>> b.max(axis=0) #Maximum value of an array row
>>> b.cumsum(axis=l) #Cumulative sum of the elements
>>> a.mean() #Mean
>>> b.median() #Median
>>> a.corrcaef() #Correlation coefficient
>>> np.std(b) #Standard deviation


### Copying Array

>>> h = a.view() #Create a view af the array with the some data
>>> np.capy(a) #Create a copy of the array
>>> h = a.copy() #Create a deep copy of the array


### Sorting Array

Subsetting

>>> a[2] #Select the element at the 2nd index
3


>>> b[1,2] #Select the element at row 1 column 2 (equivalent to b[1][2])
6.0


Slicing

>>> a[0:2] #Select items at index 0 and 1
array([1, 2])


>>> b[0:2,1] #Select items at rows 0 and 1 in column 1
array([ 2., 5.])

>>> b[:1] #Select all items at row 0 (equivalent to b[0:1, :])
array([[1.5, 2., 3.]])


>>> c[1,... ] #Same as [1,:,:]
array([[[ 3., 2., 1.],
[  4., 5.,	6.]]])
>>> a[: :-1] #Reversed array a array([3, 2, 1])


Boolean Indexing

>>> a[a<2] #Select elements from a less than 2
array([1])


Fancy Indexing

>>> b[[l, 0, 1, 0],[0, 1, 2, 0]] #Select elements (1,0),(0,1),(1,2) and (0,0)
array([ 4. , 2. , 6. , 1.5])
>>> b[[l, 0, 1, 0]][:,[0,1,2,0]] #Select a subset of the matrix's rows and columns array([[ 4. ,5. , 6. , 4. ],
[ 1.5, 2.3,	1.5],


### Array Manipulation

Transposing Array

>>> i = np.transpose(b) #Permute array dimensions
>>> i.T #Permute array dimensions


Changing Array Shape

>>> b.ravel() #Flatten the array
>>> g.reshape(3,-2) #Reshape, but don't change data


>>> h.resize((2,6)) #Return a new array with shape (2,6)
>>> np.append(h,g) #Append items to an array
>>> np.insert(a, 1, 5) #Insert items in an array
>>> np.delete(a,[1]) #Delete items from an array


Combining Arrays

>>> np.concatenate((a,d),axis=0) #Concatenate arrays array([ 1, 2, 3, 10, 15, 20])
>>> np.vstack((a,b)) #Stack arrays vertically (row-wise)
array([[ 1. , 2.	3. ],
[ 1.5, 2. , 3. ],
[   4.    ,   5.    ,   6.    ]])
>>> np.r_[e,f] #Stack arrays vertically (row -wise)
>>> np.hstack((e,f)) #Stack arrays horizontally (col umn-wise)
array([[ 7., 7., 1., 0.],
[ 7., 7., 0., 1.]])
>>> np.column_stack((a,d)) #Create stacked column-wise arrays
array([[ 1, 10],
[ 2,  15],
[ 3, 20]])
>>> np.c_[a ,d] #Create stacked column-wise arrays



Splitting Arrays

>>> np.hsplit(a,3) #Split the array horizontally at the 3rd index
[array([1]),array([2]),array([3])]
>>> np.vsplit(c,2) #Split the array vertically at the 2nd index
[array([[[ 1.5, 2. , 1. ],
[ 4. , 5. , 6. ]]]),
array([[[ 3., 2., 3.],
[  4.,	5.,	6.]]])]


# Section 5

## Data Visualization with Matplotlib

Matplotlib is a Python 2D plotting library which produces publication-quality figures in a variety of hardcopy formats and interactive environments across platforms.

1D Data

>>> import numpy as np
>>> X = np.linspace(0, 10, 100)
>>> y = np.cos(x)
>>> z = np.sin(x)



2D Data or Images

>>> data= 2 * np.random.random((10, 10))
>>> data2 = 3 * np.random.random((10, 10))
>>> Y, X = np.mgrid[-3:3:100j, -3:3:100j]
>>> U = -1 - X**2 + Y
>>> V = 1 +  X - Y**2
>>> from matplotlib.cbook import get_sample_data
>>> img = np.load(g et_sample_data('axes_gr id/bivar iate_normal.npy '))


### Create Plot

>>>import matplotlib.pyplot as plt


Figure

>>> fig = plt.figure()
>>> fig2 = plt.figure(figsize=plt.figaspect(2.0))


Axes

ll plotting is done with respect to an Axes. In most cases, a subplot will fit your needs. A subplot is an axes on a grid system.

>>> fig.add_axes()
>>> fig3, axes= plt.subplots(nrows=2,ncols=2)
>>> fig4, axes2 = plt.subplots(ncols=3)


Save Plot

>>> plt.savefig( 'foe.png' J #Save figures
>>> plt.savefig( 'foo.png',transparent=True) #Save transparent figures


Show Plot

>>> plt.show()


### Plotting Routines

1D Data

>>> fig, ax= plt.subp lots()
>>>lines= ax.plot(x,y) #Draw points with lines or markers connecting them
>>> ax.scatter(x,y) #Draw unconnected points, scaled or colored
>>> axes[0,0].bar([l,2,3],[3,4,5]) #Plot vertical rectangles (constant width)
>>> axes[l,0].barh([0.5,1,2.5],[0,1,2]) #Plot horiontol rectangles (constant height)
>>> axes[l,1].axhline(0.s5) #Draw a horizontal line across axes
>>> axes[0,1].axvline(0.65) #Draw a vertical line across axes
>>> ax.fill(x,y,color='blue') #Drow filled polygons
>>> ax.fill_between(x,y,color='yellow') #Fill between y-values and 0



2D Data

>>> fig, ax= plt.subplots()
>>> im = ax.imshow(img, #Colormapped or RGB arrays
cmap= 1 gist_ear th 1 ,
interpolation= 1 neare st',
vmin=-2,
vmax =2)
>>> axes2[0].pcolor(data2) #Pseudocolor plot of 2D array
>>> axes2[0].pcolormesh(data) #Pseudocolor plot of 2D array
>>>CS= plt.contour(Y,X,U) #Plot contours
>>> axes2[2].contourf(datal) #Plot filled contours
>>> axes2[2]= ax.clabel(CS) #Lobel a contour plot



Vector Fields

>>> axes[0,1].arrow(0,0,0.5,0.5) #Add an arrow to the axes
>>> axes[l,l].quiver(y,z) #Plot a 2D field of arrows
>>> axes[0,1].streamplot(X,Y,U,V) #Plot a 2D field of arrows



Data Distributions

>>> axl.hist(y) #Plot a histogram
>>> ax3.boxplot(y) #Make a box and whisker plot
>>> ax3.violinplot(z) #Make a violin plot



### Plot Anatomy

The basic steps to creating plots with matplotlib are:

1 Prepare Data 2 Create Plot 3 Plot 4 Customized Plot 5 Save Plot 6 Show Plot

>>> import matplotlib.pyplot as plt
>>> x = [1,2,3,s] #Step 1
>>> y = [ 10,20,25,30]
>>>fig= plt.figure() #Step 2
>>> ax.plot(x, y, color='lightb lue', linewidth=3) #Step 3, 4
>>> ax.scatter([2,4,6],
[5,15,25],
color='darkgreen ',
marker='^')
>>> ax.set_xlim(l, 6.5)
>>> plt.savefig('foo.png' ) #Step 5
>>> plt.show() #Step 6


Close and Clear

>>> plt.cla() #Clear on axis
>>> plt.clf() #Clear the entire figure
>>> plt.close() #Close a window


### Plotting Cutomize Plot

Colors, Color Bars & Color Map

>>> plt.plot(x, x, x, X**2, x, X**3)
>>> ax.plot(x, y, alpha = 0.s)
>>> ax.plot(x, y, c='k' )
>>> fig.colorbar(im, orientation= 'horizontal')
>>> im = ax.imshow(img,cmap=  'seismic'  )



Markers

>>> fig, ax= plt.subplots()
>>> ax.scatter(x,y,marker="."   )
>>> ax.plot(x,y ,marker="o")


Linestyles

>>> plt.plot(x,y,linewidth=4.0)
>>> plt.plot(x,y,ls='solid')
>>> plt.plot(x,y,ls='--' )
>>> plt.plot(x, y,'-- 1 ,X**2,Y** 2, '-.')
>>> plt.setp(lines,color='r',linewidth= 4.0)


Text & Annotations

>>> ax.text(1,-2.1,'Example Graph' , style='italic')
>>> ax.annotate("Sine",
xy=(8, 0),
xycoords= 'data',
xytext=(10.5, 0),
textcoords= 1 data',
arrowprops=dict(arrowstyle= "->" connectionstyle="arc3"),)


Mathtext

>>> plt.title(r'$sigma_ i=15$', fontsize=20)


Limits, Legends and Layouts

Limits & Autoscaling

>>> ax.margins(x=0.0,y=0.1) #Add padding to a plot
>>> ax.axis('equa l') #Set the aspect ratio of the plot to 1
>>> ax.set(xlim=[0,10.5],ylim=[-1.5,l.5]) #Set limits for x-and y-axis
>>> ax.set_xlim(0,10.5) #Set limits for x-axis


Legends

>>> ax.set(	title='An Example Axe s', #Set a title and x-ond y-axis labels
ylabel='Y-Axis',
xlabel='X-Axis')
>>> ax.legend( loc='best') #No overlapping plot elements


Ticks

>>> ax.xaxis.set(ticks=range(l,5), #Manually set x-ticks
ticklabels=[3,100,-12,"foo']')
#Makey-ticks longer and go in and out
>>> ax.tick_param s(axis='y',direction='inout ',length=10)



Subplot Spacing

>>> fig3.subplots_ad just(wspace=0.5, #Adjust the spacing between subplots
hspace=0.3, left=0.125, right=0.9, top=0.9, bottom=0.1)
>>> fig.tight_layout() #Fit subplot(s) in to the figure area


Axis Spines

>>> axl.spines['top'].set_visible(False)
#Make the top axis line for a plot invisible
>>> axl.spines['bottom'].set_po sition(('outward' ,10))
#Move the bottom axis line outward


# Queries in SQL

## Querying data from a table

Query data in columns c1, c2 from a table

SELECT c1, c2 FROM t;


Query all rows and columns from a table

SELECT * FROM t;


Query data and filter rows with a condition

SELECT c1, c2 FROM t
WHERE condition;


Query distinct rows from a table

SELECT DISTINCT c1 FROM t
WHERE condition;


Sort the result set in ascending or descending order

SELECT c1, c2 FROM t
ORDER BY c1 ASC [DESC];


Skip offset of rows and return the next n rows

SELECT c1, c2 FROM t
ORDER BY c1
LIMIT n OFFSET offset;


Group rows using an aggregate function

SELECT c1, aggregate(c2)
FROM t
GROUP BY c1;


Filter groups using HAVING clause

SELECT c1, aggregate(c2)
FROM t
GROUP BY c1
HAVING condition;


## Querying from multiple tables

Inner join t1 and t2

SELECT c1, c2
FROM t1
INNER JOIN t2 ON condition;


Left join t1 and t1

SELECT c1, c2
FROM t1
LEFT JOIN t2 ON condition;


Right join t1 and t2

SELECT c1, c2
FROM t1
RIGHT JOIN t2 ON condition;


Perform full outer join

SELECT c1, c2
FROM t1
FULL OUTER JOIN t2 ON condition;


Produce a Cartesian product of rows in tables

SELECT c1, c2
FROM t1
CROSS JOIN t2;


Another way to perform cross join

SELECT c1, c2
FROM t1, t2;


Join t1 to itself using INNER JOIN clause

SELECT c1, c2
FROM t1 A
INNER JOIN t1 B ON condition;


## Using SQL Operators

Combine rows from two queries

SELECT c1, c2 FROM t1
UNION [ALL]
SELECT c1, c2 FROM t2;


Return the intersection of two queries

SELECT c1, c2 FROM t1
INTERSECT
SELECT c1, c2 FROM t2;


Subtract a result set from another result set

SELECT c1, c2 FROM t1
MINUS
SELECT c1, c2 FROM t2;


Query rows using pattern matching %, _

SELECT c1, c2 FROM t1
WHERE c1 [NOT] LIKE pattern;


Query rows in a list

SELECT c1, c2 FROM t
WHERE c1 [NOT] IN value_list;


Query rows between two values

SELECT c1, c2 FROM t
WHERE  c1 BETWEEN low AND high;


Check if values in a table is NULL or not

SELECT c1, c2 FROM t
WHERE  c1 IS [NOT] NULL;


## Managing tables

Create a new table with three columns

CREATE TABLE t (
id INT PRIMARY KEY,
name VARCHAR NOT NULL,
price INT DEFAULT 0
);


Delete the table from the database

DROP TABLE t ;


Add a new column to the table

ALTER TABLE t ADD column;


Drop column c from the table

ALTER TABLE t DROP COLUMN c ;


ALTER TABLE t ADD constraint;


Drop a constraint

ALTER TABLE t DROP constraint;


Rename a table from t1 to t2

ALTER TABLE t1 RENAME TO t2;


Rename column c1 to c2

ALTER TABLE t1 RENAME c1 TO c2 ;


Remove all data in a table

TRUNCATE TABLE t;


## UsingSQL constraints

Set c1 and c2 as a primary key

CREATE TABLE t(
c1 INT, c2 INT, c3 VARCHAR,
PRIMARY KEY (c1,c2)
);


Set c2 column as a foreign key

CREATE TABLE t1(
c1 INT PRIMARY KEY,
c2 INT,
FOREIGN KEY (c2) REFERENCES t2(c2)
);


Make the values in c1 and c2 unique

CREATE TABLE t(
c1 INT, c1 INT,
UNIQUE(c2,c3)
);


Ensure c1 > 0 and values in c1 >= c2

CREATE TABLE t(
c1 INT, c2 INT,
CHECK(c1> 0 AND c1 >= c2)
);


Set values in c2 column not NULL

CREATE TABLE t(
c1 INT PRIMARY KEY,
c2 VARCHAR NOT NULL
);


## Modifying Data

Insert one row into a table

INSERT INTO t(column_list)
VALUES(value_list);


Insert multiple rows into a table

INSERT INTO t(column_list)
VALUES (value_list),
(value_list), …;


Insert rows from t2 into t1

INSERT INTO t1(column_list)
SELECT column_list
FROM t2;


Update new value in the column c1 for all rows

UPDATE t
SET c1 = new_value;


Update values in the column c1, c2 that match the condition

UPDATE t
SET c1 = new_value,
c2 = new_value
WHERE condition;


Delete all data in a table

DELETE FROM t;


Delete subset of rows in a table

DELETE FROM t
WHERE condition;


## Managing Views

Create a new view that consists of c1 and c2

CREATE VIEW v(c1,c2)
AS
SELECT c1, c2
FROM t;


Create a new view with check option

CREATE VIEW v(c1,c2)
AS
SELECT c1, c2
FROM t;
WITH [CASCADED | LOCAL] CHECK OPTION;


Create a recursive view

CREATE RECURSIVE VIEW v
AS
select-statement -- anchor part
UNION [ALL]
select-statement; -- recursive part


Create a temporary view

CREATE TEMPORARY VIEW v
AS
SELECT c1, c2
FROM t;


Delete a view

DROP VIEW view_name;



## Managing indexes

Create an index on c1 and c2 of the t table

CREATE INDEX idx_name
ON t(c1,c2);



Create a unique index on c3, c4 of the t table

CREATE UNIQUE INDEX idx_name
ON t(c3,c4)



Drop an index

DROP INDEX idx_name;



## Managing triggers

Create or modify a trigger

CREATE OR MODIFY TRIGGER trigger_name
WHEN EVENT
ON table_name TRIGGER_TYPE
EXECUTE stored_procedure;



WHEN

• BEFORE – invoke before the event occurs
• AFTER – invoke after the event occurs

EVENT

• INSERT – invoke for INSERT
• UPDATE – invoke for UPDATE
• DELETE – invoke for DELETE

TRIGGER_TYPE

• FOR EACH ROW
• FOR EACH STATEMENT

Delete a specific trigger

DROP TRIGGER trigger_name;



# Machine Learning with Scikit-Learn

Scikit-learn is an open source Python library that implements a range of machine learning, preprocessing, cross-validation and visualization algorithms using a unified interface.

Example

>>> from sklearn import neighbors, datasets, preprocessing
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.metrics import accuracy_score
>>> X, y = iris.data[:, :2], iris.target
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=33)
>>>scaler= preprocessing.StandardScaler().fit(X_train)
>>> X_train = scaler.transform(X_train)
>>> X_test = scaler.transform(X_test)
>>> knn = neighbors.KNeighborsClassifier(n_neighbors=5)
>>> knn.fit(X_train, y_train)
>>> y_pred = knn.predict(X_test)
>>> accuracy_score(y _test, y_pred)


Your data needs to be numeric and stored as NumPy arrays or SciPy sparse matrices. Other types that are convertible to numeric arrays, such as Pandas DataFrame, are also acceptable

>>> import numpy as np
>>> X = np.random.random((10,5))
>>> Y = np. array (['M', 'M' ,'F','F','M','F','M','M','F','F','F'])
>>> X[X < 0.7] = 0


### Training And Test Data

>>> from sklearn.model_selection import train_test_split
>>> X_train, X_test, y_train, y_test = train_test_split(X,y, random_state=0)



### Model Fitting

Supervised learning

>>> lr.fit(X, y) #Fit the model to the data
>>> knn.fit(X_train, y_train)
>>> svc.fit(X_train, y_train)


Unsupervised Learning

>>> k_mean s.fit(X_train) #Fit the model to the data
>>> pca_model = pca.fit_transform(X_train) #Fit to data, then transform it


### Prediction

Supervised Estimators

>>> y_pred = svc.predict(np.random.random((2,5))) #Predict labels
>>> y_pred = lr.predict(X_test) #Predict labels
>>> y_ pred = knn.predict_proba(X_test) #Estimate probability of a label


Unsupervised Estimators

>>> y_pred = k_means.predict(X_test) #Predict labels in clustering algos


### Preprocessing The Data

Standardization

>>> from sklearn.preprocessing import StandardScaler
>>> scaler= StandardScaler().fit(X_train)
>>> standardized_X = scaler.transform(X_train)
>>> standardized_X_test= scaler.transform(X _test)


Normalization

>>> from sklearn.preprocessing import Normalizer
>>>scaler= Normalizer().fit(X_train)
>>> normalized_X = scaler.transform(X _train)
>>> normalized_X_test = scaler.transform(X_test)


Binarization

>>> from sklearn.preprocessing import Binarizer
>>> binarizer = Binarizer(threshold=0.0).fit(X)
>>> binary_X = binarizer.transform(X)



Encoding Categorical Features

>>> from sklearn.preprocessing import LabelEncoder
>>>enc= LabelEncoder()
>>> y = enc.fit_transform(y)


Imputing Missing Values

>>> from sklearn.preprocessing import Imputer
>>>imp= Imputer(missing_values=0, strategy='mean ', axis=0)
>>> imp.fit_transform(X_train)


Generating Polynomial Features

>>> from sklearn.preprocessing import PolynomialFeatures
>>>poly= PolynomialFeatures(5)
>>> poly.fit_transform(X)


### Supervised Learning Estimators

Linear Regression

>>> from sklearn. linear m_ odel import LinearRegression
>>> lr = LinearRegression(normalize=True)


Support Vector Machines (SVM)

>>> from sklearn.svm import SVC
>>>SVC= SVC(kernel='linear')


Naive Bayes

>>> from sklearn.naive_bayes import GaussianNB
>>> gnb = GaussianNB()


KNN

>>> from sklearn import neighbors
>>> knn = neighbors.KNeighborsClassifier(n_neighbors=5)


### Unsupervised Learning Estimators

Principal Component Analysis (PCA)

>>> from sklearn.decomposition import PCA
>>>pea= PCA(n_components=0.95)


K Means

>>> from sklearn.cluster import KMeans
>>> k_means = KMeans(n_clusters=3, random_state=0)


### Classification Metrics

Accuracy Score

>>> knn.score(X_test, y_test) #Estimator score method
>>> from sklearn.metrics import accuracy_score #Metric scoring functions
>>> accuracy_score(y_test, y_pred)


Classification Report

>>> from sklearn.metrics import classification_report #Precision, recall, fl-score and support
>>> print(classification_report(y_test, y_pred))


Confusion Matrix

>>> from sklearn.metrics import confusion_matrix
>>> print(confusion_matrix(y_test.y_pred))


### Regression Metrics

Mean Absolute Error

>>> from sklearn.metrics import mean_absolute_error
>>> y_true = [3, -0.5,2]
>>> mean_absolute_error(y_true, y_pred)



Mean Squared Error

>>> from sklearn.metrics import mean_squared_error
>>> mean_squared _error(y_test, y_ pred)


R2 Score

>>> from sklearn.metrics import r2_score
>>> r2_score(y_true, y_ pred)


### Clustering Metrics

>>> from sklearn.metrics import adjusted_rand_score



Homogeneity

>>> from sklearn.metrics import homogeneity_score
>>> homogeneity_score(y_true, y_pred)


V-measure

>>> from sklearn.metrics import v_measure_score
>>>metrics.v_measure_score(y_true , y_pred)


### Cross-Validation

>>> from sklearn.cross_validation import cross_val_score
>>> print(cross_val_score(knn, X_train, y_train, cv=4))
>>> print(cross_val_score(lr, X, y, cv=2))


>>> from sklearn.grid_search import GridSearchCV
>>> params = {"n_neighbors "  : np.arange(l,3),
"metric "	 : [ "euclidean" , "cityblock "] }
>>>grid= GridSearchCV(estimator=knn,
param_grid=params)
>>> grid.fit(X_train, y_train)
>>> print(grid.best_score_)
>>> print(grid.best_estimator_.n_neighbors)


Randomized Parameter Optimization

>>> from sklearn.grid _search import RandomizedSearchCV
>>> params = { "n_neighbors":range(l,5), "weights": ["unifomr","distance"]}
>>> rsearch = RandomizedSearchCV(estimator=knn, param_distributions=params,
cv=4, n_iter=S, random_state=5)
>>> rsearch.fit(X_train, y_train)
>>> print(rsearch.best_score_)


# Section 8

## SciPy

The SciPy library is one of the core packages for scientific computing that provides mathematical algorithms and convenience functions built on the NumPy extension of Python.

>>> import numpy as np
>>> a= np.array([1,2,3])
>>> b = np.array([(1+5j,2j,3j), (4j,5j,6j)])
>>> c = np.array([[(1.5,2,3), (4,5,6)], [(3,2,1), (4,5,6)]])


Index Tricks

>>> np.mgrid[0:5,0:5] #Create a dense meshgrid
>>> np.ogrid[0:2,0:2] #Create an open meshgrid
>>> np.r_[[3,[0]*5,-1:1:10j] #Stack arrays vertically (row-wise)
>>> np.c_[b,c] #Create stocked column-wise arrays


Shape Manipulation

>>> np.transpose(b) #Permute array dimensions
>>> b.flatten() #Flatten the array
>>> np.hstack((b,c)) #Stack arrays horizontally (column-wise)
>>> np.vstack((a,b)) #Stack arrays vertically (row-wise)
>>> np.hsplit(c,2) #Split the array horizontally at the 2nd index
>>> np.vpslit(d,2) #Split the array vertically at the 2nd index


Polynomials

>>> from numpy import polyld
>>> p = poly1d([3,4,5]) #Create a polynomial object


Vectorizing Functions

>>> def myfunc(a): if a< 0:
return a*2
else:
return a/2
>>> np.vectorize(myfunc) #Vectorize functions



Type Handling

>>> np.real(c) #Return the real part of the array elements
>>> np.imag(c) #Return the imaginary part of the array elements
>>> np.real_if_close(c,tol=1000) #Return a real array if complex parts close to 0
>>> np.cast['f'](np.pi) #Cast object to a data type


Other Useful Functions

>>> np.angle(b,d eg=True) #Return the angle of the complex argument
>>> g = np.linspace(0,np.pi,num=5) #Create an array of evenly spaced values(number of samples)
>>> g [3:] += np.pi
>>> np.unwrap(g) #Unwrap
>>> np.logspace(0,10,3) #Create an array of evenly spaced values (log scale)
>>> np.select([c<li],[c*2]) #Return values from a list of arrays depending on conditions
>>> misc.factorial(a) #Factorial
>>> misc.comb( 10,3,exact=True) #Combine N things taken at k time
>>> misc.central_diff_weights(3) #Weights for Np-point central derivative
>>> misc.derivative(myfunc,1.0) #Find then-th derivative of a function at a point



### Linear Algebra

You’ll use the linalg and sparse modules. Note that scipy. linalg contains and expands on numpy. linalg.

>>> from scipy import linalg, sparse


Creating Matrices

>>> A = np.matrix(np.random.random((2,2)))
>>> B = np.asmatrix(b)
>>> C = np.mat(np.random.random((10,5)))
>>> D = n p.mat([[3,Ii],  [5,6]])


Basic Matrix Routines

>>> A.I #Inverse
>>> linalg.inv(A)  #Inverse
>>> A.T #Tranpose matrix
>>> A.H #Conjugate transposition
>>> np.trace(A) #Trace


Norm

>>> linalg.norm(A) #Frobenius norm
>>> linalg.norm(A,1) #Ll norm (max column sum)
>>> linalg.norm(A,np.inf) #L inf norm (max row sum)


Rank

>>> np.linalg.matrix_rank(C) #Matrix rank


Determinant

>>> linalg.det(A) #Determinant


Solving linear problems

>>> linalg.solve(A,b) #Solver for dense matrices
>>> E = np.mat(a).T #Solver for dense matrices
>>> linalg.lstsq(D,E) #Le ast-squares solution to linear matrix equation


Generalized inverse

>>> linalg.pinv(C) #Compute the pseudo-inverse of a matrix (least-squares solver)
>>> linalg. pinv2(C) #Compute the pseudo-inverse of a matrix (SVD)


### Creating Sparse Matrices

>>> F = np.eye(3, k=l) #Create a 2X2 identity matrix
>>> G = np.mat(np.identity(2)) #Create a 2x2 identity matrix
>>> C[C > 0.5] = 0
>>> H = sparse.csr_matrix(C) #Compressed Sparse Row matrix
>>> I=	sparse.csc_matrix(D) #Compressed Sparse Column matrix
>>> J = sparse.dok_matrix(A) #Dictionary Of Keys matrix
>>> E.tadense() #Sparse matrix to full matrix
>>> sparse.isspmatrix_csc(A)


Sparse Matrix Routines

Inverse

>>> sparse.linalg.inv(I) #Inverse


Norm

>>> sparse.linalg.norm(I) #Norm


Solving linear problems

>>> sparse.linalg.spsolve(H,I) #Solver for sparse matrices


Sparse Matrix Functions

>>> sparse.linalg.expm(I) #Sparse matrix exponential


Sparse Matrix Decompositions

>>> la, v = sparse.linalg.eigs(F,1) #Eigenvalues and eigenvectors
>>> sparse.linalg.svds(H, 2) #SVD


## Matrix Function

>>> np.add(A,D) #Addition


Subtraction

>>> np.subtract(A,D) #Subtraction


Division

>>> np.divide(A,D) #Division


Multiplication

>>> np.multiply(D,A) #Multiplication
>>> np.dot(A,D) #Dot product
>>> np.vdot(A,D) #Vector dot product
>>> np.inner(A,D) #Inner product
>>> np.outer(A,D) #Outer product
>>> np.tensardat(A,D) #Tensor dot product
>>> np.kron(A,D) #Kronecker product


Exponential Functions

>>> linalg.expm(A) #Matrix exponential
>>> linalg.expm2(A) #Matrix exponential (Taylor Series)
>>> linalg.expm3(D) #Matrix exponential (eigenvalue decomposition)


Logarithm Function

>>> linalg.lagm(A) #Matrix logarithm


Trigonometric Functions

>>> linalg.sinm(D) Matrix sine
>>> linalg.cosm(D) Matrix cosine
>>> linalg.tanm(A) Matrix tangent


Hyperbolic Trigonometric Functions

>>> linalg.sinhm(D) #Hypberbolic matrix sine
>>> linalg.coshm(D) #Hyperbolic matrix cosine
>>> linalg.tanhm(A) #Hyperbolic matrix tangent


Matrix Sign Function

>>> np.sigm(A) #Matrix sign function


Matrix Square Root

>>> linalg.sqrtm(A) #Matrix square root


Arbitrary Functions

>>> linalg.funm(A, lambda x: X*X) #Evaluate matrix function


Eigenvalues and Eigenvectors

>>> la, v = linalg.eig(A) #Solve ordinary or generalized eigenvalue problem for square matrix
>>> l1, l2 = la #Unpack eigenvalues
>>> v[:,0] #First eigenvector
>>> v[:,1] #Second eigenvector
>>> linalg.eigvals(A) #Unpack eigenvalues


Singular Value Decomposition

>>> U,s,Vh = linalg.svd(B) #Singular Value Decomposition (SVD)
>>> M,N = B.shape
>>>Sig= linalg.diagsvd(s,M,N) #Construct sigma matrix in SVD


LU Decomposition

>>> P,L,U = linalg.lu(C) #LU Decomposition


# Section 9

## Neural Networks with Keras

Keras is a powerful and easy-to-use deep learning library for Theano and TensorFlow that provides a high-level neural networks API to develop and evaluate deep learning models.

A Basic Example

>>> import numpy as np
>>> from keras.models import Sequential
>>> from keras.layers import Dense
>>>data= np.random.random((1000,100))
>>>labels= np.random.randint(2,size=(1000,1))
>>>model= Sequential()
activation='relu', input_dim=100))
>>> model.compile(optimizer='rmsprop' ,
loss='binary_crossentropy',
metrics=['accuracy' ])
>>> model.fit(data,labels,epochs=10,batch_size=32)
>>> predictions= model.predict(data)


### Data

Your data needs to be stored as NumPy arrays or as a list of NumPy arrays. Ideally, you split the data in training and test sets, for which you can also resort to the train_test_split module of sklearn. cross_ validation.

Keras Data Sets

>>> from keras.datasets import boston_housing, mnist, cifar10, imdb
>>> num_classes = 10


Other

>>> from urllib.request import urlopen
>>> data =
>>> X = data[:,0:8]
>>> y = data [:,8]


### Preprocessing

>>> from keras.preprocessing import sequence


One-Hot Encoding

>>> from keras.utils import to_categorical
>>Y_train = to_categorical(y_train,num_classes)
>>> Y_test = to_categorical(y_test,num_classes)
>>> Y_train3 = to_categorical(y_train3,num_classes
>>> Y_test3 = to_categorical(y_test3,num_classes)


### Model Architecture

Sequential Model

>>> from keras.models import Sequential
>>> model= Sequential()
>>> model2 = Sequential()
>>> model3 = Sequential()


### Multilayer Perceptron (MLP)

Binary Classification

>>> from keras.layers import Dense
input _dim=8,
kernel_initializer='uniform',
activation='relu') )


Multi-Class Classification

>>> from keras.layers import Dropout



Regression

>>> model.add(Dense(64,activation='relu',input_dim=train_data.shape[1]))


## Convolutional Neural Network (CNN)

>>> from keras.layers import Activation,Conv2D,MaxPooling20,Flatten



### Recurrent Neural Network (RNN)

>>> from keras.klayers import Embedding,LSTM


### Prediction

>>> model3.predict(x_test4, ba tch_size=32)
>>> model3.predict_classes(x_test4,batch _size=32)


Train and Test Sets

>>> from sklearn.mode l_selection import train_test_split
>>> X_train5,X_test5,y_train5,y_test5 = train_test_split(X, y,
test_size=0.33, random_state=42)


Standardization/Normalization

>>> from sklearn.preprocessing import StandardScaler
>>> scaler= StandardScaler().fit(x_train2)
>>> standa rdized_X = scaler.transform(x _train2)
>>> standardized X test= scaler.transform(x_test2


### Inspect Model

>>> model.output_shape #Model output shape
>>> model.summary() #Model summary representation
>>> model.get_config() #Model configuration
>>> model.get_weights()#List all weight tensors in the model


### Compile Model

MLP: Binary Classification

>>> model.compile(optimizer='adam' ,
loss= 'binary_crossentropy',
metrics=['accuracy'])


MLP: Multi-Class Classification

>>> model.compile(optimizer='rmsprop',
loss='categorical_crossentropy   ,
metrics=[ 'accuracy'])


MLP: Regression

>>> model.compile(optimizer='rmsprop',
loss= 'mse,' metrics=['mae'])


Recurrent Neural Network

model3.compile(loss='binary_crossentropy ',
metrics=['accuracy'])


Model Training

model3.fit(x_train4,
y_train4,
batch_size=32,
epochs=15,
verbose=1,
validation_data=(x_test4,y_test4))


>> score = model3.evaluate(x_test,
y_test,
batch_size=32)


>>> from keras.models import load_model
>>> model3.save( )


Model Fine-tuning

Optimization Parameters

>> from keras.optimizers import RMSprop
>>> opt = RMSprop(lr=0.0001, decay=1e-6)
>>> model2.compile(loss= , 'categorical_crossentropy'
optimizer=opt,
metrics=[ 'accuracy'])


Early Stopping

>>> from keras.callbacks import EarlyStopping
>>> early_stopping_monitor = EarlyStopping(patience=2)
>>> model3.fit(x_train4,
y_train4,
batch_size=32,
epochs=15,
validation_data=(x_test4,y_test4),
callbacks=[early_stopping_monitor])


# PySpark & Spark SQL

What is PySpark?

PySpark is an Apache Spark interface in Python. It is used for collaborating with Spark using APIs written in Python. It also supports Spark’s features like Spark DataFrame, Spark SQL, Spark Streaming, Spark MLlib and Spark Core.

What is PySpark SparkContext?

PySpark SparkContext is an initial entry point of the spark functionality. It also represents Spark Cluster Connection and can be used for creating the Spark RDDs (Resilient Distributed Datasets) and broadcasting the variables on the cluster.

>>> from pyspark.sql import SparkSession
>>> spark = SparkSession \
.builder \
.appName( ) \
.config( , ) \
.getOrCreate()



What is spark-submit?

Spark-submit is a utility to run a pyspark application job by specifying options and configurations.

spark-submit \
--master <master-url> \
--deploy-mode <deploy-mode> \
--conf <key<=<value> \
--driver-memory <value>g \
--executor-memory <value>g \
--executor-cores <number of cores> \
--jars <comma separated dependencies> \
--packages <package name> \
--py-files \
<application> <application args>


where

–master : Cluster Manager (yarn, mesos, Kubernetes, local, local(k))
–deploy-mode: Either cluster or client
–conf: We can provide runtime configurations, shuffle parameters, application configurations using –conf. Ex: –conf spark.sql.shuffle.partitions = 300
–driver-memory : Amount of memory to allocate for a driver (Default: 1024M).
–executor-memory : Amount of memory to use for the executor process.
–executor cores : Number of CPU cores to use for the executor process.

What are RDDs in PySpark?

RDDs expand to Resilient Distributed Datasets. These are the elements that are used for running and operating on multiple nodes to perform parallel processing on a cluster. Since RDDs are suited for parallel processing, they are immutable elements. This means that once we create RDD, we cannot modify it. RDDs are also fault-tolerant which means that whenever failure happens, they can be recovered automatically. Multiple operations can be performed on RDDs to perform a certain task.

• Data Representation: RDD is a distributed collection of data elements without any schema
• Optimization: No in-built optimization engine for RDDs
• Schema: we need to define the schema manually.
• Aggregation Operation: RDD is slower than both Dataframes and Datasets to perform simple operations like grouping the data

Creation of RDD using textFile API

rdd = spark.sparkContext.textFile('practice/test')
rdd.take(5)
for i in rdd.take(5): print(i)


Get the Number of Partitions in the RDD

rdd.getNumPartitions()


Get the Number of elements in each partition

rdd.glom().map(len).collect()


Create RDD using textFile API and a defined number of partitions

rdd = spark.sparkContext.textFile('practice/test',10)


Create a RDD from a Python List

lst = [1,2,3,4,5,6,7]
rdd = spark.sparkContext.parallelize(lst)
for i in rdd.take(5) : print(i)


Create a RDD from a Python List

lst = [1,2,3,4,5,6,7]
rdd = spark.sparkContext.parallelize(lst)
for i in rdd.take(5) : print(i)


Create a RDD from local file

lst = open('/staging/test/sample.txt').read().splitlines()
lst[0:10]
rdd = spark.sparkContext.parallelize(lst)
for i in rdd.take(5) : print(i)


Create RDD from range function

lst1 = range(10)
rdd = spark.sparkContext.parallelize(lst1)
for i in rdd.take(5) : print(i)


Create RDD from a DataFrame

df=spark.createDataFrame(data=(('robert',35),('Mike',45)),schema=('name','age'))
df.printSchema()
df.show()
rdd1= df.rdd
type(rdd1)
for i in rdd1.take(2) : print(i)


What are Dataframes?

It was introduced first in Spark version 1.3 to overcome the limitations of the Spark RDD. Spark Dataframes are the distributed collection of the data points, but here, the data is organized into the named columns

• Data Representation:It is also the distributed collection organized into the named columns
• Optimization: It uses a catalyst optimizer for optimization.
• Schema: It will automatically find out the schema of the dataset.
• Aggregation Operation: It performs aggregation faster than both RDDs and Datasets.

What are Datasets?

Spark Datasets is an extension of Dataframes API with the benefits of both RDDs and the Datasets. It is fast as well as provides a type-safe interface.

• Data Representation:It is an extension of Dataframes with more features like type-safety and object-oriented interface.
• Optimization:It uses a catalyst optimizer for optimization.
• Schema: It will automatically find out the schema of the dataset.
• Aggregation Operation:Dataset is faster than RDDs but a bit slower than Dataframes.

What type of operation has Pyspark?

The operations can be of 2 types, actions and transformation.

What is Transformation in Pyspark?

Transformation: These operations when applied on RDDs result in the creation of a new RDD. Some of the examples of transformation operations are filter, groupBy, map. Let us take an example to demonstrate transformation operation by considering filter() operation:

from pyspark import SparkContext
sc = SparkContext("local", "Transdormation Demo")
words_list = sc.parallelize (
["pyspark",
"interview",
"questions"]
)
filtered_words = words_list.filter(lambda x: 'interview' in x)
filtered = filtered_words.collect()
print(filtered)


The output of the above code would be:

[
"interview"
]


What is Action in Pyspark?

Action: These operations instruct Spark to perform some computations on the RDD and return the result to the driver. It sends data from the Executer to the driver. count(), collect(), take() are some of the examples. Let us consider an example to demonstrate action operation by making use of the count() function.

from pyspark import SparkContext
sc = SparkContext("local", "Action Demo")
words = sc.parallelize (
["pyspark",
"interview",
"questions"]
)
counts = words.count()
print("Count of elements in RDD -> ",  counts)


we count the number of elements in the spark RDDs. The output of this code is Count of elements in RDD -> 3

## Creating DataFrame

From RDDs

>>> from pyspark.sql.types import*


Infer Schema

>> sc = spark.sparkContext
>>> lines = sc.textFile("people.txt" )
>>> parts = lines.map(lambda l: l.split(","))
>>> people = parts.map(lambda p: Row(name=p[0],age=int(p[1])))
>>> peopledf = spark.createDataFrame(people)


Specify Schema

>>> people = parts.map(lambda p: Row(name=p[0],
age=int(p[1].strip())))
>>> schemaString = "name age"
>>> fields = [StructField(field_name, StringType(), True) for
field_name in schemaString.split()]
>>> schema = StructType(fields)
>>> spark.createDataFrame(people, schema).show()


From Spark Data Sources

JSON

>>> df = spark.read.json( "customer.json")
>>> df.show()


>>> df2 = spark.read.load( "people.json" , format= "json")


Parquet files

>>> df3 = spark.read.load("people.parquet" )


TXT files

>>> df4 = spark.read.text( "people.txt")


### Filter

Filter entries of age, only keep those records of which the values are >24

>>> df.filter(df["age"] >24).show()


Duplicate Values

>>> df = df.dropDuplicates()


### Queries

What is PySpark SQL? PySpark SQL is the most popular PySpark module that is used to process structured columnar data. Once a DataFrame is created, we can interact with data using the SQL syntax. Spark SQL is used for bringing native raw SQL queries on Spark by using select, where, group by, join, union etc. For using PySpark SQL, the first step is to create a temporary table on DataFrame by using createOrReplaceTempView()

>>> from pyspark.sql import functions as F


Select

>>> df.select( "firstName").show() #Show all entries in firstNome column
>>> df.select( "firstName","lastName") \
.show()
>>> df.select( "firstName", #Show all entries in firstNome, age and type
"age" ,
explode(''phoneNumber'') \
.alias(''contactlnfo')') \
.select("contactlnfo.type",
"firstName",
"age" ) \
.show()
>>> df.select(df["firstName",df[ "age" ]+ 1)  #Show all entries in firstName and age,
.show()	#add 1 to the entries of age
>>> df.select(df['age'] > 24).show() #Show all entries where age >24



When

>>> df.select( "firstName", #Show firstName and 0 or 1 depending on age >30
F.when(df.age > 30, 1) \
.otherwise(0)) \
.show()
>>> df[ df.firstName.isin( "Jane" ,"Boris") ] #Show firstName if in the given options
.collect()


Like

#Show firstName, and lastName is TRUE if lastName is like Smith
>>> df.select( "firstName",
df.lastName .like(''Smith')') \
.show()


Startswith - Endswith

>>> df.select( "firstName", #Show firstName, and TRUE if lastName starts with Sm
df.lastName \
.startswith("Sm")) \
.show()
>>> df.select(df.lastName.endswith("th"))\ #Show last names ending in th
.show()



Substring

>>> df.select(df.firstName.substr(l, 3) \ #Return substrings of firstName
.alias(''name')') \
.collect()



Between

>>> df.select(df.age.between(22, 2s)) \ #Show age: values are TRUE if between 22 and 24


### Add, Update & Remove Columns

>>> df = df.withColumn( 'city',df.address.city) \
.withColumn( 'telePhoneNumber ',explode(df.phoneNumber.number)) \
.withColumn( 'telePhone Type',explode(df.phoneNumber.type))



Updating Columns

>>> df=df.withColumnRenamed('telePhoneNumber ','phoneNumber' )


Removing Columns

>>> df = df.drop ("address","phoneNumber")


### Missing & Replacing Values

>>> df.na.fill(50).show() #Replace null values
>>> df.na.drop().shaw() #Return new df omitting rows with null values
>>> df.na \ #Return new df replacing one value with another
.replace(10, 20) \
.show()


### GroupBy

>>> df.groupBy("age")\ #Group by age, count the members in the groups
.count() \
.show()


### Sort

>>> peopledf.sort(peopledf.age.desc()).collect()
>>> df.sort("age" , ascending=False).collect()
>>> df.orderBy([ "age", "city" ],ascending=[0,1])\
.collect()


### Repartitioning

>>> df.repartitian(10)\ #df with 10 partitions
.rdd \
.getNumPartitions()
>>> df.coalesce(1).rdd.getNumPartitions() #df with 1 partition


### Running Queries Programmatically

>>> peopledf.createGlobalTempView( "people")
>>> df.createTempView ("customer")
>>> df.createOrReplaceTempView( "customer")


Query Views

>>> df5 = spark.sql("SELECT * FROM customer").show()
>>> peopledf2=spark.sql( "SELEC T* FROM global_ temp.people")\
.show()


### Inspect Data

>>> df.dtypes #Return df column names and data types
>>> df.show() #Display the content of df
>>> df.head() #Return first n raws
>>> df.first() #Return first row
>>> df.take(2) #Return the first n rows
>>> df.schema Return the schema of df
>>> df.describe().show() #Compute summary statistics
>>> df.columns Return the columns of df
>>> df.count() #Count the number of rows in df
>>> df.distinct().count() #Count the number of distinct rows in df
>>> df.printSchema() #Print the schema of df
>>> df.explain() #Print the (logical and physical) plans


### Output

Data Structures

>>> rddl = df.rdd #Convert df into an ROD
>>> df.taJSON().first() #Convert df into a ROD of string
>>> df.toPandas() #Return the contents of df as Pandas DataFrame


Write & Save to Files

>>> df.select( "firstName", "city")\
.write \
.save("nameAndCity.parquet" )
>>> df.select("firstName", "age") \
.write \
.save( "namesAndAges.json",format="json")


### Stopping SparkSession

>> spark.stop()


## PySpark RDD

PySpark is the Spark Python API that exposes the Spark programming model to Python.

Inspect SparkContext

>>> sc.version #Retrieve SparkContext version
>>> sc.pythonVer #Retrieve Python version
>>> sc.master #Master URL to connect to
>>> str(sc.sparkHome) #Path where Spark is installed an worker nodes
>>> str(sc.sparkUser()) #Retrieve name of the Spark User running SparkContext
>>> sc.appName #Return application name
>>> sc.applicationld #Retrieve application ID
>>> sc.defaultParallelism #Return default level of parallelism
>>> sc.defaultMinPartitions #Default minimum number of partitions for RDDs



Configuration

>>> from pyspark import SparkConf, SparkContext
>>> conf = (SparkConf()
.setMaster("local")
.setAppName("My app")
.set("spark.executor.memory","1g" ) )
>>> sc = SparkContext(conf = conf)


Using The Shell

In the PySpark shell, a special interpreter aware SparkContext is already created in the variable called sc.

$./bin/spark shell --master local[2]$ ./bin/pyspark --master local[4] --py files code.py


Set which master the context connects to with the –master argument, and add Python .zip, .egg or .py files to the runtime path by passing a comma separated list to –py-files

Parallelized Collections

>>> rdd = sc.parallelize([('a',7),('a',2),('b',2)])
>>> rdd = sc.parallelize([('a',2),('d',1),('b',1)])
>>> rdd3 = sc.parallelize(range(100))
>>> rdd4 = sc.parallelize([("a",["x","y","z"]),
("b",["p","r"])]


External Data

Read either one text file from HDFS.a local file system or or any Hadoop-supported file system URI with textFile(). or read in a directory of text files with wholeTextFiles()

>>> textFile = sc.textFile("/my/directory/*.txt")
>>> textFile2 = sc.wholeTextFiles( "/my/directory/")


### Retrieving RDD Information

Basic Information

>>> rdd.getNumPartitions() #List the number of partitions
>>> rdd.count() #Count ROD instances 3
>>> rdd.countByKey() #Count ROD instances by key
defaultdict(<type 'int'>,{'a':2,'b' :1})
>>> rdd.countByValue() #Count ROD instances by value
defaultdict(<type 'int'>,{('b',2):1,'(a',2):1,('a',7):1})
>>> rdd.collectAsMap() #Return (key,value) pairs as a dictionary
{'a':2,1b':2}
>>> rdd3.sum() #Sum of ROD elements 4950
>>> sc.parallelize([]).isEmpty() #Check whether ROD is empty
True



Summary

>>> rdd3.max() #Maximum value of ROD elements 99
>>> rdd3.min() #Minimum value of ROD elements
0
>>> rdd3.mean() #Mean value of ROD elements
,9.5
>>> rdd3.stdev() #Standard deviation of ROD elements 2a.8660700s772211a
>>> rdd3.variance() #Compute variance of ROD elements 833.25
>>> rdd3.histogram(3) #Compute histogram by bins
([0,33,66,991,[33,33,3,])
>>> rdd3.stats() #Summary statistics (count, mean, stdev, max & min)



### Applying Functions

#Apply a function to each ROD element
>>> rdd.map(lambda x: x+(x[l],x[0])).callect()
[('a',7,7,'a'),('a',2,2,'a'),('b',2,2,'b')]
#Apply a function to each ROD element and flatten the result
>>> rdd5 = rdd.flatMap(lambda x: x+(x[l],x[0]))
>>> rdd5.collect()
['a',7,7'a','a',2,2'a','b',2,2'b']
#Apply a flatMap function to each (key,value) pair of rdd4 without changing the keys
>>> rdds.flatMapValues(lambda x: x).callect()
[('a','x'),('a','y'),('a','z'),('b','p'),('b','r')]



### Selecting Data

Getting

>>> rdd.collect() #Return a list with all ROD elements
[('a',7),('a',2),('b',2)]
>>> rdd.take(2) #Take first 2 ROD elements
[('a',7),('a',2)]
>>> rdd.first() #Toke first ROD element
[('a',7),('a',2)]
>>> rdd.top(2) #Take top 2 ROD elements
[('b',2),('a',7)]


Sampling

>>> rdd3.sample(False, 0.15, 81).collect() #Return sampled subset of rdd3
[3,4,27,31,40,41,42,43,60,76,79,80,86,97]


Filtering

>>> rdd.filter(lambda x: "a" in x).collect() #Filter the ROD
[( 'a',7),('a',2)]
>>> rdd5.distinct().callect() #Return distinct ROD values
['a',2,'b',7]
>>> rdd.keys().collect() #Return (key,value) RDD's keys
['a','a','b']


>>> def g(x): print(x)
>>> rdd.foreach(g) #Apply a function to all ROD elements
('a',7)
('b',2)
('a',2)



### Reshaping Data

Reducing

>>> rdd.reduceByKey(lambda x,y : x+y).callect() #Merge the rdd values for each key
[('a',9),('b',2)]
>>> rdd.reduce(lambda a, b: a+	b) #Merge the rdd values
('a',7,'a',2,'b',2)



Grouping by

>>> rdd3.groupBy(lambda x: x % 2)
.mapValues(list)
.collect()
>>> rdd.groupByKey()
.mapValues(list)
.collect()
[('a',[7,2]),('b',[2])]


Aggregating

>>> seqOp = (lambda x,y: (x[0]+y,x[1]+1))
>>> combOp = (lambda x,y:(x[0]+y[0],x[1]+y[1]))
#Aggregate RDD elements of each partition and then the results
>>> rdd3.aggregate((0,0),seqOp,combOp)
(4950,100)
#Aggregate values of each RDD key
>>> rdd.aggregateByKey((0,0),seqop,combop).collect()
[('a',(9,2)),('b',(2,1))]
#Aggregate the elements of each partition, and then the results
4950
#Merge the values for each key
[('a',9),('b',2)]
#Create tuples of RDD elements by applying a function
>>> rdd3.keyBy(lambda x: x+x).collect()


### Mathematical Operations

>>> rdd.subtract(rdd2).collect() #Return each rdd value not contained in rdd2
[('b',2),('a',7)]
#Return each (key,value) pair of rdd2 with no matching key in rdd
>>> rdd2.subtractByKey(rdd).collect()
[('d',1)l
>>> rdd.cartesian(rdd2).callect() #Return the Cartesian product of rdd and rdd2


Sort

>>> rdd2.sortBy(lambda x: x[l]).collect() #Sort ROD by given function
[('d',1),('b',1),('a',2)]
>>> rdd2.sartByKey().collect() #Sort (key, value) ROD by key
[('a',2),('b',1),('d',1)]


Repartitioning

>>> rdd.repartitian(4) #New ROD with 4 partitions
>>> rdd.caalesce(1) #Decrease the number of partitions in the ROD to 1



Saving

>>> rdd .saveA sTextFile("rdd.txt")



Execution

\$ ./bin/spark submit examples/src/main/python/pi.py


Does PySpark provide a machine learning API?

Similar to Spark, PySpark provides a machine learning API which is known as MLlib that supports various ML algorithms like:

• mllib.classification − This supports different methods for binary or multiclass classification and regression analysis like Random Forest, Decision Tree, Naive Bayes etc.
• mllib.clustering − This is used for solving clustering problems that aim in grouping entities subsets with one another depending on similarity.
• mllib.fpm − FPM stands for Frequent Pattern Matching. This library is used to mine frequent items, subsequences or other structures that are used for analyzing large datasets.
• mllib.linalg − This is used for solving problems on linear algebra.
• mllib.recommendation − This is used for collaborative filtering and in recommender systems.
• spark.mllib − This is used for supporting model-based collaborative filtering where small latent factors are identified using the Alternating Least Squares (ALS) algorithm which is used for predicting missing entries.
• mllib.regression − This is used for solving problems using regression algorithms that find relationships and variable dependencies.
• Is PySpark faster than pandas?

PySpark supports parallel execution of statements in a distributed environment, i.e on different cores and different machines which are not present in Pandas. This is why PySpark is faster than pandas.

Broadcast variables: These are also known as read-only shared variables and are used in cases of data lookup requirements. These variables are cached and are made available on all the cluster nodes so that the tasks can make use of them. The variables are not sent with every task. They are rather distributed to the nodes using efficient algorithms for reducing the cost of communication. When we run an RDD job operation that makes use of Broadcast variables, the following things are done by PySpark:

The job is broken into different stages having distributed shuffling. The actions are executed in those stages. The stages are then broken into tasks. The broadcast variables are broadcasted to the tasks if the tasks need to use it. Broadcast variables are created in PySpark by making use of the broadcast(variable) method from the SparkContext class. The syntax for this goes as follows:

broadcastVar = sc.broadcast([10, 11, 22, 31])


An important point of using broadcast variables is that the variables are not sent to the tasks when the broadcast function is called. They will be sent when the variables are first required by the executors.

What is Accumulator variable?

Accumulator variables: These variables are called updatable shared variables. They are added through associative and commutative operations and are used for performing counter or sum operations. PySpark supports the creation of numeric type accumulators by default. It also has the ability to add custom accumulator types. The custom types can be of two types:

What is PySpark Architecture?

PySpark similar to Apache Spark works in master-slave architecture pattern. Here, the master node is called the Driver and the slave nodes are called the workers. When a Spark application is run, the Spark Driver creates SparkContext which acts as an entry point to the spark application. All the operations are executed on the worker nodes. The resources required for executing the operations on the worker nodes are managed by the Cluster Managers

What is the common workflow of a spark program?

The most common workflow followed by the spark program is: The first step is to create input RDDs depending on the external data. Data can be obtained from different data sources. Post RDD creation, the RDD transformation operations like filter() or map() are run for creating new RDDs depending on the business logic. If any intermediate RDDs are required to be reused for later purposes, we can persist those RDDs. Lastly, if any action operations like first(), count() etc are present then spark launches it to initiate parallel computation.

Congratulations! You have read an small summary about important things in Data Science.

Posted: