# Neural Networks with XGBoost - A simple classification

Artificial neural networks, usually simply called neural networks, are computing systems inspired by the biological neural networks that constitute animal brains.

An ANN is based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain.

XGBoost is an open-source software library that implements optimized distributed gradient boosting machine learning algorithms under the Gradient Boosting framework.

XGBoost, which stands for Extreme Gradient Boosting, is a scalable, distributed gradient-boosted decision tree (GBDT) machine learning library. It provides parallel tree boosting and is the leading machine learning library for regression, classification, and ranking problems.

It’s vital to an understanding of XGBoost to first grasp the machine learning concepts and algorithms that XGBoost builds upon: supervised machine learning, decision trees, ensemble learning, and gradient boosting.

Supervised machine learning uses algorithms to train a model to find patterns in a dataset with labels and features and then uses the trained model to predict the labels on a new dataset’s features.

## Introduction

We will build a neural network model using Keras. So the first model that we’re going to use here to train our classifier to detect diabetes. We are going to use artificial neural networks and artificial neural networks. These are information processing models that essentially mimic the human brain and try to map the inputs to the outputs.

This dataset is used to predict whether or not a patient has diabetes, based on given features/diagnostic measurements.

Only female patients are considered with at least 21 years old of Pima Indian heritage.

Inputs:

• Pregnancies: Number of times pregnant
• GlucosePlasma: glucose concentration a 2 hours in an oral glucose tolerance test
• BloodPressure: Diastolic blood pressure (mm Hg)
• Skin: ThicknessTriceps skin fold thickness (mm)
• Insulin: 2-Hour serum insulin (mu Wm!)
• BMI: Body mass index (weight in kg/(height in m)^2)
• DiabetesPedigreeFunction: Diabetes pedigree function
• Age: Age (years)
• Outputs: Diabetes or no diabetes (0 or 1)

Basically I’m going to build think of it as a mini artificial brain that can consume or use all these different features that we have in the input in X train and it’s going to learn all the different features within that data and map it to the output, which is going to be my white train.

Let us first load our libraries and analyze the input data.

import tensorflow as tf
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

# You have to include the full link to the csv file containing your dataset

diabetes

Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigreeFunction Age Outcome
0 6 148 72 35 0 33.6 0.627 50 1
1 1 85 66 29 0 26.6 0.351 31 0
2 8 183 64 0 0 23.3 0.672 32 1
3 1 89 66 23 94 28.1 0.167 21 0
4 0 137 40 35 168 43.1 2.288 33 1
... ... ... ... ... ... ... ... ... ...
763 10 101 76 48 180 32.9 0.171 63 0
764 2 122 70 27 0 36.8 0.340 27 0
765 5 121 72 23 112 26.2 0.245 30 0
766 1 126 60 0 0 30.1 0.349 47 1
767 1 93 70 31 0 30.4 0.315 23 0

768 rows × 9 columns

diabetes.head()

Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigreeFunction Age Outcome
0 6 148 72 35 0 33.6 0.627 50 1
1 1 85 66 29 0 26.6 0.351 31 0
2 8 183 64 0 0 23.3 0.672 32 1
3 1 89 66 23 94 28.1 0.167 21 0
4 0 137 40 35 168 43.1 2.288 33 1
diabetes.tail()

Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigreeFunction Age Outcome
763 10 101 76 48 180 32.9 0.171 63 0
764 2 122 70 27 0 36.8 0.340 27 0
765 5 121 72 23 112 26.2 0.245 30 0
766 1 126 60 0 0 30.1 0.349 47 1
767 1 93 70 31 0 30.4 0.315 23 0
diabetes.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 768 entries, 0 to 767
Data columns (total 9 columns):
#   Column                    Non-Null Count  Dtype
---  ------                    --------------  -----
0   Pregnancies               768 non-null    int64
1   Glucose                   768 non-null    int64
2   BloodPressure             768 non-null    int64
3   SkinThickness             768 non-null    int64
4   Insulin                   768 non-null    int64
5   BMI                       768 non-null    float64
6   DiabetesPedigreeFunction  768 non-null    float64
7   Age                       768 non-null    int64
8   Outcome                   768 non-null    int64
dtypes: float64(2), int64(7)
memory usage: 54.1 KB

diabetes.describe()

Pregnancies Glucose BloodPressure SkinThickness Insulin BMI DiabetesPedigreeFunction Age Outcome
count 768.000000 768.000000 768.000000 768.000000 768.000000 768.000000 768.000000 768.000000 768.000000
mean 3.845052 120.894531 69.105469 20.536458 79.799479 31.992578 0.471876 33.240885 0.348958
std 3.369578 31.972618 19.355807 15.952218 115.244002 7.884160 0.331329 11.760232 0.476951
min 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.078000 21.000000 0.000000
25% 1.000000 99.000000 62.000000 0.000000 0.000000 27.300000 0.243750 24.000000 0.000000
50% 3.000000 117.000000 72.000000 23.000000 30.500000 32.000000 0.372500 29.000000 0.000000
75% 6.000000 140.250000 80.000000 32.000000 127.250000 36.600000 0.626250 41.000000 1.000000
max 17.000000 199.000000 122.000000 99.000000 846.000000 67.100000 2.420000 81.000000 1.000000

# PERFORM DATA VISUALIZATION

plt.figure(figsize = (12, 7))
sns.countplot(x = 'Outcome', data = diabetes)

<AxesSubplot:xlabel='Outcome', ylabel='count'>


sns.pairplot(diabetes, hue = 'Outcome', vars = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age'])

<seaborn.axisgrid.PairGrid at 0x1abf1ceddc0>


# SPLIT THE DATA AND PREPARE IT FOR TRAINING

X = diabetes.iloc[:, 0:8].values

X

array([[  6.   , 148.   ,  72.   , ...,  33.6  ,   0.627,  50.   ],
[  1.   ,  85.   ,  66.   , ...,  26.6  ,   0.351,  31.   ],
[  8.   , 183.   ,  64.   , ...,  23.3  ,   0.672,  32.   ],
...,
[  5.   , 121.   ,  72.   , ...,  26.2  ,   0.245,  30.   ],
[  1.   , 126.   ,  60.   , ...,  30.1  ,   0.349,  47.   ],
[  1.   ,  93.   ,  70.   , ...,  30.4  ,   0.315,  23.   ]])

y = diabetes.iloc[:, 8].values

# Feature Scaling is a must in ANN
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X = sc.fit_transform(X)

X

array([[ 0.63994726,  0.84832379,  0.14964075, ...,  0.20401277,
0.46849198,  1.4259954 ],
[-0.84488505, -1.12339636, -0.16054575, ..., -0.68442195,
-0.36506078, -0.19067191],
[ 1.23388019,  1.94372388, -0.26394125, ..., -1.10325546,
0.60439732, -0.10558415],
...,
[ 0.3429808 ,  0.00330087,  0.14964075, ..., -0.73518964,
-0.68519336, -0.27575966],
[-0.84488505,  0.1597866 , -0.47073225, ..., -0.24020459,
-0.37110101,  1.17073215],
[-0.84488505, -0.8730192 ,  0.04624525, ..., -0.20212881,
-0.47378505, -0.87137393]])

# Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)

X_train.shape

(614, 8)

X_test.shape

(154, 8)


We have been able to change the split between training and testing and we allocated 20% for testing and the remaining which is 80% have been allocated for training. And before that we also have been able to prepare our data.

# BUILD A NEURAL NETWORK MODEL USING KERAS

Tensorflow as google’s framework to build train test and deploy AI and ml models at scale and keras is simply a high level api that could be used to kind of make our life extremely easy.

So instead of building all the math and writing all the equations for example, that describe artificial neural networks, I’m just gonna use Keras which is high level API.

Let use sequential mode to build our artificial neural network model object or instance.

To this object we add a layers dense with 400 neurons. Due to the the non linearity of the network. We specify the activation function to be a ReLu activation function and ReLu stands for rectified linear units.

We need to specify the shape here and the shape is eight because we have eight inputs in the model. Next we add what we call it, a dropout layer after you add this layer here which contains 400 neurons.

Finally we add another layer with dropout, the dropout make sure that the network is able to generalize and not memorize , it is a regularization technique.

We drop a couple of neurons along with their connected weight, we make sure that the neurons do not develop any dependency.

When we add a dense layer is commonly followed by dropout. We add another dense layer followed by another dropout. The next dense layer will to contain 400 neurons with again, activation function ReLu.

And then add the dropout afterwards. And then finally the output include an activation function of sigmoid and the units here is only one neuron. And the reason is is because the output is going to simply going to be a binary values either zero or one.

# !pip install tensorflow
import tensorflow as tf
ANN_model = tf.keras.models.Sequential()

ANN_model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense (Dense)                (None, 400)               3600
_________________________________________________________________
dropout (Dropout)            (None, 400)               0
_________________________________________________________________
dense_1 (Dense)              (None, 400)               160400
_________________________________________________________________
dropout_1 (Dropout)          (None, 400)               0
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 401
=================================================================
Total params: 164,401
Trainable params: 164,401
Non-trainable params: 0
_________________________________________________________________


To summarize the network have 164 trainable parameters. It is a optimization problem where we are trying to connect all these neurons have weights or connections between all these neurons

There are 164 weights or connections than trying to optimize. Which we try to find the best values that could map the inputs to the outputs.

# COMPILE AND TRAIN THE ANN MODEL

ANN_model.compile(optimizer='Adam', loss='binary_crossentropy', metrics = ['accuracy'])

epochs_hist = ANN_model.fit(X_train, y_train, epochs = 200)

Epoch 1/200
20/20 [==============================] - 2s 6ms/step - loss: 0.5430 - accuracy: 0.7427
Epoch 2/200
.
.
.
Epoch 200/200
20/20 [==============================] - 0s 5ms/step - loss: 0.0335 - accuracy: 0.9951

y_pred = ANN_model.predict(X_test)

#y_pred

y_pred = (y_pred > 0.5)

#y_pred


# EVALUATE TRAINED MODEL PERFORMANCE

epochs_hist.history.keys()

dict_keys(['loss', 'accuracy'])

plt.plot(epochs_hist.history['loss'])
plt.title('Model Loss Progress During Training')
plt.xlabel('Epoch')
plt.ylabel('Training and Validation Loss')
plt.legend(['Training Loss'])

<matplotlib.legend.Legend at 0x1abf102b700>


# Training set Performance
from sklearn.metrics import confusion_matrix
# Testing Set Performance
cm = confusion_matrix(y_test, y_pred)
sns.heatmap(cm, annot=True)


<AxesSubplot:>


from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred))

              precision    recall  f1-score   support

0       0.75      0.80      0.77        98
1       0.60      0.54      0.57        56

accuracy                           0.70       154
macro avg       0.68      0.67      0.67       154
weighted avg       0.70      0.70      0.70       154


# TRAIN AND EVALUATE AN XG-BOOST ALGORITHM

#!pip install xgboost

# Train an XGBoost classifier model
import xgboost as xgb
XGB_model = xgb.XGBClassifier(learning_rate = 0.1, max_depth = 5, n_estimators = 10)
XGB_model.fit(X_train, y_train)

result_train = XGB_model.score(X_train, y_train)
print("Accuracy : {}".format(result_train))

Accuracy : 0.8729641693811075

# predict the score of the trained model using the testing dataset

result_test = XGB_model.score(X_test, y_test)
print("Accuracy : {}".format(result_test))

Accuracy : 0.7662337662337663

# make predictions on the test data
y_predict = XGB_model.predict(X_test)


from sklearn.metrics import confusion_matrix, classification_report
print(classification_report(y_test, y_predict))

              precision    recall  f1-score   support

0       0.79      0.87      0.83        98
1       0.72      0.59      0.65        56

accuracy                           0.77       154
macro avg       0.75      0.73      0.74       154
weighted avg       0.76      0.77      0.76       154

cm = confusion_matrix(y_predict, y_test)
sns.heatmap(cm, annot = True)
plt.ylabel('Predicted class')
plt.xlabel('Actual class')

Text(0.5, 34.0, 'Actual class')