Feature Selection In Machine Learning: All You Need to Know

Feature Selection Models

Feature selection models are of two types:

1. Supervised Models: Supervised feature selection refers to the method which uses the output label class for feature selection. They use the target variables to identify the variables which can increase the efficiency of the model
2. Unsupervised Models: Unsupervised feature selection refers to the method which does not need the output label class for feature selection. We use them for unlabelled data.

Figure 4: Feature Selection Models

We can further divide the supervised models into three :

1. Filter Method: In this method, features are dropped based on their relation to the output, or how they are correlating to the output. We use correlation to check if the features are positively or negatively correlated to the output labels and drop features accordingly. Eg: Information Gain, Chi-Square Test, Fisher’s Score, etc. 

Figure 5: Filter Method flowchart                         

2. Wrapper Method: We split our data into subsets and train a model using this. Based on the output of the model, we add and subtract features and train the model again. It forms the subsets using a greedy approach and evaluates the accuracy of all the possible combinations of features. Eg: Forward Selection, Backwards Elimination, etc.

                                              Figure 6: Wrapper Method Flowchart

Your AI/ML Career is Just Around The Corner!

AI Engineer Master's ProgramExplore Program

3. Intrinsic Method: This method combines the qualities of both the Filter and Wrapper method to create the best subset.

                                                Figure 7: Intrinsic Model Flowchart

This method takes care of the machine training iterative process while maintaining the computation cost to be minimum. Eg: Lasso and Ridge Regression.

How to Choose a Feature Selection Model?

How do we know which feature selection model will work out for our model? The process is relatively simple, with the model depending on the types of input and output variables.

Variables are of two main types:

• Numerical Variables: Which include integers, float, and numbers.
• Categorical Variables: Which include labels, strings, boolean variables, etc.

Based on whether we have numerical or categorical variables as inputs and outputs, we can choose our feature selection model as follows:

                                            Table 1: Feature Selection Model lookup

Your AI/ML Career is Just Around The Corner!

AI Engineer Master's ProgramExplore Program

Feature Selection With Python

Let’s get hands-on experience in feature selection by working on the Kobe Bryant Dataset which analyses shots taken by Kobe from different areas of the court to determine which ones will go into the basket. 

The dataset is as shown:

                                                        Figure 8: Kobe Bryant Dataset

As we can see, the dataset has 25 different columns. We will not need all of them. 

We first begin by loading in the necessary modules. 

                                                        Figure 9: Importing modules

First, let's check out the loc_x and loc_y columns. They probably represent longitude and latitude. 

                                     Figure 10: Plotting the latitude and longitude columns in our dataset

The figure is as shown:

                                                 Figure 11: Plotting Latitude and Longitude  

From the above figures, we can see that they resemble the two ‘D’s on a basketball court. Instead of having two separate columns, we can change the coordinates into polar form and have a single column [‘angle’].

                                          Figure 12: Changing Latitude and Longitude into polar form 

Your AI/ML Career is Just Around The Corner!

AI Engineer Master's ProgramExplore Program

We can combine the minutes and seconds columns into a single column for time. 

                                           Figure 13: Combining two columns 

Let’s look at the unique values in the ‘team_id’ and ‘team_name’ columns:

                                          Figure 14: Unique values in ‘team_id’ and ‘team_name’

The entire column contains only one value and can be dropped. Let’s take a look at the ‘match_up’ and ‘opponent’ columns : 

                                          Figure 15: ‘match_up’ and ‘opponent’ columns

Again, they contain the same information. Let’s plot the values of ‘dist’ and ‘shot_distance’ columns on the same graph to see how they differ:

                                                   Figure 16: Plotting ‘dist’ and ‘shot_distance’ columns

Again, they contain exactly the same information. Let’s take a look at columns shot_zone_area, shot_zone_basic and shot_zone_range.

                                                Figure 17: Plotting the different shot zones columns

The figure depicted below shows the plots :

                                                        Figure 18: Different shot zones

We can see that they contain the different parts of the court from where the shots were taken. This information already exists in the angle and dist columns. 

Now, let’s drop all the useless columns.

                                                        Figure 19: Dropping Columns

After merging columns and removing useless columns, we get a dataset that contains only 11 important columns.

                                                             Figure 20: Final Dataset

Learn the essentials of object-oriented programming, web development with Django, and more with the Caltech Post Graduate Program In AI And Machine Learning. Enroll now!

Your AI/ML Career is Just Around The Corner!

AI Engineer Master's ProgramExplore Program

Conclusion

In this article titled ‘Everything you need to know about Feature Selection’, we got an idea of how important it is to select the best features for our machine learning model. We then took a look at what feature selection is and some feature selection models. We then moved onto a simple way to choose the right feature selection model based on the input and output values. Finally, we saw how to implement feature selection in Python with a demo. If you are looking to learn more about feature selection and related fundamental features of Python, Simplielarn’s Python Certification Course would be ideal for you. This python certification course covers the basics fundamentals of python including data operations, conditional statements, shell scripting, and Django and much more, and prepares you for a rewarding career as a professional Python programmer.

Was this article on feature selection useful to you? Do you have any doubts or questions for us? Mention them in this article's comments section, and we'll have our experts answer them for you at the earliest!