Python Remove Duplicates From a List

What Is a Python list?

The list is the most important data type in the Python programming language. The list is written in Python as a list of comma-separated values enclosed by square brackets. The most significant advantage of the list is that the elements within it do not have to be of the same data type and can be indexed negatively. Furthermore, all string operations, such as slicing and concatenation, are also applicable to list data types. We can also make a nested list, which is a list that contains another list.

How to Remove Duplicates From a Python list?

In Python, there are numerous methods for removing duplicates from a list. To remove duplicates from a given list, you can make use of these methods and get your work done. Let's take a look at them: 

Method 1 - Naive Method 

To remove duplicates from a list in Python, iterate through the elements of the list and store the first occurrence of an element in a temporary list while ignoring any other occurrences of that element.

The basic approach is implemented in the naive method by:

• Using a For-loop to traverse the list
• If the elements do not already exist in a temporary list, they are added to it
• The temporary list has been assigned to the main list

Example: 

# removing duplicates from the list using naive methods 

# initializing list 

sam_list = [11, 13, 15, 16, 13, 15, 16, 11] 

print ("The list is: " + str(sam_list)) 

# remove duplicates from list

result = [] 

for i in sam_list: 

if i not in result: 

result.append(i) 

# printing list after removal 

print ("The list after removing duplicates : " + str(result))

Output: 

 The list is: [11, 13, 15, 16, 13, 15, 16, 11]

 The list after removing duplicates: [11, 13, 15, 16]

Method 2 - Using List Comprehension

Instead of using the For-loop to implement the Naive method of removing duplicates from a list, we can use Python's List comprehension functionality to do so in just one line of code.

Example:

# removing duplicates from the list using list comprehension 

# initializing list 

sam_list = [11, 13, 15, 16, 13, 15, 16, 11] 

print ("The list is: " + str(sam_list)) 

# to remove duplicates from list 

result = [] 

[result.append(x) for x in sam_list if x not in result] 

# printing list after removal 

print ("The list after removing duplicates: " + str(result))

Output:

The list is: [11, 13, 15, 16, 13, 15, 16, 11]

The list after removing duplicates: [11, 13, 15, 16]

Prepare Yourself to Answer All Questions!

Automation Testing Masters ProgramExplore Program

Method 3 - Using set()

This is the most commonly used method for removing a duplicate from a Python list. This is due to the fact that duplicates are not permitted in the set data structure. However, the order of the elements is lost when using this method.

Example: 

# removing duplicates from the list using set() 

# initializing list 

sam_list = [11, 15, 13, 16, 13, 15, 16, 11] 

print ("The list is: " + str(sam_list)) 

# to remove duplicates from list 

sam_list = list(set(sam_list)) 

# printing list after removal 

# ordering distorted

print ("The list after removing duplicates: " + str(sam_list))

Output:

The list is: [11, 15, 13, 16, 13, 15, 16, 11]

The list after removing duplicates: [16, 11, 13, 15]

Method 4 - Using list comprehension + enumerate ()

We find the distinct elements and store them in a temporary list using the List comprehension method. When we use the List comprehension in conjunction with the enumerate() function, the programme checks for previously encountered elements and skips adding them to the temporary list. The enumerate function takes an iterable as an argument and returns it as an enumerating object (index, element), which adds a counter to each iterable element.

Example:

# removing duplicates from the list using list comprehension + enumerate() 

# initializing list 

sam_list = [11, 15, 13, 16, 13, 15, 16, 11] 

print ("The list is: " + str(sam_list)) 

# to remove duplicates from list 

result = [i for n, i in enumerate(sam_list) if i not in sam_list[:n]] 

# printing list after removal 

print ("The list after removing duplicates: " + str(result))

Output:

The list is: [11, 13, 15, 16, 13, 15, 16, 11]

The list after removing duplicates: [11, 13, 15, 16]

Method 5 - Using collections.OrderedDict.fromkeys()

This is the quickest way to accomplish the goal of removing duplicates from the Python list. This method first removes duplicates before returning a dictionary that has been converted to a list. In addition, this method works well with strings.

Example:

# removing duplicates from list using collections.OrderedDict.fromkeys() 

from collections import OrderedDict 

# initializing list 

sam_list = [11, 15, 13, 16, 13, 15, 16, 11] 

print ("The list is: " + str(sam_list)) 

# to remove duplicates from list 

result = list(OrderedDict.fromkeys(sam_list)) 

# printing list after removal 

print ("The list after removing duplicates: " + str(result))

Output:

The list is: [11, 15, 13, 16, 13, 15, 16, 11]

The list after removing duplicates: [11, 15, 13, 16]

Conclusion

Sets, built-in functions, and iterative methods can be used to remove duplicates from a list. If the list's elements are non-hashable, always take an iterative approach, traversing the list and extracting unique items. The Naive Method, List comprehensions, and List.count() methods are examples of iterative approaches. If the order of the elements is not critical, we can remove duplicates using the Set method and the Numpy unique() function. We can use Pandas functions, OrderedDict, reduce() function, Set + sort() method, and iterative approaches to keep the order of elements.

To learn and make a successful career as a Python programmer, enroll in Simplilearn’s PGP Full Stack Development Course. With this course you will also master data operations, conditional statements, shell scripting, Django and so much more.