How To Remove Duplicates From An Array In Python?

In this tutorial, I will explain how to remove duplicate elements from an array in Python. As a developer, you may often encounter situations where you need to eliminate repeated values from a list or array. We will explore more about this topic with examples and screenshots of executed example codes.

Table of Contents

Methods to Remove Duplicates from an Array in Python

Imagine you are working on a project for a client in New York City. You have a Python array of customer names, but you found that some names appear multiple times. and you need to remove these duplicate names from the array. Python provides several methods to achieve this task.

Let’s say you have the following array:

customer_names = ["John Smith", "Emily Johnson", "Michael Davis", "John Smith", "Emma Brown", "Emily Johnson"]

Your goal is to create a new array that contains unique names, like this:

unique_customer_names = ["John Smith", "Emily Johnson", "Michael Davis", "Emma Brown"]

Check out How to Sort an Array in Python

Method 1: Using a Set

One of the simplest and most efficient ways to remove duplicates from an array is by converting it to a set. Python set is an unordered collection of unique elements. By converting an array to a set, duplicates are automatically eliminated. Here’s how you can do it:

customer_names = ["John Smith", "Emily Johnson", "Michael Davis", "John Smith", "Emma Brown", "Emily Johnson"]
unique_customer_names = list(set(customer_names))
print(unique_customer_names)

Output:

['John Smith', 'Emma Brown', 'Emily Johnson', 'Michael Davis']

I have executed the above example code, you can refer to the screenshot below.

Remove Duplicates from an Array in Python

As you can see, the duplicates “John Smith” and “Emily Johnson” are removed, and the resulting array unique_customer_names contains unique names. However, converting an array to a set may change the order of the elements since sets are unordered.

Read How to Use Python Array Index -1

Method 2: Using a For Loop

If you need to maintain the original order of the array while removing duplicates, you can use a for loop along with a temporary list. This approach iterates through each element in the array and adds it to the temporary list only if it hasn’t been encountered before. Here’s an example:

customer_names = ["John Smith", "Emily Johnson", "Michael Davis", "John Smith", "Emma Brown", "Emily Johnson"]
unique_customer_names = []

for name in customer_names:
    if name not in unique_customer_names:
        unique_customer_names.append(name)

print(unique_customer_names)

Output:

['John Smith', 'Emily Johnson', 'Michael Davis', 'Emma Brown']

I have executed the above example code, you can refer to the screenshot below.

How to Remove Duplicates from an Array in Python

In this example, we start with an empty list unique_customer_names. We iterate over each name in the customer_names array and check if it already exists in unique_customer_names. If it doesn’t, we append it to the list. This way, we maintain the original order while eliminating duplicates.

Check out How to Save an Array to a File in Python

Method 3: Using a Dictionary

Another approach to remove duplicates while preserving the order is to use a dictionary. In Python, dictionaries are key-value pairs where keys are unique. By using the array elements as keys in a dictionary, we can effectively remove duplicates. Here’s how it works:

customer_names = ["John Smith", "Emily Johnson", "Michael Davis", "John Smith", "Emma Brown", "Emily Johnson"]
unique_customer_names = list(dict.fromkeys(customer_names))
print(unique_customer_names)

Output:

['John Smith', 'Emily Johnson', 'Michael Davis', 'Emma Brown']

I have executed the above example code, you can refer to the screenshot below.

Remove Duplicates from an Array in Python Using a Dictionary

In this method, we use the dict.fromkeys() function to create a dictionary with the elements of customer_names as keys. Since dictionary keys are unique, duplicates are automatically removed. Finally, we convert the dictionary keys back to a list list() to get the desired result while maintaining the original order.

Read How to Find the Number of Elements in a Python Array

Handling Complex Duplicates

In some cases, you may have more complex duplicates in your array, such as duplicates within nested arrays or duplicates based on specific criteria. Let’s consider an example where you have an array of customer information, including their name, age, and city:

customer_data = [
    {"name": "John Smith", "age": 35, "city": "New York"},
    {"name": "Emily Johnson", "age": 28, "city": "Los Angeles"},
    {"name": "Michael Davis", "age": 42, "city": "Chicago"},
    {"name": "John Smith", "age": 35, "city": "New York"},
    {"name": "Emma Brown", "age": 31, "city": "Houston"},
    {"name": "Emily Johnson", "age": 28, "city": "Los Angeles"}
]

To remove duplicates based on specific fields, you can use a combination of a for loop and a temporary list of dictionaries:

unique_customer_data = []
seen = set()

for customer in customer_data:
    key = (customer["name"], customer["age"])
    if key not in seen:
        seen.add(key)
        unique_customer_data.append(customer)

print(unique_customer_data)

Output:

[{'name': 'John Smith', 'age': 35, 'city': 'New York'}, 
 {'name': 'Emily Johnson', 'age': 28, 'city': 'Los Angeles'}, 
 {'name': 'Michael Davis', 'age': 42, 'city': 'Chicago'}, 
 {'name': 'Emma Brown', 'age': 31, 'city': 'Houston'}]

In this approach, we create a temporary set seen to keep track of the unique combinations of name and age. We iterate over each customer dictionary customer_data and check if the combination of name and age exists in the seen set. If it doesn’t, we add the combination seen and append the customer dictionary to unique_customer_data. By this duplicates based on the specified criteria are removed.

check out How to Transpose an Array in Python

Conclusion

In this tutorial, I have explained how to remove duplicates from an array in Python and I showed several methods to achieve this task, One of the methods is using a Set in which the array is converted into a set, by using for loop to iterate through each element in the array, and by using dictionary. We also discussed how to handle complex duplicates within nested arrays.