How To Remove Duplicates From A Sorted Array In Python?

In this tutorial, I will help you understand how to remove duplicate elements from a sorted array in Python. As a software developer, I faced an issue while working on a project for a client in New York. The dataset contained a sorted list of customer IDs, and I needed to ensure that each ID appeared only once. I will show you how to remove duplicates from a sorted array in Python with some examples.

Table of Contents

Problem Statement

Given a sorted array, the goal is to remove duplicates in place such that each element appears only once. The relative order of the elements should be maintained, and the function should return the new length of the array after removing duplicates.

For example, consider the following sorted array of customer IDs:

customer_ids = [1001, 1001, 1002, 1003, 1003, 1004]

After removing duplicates, the array should look like this:

customer_ids = [1001, 1002, 1003, 1004]

The function should return the new length, which is 4.

check out How to Sort an Array in Python

Python Implementation

Here is the Python code to remove duplicates from a sorted array using the two-pointer technique:

def remove_duplicates(customer_ids):
    if not customer_ids:
        return 0

    i = 0
    for j in range(1, len(customer_ids)):
        if customer_ids[j] != customer_ids[i]:
            i += 1
            customer_ids[i] = customer_ids[j]

    return i + 1

# Example usage
customer_ids = [1001, 1001, 1002, 1003, 1003, 1004]
new_length = remove_duplicates(customer_ids)
print("New length:", new_length)
print("Array after removing duplicates:", customer_ids[:new_length])

Output:

New length: 4
Array after removing duplicates: [1001, 1002, 1003, 1004]

I executed the above example code and added its screenshot below.

Remove Duplicates from a Sorted Array in Python

The function first checks if the input array is empty. If it is, the function returns 0. The pointer i is initialized to 0. The loop starts from the second element (index 1) and goes through the entire array. Inside the loop, if the current element customer_ids[j] is different from customer_ids[i], increment i and copy the current element to customer_ids[i]. After the loop, the function returns i + 1, which is the new length of the array with unique elements.

Read How to Use Python Array Index -1

Example

Let’s consider a real-world example where we have a sorted list of customer IDs from different states in the USA. We need to remove duplicates to maintain a unique list of customer IDs.

customer_ids = [
    1001, 1001, 1002, 1003, 1003, 1004, 1005, 1005, 1006, 1007, 1007, 1008
]

new_length = remove_duplicates(customer_ids)
print("New length:", new_length)
print("Array after removing duplicates:", customer_ids[:new_length])

Output:

New length: 8
Array after removing duplicates: [1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008]

I executed the above example code and added its screenshot below.

How to Remove Duplicates from a Sorted Array in Python

In this example, the function successfully removes duplicates and returns the new length of the unique customer ID list.

Check out How to Save an Array to a File in Python

Handle Large Datasets

When dealing with larger datasets, the two-pointer technique remains efficient. However, it is important to ensure that the input array is sorted. If the Python array is not sorted, you need to sort it first before applying the two-pointer technique.

# Example with unsorted array
unsorted_customer_ids = [1004, 1001, 1003, 1001, 1002, 1003]
sorted_customer_ids = sorted(unsorted_customer_ids)
new_length = remove_duplicates(sorted_customer_ids)
print("New length:", new_length)
print("Array after removing duplicates:", sorted_customer_ids[:new_length])

Output:

New length: 4
Array after removing duplicates: [1001, 1002, 1003, 1004]

I executed the above example code and added its screenshot below.

Remove Duplicates from a Sorted Array in Python by Handling Large Datasets

Read How to Find the Number of Elements in a Python Array

Conclusion

In this tutorial, I have explained how to remove duplicate elements from a sorted array in Python. I have given a problem statement to understand the requirement and I showed the Python implementation using the two-pointer technique, I have taken a real-world example to make you understand better, We have also discussed how to handle large datasets with an example and screenshot of executed code.