Member-only story
Using Pandas’ .loc and .isin() to Filter for a List of Values in Python
Documentation for using both .isin() and .loc to filter a Pandas Dataframe on a list of values
I use .loc on a daily basis. It’s like using the filter function on a spreadsheet.
It’s an effortless way to filter down a Pandas Dataframe into a smaller chunk of data.
It typically works like this:
new_df = df.loc[df.column == 'value']Sometimes, you’ll want to filter by a couple of conditions. Let’s pretend you want to filter down where this is true and that is true. You might write:
new_df = df.loc[(df.this == True) & (df.that == True)]Now let’s pretend that list of filter attributes grows a little bit. Instead of 2 attributes to filter on, you have 4:
list_of_data = ['this','that','something else','another thing']Using the .loc approach, you might come up with something like this:
new_df = df.loc[(df.column == 'this') | (df.column == 'that') | (df.column == 'something else') | (df.column == 'another thing')]This approach above is doable for 1–2 attributes — maybe even 3 — but becomes much too time consuming for larger lists of filter attributes.
What if we wanted to filter a dataframe with 10 different conditions? It’s not feasible to use .loc anymore. It’s time consuming and it’s hard to read.
So what can we do?
You probably have an answer for this type of problem in SQL, but what can you do in Pandas? Well, it’s similar…
This is where .isin() comes into play.
filter_list = ['this','that','something else','another thing', ... ]new_df = df.loc[df['column'].isin(filter_list)]
Using isin() you can easily filter a dataframe by a list of filter conditions/attributes. I’ve found it this to be a helpful function.
But it gets better.
What if we want to filter OUT these conditions? All it takes is a simple ~ :
new_df = df.loc[~df['column'].isin(filter_list)]Boom.
If you enjoy reading stories like these and want to support me as a writer, consider signing up to become a Medium member. It’s $5 a month, giving you unlimited access to thousands of Python guides and Data science articles. If you sign up using my link, I’ll earn a small commission with no extra cost to you.
