As part of this interview cycle I have been exploring pola.rs, a pandas like library for python and rust. I used it for python, and I have some experience with pandas to compare.

To me the main differences between both libraries are performance , the different ways to group operations and general cleanliness of the interface.

On performance I would argue that pola.rs feels much faster than pandas for objects with 1M rows, and I did not notice any massive memory spikes compared to pandas.

With pandas 1.x I had issues with regular code just because I imported pandas. E.g.:

# import pandas
NCOLS = 200  
NROWS = 10**5
def generate_collection():
    for x in range(NROWS):                
        to_store = {'column':str(x) + "a"}
        for col in range(NCOLS):
            to_store['column{}'.format(col)] = x+col
        yield to_store
collection = list(generate_collection())
del collection

This was a difficult problem to explore, but I managed to find the problem in the python heap, and accidentally discovered that glibc.malloc.trim_threshold=128 fixes it . This is a problem for another post, but the idea is that pandas requires many tricks to stay performant.

Most of this is gone with pola.rs, just because the move to rust and back avoids tweaking in python.

On grouping operations, I really liked the df.lazy().operation1().operation2().operation3().collect() It creates an in memory representation of the operations and applies all of them in bulk docs.

This is not the same than pandas in_place() but serves a similar purpose of avoiding copying data around.

On general cleaniness My feeling is that the polars API benefited immensely of starting from scratch, and avoided most of the API changes that made pandas upgrades really painful. Although I acknowledge that I am a bit biased against pandas as upgrading it has been a big fraction of my experience and it isn’t as pleasant as upgrading something like django.

Some things I learned specifically

joins act very similar to SQL so for example a right join would have nulls for the ids of the left table docs

with_columns and joins are the main way to change the shape of the data. filter and [] (getitem) are the ways to change the rows or columns. concat also works

coalesce for full joins are not set by default so columns get duplicated, although I am grateful that the suffix for the right side can be changed.

expressions are similar to pandas. The when-then-otherwise allows some logic, but not very powerfull overall.

Some expressions required the horizontal call so for example:

expr = pl.min_horizontal(1, bounded_expr / pl.col("total"))

pandera link did not work very well with polars

Some thoughts on “DataFrame programming”

As I was working with dataframes I found myself trying to write the code in regular python, where things like entities and services can be named and executed. But eventually dataframes require to switch to a more constrained way of programming. Here are two observations:

1

DataFrame programming is very similar to SQL programming. The data lives in the programming language memory instead of the database, and the support for relations is mediated by the programming language (joins are in memory instead of FKs). The power of the expressions in pandas is not too different from what is possible in something like postgres.

2

Compared to regular python programming, dataframes require a change of mindset. Something as simple like an aggregation:

class Aggregator:
   def __init__(self):
      aggregation = []
   
   @property
   def property_1(self):
      """aggregation property like sum"""
      ...

class Aggregated:
   def __init__(self, aggregator):
      self.aggregator = aggregator

   @property
   def property_2(self):
      ...

cannot be translated easily without joining the dataframes

property_1 = aggregated_df["property_2", "aggregator_id"].group_by("aggregator_id").sum()
aggregator_df = aggregator_df.join(aggregated_df, on="aggregator_id", how="left")
aggregated_df = aggregated_df.join(property_1, on="aggregator_id")
coalesce="true")

The patterns that exist in a general purpose like python don’t quite exist in dataframe programming.

  • Services remain away of the data, and typing is poorly integrated with python (even with pandera)
  • Entities cannot be named within a dataframe other than by prefixing or suffixing
  • Things like a exception becomes an error value or null.
  • Loops don’t quite exist and need to be calculated elsewhere
  • Class relationships like inheritance need to be flattened into a dataframe to perform an operation. Even SQL can do this better

Dataframes become this array of struct that can be mutated in shape and content, but in order to be processed efficiently it lacks the ability to traverse or separate to emulate composition. .