The problem

The main objective was to develop something that could get a high F1-score for classifing non-human traffic while investing low time developing it.

I was also given a really powerful virtual machine from Azure with the following specs:

type: M64
cores: 64 vCPUs
RAM: 1024 GB

storage: ~ 20 TB with SSD
cost: ~ 10 000 $ / month

This allowed me to work with pandas which was one of the packages I was more comfortable with. If I had to do it again I would do it with pyspark or dask.

Solution

So the idea was to rely on pandas for reading and processing the data and use numba were it was possible to make it all faster.

Then I used PCA and Random Forests (RF) from scikit for the predictions itself.

The pipeline looks like this:

This combination allowed for a relatively fast solution and a really fast development time.

Since the development time was an important constrain I tried to only extract simple data and avoid feature engineering. Even with those constrains the F1-score was always way above 90%.

Deep Learning (DL)

I also trained a Direct Neural Network but the results were not really much better. However the training and prediction time were way slower with DL compared to the RF.

Interesting facts discovered

The input data was a csv with around 650 - 800 million rows. Using the powerful Azure M64 pandas was able to read 10 million rows per minute of the csv.

And then to export the dataframe with 650 million rows it took 109 minutes to export a csv (and used 173 GB) and only 26 minutes to export a parquet (which used 42 GB). This also shows how efficient parquet is for storing data. More info here.