Personal Assistant

How to automate tasks with Luigi and an AWS EC2 instance.

luigi
Python Luigi AWS Automation Dropbox

Brief

The idea was to create an assistant that would work in an AWS EC2 instance. This assistant would allow me to easily automate tasks that will be useful for me.

As an example I use this assistant to create reports about my personal finances and to help me book cheap flights.

Motivation

I am a huge fan of automation and I wanted a bot that worked for me.

The idea was to create an assistant that allowed me to avoid repetitive tasks. It was a must that new tasks could be added easily to the assistant.

How to automate tasks

I wanted to use a cloud provider so the first thing I did was to create an AWS EC2 instance (more info).

Then I needed an orchestrator that handled the tasks and I was between Airflow and Luigi.

I started working with Airflow (more info) because it has more features. But I didn't need a lot of the features that Airflow offered and it was something more complex that what I wanted.

So I switched to Luigi and a created a method that made really easy to integrate new tasks.

The tasks

1. Expenses/Incomes Report

Since 2010 I have been recording all expenses and incomes. For the last years I have been using Money Lover to do so. This app has a way to export the data as an Excel file. But the problem is that it does not have the format I want. So the first task is to clean that data (clean data).

With the data cleaned the idea is to create a custom html report using Jinja2 templates.

In order to make it more flexible there are two tasks:

  1. Extract info that creates a yaml with the info
  2. Create Report that creates the html report with the yaml data

With this approach is easy to modify the data or the template without needing to modify the other.

So the pipeline is as follows:

luigi reports

As you can see I use dropbox for storing all data this way is easy for me to access or modify it.

The report itself uses W3css for the layout and Highcharts for charts.

Here you can see some of the pages the report have:

luigi report dashboard
luigi report comparison
luigi report pies
luigi report liquid
luigi report sankey

And of course the report is responsive:

luigi report nexus 5X

Flights tracker

I regularly travel to Italy and I want to do it as cheap as possible. So I thought the best way to do it was to let the assistant track all the prices between the airports I wanted and store that data. Then I could use Data Analysis or Machine Learning to minimize the price I pay for the flights.

To do so I used the Rapid API Flight Search app. This API allowed me to query some pairs of airports daily for free. So right now the assistant is storing a year of data each day so that I can see prices changes and which company offers cheaper flights each day.

luigi flights

As you can see Rapid API is getting their data from Skyscanner.

A sample of the data:

luigi flights data

Getting info

The assistant is connected to slack so that is able to send messages of each task it completes. It will also give information about any task failure that might happen.

Nexts steps

The idea is to extend the slack integration by creating a chatbot.

This chatbot would allow me to:

Github
Lights

More Projects

More Projects