Poetry package manager

Table of Contents

0. Overview

Poetry is a python packaging and dependency manager. It makes it really easy to manage packages while using environments under the hood. It also allows build and publishing packages to PyPI (or other sites).

1. Alternatives. Why Poetry?

Before viewing why is Poetry a good solution let's check the more common alternatives.

1.1. Raw

The easiest and least elegant why of handling dependencies is by simply installing all packages in the main environment. In other words this means installing packages only using:

pip install example_package

This is usually the way new python developers install programs. In order to track the installed libraries users could add them manually in the requirements.txt file or export it using:

pip freeze > requirements.txt

This method will have some major problems:

1. It will be difficult to install different versions for different projects (for example pandas 0.25 for one project and 1.0 for another.)
2. If requirements are added manually it is possible to forget some (when adding new ones or deleting unused)
3. If requirements are added with pip freeze it will be difficult to differenciate between real dependencies and dependencies of the installed packages
4. There will be no differenciation between development dependencies and production dependencies

It is true that you can create a requirements_dev.txt file to handle development dependencies but you will need to manually mantain it.

1.2. Virtualenv / Conda

With both Virtualenv or Conda it is possible to create environment. This will allow to have different versions for different projects.

But except from that point there will still be the other problems.

2. How Poetry works?

With Poetry you create a project (or import one) and it automatically creates/uses a virtual environment under the hood.

Poetry will only use two files the pyproject.toml and poetry.lock.

The pyproject.toml is a standard python file for configuring a project. It aims to have all information about the project.

As an example:

[tool.poetry]
name = "test_project"
version = "0.1.0"
description = ""
authors = ["villoro7 <villoro7@gmail.com>"]

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]
pytest = "^5.2"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

By using the pyproject.toml file you will not need the setup.py file.

Finally the poetry.lock is a snapshot with the exact versions install. It can be updated from the pyproject.toml.

2. Creating a project

2.1. Init a new project

The first thing to do with python is to create a new project. You can do it with:

poetry new projectname
# Replace projectname for the name you want

2.2. Project structure

Creating a new project will create the following files:

- /projectname
    ├── /projectname
    │   └── __init__.py
    ├── /tests
    │   ├── __init__.py
    │   └── test_projectname.py
    ├── pyproject.toml
    └── README.rst

There will be a dummy test inside the tests folder using the pytest package.

3. Managing dependencies

The idea is to use poetry to add or remove packages and let it handle the pyproject.toml file.

3.1. Add/remove packages

To add or remove packages you can use:

poetry add pandas
poetry remove pandas

You can also specify a development dependency with:

# Both are equivalent
poetry add -D black
poetry add --dev black

# And for remove
poetry remove -D black
poetry remove --dev black

Poetry is more has more ways of specifing the versions of the dependencies. The default one is:

pandas = "^0.25"

Wich translates to >=0.25.0 <1.0.0

You can check all the possible ways in the poetry doc.

3.2. Install update dependencies

Once there is a pyproject.toml file you can install/update all the specified dependencies.

Those are also the commands you should run if the project was created and you are fetching it from git.

poetry install
poetry update

# for production
poetry install --no-dev

Both commands will create a the poetry.lock file with a snapshot of the installed packages (or update it if already present).

3.3. See dependencies

With poetry you easily see the dependencies of your project as well as the dependencies of those dependencies.

# Show all dependencies
poetry show -t

# Show production dependencies (no development)
poetry show -t --no-dev

As an example:

3.4. Check

With poetry check you can check if the pyproject.toml file has any errors.

4. Runing code with Poetry

Since poetry uses a virtual environment and you are not activating/sourcing it has commands to actually work with the environment.

With poetry shell you can have a shell using the environment created with poetry.

And poetry run xxxx will allow you to run the command you want using the environment. As an example instead of running:

luigid

python src/master.py

You should do:

poetry run luigid

poetry run python src/master.py

5. Create and publish packages

You can use poetry to build python packages with:

poetry build

This will create both a whl and .tar.gz files.

And then you can also upload it to PyPI with:

poetry publish

6. Poetry compatibility

Finally you can use poetry and allow other people to use the old requirements.txt file.

You will only need to export it with:

poetry export -f requirements.txt > requirements.txt

7. More info

If you want to read more about poetry check out the following links:

1. Poetry documentation
2. Solving dependency management in Python with Poetry
3. Package Python Projects the Proper Way with Poetry