Cache .tox and possible apt packages

[MischaPanch]

This is already done in the gitlab pipeline, we should also to it on github. Probably won't be totally trivial though

[opcode81]

Why?

[MischaPanch]

It would significantly speed up the build, no need to recreate the environments and download the dependencies each time. I might make a PR today or tomorrow

[opcode81]

Build speed is not really important, so only do this if you can guarantee that there won't be adverse effects.