Add Wal-E to my home server

I need to setup wal-e on my home server because recovery plans are good.
First I setup a new S3 bucket and IAM user for it
Then I started following the README and the following blog post.

Links:

https://github.com/wal-e/wal-e#installation

https://coderwall.com/p/cwe2_a/backup-and-recover-a-postgres-db-using-wal-e
https://web.archive.org/web/20200208221334/https://coderwall.com/p/cwe2_a/backup-and-recover-a-postgres-db-using-wal-e

https://gist.github.com/ruckus/2293434
https://web.archive.org/web/20200208224002/https://gist.github.com/ruckus/2293434

I have modified some of the instructions to use a python virtualenv because I like clean environments.

umask u=rwx,g=rx,o=
sudo mkdir -p /etc/wal-e.d/vars
sudo su
echo "my secret" > /etc/wal-e.d/vars/AWS_SECRET_ACCESS_KEY
echo "my access key" > /etc/wal-e.d/vars/AWS_ACCESS_KEY_ID
echo "us-west-1" > /etc/wal-e.d/vars/AWS_REGION
echo 's3://mybucket/wal-e' > /etc/wal-e.d/vars/WALE_S3_PREFIX

virtualenv env -p python3.8
source env/bin/activate
pip install wal-e[aws] envdir

chown -R root:postgres /etc/wal-e.d

First and foremost I want to make sure I can push a full backup via cron

sudo su postgres
/etc/wal-e.d/env/bin/envdir /etc/wal-e.d/vars /etc/wal-e.d/env/bin/wal-e backup-push /virt/postgresql_data/postgresql/10/main/

I got this error, so I installed lzpop and pv

postgres@pelennor5:/etc/wal-e.d$ /etc/wal-e.d/env/bin/envdir /etc/wal-e.d/vars /etc/wal-e.d/env/bin/wal-e backup-push /virt/postgresql_data/postgresql/10/main/

wal_e.main   INFO     MSG: starting WAL-E
        DETAIL: The subcommand is "backup-push".
        STRUCTURED: time=2020-02-08T22:28:37.586041-00 pid=25452
wal_e.main   ERROR    MSG: could not run one or more external programs WAL-E depends upon
        DETAIL: Could not run the following programs, are they installed? lzop, pv
        STRUCTURED: time=2020-02-08T22:28:37.693263-00 pid=25452
postgres@pelennor5:/etc/wal-e.d$ exit
root@pelennor5:/etc/wal-e.d# sudo apt install ^C
root@pelennor5:/etc/wal-e.d# apt install lzop pv

Next time my cron-capable command worked from the postgres user, which suggests a few things are going well.

Now I need to edit postgresql.conf

archive_mode = on
archive_command = '/etc/wal-e.d/env/bin/envdir /etc/wal-e.d/vars /etc/wal-e.d/env/bin/wal-e wal-push %p'
archive_timeout = 60

And sudo systemctl restart postgresql then seemed to work…

After that, I setup the base backup on crontab for postgres. All set on the backup side.

Now I need to test a restore to prove to myself this is reasonable, so I am going to setup postgres10 in docker on my laptop and try to restore from S3.

8th February 2020