Using the redirect feature does not work for URLs with spaces in them. Only when manually embedding the regex in quotes in the nginx redirect config file, the redirect works.
e.g. redirecting from /an example page to /example does not work. Trying /an%20example%20page does not work either.
Looking at the redirect nginx config it contains:
rewrite ^/an example page$ /example permanent;
while it should be:
rewrite "^/an example page$" /example permanent;
Adding the quotes manually fixes the redirect.
On another note, it should be possible to edit redirects and not only delete them.