In the original discussion about this [1], I initially leaned towards
"they should both fail", but I reconsidered: there doesn't seem to be
any harm in allowing ALTER SYSTEM SET to succeed for any custom GUC
name, as long as you're superuser.
+1 for allowing non-existent custom GUCs.
From time to time we have to roll out custom binaries controlled by GUCs that do not exist in normal binaries. Juggling with postgresql.conf would be painful in this case.
Best regards, Andrey Borodin.