It is possible to completely disable the workspaces transition delay on M1.
See here:
I can confirm that this works on my M1 mac running macOS 11.4.
Would love to see this functionality come to total spaces