sorry that I don't have a perfect solution for you.
When it fails in the way you describe, I get in the browser(chrome) inspector, a javascript console error message about internal server error 500. This is quite ambiguious and might have many causes.... Can you go to your dashboard access this stack0 app and get the log ? perhaps there are R errors thrown we could investigate.
I did the following tests.
I succesfully uploaded a simple .R text script but renamed it to .mat this loaded fine
I also sucessfully uploaded your example .mat by first zipping it and uploading the zip.