It is not easy to troubleshoot your code without access to your data.
For a reproducible example of an animated map built with {ggplot2} and {gganimate} on top of a spatial object consider this post:
It is a part of a longer thread about visualizing spatial data and it is built on the widely available North Carolina shapefile (it ships with {sf}, so just about everybody has access to it).
Have a look at it and if your doubts remain come back to me, I would be happy to help.