When you work with data from gadm.org, you are downloading rds file that contains sf or sp object.
I think you need to convert also the data you read from csv to the corresponding format (sf or sp) and work with sf 
to merge the data.
Also, Could you ask this with a minimal REPRoducible EXample (reprex)? A reprex makes it much easier for others to understand your issue and figure out how to help.