First, if using read.csv(), I would set stringsAsFactors = FALSE, which will result in the data coming in as Character instead of Factor. To move beyond that, we have to know what it is about the data that prevent it being interpreted as a number. Can you post a small sample of the data, preferably as a reproducible example (reprex).
As a guess, you may be able to use the sub() or gsub() functions to replace undesirable characters with empty strings and then use as.numeric().