Not able to use badRecordsPath in spark_read_csv. Is there any limitations to it?

General
, ,
1

I am trying to load below data into spark-->
id,name,age,sal
1,a,12,100
2,b,24,200
3,c,36,300
4,d,48,15k

df <- spark_read_csv(sc,"tempData","path/temp.csv",columns = list(id = "integer",name = "character",age = "integer",sal = "integer"),overwrite = TRUE,badRecordsPath = "path/sosharm/")

I was expecting the last record should be output in badRecordsPath but it is outputting as NA in df.

Is there any reason badRecordsPath is not working?

2

This topic was automatically closed 42 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.