Reticulate::py_to_r How to convert a pandas DataFrame to a R data.frame

hitfuture · July 20, 2018, 6:16pm

I am using the reticulate package to integrate Python into an R package I'm building. One of the capabilities I need is to return R data.frames from a method in the R6 based object model I'm building. I utilize Python Pandas package to create a DataFrame in the reticulate python environment. My objective is to return this an R data.frame. The issue I'm seeing is that when I used reticulate::py_to_r(df) it does not convert to R and instead it returns a python DataFrame object. Below is a simple test I'm doing:

# platform       x86_64-pc-linux-gnu         
# arch           x86_64                      
# os             linux-gnu                   
# system         x86_64, linux-gnu           
# status                                     
# major          3                           
# minor          4.4                         
# year           2018                        
# month          03                          
# day            15                          
# svn rev        74408                       
# language       R                           
# version.string R version 3.4.4 (2018-03-15)
# nickname       Someone to Lean On


 
library(reticulate)
pd <- import("pandas",as = "pd",convert = FALSE)
aa <- pd$read_csv("CDM_LOCATION.csv")
df <-  reticulate::py_to_r(aa)
print(class(df))

df

[1] "pd.core.frame.DataFrame" "pd.core.generic.NDFrame" "pd.core.base.PandasObject"
[4] "pd.core.base.StringMixin" "pd.core.accessor.DirNamesMixin" "pd.core.base.SelectionMixin"
[7] "python.builtin.object"

Am I using the wrong method of transforming a DataFrame from Python to R? Feedback will be appreciated!

Thank you,
Brett

mara · July 21, 2018, 11:04am

Possibly related?

github.com/rstudio/reticulate

cannot coerce pandas dataframe to R dataframe

opened 10:01PM - 21 Sep 17 UTC

closed 03:33PM - 03 Aug 23 UTC

VyshaliEnukonda

`````` > use_virtualenv("C:\\Users\\venukond\\AppData\\Local\\Continuum\\Anac…onda3\\envs\\py35\\") > py_config() python: C:\Users\venukond\AppData\Local\CONTIN~1\ANACON~1\python.exe libpython: C:/Users/venukond/AppData/Local/CONTIN~1/ANACON~1/python35.dll pythonhome: C:\Users\venukond\AppData\Local\CONTIN~1\ANACON~1 version: 3.5.2 |Anaconda 4.2.0 (64-bit)| (default, Jul 5 2016, 11:41:13) [MSC v.1900 64 bit (AMD64)] Architecture: 64bit numpy: C:\Users\venukond\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\numpy numpy_version: 1.11.1 pandas: C:\Users\venukond\AppData\Local\CONTIN~1\ANACON~1\lib\site-packages\pandas python versions found: C:\Users\venukond\AppData\Local\CONTIN~1\ANACON~1\python.exe C:\Users\venukond\AppData\Roaming\BLACKR~1\python\3420B8~1.3\python.exe C:\Users\venukond\AppData\Local\Continuum\Anaconda3\envs\py34\python.exe C:\Users\venukond\AppData\Local\Continuum\Anaconda3\envs\py34_test\python.exe C:\Users\venukond\AppData\Local\Continuum\Anaconda3\envs\py35\python.exe > pd <- import("pandas") > library(reticulate) > pd <- import("pandas") > np <- import("numpy") > > > df <- pd$DataFrame( + list( + 'A' = 1., + 'B' = pd$Timestamp('20130102'), + 'C' = pd$Series(1, index = seq(4), dtype = 'float32'), + 'D' = np$array(rep(3L, 4), dtype='int32'), + 'E' = pd$Categorical(c("test","train","test","train")), + 'F' = 'foo' + ) + ) > class(df) [1] "pandas.core.frame.DataFrame" "pandas.core.generic.NDFrame" "pandas.core.base.PandasObject" "pandas.core.base.StringMixin" [5] "python.builtin.object" > as.data.frame(df) Error in as.data.frame.default(df) : c("cannot coerce class \"c(\"pandas.core.frame.DataFrame\", \"pandas.core.generic.NDFrame\", \" to a data.frame", "cannot coerce class \"\"pandas.core.base.PandasObject\", \"pandas.core.base.StringMixin\", \" to a data.frame", "cannot coerce class \"\"python.builtin.object\")\" to a data.frame") ``` ``` > sessionInfo() R version 3.3.0 (2016-05-03) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 7 x64 (build 7601) Service Pack 1 locale: [1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252 [4] LC_NUMERIC=C LC_TIME=English_United States.1252 attached base packages: [1] stats graphics grDevices datasets utils methods base other attached packages: [1] reticulate_1.1 loaded via a namespace (and not attached): [1] tools_3.3.0 Rcpp_0.12.9 jsonlite_1.2 > ``` ```

jdlong · July 22, 2018, 3:03pm

I had some snags doing object conversion in with the CRAN version of Reticulate. If you are running the CRAN version, try using the dev version:

devtools::install_github("rstudio/reticulate")

hitfuture · July 22, 2018, 9:57pm

Hi mara and jdlong,
Thank both of you for the feedback. I had forked reticulate into my github repository so I am using the latest version. I also see that there are well defined S3 methods to handle pandas DataFrame conversion in the reticulate py_to_r() S3 class (e.g. py_to_r.pandas.core.frame.DataFrame). I have identified the problem. The following test executes correctly in a new R session.

library(reticulate)
library(testthat)
 
pd <- import("pandas")
py_config()

## python:         /usr/local/bin/python3
## libpython:      /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6.dylib
## pythonhome:     /Library/Frameworks/Python.framework/Versions/3.6:/Library/Frameworks/Python.framework/Versions/3.6
## version:        3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 05:52:31)  [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
## numpy:          /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy
## numpy_version:  1.14.5
## pandas:         /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas
## 
## NOTE: Python version was forced by RETICULATE_PYTHON

before <- iris
expect_is(before,class = "data.frame")
convert <- r_to_py(before)
expect_is(convert,class = "pandas.core.frame.DataFrame")

after  <- py_to_r(convert)

expect_is(after,class = "data.frame")

The failure occurs when I utilize the function 'reticulate::import("pandas", as="pd")' with the as parameter. You can see below that the pandas.DataFrame is not converted into an R data.frame. So the problem is related to the S3 method for the pandas DataFrame not matching based on the name of the python module.

pd <- import("pandas",as = "pd")
py_config()

## python:         /usr/local/bin/python3
## libpython:      /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/config-3.6m-darwin/libpython3.6.dylib
## pythonhome:     /Library/Frameworks/Python.framework/Versions/3.6:/Library/Frameworks/Python.framework/Versions/3.6
## version:        3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 05:52:31)  [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
## numpy:          /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy
## numpy_version:  1.14.5
## pandas:         /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas
## 
## NOTE: Python version was forced by RETICULATE_PYTHON

before <- iris 
expect_is(before,class = "data.frame")

convert <- r_to_py(before)
expect_is(convert,class = "pd.core.frame.DataFrame")

after  <- py_to_r(convert)
print("after class: ")

## [1] "after class: "

print(class(after))

## [1] "pd.core.frame.DataFrame"        "pd.core.generic.NDFrame"       
## [3] "pd.core.base.PandasObject"      "pd.core.base.StringMixin"      
## [5] "pd.core.accessor.DirNamesMixin" "pd.core.base.SelectionMixin"   
## [7] "python.builtin.object"

#expect_is(after,class = "data.frame")

I have tested this on two different Docker containers, and also on my MacBook Pro and the same error occurs. I think this should be addressed in the reticulate package.
Thanks,
Brett

mara · July 23, 2018, 11:39am

Hi Brett,

If there isn't an open issue in the reticulate repo, then I suggest you file one! You've done a great job of prepping the problem, so hopefully it can get resolved soon.

Mara

hitfuture · July 23, 2018, 3:21pm

Hi Mara,
I just created an issue in the reticulate Github repository. If I were the developers of reticulate, I would start by just creating documentation in this area. I hope the Rstudio community knows that reticulate enables a great capability for R programmers to utilize Python when necessary. The package I'm building right now is Neo4jDriveR which will enable use of the Neo4j Python library which is supported by Neo4j and it will provide the correct access to the Graph Database. I wouldn't take this on without the reticulate package Rstudio's team has developed. Great work!

https://www.hitfuturenow.com/blog/2018/05/17/2018-05-14-leveraging-python-in-r-to-access-the-bolt-protocol-of-neo4j/

mara · July 23, 2018, 4:18pm

Thanks, Brett. Looks like a really neat project!

Would you mine linking the issue back to this thread so others who run into the same problem can follow along?

Thanks again.

Mara

hitfuture · July 23, 2018, 4:32pm

The reticulate::py_to_r() issue is posted on Github at https://github.com/rstudio/reticulate/issues/319