RMarkdown with Python Script

Publishing Your Results
1

I am able to execute Python scripts inside R Markdown. However, when it comes to the widgets portions to display those UI elements, those cannot be displayed. It outputs as below:

HTML(value="<h2 style='color:DarkBlue';>Parameters for data cleaning and model selection</h2>")
DatePicker(value=None, description='From Date')
DatePicker(value=None, description='To Date')
HBox(children=(VBox(children=(FloatText(value=0.09, description='dist_ex_min'), FloatText(value=3.0, description='dist_ex_max'))), VBox(children=(FloatText(value=2.0, description='dist_cal_max'),)), VBox(children=(FloatText(value=95.0, description='grade_min'),))))

VBox(children=(HTML(value="<h4 style='color:DarkBlue';>Model</h4>"), GridspecLayout(children=(Label(value='Cal0', layout=Layout(grid_area='widget001')), Label(value='Cal1', layout=Layout(grid_area='widget002')), Label(value='Cal2', layout=Layout(grid_area='widget003')), Label(value='Cal3', layout=Layout(grid_area='widget004')), Label(value='Ex1', layout=Layout(grid_area='widget005')), Label(value='Ex2', layout=Layout(grid_area='widget006')), Label(value='Ex3', layout=Layout(grid_area='widget007')), Label(value='Ex4', layout=Layout(grid_area='widget008')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget009', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget010', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget011', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget012', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget013', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget014', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget015', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget016', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget017', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget018', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget019', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget020', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget021', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget022', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget023', width='10%')), Checkbox(value=True, indent=False, layout=Layout(grid_area='widget024', width='10%'))), layout=Layout(align_items='center', grid_template_areas='". widget001 widget002 widget003 widget004"\n"widget005 widget009 widget010 widget011 widget012"\n"widget006 widget013 widget014 widget015 widget016"\n"widget007 widget017 widget018 widget019 widget020"\n"widget008 widget021 widget022 widget023 widget024"', grid_template_columns='repeat(5, 1fr)', grid_template_rows='repeat(5, 1fr)', width='50%'))))
HTML(value="<h4 style='color:DarkBlue';>Select Section</h4>")
interactive(children=(Dropdown(description='Section', index=4, options=('1234', '5678'), Output()), _dom_classes=('widget-interact',))
HBox(children=(Button(description='Clean Raw Data', style=ButtonStyle()), Button(description='Train Model', disabled=True, style=ButtonStyle())))
HBox(children=(Output(), Output()))
Button(description='Save Model', disabled=True, style=ButtonStyle()) Output()

Any help or suggestion is greatly appreciated.

2

Can you provide a full example, and reprocubible would be better ?

It is hard to tell which package you are using (for DatePicker, HBox and all...), which type of document you are using and what your Rmd file looks like.

Thank you.

3 
---
title: "Running Python Scripts inside RStudio"
author: "TANTH"
date: "10/12/2020"
output:
  pdf_document: default
  html_document: default
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
library(reticulate)
use_python("/usr/bin/python3")
```

```{python}
# General
import os
import sys
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pickle
from pathlib import Path
import ipywidgets as widgets
from IPython.display import display
# # sklearn
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
pd.set_option('display.max_columns', 500)
```
```{python}
# Interactive Widgets
# retrieve section


## section dropdown
##############################################################################################################################################################
# section_dropdown = widgets.Dropdown(options=['1234', '5678'],
#                               value='1234',
#                               description='Section:',
#                               disabled=False,
#                              )
title_section = widgets.HTML(value="<h4 style='color:DarkBlue';>Select Section</h4>")

def selected_section_change_function(Section):
    output_clean.clear_output()
    output_train.clear_output()
    output_save.clear_output()
    button_train.disabled = True
    button_save.disabled = True
    return Section

section_list = ['1234', '5678']
section_dropdown = widgets.interactive(selected_section_change_function, Section=widgets.Dropdown(options=section_list, value='1234'))
# display(section_dropdown)
##############################################################################################################################################################

## date picker
##############################################################################################################################################################
from_date_picker = widgets.DatePicker(description='From Date',
                               disabled=False
                              )

to_date_picker = widgets.DatePicker(description='To Date',
                               disabled=False
                              )
##############################################################################################################################################################

## data outlier distances selection parameters
##############################################################################################################################################################
dist_ex_min_input = widgets.FloatText(value=default_dist_ex_min,
                                      description='dist_ex_min',
                                      disabled=False
                                     )

dist_ex_max_input = widgets.FloatText(value=default_dist_ex_max,
                                      description='dist_ex_max',
                                      disabled=False
                                     )

dist_cal_max_input = widgets.FloatText(value=default_dist_cal_max,
                                 description='dist_cal_max',
                                 disabled=False
                                )

grade_min_input = widgets.FloatText(value=default_grade_min,
                                 description='grade_min',
                                 disabled=False
                                )

dist_ex_input_box = widgets.VBox([dist_ex_min_input, dist_ex_max_input])
dist_cal_input_box = widgets.VBox([dist_cal_max_input])
grade_min_input_box = widgets.VBox([grade_min_input])
dist_input_box = widgets.HBox([dist_ex_input_box, dist_cal_input_box, grade_min_input_box])
##############################################################################################################################################################

## Model widget 
##############################################################################################################################################################
cal_header = [widgets.Label('Cal'+str(i)) for i in range(4)] 
ex_header = [widgets.Label('Ex'+str(i+1)) for i in range(4)] 
checkboxes = {}
for ex in range(1, 5):
    checkboxes[ex] = {}
    for cal in range(4):
        checkboxes[ex][cal] = widgets.Checkbox(value = bool(1), disabled=False, indent=False)
        checkboxes[ex][cal].layout.width = '10%'
        
grid = widgets.GridspecLayout(5, 5, align_items='center', width = '50%')
for i in range(1,5):
    grid[0, i] = cal_header[i-1]
for i in range(1,5):
    grid[i, 0] = ex_header[i-1]    
for ex in range(1,5):
    for cal in range(4):
        grid[ex, cal+1] = checkboxes[ex][cal] 
    
title = widgets.HTML(value="<h2 style='color:DarkBlue';>Parameters for data cleaning and model selection</h2>")
title_model = widgets.HTML(value="<h4 style='color:DarkBlue';>Model</h4>")
model_widget = widgets.VBox([title_model, grid])
##############################################################################################################################################################


# CLean Data Button 
##############################################################################################################################################################
button_clean = widgets.Button(description="Clean Raw Data")
output_clean = widgets.Output()

def on_button_clean_clicked(b):
    with output_clean:
        output_clean.clear_output()
        print('section', section_dropdown.result)
        section = section_dropdown.result
        
        # Initialization of clean dataset
        global clean_dataset_df
        clean_dataset_df = raw_dataset_df.copy()

        # Missing values
        clean_dataset_df = clean_missing_values(clean_dataset_df, section)

        # Time range selection
        clean_dataset_df = clean_time_range(clean_dataset_df)

        # Filter out unwanted samples
        clean_dataset_df = clean_outliers(clean_dataset_df, section)
        
        print('Cleaned Data Shape:', clean_dataset_df.shape)
        
        button_train.disabled = False


button_clean.on_click(on_button_clean_clicked)
##############################################################################################################################################################

# Train Button 
##############################################################################################################################################################
button_train = widgets.Button(description="Train Model", disabled=True)
output_train = widgets.Output()

def on_button_train_clicked(b):
    with output_train:
        output_train.clear_output()
        section = section_dropdown.result
        ex_cal_map_df = ex_cal_map_df_gen(section)
#         print(ex_cal_map_df)
        coef = train_model(ex_cal_map_df)
#         print(coef)
        global X, model
        X, model = model_validation(coef)
        
        button_save.disabled = False


button_train.on_click(on_button_train_clicked)
##############################################################################################################################################################

# Save Button 
##############################################################################################################################################################
# saving of the model
button_save = widgets.Button(description="Save Model", disabled=True)
output_save = widgets.Output()

def on_save_button_clicked(b):
    with output_save:
        output_save.clear_output()
        
        meta_data = {}
        if not pd.isnull(bw_id_dropdown.result):
            meta_data['selected_file_id'] = int(bw_id_dropdown.result)
        else:
            meta_data['selected_file_id'] = np.nan

        pickle_dict = {}
        pickle_dict['features'] = X.columns
        pickle_dict['estimator'] = model
        pickle_dict['data'] = meta_data
        
        section = section_dropdown.result
        filehandler = open('models/{}'.format(file_name), 'wb')
        pickle.dump(pickle_dict, filehandler)
        filehandler.close()
        
        print('Model saved to pickle file {}.'.format(file_name))
        
button_save.on_click(on_save_button_clicked)
##############################################################################################################################################################
```

```{python}
# Display
display(title)
display(from_date_picker)
display(to_date_picker)
display(dist_input_box)
print('')
display(model_widget)
display(title_section)
display(section_dropdown)
display(bw_id_dropdown)
display(widgets.HBox([button_clean, button_train]))
display(widgets.HBox([output_clean, output_train]))
display(button_save, output_save)
```
4

Oh so you are using widget from python !

I am not sure the reticulate python engine support widget-like output yet. See for example the open issue on plotly where jupyter widgets are mentioned: https://github.com/rstudio/reticulate/issues/269#issuecomment-390156263

That would be a new feature for widget support in reticulate python engine for Knitr/Rmarkdown

5

Thanks, does that mean at present, reticulate python engine doesn't support widget-like output?

6

I think so but maybe we should get confirmation from @kevinushey ?

1 Like
7

That's correct, it does not.

2 Likes
8

How can we put a request for that reticulate R package to support widget-like output from Python? It would be very useful. Thanks.

9

I guess you can open a specific issue in the Github repository of reticulate or just add a :+1: to the issue mentioned above.

10

I went through this website, https://rviews.rstudio.com/2018/04/17/reticulated-shiny/
and create its virtual environment as follows:

library(reticulate)
reticulate::virtualenv_create(envname = '/srv/shiny-server/MyApplication', 
                              python= '/usr/bin/python3')

where the directory '/srv/shiny-server/MyApplication' contains server.R and ui.R

> reticulate::py_config()
python:         /usr/bin/python3
libpython:      /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so
pythonhome:     //usr://usr
version:        3.6.9 (default, Oct  8 2020, 12:12:24)  [GCC 8.4.0]
numpy:          /home/XXX/.local/lib/python3.6/site-packages/numpy
numpy_version:  1.19.2

python versions found: 
 /usr/bin/python3
 /usr/bin/python

Above does NOT show that the virtual environment was created, as what was described in the above link. However, I did see some directories (bin,include,lib,lib64,share,pyvenv.cfg) being added to my directory '/srv/shiny-server/MyApplication'.

Going through the above does NOT help too.

11

Can I ask about running the Python Script inside R/RStudio again?
I have no issue when running a test.R script which contains

library(reticulate)
use_python("/usr/bin/python3")

# Sourcing Python scripts — The source_python() function enables you to source a Python script the same way you would source() an R script 
# (Python functions and objects defined within the script become directly available to the R session).

source_python("/srv/shiny-server/CelayaPlant/Python/imports.py")

Above is able to load in various Python packages.
However, when I am doing some Shiny application, I need to load similar thing inside the server side.
It creates error as below:

Listening on http://127.0.0.1:43541
Warning: Error in py_run_file_impl: ModuleNotFoundError: No module named 'pandas'

Detailed traceback: 
  File "<string>", line 5, in <module>

Any help is appreciated. Thanks.

12

Do you see your environment in reticulate::virtualenv_list() ?

You need to specifically tell reticulate to choose this virtual environment using reticulate::use_virtualenv() or by setting RETICULATE_PYTHON_ENV. You can also set RETICULATE_PYTHON to the path of the python binary inside your virtualenv.

Check after with py_discover_config() what is found.

See about Python version configuration here: Python Version Configuration • reticulate

13

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.

If you have a query related to it or one of the replies, start a new topic and refer back with a link.