I've been struggling to figure out what is the 'correct' way of determining file paths for a python (streamlit in this case) project. I have resources (data, images) that I'd like to access and open from the code. It's a medium-sized project so the code is organized into multiple folders and modules. Both code and data files are correctly included in the bundle, so this is not the case of trying to access files outside of the project reach.
I was hoping that the working directory would be set to the bundle root so I'd use relative paths from there to reach each data file, but when run, I see the os.getcwd() returns the python executable path (something like /opt/rstudio-connect/mnt/app/python/env/lib/python3.11/).
Is there something I'm overlooking and there is an easier way to reliably organize project navigation?
I've always used Pathlib to programmatically determine where I am in the filetree. What sort of issues are you finding, problems with relative import statements or passing file references to functions (or both)?
Relative imports took some head-scratching, but I managed to make it work. Perhaps how I organized this has impacted my other issues, so let me describe the full picture:
I have part of the functionality as a package that I'd like to share between 3-4 apps. The server is managed by another team so I haven't (yet) considered deploying this as a local package to the server, but instead include in the bundle.
Then I have the app code organized within folders/modules.
And the data files.
So the structure of the bundle (I manage the deployment using a manifest file) is something like this:
./package_src/
./app_code/
./data/
./setup.py
./requirements.txt
requirements.txt contains ./package_src/ line to install this package from the local storage once on the server.
So this solves the problem of importing and using the package from the app_code. Relative imports of other modules located within app_code work es expected.
But then the different modules/submodules inside the package and app_code need to locate and open files from /data/.. folder, and here is where I'm stuck ...
It's hard to understand what kind of problem you're seeing without some kind of reproducible example, but as Randy suggests, some kind of path helper library may help you make sure you come up with paths than can be resolved correctly from anywhere in your project.
I finally figured out the root of the problem - a somewhat not obvious side-effect of installing part of the application as a separate local project - although I was bundling and uploading changed code for both the package and the main app, posit was only deploying the app part but re-using the existing environment with the old version of my package.
So the code of locating the file resources (using pathlib etc) was in the package and correct, but in my attempt to figure out why it was not working on the server, I kept making modifications, uploads, modifications, uploads etc and still not seeing the expected behavior, thinking I had an error in my code, not realizing why the old code was still running.
Of course, now that I know a bit more, I can reorganize the whole project to avoid such pitfalls, but this is my first experience of having an existing project (developed by someone else) to be 'simply' migrated to posit connect, but the process was everything but simple...