Hi,
based on your advice, I checked the performance. Then I realized, that the Shiny server automatically turns off the app when not used for 1 minute (default settings). Then, when accessed next time, a new R session is initialized which in this case means loading quite big .RDS files. The result is a long waiting time for the server response. So my initial guess that there is some connection with the data caching was wrong.
I solved the issue by turning the idle timeout off as described here or here.
Thank you for your help.