Hi all,
Unfortunately, at this time, there isn't an ability to modify the robots.txt for shinyapps.io applications. We currently block crawling deliberately because we found that the robot crawlers would cause applications to start, but would not follow the sockjs connections to get actual content. Because of this, we were paying the cost of starting the application, waiting for it to become idle, and then stopping, without getting any benefit from the crawl.
That said, we are looking into options to allow this in the future in some fashion and we'll see what we can do going forward.