I anyone is still interested, I did decide to leave travis for Docker, and I have gotten part of my life back .
This process has been made a lot easier and more appealing by the recently announced GitHub actions, which allows you to run arbitrary Docker containers as a CI/CD of sorts directly on GitHub.
This by no means solves all of the limitations of Docker mentioned by @cole, but has greatly reduced the friction around using Docker, and thus made it yet more appealing.
I now develop inside the same container that I also use for CI/CD on GitHub actions, which is awesome. No more duplicate environment provisioning. (Of course, many of the caveats remain: Docker images are reproducible, but not Dockerfiles
per se).
Anyway, I went ahead and filled the gaping whole left in my life by no longer wrestling with TravisCI by developing some GitHub actions for R and an accompanying package:
Announcement here: