I suspect that this change is the cause:
rstudio-user@71439faf81d5:/tmp/rgeos$ geos-config --libs
-L/usr/lib/x86_64-linux-gnu -lgeos-3.5.1
rstudio-user@71439faf81d5:/tmp/rgeos$ geos-config --cclibs
-L/usr/lib/x86_64-linux-gnu -lgeos
rstudio-user@71439faf81d5:/tmp/rgeos$ grep GEOS_LIBS configure
GEOS_LIBS=`${GEOS_CONFIG} --cclibs`
PKG_LIBS="${PKG_LIBS} ${GEOS_LIBS} ${GEOS_CLIBS}"
rstudio-user@71439faf81d5:/tmp/rgeos$ dpkg-query --listfiles libgeos-3.5.1
/.
/usr
/usr/share
/usr/share/doc
/usr/share/doc/libgeos-3.5.1
/usr/share/doc/libgeos-3.5.1/changelog.Debian.gz
/usr/share/doc/libgeos-3.5.1/copyright
/usr/share/doc/libgeos-3.5.1/NEWS.Debian.gz
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libgeos-3.5.1.so
Try using the older version of rgeos
for now.