Block direct access to 3838 shiny server port

I have managed to setup up an Auth0 authentication proxy on my shiny server (ubuntu, digital ocean). It works well when accessing the applications via their nginx proxies but it seems that the authentication step can be bypassed by accessing the shiny port directly e.g.

Is there any way to block direct access to the port in the shiny server config, or limit the access to certain IP addresses?