If you're willing to add another dependency, take a look at the xaringanExtra package https://github.com/gadenbuie/xaringanExtra#-logo
You're able to see what it would look like here https://pkg.garrickadenbuie.com/xaringanExtra/logo/#1
The use_logo function from the package would add the logo to every slide, but you can probably use CSS and set the first and inverted slides to display: none. Maybe @grrrck already has thought of that within the package.