In @FJCC 's solution to that question, setting the geom_text() parameter hjust to 0 did the trick, but if I replace geom_text() with geom_label(), we can see more easily that the text box is also shifted in the process:
Looking at the source code for geom_label(), it looks like the same parameters (hj, and vj) are used to calculate justification of the text and the rounded rectangle when constructing the grob that is plotted:
You might try using some of the annotation functions in {cowplot}: