I dont know what modelplot is , or how it constructs a ggplot. I assumed it would set x and y aes() , but maybe it doesnt; or maybe it does and then unsets them. if it doesnt it might be up to you to set it.
aes(x=x,y=y)
In general its hard to answer technical questions about what code should be on data we don't have. Reprex is best practice in question answering as it reduces friction in finding solutions.