I was working from the OP, which was good, but having to bring in the data separately slowed me down. Please use a complete reprex (see the FAQ).
I saw that the combination of points, error bars and line aesthetics was going to be problematic, but thought to let that question bubble.
I see that you've fixed the first issue, that the error bars extended beyond the range of the y-axis. The second issue, the mess with the line aes raises the question of what useful information the slope of change of Score from Contrast Ratio to Contrast Ratio provides. If it is something to draw attention to, a geom_smooth() aes with or without captures the movement without creating a lot of discontinuities. Finally, you might consider a log scale o the x-axis to allow the relatively greater number of observations below a contrast of 7.5 to have more breathing room.