Following along with @HanOostdijk, the difference is whether R thinks it's dealing with a floating point number or an integer. In general, a fractional power of a negative number is undefined.

I see. Trying the examples of @HanOostdijk I received NaNs. I think I can deal with it multiplying log10(x) by -1 before and then multiplying again by -1 after ^0.5
Thanks!

I wanted to produce an adsorption-isotherm after Freundlich, that has the form Y = Kfr * X^(1/n)
with the nlsLM()function. Since I wanted to add the resulting fit into a log-scaled figure, I adjusted the formula like this: log10(Y) = Kfr *log10(X)^(1/n)
The input values of X and Y lay between zero and one. Anyway I'm also having troubles finding good starting points for Kfr and n, but that's another issue. Now seeing that log10(0<X<1)is a negative number and the fractional power of a negative number is undefined - as mentioned by @startz - I thought I can avoid it by doing this: log10(Y) ~ Kfr * (-1*((-1*log10(X))^(1/n)))

Haha I wasn't sure to be honest. But forget about it. With log-scaled axes I can simply use a linear model to fit the data and it works just fine. Anyway thanks for your help!