\(U = \frac a{x^2} - \frac b{x^3}\)
At equilibrium Net force
\(F_{net} (x) = 0\) .....(i)
\(F(x) = \frac d{dx} \left(\frac a{x^2} - \frac b{x^3}\right)\)
\(F(x) = -\frac{2a}{x^3} - \frac{(-3)b}{x^4}\)
\(F(x) = \frac{3b}{x^4} - \frac {2a}{x^3}\)
From (i),
\(\frac{3b}{x^4} - \frac{2a}{x^3} = 0\)
\(\frac{3b - 2ax}{x^4} = 0\)
\( 3{b} = 2ax\)
\(x = \frac{3b}{2a}\)