(a^8/3 x b^5/7)^7/8
simplify (a^(8/3) * b^(5/7))^(7/8) Look at how I have bracketted your terms. If this is what you meant, then: (a^(8/3) * b^(5/7))^(7/8) = a^(7/3) b^(5/8)
