There must be 2 math books on either side, and there are 3 to choose from, so there are 3 possible books for one end, and 2 for the other. Therefore there are 6 possible ways to arrange the books.
For every possible way to arrange the two math books at either end, there is a number of possibilities to sort the 1 math textbook and 5 history textbooks. Then to find the possible ways of arranging you would do 6! = 720
720 x 6 = 4320 ways to arrange the books
https://web2.0calc.com/questions/more-questions-halp has the same problem