Hint:
the integral is lnx = the integral of (1*lnx)
do that first.
You need to use integration by parts twice.
I have not done it, I don't have time at present, but that is how I would proceed.
\displaystyle \int [ln(x)]^2 dx
\(\displaystyle \int [ln(x)]^2 dx\\ =\displaystyle \int [ln(x)] [ln(x)] dx\\ u=lnx\qquad v'=lnx\\ u'=\frac{1}{x}\qquad v=?\\~\\ ---------------------\\ \displaystyle \int [ln(x)] dx\\ =\displaystyle \int 1*[ln(x)] dx\\ p=lnx\qquad q'=1\\ p'=\frac{1}{x}\qquad q=x\\ so\\ \displaystyle \int 1*[ln(x)] dx\\ =xlnx-\int \frac{1}{x}*x\;dx\\ =xlnx-\int 1\;dx\\ =xlnx-x\\~\\ \text {So v=xlnx-x}\\~\\ ----------------\\ \)
Going back to the beginning
\(\displaystyle \int [ln(x)]^2 dx\\ =\displaystyle \int [ln(x)] [ln(x)] dx\\ \qquad u=lnx\qquad v'=lnx\\ \qquad u'=\frac{1}{x}\qquad v=xlnx-x\\~\\ =uv-\int vu' dx \;\;+c \)
And you can finish it
LaTex:
\displaystyle \int [ln(x)]^2 dx\\
=\displaystyle \int [ln(x)] [ln(x)] dx\\
u=lnx\qquad v'=lnx\\
u'=\frac{1}{x}\qquad v=?\\~\\
---------------------\\
\displaystyle \int [ln(x)] dx\\
=\displaystyle \int 1*[ln(x)] dx\\
p=lnx\qquad q'=1\\
p'=\frac{1}{x}\qquad q=x\\
so\\
\displaystyle \int 1*[ln(x)] dx\\
=xlnx-\int \frac{1}{x}*x\;dx\\
=xlnx-\int 1\;dx\\
=xlnx-x\\~\\
\text {So v=xlnx-x}\\~\\
----------------\\
\(\newcommand{\dint}{\displaystyle\int} \begin{array}{rcl} \dint (\ln x)^2 \,dx &=& x(\ln x)^2 - \dint x\,d\left((\ln x)^2\right)\\ &=& x(\ln x)^2 - \dint x\cdot \dfrac{2\ln x}{x}\,dx\\ &=& x(\ln x)^2 - 2\dint \ln x\,dx\\ &=& x(\ln x)^2 - 2\left(x\ln x - \dint x\,d(\ln x)\right)\\ &=& \cdots \end{array}\)
I will leave the rest to you.