This kind of stuff - called "composite functions" - used to give me trouble, as well. I couldn't "see it" for awhile.
Let me see if I can help you understand it the way I finally saw it.
Suppose we had f(x) = 2x, and I wanted to know what f(2) was?? Well, x is like a "box" that's holding something inside. What is this something??
We could write f(x) = 2[ ] where the [ ] is the "box"......So. what are we putting in this box?? The answer is the "2" in f(2) !!!
So we have f(2) = 2[ ] = 2[2] = 4....and that's what f(2) "evaluates to!!
Okay....suppose we had f(x) = x^2 + 3x + 4, and I wanted to know what f(3) was??
So we have f(3) = [3]^2 + 3[3] + 4 = 9 + 9 + 4 = 22......so f(3) would evaluate to 22 .......note that, the "4" doesn't have any "box" associated with it because there is no "x" on this term !!!!
With me so far??
Now........ suppose instead that I was given two functions "f" and "g" so that f(x) = 3x and g(x) = 2x
We can rewrite these using our "boxes" as f(x) = 3[ ] and g(x) = 2[ ]
OK......here comes the "punchline"...so to speak!!
What would f(g) be?? Well...if f(3) meant that I was putting "3" into the box in "f,' then f(g) must mean that I'm putting the function "g" into the "box." in function "f" !!!!!
So we have f(g) =3[g] = 3[2x] = 6x........!!!
Alright....what would g(f) be??.......
We have..........g[f] = 2[f[ = 2[3x] = 6x........Note that all I did was to "stick" the function "f" - in this case, 3x - into the "box" in function "g" .......
So...what we're REALLY doing is, instead of putting a NUMBER into the "x's," we're putting a "function' into them!!!
Let's look at your problem, now. (finally!!!)
find [goh](x)
g(x)=x+3
h(x)=x^2+5x-7
OK.........let g(x) be written as [ ] + 3
Now......the notation (g o h)(x) just means the same thing as g(h).....Notice that the SECOND function listed - (h) - is being put into the FIRST one - (g)!!
Well.....let's do that now!!!
We have.........g(h) = [h] + 3 = [x^2 + 5x - 7] + 3 = x^2 + 5x - 4.......!!!!........ And that's your answer
Additionally. what if we wanted to know (h o g)(x) = h(g), instead??
We have h(g) = [g]^2 + 5[g] - 7 = [x + 3]^2 + 5[x + 3] - 7 = ( x^2 + 6x + 9) + (5x + 15) - 7 = x^2 + 11x + 17 !!!!!!........ Notice, I just "stuck" "g" into "h'
And that's it!!!
I hope you see it, now. It's kind of a tough concept to master, at first!!