Been a while since I did calculus, but here goes.
(i) First prove that Ψ is an isomorphism and infinitely differentiable. Isomorphism follows by writing Ψ(x)={{1,0},{-1,1}}*x and noting that said matrix is invertible. Differentiability follows by Ψ being linear. V is given by finding the preimage Ψ
-1(U), that is, the set of points x in ℝ
2 for which {1,1}*Ψ(x)<u. Simple matrix multiplication!
(ii) This is just a change of variables using Ψ. The earlier matrix is the Jacobian of Ψ, which you can plug into the
usual formula.
(iii) Apply the equation of (ii) and note that the integrand is convolution. Write h(y2)=∫ f(y1)f(y2-y1)dy1, where y1 is integrated over ℝ. Denote the Fourier transform of h by H and recall that the transform of (1) convolution is multiplication and (2) exp(-x^2) is sqrt(pi)*exp(-pi^2*x^2). Therefore H(x)=pi*exp(-2*pi^2*x^2), which is easily inverse transformed into h(x)=sqrt(pi/2)*exp(-x^2/2). In all, ∫f(x1)f(x2)dx = ∫∫f(y1)f(y2-y1)dy1dy2 = ∫h(y2)dy2 = sqrt(pi/2) * ∫ f(y2/2)dy2. Substituting y2 ↦ y2*sqrt(2) yields the result.