Since you mentioned calculus, perhaps you're supposed to find the area by integration.
The square is circumscribed by a circle of radius 6, so its diagonal (equal to the diameter) has length 12. The lengths of a square's side and its diagonal occur in a ratio of 1 to sqrt(2), so the square has side length 6sqrt(2). This means its sides occur on the lines
and
.
Let
be the region bounded by the line
and the circle
(the rightmost blue region). The right side of the circle can be expressed in terms of
as a function of
:
![x^2+y^2=36\implies x=√(36-y^2)](https://img.qammunity.org/2020/formulas/mathematics/college/9tl3q8lhncoq8plfzu8fs91njm7jpn3j4k.png)
Then the area of this circular segment is
![\displaystyle\iint_R\mathrm dA=\int_(-3\sqrt2)^(3\sqrt2)\int_(3\sqrt2)^(√(36-y^2))\,\mathrm dx\,\mathrm dy](https://img.qammunity.org/2020/formulas/mathematics/college/n2xouqvktr22ti8tg1u30r2dozqy0p7l2e.png)
![=\displaystyle\int_(-3\sqrt2)^(3\sqrt2)(√(36-y^2)-3\sqrt2)\,\mathrm dy](https://img.qammunity.org/2020/formulas/mathematics/college/tkvsjru77trkul15civtm7socxvgkhcuss.png)
Substitute
, so that
![\mathrm dy=6\cos t\,\mathrm dt](https://img.qammunity.org/2020/formulas/mathematics/college/7tuavg5239pbmrt4vwj24jhkks1cyfga3o.png)
![=\displaystyle\int_(-\pi/4)^(\pi/4)6\cos t(√(36-(6\sin t)^2)-3\sqrt2)\,\mathrm dt](https://img.qammunity.org/2020/formulas/mathematics/college/g2h8lfi0hbvgj37yj9ux17e18bo7nd3gab.png)
![=\displaystyle\int_(-\pi/4)^(\pi/4)(36\cos^2t-18\sqrt2\cos t)\,\mathrm dt=9\pi-18](https://img.qammunity.org/2020/formulas/mathematics/college/e4vw7huqenglxawudhy1xq1ahwwhxam6tk.png)
Then the area of the entire blue region is 4 times this, a total of
.
Alternatively, you can compute the area of
in polar coordinates. The line
becomes
, while the circle is given by
. The two curves intersect at
, so that
![\displaystyle\iint_R\mathrm dA=\int_(-\pi/4)^(\pi/4)\int_(3\sqrt2\sec\theta)^6r\,\mathrm dr\,\mathrm d\theta](https://img.qammunity.org/2020/formulas/mathematics/college/crlt1bs5mnwmg84lq090d7d1ezo7mo8xww.png)
![=\displaystyle\frac12\int_(-\pi/4)^(\pi/4)(36-18\sec^2\theta)\,\mathrm d\theta=9\pi-18](https://img.qammunity.org/2020/formulas/mathematics/college/1hcwxf4rv65st7ua24mxtwdgfrq6ky623a.png)
so again the total area would be
.
Or you can omit using calculus altogether and rely on some basic geometric facts. The region
is a circular segment subtended by a central angle of
radians. Then its area is
![\frac{6^2\left(\frac\pi2-\sin\frac\pi2\right)}2=9\pi-18](https://img.qammunity.org/2020/formulas/mathematics/college/9ecr3c6qr5ee4n8e5uf9tn58enibq31hxf.png)
so the total area is, once again,
.
An even simpler way is to subtract the area of the square from the area of the circle.
![\pi6^2-(6\sqrt2)^2=36\pi-72](https://img.qammunity.org/2020/formulas/mathematics/college/erjk63s3vozvlu9yr6lfbl71wrb8tnmy9c.png)