Since density is the ratio of mass to (in this case) area, we can find the mass of the triangular region [tex]\mathcal T[/tex] by computing the double integral of the density function over [tex]\mathcal T[/tex]:
[tex]\mathrm{mass}=\displaystyle\iint_{\mathcal T}\rho(x,y)\,\mathrm dx\,\mathrm dy[/tex]
The boundary of [tex]\mathcal T[/tex] is determined by a set of lines in the [tex]x,y[/tex] plane. One way to describe the region [tex]\mathcal T[/tex] is by the set of points,
[tex]\mathcal T=\left\{(x,y)\mid0\le x\le 3\,\land\,0\le y\le1-\dfrac x3\right\}[/tex]
So the mass is
[tex]\mathrm{mass}=\displaystyle\int_{x=0}^{x=3}\int_{y=0}^{y=1-x/3}(x^2+y^2)\,\mathrm dy\,\mathrm dx[/tex]
[tex]=\displaystyle\int_{x=0}^{x=3}\left(x^2y+\frac{y^3}3\right)\bigg|_{y=0}^{y=1-x/3}\,\mathrm dx[/tex]
[tex]=\displaystyle\int_{x=0}^{x=3}\left(x^2\left(1-\frac x3\right)+\frac{\left(1-\frac x3\right)^3}3\right)\,\mathrm dx[/tex]
[tex]=\displaystyle\frac1{81}\int_0^3(27-27x+90x^2-28x^3)\,\mathrm dx=\frac52[/tex]