The vectors A and B span a parallelogram. The area of this parallelogram is equal to the magnitude of the cross product, A × B. Cut this area in half and you get the area of the triangle of interest.
Recall that for any two vectors x and y, we have
||x × y|| = ||x|| ||y|| sin(θ)
where θ is the angle between the two vectors; also, recall that
x • y = ||x|| ||y|| cos(θ)
Use the dot-product identity to find the angle between A and B. Compute the dot product and magnitudes:
A • B = (3i + 10j - k) • (-i + 2j + 11k) = -3 + 20 - 11 = 6
||A|| = √(3² + 10² + (-1)²) = √110
||B|| = √((-1)² + 2² + 11²) = √126 = 3√14
Solve for the angle:
cos(θ) = 6/(√110 • 3√14) = 1/√385
θ = arccos(1/√385)
Now using the cross-product identity, we have
||A × B|| = √110 • 3√14 sin(arccos(1/√385)) = 3√1540 • √(384/385) = 48√6
and this is the area of the parallelogram. The area of the triangle in question is half of this, 24√6.
Alternatively, you can compute the cross product directly.
Recall that for any two vectors x and y,
x × x = 0
x × y = - (y × x)
and that the cross product is defined by the following rules:
i × j = k
j × k = i
k × i = j
Then the cross product is
A × B = (3i + 10j - k) × (-i + 2j + 11k)
A × B = -3 (i × i) - 10 (j × i) + (k × i)
… … … … … + 6 (i × j) + 20 (j × j) - 2 (k × j)
… … … … … + 33 (i × k) + 110 (j × k) - 11 (k × k)
A × B = 10k + j + 6k + 2i - 33j + 110i
A × B = 112i - 32j + 16k
Compute its magnitude:
||A × B|| = √(112² + (-32)² + 16²) = √288 = 48√6
and cut it in half to get the area of the triangle; again, you end up with 24√6.