Ancient Berland Circus

Observation 1: the circle is forced

Any regular polygon is cyclic, so all its vertices lie on one circle. Therefore the unknown arena must use the circumcircle of the triangle formed by the three found pillars. Its circumradius $R$ is completely determined by the input.

Observation 2: count gaps, not vertices

Assume the answer is a regular $n$-gon. Walking around the polygon, the three known vertices split the boundary into three chains of lengths $k_1, k_2, k_3 \ge 1$, with $k_1 + k_2 + k_3 = n.$

So the corresponding arcs on the circle have measures $2\pi k_1 / n, \quad 2\pi k_2 / n, \quad 2\pi k_3 / n.$

Now let the triangle angles be $A, B, C$. By the inscribed angle theorem, each triangle angle equals half of the opposite arc. Hence, after matching the order correctly, $A = \pi k_1 / n, \quad B = \pi k_2 / n, \quad C = \pi k_3 / n.$

So a candidate $n$ works iff each of $A, B, C$ is an integer multiple of $\pi / n$.

Conversely, if $A, B, C$ are multiples of $\pi/n$, then the three arcs $2A, 2B, 2C$ are multiples of $2\pi/n$, so the three given points are vertices of some regular $n$-gon on the same circumcircle.

That is the whole trick. The rest is just not messing up the floating point.

Observation 3: brute force $n$

The statement literally tells us the optimal polygon has at most $100$ sides. So don't invent a floating-point gcd ritual unless you enjoy pain. Just try every $n = 3, 4, \dots, 100$ and check whether $A / (\pi / n), \quad B / (\pi / n), \quad C / (\pi / n)$ are integers up to some small $\varepsilon$.

The first valid $n$ is optimal. For a fixed circumradius $R$, the area of an inscribed regular $n$-gon is $S_n = n R^2 \sin(2\pi / n) / 2.$

Let $x = 2\pi / n$. Then $S_n = \pi R^2 \cdot \sin x / x.$

On $(0, \pi)$, the function $\sin x / x$ decreases as $x$ increases. When $n$ grows, $x$ shrinks, so $S_n$ grows. Therefore the smallest valid $n$ gives the smallest area. Nice and clean.

Computing the pieces

Let the side lengths of the triangle be $a = |BC|, \quad b = |CA|, \quad c = |AB|.$

Compute the triangle angles with the law of cosines: $A = \arccos((b^2 + c^2 - a^2) / (2bc)),$ $B = \arccos((c^2 + a^2 - b^2) / (2ca)),$ $C = \arccos((a^2 + b^2 - c^2) / (2ab)).$

Clamp the cosine value to $[-1, 1]$ before calling $\arccos$, because floating point likes to be annoying.

Next, compute the triangle area $\Delta$ from the cross product: $\Delta = |(P_2 - P_1) \times (P_3 - P_1)| / 2.$

Then the circumradius is $R = abc / (4\Delta).$

Finally, once the smallest valid $n$ is found, the answer is $\text{area} = n R^2 \sin(2\pi / n) / 2.$

Complexity

We try at most $98$ values of $n$, each in $O(1)$. So the total complexity is $O(100),$ with $O(1)$ memory.

In other words: geometry problem, but the algorithm is basically constant time. Very Codeforces.