Bridging Theory and Expert Judgment: Advances in Polarization Metrics and Their Empirical Correlations

In our investigation of opinion divergence dynamics, we conducted a comprehensive experimental study that bridges theoretical models with expert evaluation. Our objective was to elucidate the relationships among various polarization measures and to assess their alignment with expert judgment. I personally carried out all the experiments and authored this article as a means of both disseminating our findings and maintaining a detailed record for future reference. It is important to note, however, that this work is the result of a collaborative effort by the polarization measures team from the Promueva group, which includes Frank D. Valencia, Juan Francisco Díaz, Jesús Aranda, and Carlos Pinzón. Each of these team members has significantly contributed to guiding the experimental process, providing critical feedback and advice that have shaped the direction of this project. I am truly grateful for the opportunity to work alongside such dedicated colleagues.

We begin by formally defining a polarization measure as a function $P:\mathscr{D}\to\Reals_{\geq 0}$, where for any consensus distribution $\pi\in \mathscr{D}$, we have $P(\pi)=0$, and for any extremal distribution $\pi \in \mathscr{D}_k$, we have $P(\pi) \geq P(\pi')$ for all $\pi'\in \mathscr{D}_k$. Here, $\mathscr{D}$ denotes the set of all opinion distributions. An opinion distribution is defined as a function $\pi: [0,1]\to \Reals_{\geq 0}$ with finite, non-empty support $\text{s}(\pi)=\{ x\in [0,1]\mid \pi(x)>0 \}$. For each opinion value $x$, $\pi(x)$ represents its associated weight, and the total mass of $\pi$ is defined as

$$\Sigma \pi = \sum_{x \in \text{s}(\pi)} \pi(x).$$

For notational clarity, we represent an opinion distribution as an ordered pair

$$(\vec{\pi},\vec{x})=((\pi_1,\ldots,\pi_n),(x_1,\ldots,x_n))\in \Reals^{n}_{\geq 0} \times [0,1]^n,$$

where $x_1 < \cdots < x_n$ and $\Sigma\pi=1$. This representation uniquely characterizes the opinion distribution $\pi$ through $\pi(x_i)=\pi_i$ for each $i$, with total mass $\Sigma \pi = \sum_{i=1}^n \pi_i$.

To analyze the correlation between various polarization measures from the literature (implemented in our package), we constructed an experimental framework examining all possible opinion distributions $\pi\in\mathscr{D}_{100}$ corresponding to a population of size $k = 100$ distributed across $n = 5$ bins, representing a Likert scale. The generation of these distributions employed the classical Stars and Bars combinatorial method: for any pair of positive integers $k$ and $n$, the number of $n$-tuples of non-negative integers summing to $k$ is given by

$$\binom{k+n-1}{n-1},$$

corresponding to the weak compositions of $k$. For fixed $n$, this sequence appears along the $(n-1)$st diagonal of Pascal's triangle.

In our analysis of these polarization measures, we examined their inter-correlations using Kendall's tau-b coefficient. Notably, since all measures under consideration satisfy a symmetry property where mirror images defined by $\text{tuple}(\pi)\leqslant\text{tuple}(\pi^{\mathrm{rev}})$ are equivalent, we reduced our analysis from 4,598,126 to 2,299,063 unique distributions. The Kendall tau-b coefficient, introduced by Kendall (1945), was specifically chosen for its ability to handle ties in ranking data.

$$\tau_B = \frac{N_c - N_d}{\sqrt{(N_c + N_d + T_x)(N_c + N_d + T_y)}},$$

where $N_c$ represents the number of concordant pairs, $N_d$ the number of discordant pairs, and $T_x$, $T_y$ account for ties in the first and second quantity respectively. Two pairs of observations $(x_i,y_i)$ and $(x_j,y_j)$ are considered concordant if both $x_i > x_j$ and $y_i > y_j$, or if both $x_i < x_j$ and $y_i < y_j$. Conversely, they are discordant if $x_i > x_j$ and $y_i < y_j$, or if $x_i < x_j$ and $y_i > y_j$.

This coefficient provides several advantages over other correlation measures. First, it is robust to outliers since it depends only on the relative ordering of values rather than their magnitudes. Second, it explicitly accounts for ties in the data through the terms $T_x$ and $T_y$, making it particularly suitable for ordinal data or data with repeated values. Third, it has a clear probabilistic interpretation: $\tau_B$ represents the difference between the probability of observing concordant and discordant pairs in the data.

The coefficient ranges from -1 to 1, where 1 indicates perfect positive correlation (the rankings are identical), -1 indicates perfect negative correlation (the rankings are exactly opposite), and 0 indicates no correlation (the rankings are independent). In the context of our analysis, Kendall's tau-b provides a robust way to assess how different polarization measures rank distributions, accounting for the possibility of measures assigning equal polarization values to different distributions.

Correlation Analysis of Polarization Measures

To implement our analysis, we developed the following computational framework:

def main(k: int = 100, n: int = 5):
    calculator = MeasureCalculator()
    total_distributions = count_distributions(k, n)

    for i, (x, weights) in enumerate(generate_distributions(k, n)):
        calculator.process_distribution(x, weights)

    values = calculator.get_values()
    correlation_matrix = compute_kendall_matrix(values)

Our computational analysis revealed several significant patterns in the relationships among polarization measures. Most notably, we confirmed empirically the theoretical equivalence previously demonstrated between $\text{EMD}$ and $\text{MEC}_{1,1}$, as both measures exhibit identical behavior across all distributions. Furthermore, our findings align with previous research presented in "A New Opinion Polarization Index Developed by Integrating Expert Judgments" (Frontiers in Psychology, 2021), regarding the diminished effectiveness of the $\text{ER}_{\alpha}$ measure when employing high values of the identification parameter $\alpha$.

The correlation between $\text{MEC}_{2,1.15}$ and its normalized variant shows an expected near-perfect correlation ($\tau_B = 0.999$), with the negligible difference likely attributable to floating-point arithmetic precision limitations in numerical computations. A particularly notable observation is the strong correlation between $\text{BiPol}$ and $\text{Shannon}$'s polarization measure ($\tau_B = 0.921$). While $\text{BiPol}$ maximizes the weighted sum of pairwise distances between two disjoint subsets of opinions, $\text{Shannon}$'s measure quantifies polarization through the entropy of opinion distances from the mean, both capturing different mathematical aspects of opinion dispersion. The $\text{ER}_{1.6}$ measure shows consistently lower correlations with other measures ($\tau_B < 0.4$ in most cases), indicating it captures fundamentally different aspects of polarization. The expert-based interpolated measure, given by

$$\text{Experts}(\vec{x}) = \frac{2.14x_2x_4 + 2.70(x_1x_4 + x_2x_5)+3.96x_1x_5}{0.0099n^2},$$

shows moderate correlations with most measures ($0.5 < \tau < 0.7$), with strongest agreement with $\text{BiPol}$ ($\tau_B = 0.705$) and $\text{Shannon}$ ($\tau_B = 0.697$). However, this measure exhibits several theoretical shortcomings. Most notably, it yields zero polarization when either extreme pairs $(x_1,x_2)$ or $(x_4,x_5)$ are zero, contradicting basic polarization principles. Such behavior, among other issues, highlights the limitations of direct linear interpolation approaches in capturing the complex nature of opinion polarization.

These findings suggest that $\mathrm{MEC}_{\alpha,\beta}$'s parameters could potentially be tuned to approximate other polarization measures. Future work could explore this possibility systematically, aiming to find parameter combinations that minimize the distance to target measures while maintaining desirable theoretical properties. Such an investigation could lead to a unified framework where $\mathrm{MEC}_{\alpha,\beta}$ serves as a generalized polarization measure, capable of reproducing various existing indices through appropriate parameter selection.

Expert Validation of Polarization Measures

To validate different polarization measures against expert judgment, we implemented the following analysis framework:

def validate_with_experts():
    # Load expert data and distributions
    data = ValidationData()
    distributions = data.get_normalized_distributions()
    expert_scores = data.expert_scores
    
    # Calculate polarization values for each measure
    calculator = ValidationCalculator()
    calculator.process_distributions(data.x_values, distributions)
    measure_values = calculator.get_values()
    
    # Compute Kendall's tau-B correlations with expert scores
    correlations = {}
    for measure_name, values in measure_values.items():
        tau, _ = kendalltau(expert_scores, values)
        correlations[measure_name] = tau

A fundamental aspect in validating polarization measures is their ability to capture expert judgment regarding what constitutes a polarized state of opinions. In this context, a seminal study by Koudenburg et al. (2021) collected evaluations from 60 international experts on polarization for a set of 15 opinion distributions represented as histograms.

The experts rated each distribution on a scale from 0 (not polarized) to 100 (completely polarized), thereby establishing a reference order to validate different polarization measures. This original order of the histograms $(3, 10, 1, 12, 13, 15, 5, 6, 9, 11, 2, 7, 8, 4, 14)$ reflects the expert consensus on the relative levels of polarization for each distribution. To validate the various polarization measures available in the literature, we compared how each measure would order these same 15 histograms and computed the Kendall $\tau_B$ between these orders and the original order provided by the experts. This methodology allows us to assess how well each measure captures the expert intuition about polarization.

As can be observed in the graph, some measures exhibit a stronger correlation with expert judgment than others. Notably, $\text{MEC}_{2,1.15}$ and $\text{ER}_{0.8}$ with correlations of approximately 0.89. The $\text{BiPol}$ measure shows a moderate correlation of 0.77, while other measures such as $\text{VanDerEijk}$ and $\text{ER}_{1.6}$ exhibit lower correlations of 0.67 and 0.52 respectively.

These results suggest that not all polarization measures capture expert intuition equally well regarding what constitutes a polarized state. The differences in correlations reveal the strengths and limitations of each measure in reflecting the specific aspects of polarization that experts consider important. Further research remains to be done to determine the optimal $\alpha$ and $\beta$ parameters for $\text{MEC}_{\alpha,\beta}$, both theoretically and experimentally, that could potentially better align with expert judgment while maintaining desirable mathematical properties.

Axiomatic Foundations and Measure Properties

Having examined the empirical correlations between different polarization measures and explored their alignment with expert judgment, we now turn to a more theoretical analysis of how well existing measures satisfy the fundamental axioms proposed by Esteban and Ray for polarization measurement. These axioms provide a rigorous mathematical framework for evaluating the theoretical soundness of polarization measures and their ability to capture key properties of opinion distributions.

The Esteban-Ray measure represented a breakthrough in the formal measurement of polarization by proposing a theoretically grounded axiomatic framework. Their measure takes the form:

$$\text{ER}_{\alpha}(\boldsymbol{\pi},\boldsymbol{y}) = K \sum_{x \in s(\pi)} \sum_{y \in s(\pi)} \pi(x)^{1+\alpha}\pi(y)|x - y|$$

for some constants $K > 0$ and $\alpha \in (0,\alpha^*]$ where $\alpha^* \approx 1.6$. This measure was characterized by three fundamental axioms that capture key intuitions about what constitutes polarization:

Axiom 1. For any $p > 0$ and any $x > 0$, there exist $\varepsilon > 0$ and $\mu > 0$ such that, for any $y > x$ and any $q < p$ with $y - x < \varepsilon$ and $0 < q < \mu p$,

$$P(((p,q,q),(0,x,y))) < P\left(\left(p,2q\right),\left(0,\frac{x+y}{2}\right)\right).$$

Axiom 2. For any $p,q,r > 0$ with $p > r$, any $x,y > 0$ with $|y-x| < x < y$, and any $\Delta \in (0,y-x)$,

$$P((p,q,r),(0,x,y)) < P((p,q,r),(0,x+\Delta,y)).$$

Axiom 3. For any $p,q > 0$, any $x,y > 0$ with $x = y-x$, and any $\Delta \in (0,q/2)$,

$$P((p,q,p),(0,x,y)) < P((p+\Delta,q-2\Delta,p+\Delta),(0,x,y)).$$

In addition to these foundational axioms, we also include a property proposed by Professor Jesús Aranda. Derived as an intermediate point from Axiom 3, this property—temporarily referred to as Property 1—captures an additional nuance in the behavior of polarization measures.

Property 1. For any $p,q \ge 0$, any $x,y,z \ge 0$ with $x = y - x$ and $x \ge z$, and any $\Delta \in (0, q/2)$,

$$P((p,q,p), (0,x,y)) < P((p, \Delta, q - 2\Delta, \Delta, p), (0,x-z, x, x+z, y)).$$

Our goal is to analyze how different measures from the literature align with these fundamental axioms and with Property 1. This analysis will provide insights into the theoretical underpinnings of each measure and their ability to capture the essential properties of polarization as formalized by Esteban and Ray, as well as the additional nuance proposed by Professor Aranda.

For the moment, we conducted a preliminary study using our set of opinion distributions. In this study, each polarization measure (namely MEC, EMD, ER, Shannon, VanDerEijk, and BiPol) was evaluated against several theoretical conditions—namely, Condition H, Axiom 1, Axiom 2, Axiom 3, and Property 1. (For a detailed implementation, please refer to our Colab Notebook

The framework of the test is summarized as follows:

def test_all_measures():
    measures = {
        'MEC': MECNormalized(2, 1.15),
        'EMD': EMDPol(),
        'ER': EstebanRay(0.8),
        'Shannon': ShannonPol(),
        'VanDerEijk': VanDerEijkPol(),
        'BiPol': BiPol()
    }
    results = {}
    for name, measure in measures.items():
        results[name] = {
            'Axiom 1': test_axiom1(measure)['passes'],
            'Axiom 2': test_axiom2(measure)['passes'],
            'Axiom 3': test_axiom3(measure)['passes'],
            'Property 1': test_property1(measure)['passes']
        }
    df_results = pd.DataFrame(results).T
    return results

Below is the summary table of the test results:

Measure	Axiom 1	Axiom 2	Axiom 3	Property 1
$\text{MEC}_{2,1.15}$	✓	✓	✓	✓
$\text{EMD}$	✗	✓	✓	✓
$\text{ER}_{0.8}$	✓	✓	✓	✗
$\text{Shannon}$	✗	✓	✓	✓
$\text{VanDerEijk}$	✓	✓	✓	✓
$\text{BiPol}$	✗	✓	✓	✓

Although this study is preliminary, it already provides counterexamples that show some measures do not fully satisfy the axioms. For instance, $\text{EMD}$ fails Axiom 1 since its polarization value remains $0.3$ when comparing the original distribution $[0.6,\;0,\;0.2,\;0,\;0.2]$ with the modified distribution $[0.6,\;0,\;0,\;0.4,\;0]$. Similarly, $\text{ER}$ violates Property 1 as its value drops from $0.53$ to $0.44$ when the distribution $[0.25,\;0,\;0.5,\;0,\;0.25]$ is altered to $[0.25,\;0.1,\;0.3,\;0.1,\;0.25]$ under parameters $p=0.25$, $q=0.5$, $\Delta=0.1$, and $z=0.1$; likewise, $\text{Shannon}$ exhibits an Axiom 1 failure (from $0.72$ to $0.65$), and $\text{BiPol}$ does not preserve the order of ties. In contrast, $\text{MEC}$ satisfies all the axioms and Property 1—aligning with expert opinion at 88%—whereas $\text{VanDerEijk}$, despite meeting the axioms, corresponds to experts only 67% of the time. Proofs for $\text{MEC}$’s compliance are pending publication, and future studies will investigate these properties in more depth, including Condition H.

Threshold Analysis for Polarization Classification

A preliminary analysis was performed to establish thresholds for classifying polarization measure values into low, medium, and high categories. In this study, three measures were considered: $\text{MEC}_{2,1.15}$, $\text{BiPol}$, and $\text{Esteban--Ray}_{0.8}$. For each measure, thresholds were derived using multiple percentile-based schemes. In particular, the 33rd/66th split, the 25th/50th/75th split, and the 20th/40th/60th/80th split were applied to the computed values across all distributions.

The image illustrates the distribution of polarization values obtained for three measures. The first row corresponds to $\text{MEC}_{2,1.15}$, the second row to $\text{BiPol}$, and the third row to $\text{ER}_{0.8}$. In this context, the first set of results in the table (the first "column" of splits) represents the 33rd/66th split, the next set corresponds to the 25th/50th/75th split, and the final set pertains to the 20th/40th/60th/80th split. This arrangement allows us to compare how the thresholds, descriptive statistics, and counts change with each percentile partitioning strategy.

Below is the table summarizing these results (excluding the value ranges):

Measure & Scheme	Thresholds	Descriptive Stats	Counts
MEC, 33rd/66th	0.1971 / 0.2922	[0.0000, 0.2574, 0.2408, 1.0000]	(758910, 758909, 781907)
BiPol, 33rd/66th	0.4726 / 0.5994	[0.0000, 0.5379, 0.5376, 1.0000]	(758979, 758859, 781888)
EstebanRay, 33rd/66th	0.3609 / 0.4466	[0.0000, 0.4157, 0.4031, 1.0000]	(758910, 758909, 781907)
MEC, 25th/50th/75th	0.1747 / 0.2408 / 0.3274	[0.0000, 0.2574, 0.2408, 1.0000]	(574932, 574931, 574931, 574932)
BiPol, 25th/50th/75th	0.4380 / 0.5376 / 0.6383	[0.0000, 0.5379, 0.5376, 1.0000]	(575029, 574883, 574967, 574847)
EstebanRay, 25th/50th/75th	0.3407 / 0.4031 / 0.4761	[0.0000, 0.4157, 0.4031, 1.0000]	(574932, 574931, 574931, 574932)
MEC, 20th/40th/60th/80th	0.1581 / 0.2141 / 0.2716 / 0.3503	[0.0000, 0.2574, 0.2408, 1.0000]	(459946, 459945, 459945, 459945, 459945)
BiPol, 20th/40th/60th/80th	0.4140 / 0.5000 / 0.5756 / 0.6627	[0.0000, 0.5379, 0.5376, 1.0000]	(460296, 459784, 460139, 459566, 459941)
EstebanRay, 20th/40th/60th/80th	0.3275 / 0.3781 / 0.4295 / 0.4965	[0.0000, 0.4157, 0.4031, 1.0000]	(459946, 459945, 459945, 459945, 459945)

In this table, the "Descriptive Stats" column follows the format [Min, Mean, Median, Max]. The "Counts" column indicates the number of distributions falling within each category defined by the respective thresholds. For instance, under a 33rd/66th split, there are three categories (Low, Medium, High), whereas the 25th/50th/75th and 20th/40th/60th/80th splits yield four and five categories, respectively.

The detailed percentile analyses have provided valuable insight into the overall dispersion and central tendency of each measure’s distribution. However, they also reveal certain limitations inherent in a uniform, percentile-based approach. While percentiles offer a useful general intuition, they impose rigid, pre-determined divisions on the data that may not capture the natural clustering or subtle nuances present in the distribution of polarization values. This is particularly evident for measures such as $\text{MEC}_{2,1.14}$, $\text{ER}_{0.8}$, and $\text{BiPol}$, where extreme polarization values tend to be concentrated within a much narrower interval than a uniform split would suggest.

These disadvantages motivated us to explore an alternative approach using the $k$-means clustering algorithm. The $k$-means algorithm is a well-established iterative refinement technique that partitions data into $k$ clusters. It begins with an initial set of $k$ means, denoted by $m_1^{(1)}, m_2^{(1)}, \dots, m_k^{(1)}$, and then alternates between two key steps. In the assignment step, each observation is allocated to the cluster with the nearest mean based on the least squared Euclidean distance. This can be mathematically expressed for iteration $t$ as

$$S_i^{(t)} = \left\{ x_p : \| x_p - m_i^{(t)} \|^2 \le \| x_p - m_j^{(t)} \|^2 \quad \forall\, j,\, 1 \le j \le k \right\},$$

where each observation $x_p$ is assigned exclusively to one cluster, even if it might be equidistant from more than one mean. Following this, the update step recalculates the means (or centroids) for each cluster using the formula

$$m_i^{(t+1)} = \frac{1}{|S_i^{(t)}|} \sum_{x_j \in S_i^{(t)}} x_j.$$

With each iteration, the algorithm minimizes the within-cluster sum of squares (WCSS), ensuring a nonnegative, monotonically decreasing sequence that leads to convergence, although not necessarily to the global optimum. This iterative process, often referred to as Lloyd's algorithm or simply as $k$-means, adapts to the intrinsic structure of the data by uncovering natural groupings that a uniform percentile-based approach might miss.

The k-means clustering results image shows how polarization values naturally group for three measures. The first row corresponds to $\text{MEC}_{2,1.15}$, the second to $\text{BiPol}$, and the third to $\text{ER}_{0.8}$. Within each row, the first column represents the case when $k=3$, the second when $k=4$, and the third when $k=5$. This presentation allows us to observe how the grouping structure evolves as the number of clusters increases. The counts of observations assigned to each cluster, presented in the same format as in the percentile analysis, provide additional insight into the distribution of values across the clusters. The table below summarizes these results.

Measure (k)	Cluster Centroids	Counts	Value Ranges
MEC (k=3)	[0.1546, 0.2919, 0.4683]	(1002929, 960752, 336045)	[0.0000, 0.2233] / [0.2233, 0.3801] / [0.3801, 1.0000]
BiPol (k=3)	[0.3652, 0.5398, 0.7137]	(649188, 1016983, 633555)	[0.0000, 0.4525] / [0.4526, 0.6267] / [0.6268, 1.0000]
Esteban-Ray (k=3)	[0.3202, 0.4380, 0.5902]	(928943, 993213, 377570)	[0.0000, 0.3791] / [0.3791, 0.5141] / [0.5141, 1.0000]
MEC (k=4)	[0.1182, 0.2255, 0.3428, 0.5114]	(554166, 911036, 633659, 200865)	[0.0000, 0.1718] / [0.1718, 0.2841] / [0.2841, 0.4271] / [0.4271, 1.0000]
BiPol (k=4)	[0.3208, 0.4672, 0.5963, 0.7442]	(376399, 736586, 760015, 426726)	[0.0000, 0.3940] / [0.3941, 0.5317] / [0.5318, 0.6702] / [0.6703, 1.0000]
Esteban-Ray (k=4)	[0.3027, 0.3975, 0.4959, 0.6361]	(659528, 859091, 578046, 203061)	[0.0000, 0.3501] / [0.3501, 0.4467] / [0.4467, 0.5660] / [0.5660, 1.0000]
MEC (k=5)	[0.1040, 0.1987, 0.2855, 0.3883, 0.5495]	(417349, 742128, 614763, 398368, 127118)	[0.0000, 0.1513] / [0.1513, 0.2421] / [0.2421, 0.3369] / [0.3369, 0.4689] / [0.4689, 1.0000]
BiPol (k=5)	[0.2895, 0.4191, 0.5269, 0.6364, 0.7686]	(237904, 523153, 654893, 581750, 302026)	[0.0000, 0.3543] / [0.3544, 0.4730] / [0.4731, 0.5816] / [0.5817, 0.7024] / [0.7025, 1.0000]
Esteban-Ray (k=5)	[0.2803, 0.3588, 0.4358, 0.5259, 0.6582]	(395345, 701921, 656490, 401449, 144521)	[0.0000, 0.3196] / [0.3196, 0.3973] / [0.3973, 0.4809] / [0.4809, 0.5921] / [0.5921, 1.0000]

In our study, the application of $k$-means clustering confirmed that extreme polarization values are relatively rare and tend to cluster within a narrow interval. This finding highlights the advantage of using $k$-means to refine threshold determination, as it more accurately reflects the underlying distribution of the polarization measures. For future studies, we plan to examine the correlation among these measures within the identified clusters, which may reveal further insights into how different indices capture the same underlying phenomena of opinion divergence. Overall, our integrated approach—combining theoretical modeling, extensive computational analysis, and expert validation—provides a robust framework for understanding and classifying polarization. By bridging theoretical rigor with empirical evidence, this work lays the groundwork for more nuanced and reliable assessments of polarization in complex social systems, and it opens promising avenues for future research in the field.