😸 LLM Personality Distribution Analysis 😽

Personality Distribution Analysis: Research Methodology & Findings

Executive Summary

This analysis challenges the conventional treatment of Large Language Models (LLMs) as monolithic entities by demonstrating that models exhibit significant personality variation across contexts. We measure and visualize how individual models adopt different emotional response patterns depending on the personality type they are instructed to adopt, and compare these patterns across model generations and configurations.

Research Hypothesis

Primary Hypothesis: LLMs are not uniform personality entities. Instead, they possess the capacity to adopt diverse personality profiles, with measurable differences in how they:

3. Maintain or abandon consistency depending on model architecture

4. Differ based on training approach (base vs. chat-tuned, distilled vs. original)

Research Question: Which models demonstrate the greatest personality malleability, and which maintain more rigid personality structures? What can this tell us about the underlying training procedures and alignment mechanisms?

Dataset Description

Data Source

Experimental Design

Key Variables

#### Personality Types (7 total)

3. Reddit - Casual internet discussion

4. LinkedIn - Professional/networking tone

5. News - Journalistic/news-focused

6. Research - Academic research style

7. Vlog - Casual creator/video blog style

#### Emotions Measured (10 total)

Positive valence: joy, excitement, curiosity, admiration, approval, caring, optimism

Neutral/negative valence: confusion, nervousness, surprise

#### Model Categories

Methodology: Data Extraction & Processing

Step 1: Data Loading

# Load the emotion probabilities for all models and personalities
df = pd.read_csv('7000_sampling_emotions.csv')
# Columns: {emotion}_prob_a, {emotion}_prob_b for each of 10 emotions

Key insight: Each row represents a comparison between two text samples (A and B), with emotional response probabilities for both. We aggregate across both conditions to get a complete distribution.

Step 2: Personality Profile Computation

#### Core Calculation: Personality Probability

For each combination of emotion, personality, and model:

def compute_personality_probability(df, emotion, personality, model=None):
    # Filter to specific personality type
    subset = df[df['personality'] == personality]
    
    # Combine both conditions (A and B)
    prob_a = subset[f'{emotion}_prob_a']
    prob_b = subset[f'{emotion}_prob_b']
    combined = pd.concat([prob_a, prob_b])
    
    # Return distribution statistics
    return {
        'mean': combined.mean(),          # Central tendency
        'std': combined.std(),             # Variability
        'median': combined.median(),       # Robustness check
        'q25': combined.quantile(0.25),   # Distribution shape
        'q75': combined.quantile(0.75),   # Distribution shape
        'iqr': q75 - q25,                 # Interquartile range
        'range': (min, max),              # Full span
        'n': len(combined)                # Sample size
    }

Rationale: By combining observations from both conditions (A and B), we create a more stable estimate of how a model expresses a given emotion under a specific personality type, reducing noise from individual comparisons.

#### Personality Profile (10-dimensional)

For each personality type, we compute this calculation for all 10 emotions, creating a "personality fingerprint":

Example: Reddit personality
joy:       0.18 ± 0.05
excitement: 0.22 ± 0.06
confusion:  0.08 ± 0.04
curiosity:  0.25 ± 0.07
[... 6 more emotions ...]

This 10-D vector represents the complete emotional signature of the Reddit personality

Step 3: Consistency Analysis

#### Personality Consistency Metric

Measures how much a model maintains consistent emotional expression across personality contexts:

def compute_personality_consistency(df, emotion, model):
    # For each personality, compute emotion probability
    profiles = []
    for personality in PERSONALITY_TYPES:
        prob = compute_personality_probability(df, emotion, model, personality)
        profiles.append(prob['mean'])
    
    # Calculate variability across personalities
    consistency_cv = std(profiles) / mean(profiles)  # Coefficient of variation
    mean_range = max(profiles) - min(profiles)        # Range
    
    return {'consistency_cv': consistency_cv, 'mean_range': mean_range}

Interpretation:

Step 4: Personality Diversity Analysis

#### Euclidean Distance in Personality Space

Measures how spread out a model's personalities are in emotional space:

def compute_personality_diversity(df, model):
    # Get emotional profiles for all 7 personalities
    profiles = {personality: get_emotional_profile(personality, model) 
                for personality in PERSONALITY_TYPES}
    
    # Compute pairwise distances between personality profiles
    distances = []
    for personality_pair in all_pairs(PERSONALITY_TYPES):
        p1, p2 = personality_pair
        dist = euclidean_distance(profiles[p1], profiles[p2])
        distances.append(dist)
    
    # Aggregate
    return {
        'mean_diversity': mean(distances),     # Average separation
        'std_diversity': std(distances)         # Variability in separation
    }

Interpretation:

This reveals whether models have truly learned to inhabit different personalities or if they superficially adjust behavior while maintaining similar underlying emotional patterns.

Step 5: Emotion Variability Analysis

#### Coefficient of Variation by Emotion

Identifies which emotions are stable vs. variable across personality contexts:

for emotion in EMOTIONS:
    for personality in PERSONALITY_TYPES:
        stats = compute_personality_probability(emotion, personality)
        cv = stats['std'] / stats['mean']  # Coefficient of variation
        std = stats['std']                  # Standard deviation

Interpretation:

Step 6: Model Type & Generation Comparisons

#### Model Type Grouping

Aggregate by model category (base, chat, distilled, MOE) to reveal systematic differences:

type_data = []
for model_type, models_list in MODEL_GROUPS.items():
    for emotion, personality in all_pairs(EMOTIONS, PERSONALITY_TYPES):
        subset = df[df['model'].isin(models_list)]
        prob = compute_combined_probability(subset, emotion, personality)
        type_data.append({'model_type': model_type, 'emotion': emotion, 
                         'personality': personality, 'mean': prob})

#### Generation Analysis

Compare by model size (8B, 32B, 70B) to see if scaling affects personality expression:

def categorize_by_size(model):
    if '70b' in model: return '70B'
    elif '32b' in model: return '32B'
    elif '8b' in model: return '8B'
    elif 'moe' in model: return 'MOE'
    else: return 'Other'

Analysis Outputs: The 7 Visualizations

Visualization 1: Personality Profiles by Model

What it shows: Each model's emotional profile across all 7 personalities

Matrix: 11 models × (10 emotions × 7 personalities)

Each cell: Mean emotion probability for that (emotion, personality, model) combination

Color scale: Yellow→Red indicates low→high probability

Research value:

Visualization 2: Personality Consistency

What it shows: How much each model changes its emotional responses when adopting different personalities

Two metrics:

Research value:

Visualization 3: Personality Diversity

What it shows: How much models differentiate between personality types in emotional space

Metric: Mean Euclidean distance between all pairs of 7-personality emotional profiles

Color coding:

Research value:

Visualization 4: Emotion Variability

What it shows: Which emotions are most stable vs. variable across personality contexts

Two heatmaps:

Research value:

Visualization 5: Model Type Comparison

What it shows: Do base, chat, distilled, and MOE models express personalities differently?

Four separate heatmaps: One for each model type, showing (emotion × personality) matrix

Research value:

Visualization 6: Model Generation Comparison

What it shows: How does model size/generation affect personality expression?

Two plots:

Research value:

Visualization 7: Personality Radar Charts

What it shows: The unique emotional signature of each personality type

Format: 7 polar plots (one per personality), showing 10-D emotional profiles

Research value:

Key Findings & Research Implications

Finding 1: Models are Personality-Responsive, Not Personality-Neutral

Evidence: The consistency and diversity visualizations show non-trivial variation across personality types.

Implication:

Research strengthening: This directly supports your core hypothesis that models should not be treated as monolithic entities.

Finding 2: Personality Responsiveness Varies Significantly Across Model Types

Evidence: Chat models typically show higher personality diversity than base models; distilled models show unique patterns

Implication:

Research strengthening: Provides evidence that training procedure (RLHF vs. base) significantly affects personality expression characteristics.

Finding 3: Emotion-Specific Stability Differences

Evidence: Some emotions (confusion, curiosity) remain stable across personalities; others (approval, caring) vary significantly

Implication:

Research strengthening: Shows that personality effects aren't uniform across emotional dimensions — some emotions are personality-driven, others are model-driven.

Finding 4: Model Generation Effects on Personality

Evidence: Comparison of 8B, 32B, 70B models shows scaling effects

Implication:

Research strengthening: Provides empirical evidence about how personality expression changes with model capacity.

How This Strengthens Your Research Argument

1. Empirical Validation of Personality Heterogeneity

Your core claim: "Models should not be treated as single entities"

2. Demonstrates Personality is Contextual, Not Intrinsic

Your claim: "Personality varies by use-case, user, context"

3. Reveals Training Effects on Personality

Your claim: "Models' personality properties result from training procedure"

4. Identifies Personality "Hardness" Dimensions

5. Provides Tools for Future Research

6. Challenges Anthropomorphic Models of AI Alignment

Limitations & Caveats

1. Emotion as Personality Proxy

2. Personality Instructions as Ground Truth

3. Limited Model Diversity

4. Static Evaluation

5. Emotion Classification Quality

Recommended Extensions

1. Statistical Tests

2. Behavioral Consistency Validation

3. Mechanistic Analysis

4. Comparative Analysis

5. Longitudinal Study

Conclusion

This analysis provides quantitative, visual evidence that LLMs are not monolithic entities with fixed personalities, but rather systems with personality-response capacities that vary significantly across contexts, models, and training procedures. The visualizations and metrics create a framework for future research into how models adopt, maintain, and express personality traits under different conditions.

This directly supports your research hypothesis while providing specific, measurable evidence for the academic argument that "models should not be treated as single entities" but rather as contextual, adaptive systems whose personality expression depends critically on training procedure, architecture, and contextual instructions.