Djinn AI for UMD

Peer-reviewed research published to arXiv effective Nov. 2025

Case Study: Predicting Big Five Personality from Text

Background

The Big Five—Open-Mindedness, Conscientiousness, Extraversion, Agreeableness, and Negative Emotionality (OCEAN)—is the most academically and clinically validated framework for categorizing and describing human personality. Inferring it from a person's own writing has real value across the Health & Wellness Industry, Intelligence Community (IC), and field of Psychology, and modern language models have made great strides in making text easy to represent and process numerically. The catch is that the quantity of text required for an accurate assessment is large, and the models that read it can only take in so much at once.

Challenge

BERT-type language models cap their input at 512 tokens, but the bodies of text provided to us by our client routinely blew past that threshold, witha median of roughly 890 tokens and a 95th percentile near 1,500. The standard fix was to truncate, throwing away most of what the author wrote. Worse, even inside a single body of text, the great majority of sentences said nothing relevant to any given trait, so feeding in all the text diluted any existing signal with noise. The data is unforgiving in the other direction too: 75% of essays contained fewer than 55 sentences–often too few to speak to all 60 survey items that a full personality profile is built from.

Solution

Developed in collaboration with the University of Maryland Applied Research Laboratory for Intelligence and Security (ARLIS), BIG5-TPoT introduces Targeted Preselection of Texts: rather than reading an entire body of text, a model first intelligently selects only the sentences that are actually relevant to the trait it is trying to score, then predicts from that distilled signal. Three pieces make it work:

Ground Truth & Data

The data consisted of 5,810 stream-of-consciousness essays written by undergraduates (Pennebaker, 2015–2023). Every author also completed the 60-item BFI-2 survey, supplying real scores for 5 traits and 15 facets. All personal information was scrubbed with Microsoft Presidio to protect any PII (Personally Identifiable Information) before any modeling.

Targeted Preselection

Each trait, facet, or item, in every sentence is scored by its semantic similarity to the survey questions that define it. Sentences below a δ = 0.2 threshold are dropped; the rest are merged into one document embedding as a similarity-weighted average. This methodology is key to presenting the model with the signal, not the noise.

Model & Representation

(1) A frozen sentence encoder (paraphrase-multilingual-mpnet-base-v2) turns each sentence into a 768-dimensional vector, (2) a lightweight 2-layer network regresses it to a score, and then (3) a final variant uses ordinal (CORAL) regression to predict each of the 60 OCEAN traits and facets directly, to produce the author's Big-5 personality profile.

Under the Hood

Trait–facet–item hierarchy: 5 broad traits break down into 15 facets measured by 60 BFI-2 items; the model can predict the finer levels and average them back up.
Relevance threshold (δ = 0.2): sentences only weakly related to a trait are discarded before pooling, so weak signal never drowns strong signal.
Ordinal item modeling: the 1–5 Likert responses are treated as ordered categories via CORAL, rather than plain regression, when predicting individual items.
10-fold cross-validation: an 80/20 train/test split per fold with 10% held out for validation—every number reported is a mean over ten folds.
Genre- and model-agnostic: the same preselection extends beyond essays to social-media posts and to generative LLMs with far larger (8,192-token) input limits.

Outcome: Less noise, better predictions!

Across 10-fold cross-validation, BIG5-TPoT beats both the mean baseline and the standard whole-text Model 1 on nearly every trait and facet—on both error (MAE) and accuracy—despite reading only a fraction of each essay. The gains are largest on Negative Emotionality, where trait MAE drops from 0.652 to 0.608 and accuracy climbs from 45.2% to 48.1%. Predicting the 60 individual items directly, then rolling them up, pushes accuracy higher still.

Trait Prediction Accuracy

Trait	MAE ↓			Accuracy ↑
Trait	Base	M1	TPoT	Base	M1	TPoT
O · Open-Mindedness	0.520	0.515	0.503	0.532	0.550	0.560
C · Conscientiousness	0.550	0.548	0.538	0.523	0.526	0.529
E · Extraversion	0.587	0.578	0.564	0.499	0.502	0.522
A · Agreeableness	0.490	0.492	0.480	0.567	0.567	0.589
N · Negative Emotionality	0.652	0.628	0.608	0.452	0.471	0.481

Trait-level scores over 10-fold cross-validation. Base = mean-predicting baseline, M1 = standard whole-text model, TPoT = BIG5-TPoT (bold). MAE is lower-is-better; Accuracy is the share of predictions within ±0.5 of ground truth (higher-is-better).

BIG5-TPoT was developed as part of the 2023 Research for Intelligence & Security Challenges (RISC) program and published to arXiv in November 2025. Beyond the trait-level results shown here, the paper reports facet and item-level performance across all 15 facets and 60 BFI-2 items, and outlines extending targeted preselection to other text genres and to generative language models. Read the full paper on arXiv (2511.09426).

Predicting Personality from Text (BIG5-TPoT)