First author and analysis contributor, published academically as Brian Droncheff.
Explainable ML & measurement
Explainable ML for Psychometric Measures
SHAP-guided item retention.
A first-authored, open-access 2024 study by Brian Theory—published academically as Brian Droncheff—testing SHAP-guided item retention for abbreviated psychometric measures.
Peer-reviewed study · Public data and analysis
Project video · 3:49
SHAP-guided psychometric item retention
A research walkthrough of neural-network and SHAP-guided item retention for abbreviated psychometric measures.
Click the picture to play. Full screen and volume are available in the controls.
Open the video file ↗3,906 adolescent responses14–19 participant ages2–8 score-bin targets100× randomized models per setting
A neural-network and SHAP item-selection study evaluated across changing score targets.
Higher-SHAP configurations performed better, while the retained ranking changed with the intended target.
A peer-reviewed open-access article, public analysis materials and aggregate results.
01 / The measurement question
Shorter for whom—and for what decision?
Shorter questionnaires can reduce respondent burden, but removing items can also change what a measure captures. The practical takeaway is simple: the “best” questions depend on what the shortened measure must predict.
The study treated retention as a feature-selection problem and tested how item importance changes when the worry-score distribution is divided into different classification targets.
02 / Study method
01Full responses
PSWQ item-level data
→02Define target
Two through eight score bins
→03Model & explain
Neural networks plus SHAP values
→04Retain & test
Compare abbreviated configurations
Higher-ranked and lower-ranked item configurations were compared. A previously published five-item factor-analytic questionnaire was also evaluated on the same dataset.
03 / Evaluation design
Separate learning
from evaluation.
Models learned from 80% of the responses and were evaluated on the remaining unseen 20%. Classification metrics were averaged across 100 randomized models for each setting rather than reported from a single favorable split.
Evaluation design
Train on one partition. Test on unseen responses.
14–19 years old13 secondary schools100 randomized models per classification setting
04 / Target sensitivity
Importance is conditional.
A feature is not simply important in the abstract. In the study, item ranking changed with the portion and granularity of the worry-score distribution being predicted.
Broader target2 score binsFewer boundaries
→More granular target8 score binsMore precise classification
→Selection consequenceRanking can shiftRetain for the intended use
Aggregate result reconstruction
The ranking changes with the target.
Only publication-level item rankings are shown; no participant records or unverified local output are used.
The analysis shows that which questionnaire items matter most depends on the prediction goal, helping researchers shorten measures without treating one ranking as universal.
Open full-size view ↗What the study found
Configurations built from higher-SHAP items produced stronger classification metrics than configurations built from lower-SHAP items. The SHAP-informed neural-network approach also outperformed the five-item factor-analytic comparison in the reported same-dataset classification metrics.
Why it matters beyond questionnaires
The broader evaluation lesson applies to AI systems generally: define the decision target, measure against that target, and check whether an explanation remains stable when the intended use changes.
Scope & limitations
One sample is not
external validation.
The study used one adolescent sample from 13 secondary schools in the Netherlands. An independent sample is needed to establish external validity, and the findings may not generalize to other populations or measures.
The paper also notes the interpretability limits of neural networks and the sensitivity of SHAP values to correlated features. The result supports this method in the studied setting, not a universal recipe for questionnaire reduction.
Inspect the evidence
Publication, materials
and research context.
Useful when
Your research or product team needs machine-learning explanations to support feature selection without treating importance as universal.
Explore technical validation →