Examining the Factor Structure of the DSM-5 Level 1 Cross-Cutting Symptom Measure

This article is a US Government work. It is not subject to copyright under 17 USC 105 and is also made available for use under a CC0 license.

Updated version available: A peer-reviewed version of this article, "Examining the factor structure of the DSM‐5 Level 1 cross‐cutting symptom measure", has been published in Int J Methods Psychiatr Res.

Associated Data

Abstract

Objectives:

The DSM-5 Level 1 Cross-Cutting Symptom Measure (DSM-XC) is a transdiagnostic mental health symptom measure that has shown promise in informing clinical diagnostic evaluations and as a screening tool for research. However, few studies have assessed the latent dimensionality of the DSM-XC. We examined the factor structure of the DSM-XC in a large convenience sample of participants with varying degrees of psychological health.

Methods:

Participants (n=3533) enrolled in a protocol conducted at the National Institute of Mental Health (NCT04339790). We used a factor analytic framework to evaluate an existing two-factor solution (Lace & Merz, 2020) and two additional candidate solutions.

Results:

The Lace and Merz solution had acceptable fit. Exploratory factor analysis yielded two candidate solutions: a six-factor (characterized as mood, worry, activation, somatic, thoughts, and substance use) and a bifactor (general factor of non-specific psychopathology, residual factors characterized as internalizing and thought disorder), which both had good fit and full measurement invariance across age, sex, and enrollment date.

Conclusions:

Our findings confirm that the DSM-XC may be conceptualized as a multidimensional instrument and provide a scoring solution for researchers who wish to measure distinct constructs. Future research on the psychometric profile of the DSM-XC is needed, focused on the validity of these candidate solutions and their performance across research populations and settings.

1. Introduction

According to the 2019 National Survey on Drug Use and Health (NSDUH) by the Substance Abuse and Mental Health Services Administration (SAMHSA), an estimated 20.6%, or 51.5 million adults aged 18 or older in the United States were diagnosed with any mental illness (AMI), defined as a mental, behavioral, or emotional disorder that can vary in impairment (( 1 )). Furthermore, fewer than half (44.8%) of those with AMI received mental health services in the past year. Given these high rates of current mental illness and low rates of treatment, there is a need to better assess mental health problems in the general population. While there are existing mental health measures in use, many are limited to symptoms within one diagnostic category, such as the Patient Health Questionnaire 9 (PHQ-9), which focuses on depression (( 2 )). In addition, because symptoms associated with discrete mental health diagnoses often co-occur, a tool that is broader and transdiagnostic would be useful both in clinical settings and in research studies.

The DSM-5 Level 1 Cross-Cutting Symptom Measures (DSM-XC) were developed by the American Psychiatric Association (APA) as transdiagnostic mental health symptom measures (adult, child and parent versions). They were developed in response to recommendations that the DSM-5 include dimensional assessments in addition to categorical diagnoses utilized in previous versions of the DSM ( 3 ). The adult measure includes 23 self-report questions covering 13 cross-diagnostic domains of psychopathology: depression, anger, mania, anxiety, somatic symptoms, suicidal ideation, psychosis, sleep problems, memory, repetitive thoughts and behaviors, dissociations, personality functioning, and substance use ( 4 ). The survey was intended to inform a clinical diagnostic evaluation and provide clinicians with a tool to track the presence, frequency, and severity of cross-cutting psychiatric symptoms ( 5 ). An early study evaluated the measure in routine clinical practice and found it to be both clinically useful and favorably viewed by clinicians and patients ( 6 ). It has also been recommended for use in clinical research ( 7 ).

In clinical practice, the DSM-XC is to be used as a Level 1 screener for mental illness followed by Level 2 measures of a specific domain depending on initial responses. The limited psychometric data available for the DSM-XC primarily pertain to this purpose. Our group used a modified version of the DSM-XC to screen volunteers for mental health research and found good specificity but poor sensitivity when used alone, but improved sensitivity when combined with additional clinical data, e.g., use of psychotropic medication ( 8 ). Conversely, among patients with substance use disorders in a correctional program, the DSM-XC had good sensitivity but poor specificity for DSM diagnoses of depression, mania, psychosis, and anxiety ( 9 ).

Though most reports on the DSM-XC described its use in clinical practice, it has also been recommended for use as a dimensional measure of psychopathology in research studies. Of note, the DSM-XC is one of the measures in the minimal list of data collection measures required by the National Institute of Mental Health (NIMH) for its grantees who conduct mental health human subjects’ research. The minimal list has been identified by NIMH, the Wellcome Trust and other funders of mental health research using the PhenX consensus process (https://www.phenxtoolkit.org/collections/view/1) or the International Consortium for Health Outcomes Measurement (ICHOC) (https://www.ichom.org/resource-library/category/condition-specific-resources/depression-anxiety). Given that the need for such a measure was the impetus behind the DSM-XC’s development, it is surprising that few published studies have assessed its latent dimensionality or provided guidance on how to interpret DSM-XC responses at the item, domain or full survey level. One study that administered the DSM-XC to non-treatment seeking college students found it to have acceptable convergent validity when the domain total scores were compared to longer measures of the intended mental health construct ( 10 ). A study conducted by the APA in concert with DSM field trials administered the survey to adult psychiatric patients and reported mean scores for the DSM-XC items in their supplementary materials. However, they made no claims that the scores within domains measured a unidimensional construct. Whether and how the items included in the DSM-XC measure dimensional mental health constructs (e.g., depression) is an empirical question that should be further investigated.

Given the transdiagnostic design of the DSM-XC survey, the latent dimensionality of the measure should be explored. To date, three studies have published on the factor structure of the DSM-XC, each with limitations. A Turkish study reported an exploratory factor analysis (EFA) on data collected from healthy volunteers and psychiatric patients (n=206) and found that a three-factor structure capturing neurosis, psychosis, and substance use best suited the data ( 11 ). Using confirmatory factor analysis (CFA), another group demonstrated adequate fit of the 13 DSM-XC domain structure to data from 362 Pakistani prisoners ( 12 ). Both studies used translated versions of the measure (Turkish and Urdu, respectively) and collected data from distinct populations (i.e., psychiatric patients and their relatives, prisoners), which could limit the generalizability of their findings. A recent study employed both EFA and CFA methods on the adult version of the DSM-XC in English ( 13 ). They recruited and collected data from 400 participants using Amazon Mechanical Turk, a crowdsourcing website (n=191 EFA, n= 209 CFA). Lace and Merz report that a two-factor model, comprising externalizing/serious mental illness and internalizing/affective constructs, best fit their data. All three studies utilized sample sizes which were small relative to recommendations by psychometric ( 14 ) and none performed invariance analyses to demonstrate robustness of the identified solution. Notably, none of the three studies described utilized a bifactor approach to analyzing their data. This was the approach described in the seminal paper by Caspi and colleagues in which they propose a general psychopathology factor (p factor) which was the most parsimonious explanation of psychiatric disorders in a large longitudinal data set ( 15 ).

In this report, we sought to address the limitations outlined above by examining the factor structure of the DSM-XC in a large sample of participants enrolled in a protocol conducted through the National Institute of Mental Health Intramural Research Program (NIMH IRP) (ClinicalTrials.gov identifier: NCT04339790). This online study was launched to examine the mental health impact of the COVID-19 pandemic among a convenience sample with a wide range of mental health histories, and the psychometric evaluation of the DSM-XC is a secondary analysis. Following an EFA to identify candidate solutions, we employed CFA to evaluate the fit of the two-factor solution proposed by Lace and Merz ( 13 ), the fit of our proposed solutions, and the measurement invariance of our proposed solutions across age, sex, and enrollment date.

2. Method

These data were collected as part of a NIMH study that was carried out in accordance with the latest version of the Declaration of Helsinki and was approved by the Institutional Review Board of the National Institutes of Health. For this purpose of this report, we will primarily focus on methodology relevant to the collection of the DSM-XC survey data.

2.1. Participants

Participants were recruited through a variety of means, including email invitations, social media ads, list-serv postings, clinicaltrials.gov and direct mail postcards. Exclusion criteria were minimal: participants were required to be 18 years or older and to speak and understand English. The sample was intended to be representative of the general population. For this analysis, we included participants who had provided responses to all 22 DSM-XC items at the time of enrollment (n=3533).

2.2. Measure

The DSM-XC was chosen for the parent study because of its cross-diagnostic approach to mental health symptoms. In addition, the DSM-XC was promoted by the NIMH as a common measure in funded research projects (NOT-MH-15–009, 2015). The measure consists of 23 items, each rated on a 5-point Likert scale (0=none – not at all, 4=severe – nearly every day). We chose to omit one item on thoughts of self-harm (question 11) because the study was conducted online and therefore participants who may endorse self-harm could not be reliably contacted for further safety evaluation.

2.3. Procedures

Study enrollment occurred from April 4 th to November 15 th , 2020. After signing an electronic consent form, participants used a secure website to complete several surveys, including the DSM-XC. Participants were then invited to complete a smaller set of surveys which included the DSM-XC every two weeks for 6 months. The study website was HIPAA compliant and did not collect personally identifiable information. Compensation was not provided for participation.

2.4. Statistical Analysis

2.4.1. Data preparation.

The analysis set of cross-sectional DSM-XC responses of 3,533 individuals at time of enrollment was separated into exploratory (n = 1,202) and confirmatory sets (n = 2,331) using a randomly generated number for each participant. Response category cell size for each item was reviewed, and adjacent categories were collapsed to achieve a minimum cell size of n=50 in the exploratory set (see Supplementary Information for a list of response categories). The same transformations were applied to the confirmatory set.

2.4.2. Exploratory analysis.

Exploratory factor analysis (EFA) was conducted in the exploratory dataset using Mplus Version 8.4, and the R package MplusAutomation ( 16 ) was used to process the output. Given the ordinal nature of the data, we employed a categorical factor model with diagonally weighted least squares estimation with mean- and variance-adjusted chi-square statistics and standard errors, and geomin (oblique) rotation. Two approaches were used: standard EFA and bifactor EFA. The bifactor model specifies a general factor (always the first factor extracted), which in this case would approximate the “p factor” described by Caspi et al., and additional factors account for the remaining shared variation. The root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker-Lewis index (TLI) were generated and interpreted as described below. We selected the best model from these candidates by first determining which solutions had relatively better fit, and then evaluating the clinical interpretability of each solution. Items were adopted onto a factor if they possessed a loading >0.30; in the standard EFA, cross-loading was allowed because it is theoretically consistent with the constructs measured by the DSM-XC.

2.4.3. Confirmatory analyses.

The confirmatory factor analysis (CFA) was conducted in the confirmatory dataset using lavaan version 0.6–7 ( 17 ) in R version 4.0.2. lavaan also employs diagonally weighted least squares estimation for ordinal data, and the Santorra-Bentler correction was applied to the test statistic. The robust fit indices (RMSEA, CFI, TLI) are reported and interpreted as described below.

2.4.4. Invariance analyses.

Measurement invariance analyses were conducted using semTools version 0.5–4 ( 18 ). We sequentially tested increasing levels of measurement invariance: configural, thresholds, metric, and scalar. Change in the relative (robust) fit indices RMSEA, CFI, and TLI were interpreted as described below. The measurement invariance of the selected solution across age (18–29 years, n = 1330; 40–59 years, n = 1370; 60+ years, n = 791) and sex (males, n=589; females, n = 2921) was evaluated using the full sample (combined testing and training, N = 3,533). Respondents missing age or sex data were excluded from the respective invariance analysis. We also evaluated the cross-sectional invariance of the solution across enrollment date, comparing baseline responses for participants whose study baseline occurred during the first month of the study (April 04 through May 04, 2020, n = 1380) to those who started during months 5 – 7 of the study (September 01 through October 31, 2020, n = 490).

2.4.5. Interpretation of fit indices.

While several sets of guidelines for the interpretation of fit indices exist (e.g. ( 19 )), so do strong arguments against the use of cutoffs, especially when diagonally weighted least squares estimation was used (e.g. ( 20 – 24 )). Without a universally agreed upon alternative, we chose to use the RMSEA, CFI, and TLI to guide our modelling decisions. Rather than classify these indices into “good” or “bad,” we evaluated each from a relative perspective. In other words, lower (scaled) chi-square to DF ratios, lower RMSEA, and higher TLI or CFI are preferred, regardless of their actual values.

The Mplus and R code files for all analyses are provided in Supplementary Information. For more information on how to interpret factor scores, please see ( 25 ).

3. Results

During the 7-month period of study recruitment, 3654 participants from all 50 U.S. states signed an electronic consent form and completed the study enrollment surveys. Of these participants, 3533 provided responses to all 22 DSM-XC items in our version of the survey and were included in this analysis. The mean age of this final sample was 46.5 +/− 14.8 with a range from 18 to 87 years. The sample was majority female (83.2%) and white (90.2%). The majority of the sample (71.6%) endorsed a history of mental health counseling. Additionally, about half (53.2%) endorsed a history of medication for a mental health condition and 13.8% endorsed a past psychiatric hospitalization ( Table 1 ).

Table 1.

Demographics of sample

Exploratory factor analysis.

Based on clinical usage of the instrument and six eigenvalues greater than 1, we explored solutions with one through six factors in the training dataset (n = 1202). The fit indices for these solutions are shown in Table 2 and the standardized loadings for each solution are shown in Supplementary Tables 1 and 2. Based on fit indices and interpretability, we carried forward the six-factor standard EFA and the two-factor bifactor solutions.

Table 2.

Notes: χ 2 – Chi Square, df – Degrees of Freedom, CFI – Comparative Fit Index, TLI – Tucker Lewis Index, RMSEA – Root Mean Square Error of Approximation, SRMR - Standardized Root Mean Square Residual. Robust indices used for Confirmatory Factor Analyses.

Confirmatory factor analysis.

The results of the CFA indicated that both the six-factor and two-factor bifactor solutions were a good fit for the testing data (n = 2331; Table 2 ). Both models had good interpretability. We propose that the six factors represent the following constructs: mood, worry, activation, somatic, thoughts, and substance use ( Figure 1 ). The bifactor model, in addition to the general psychopathology factor, represents internalizing and thought disorder constructs (Figure 2). The standardized loadings on each factor were moderate-to-high ( Table 3 ). Finally, we evaluated the existing Lace and Merz ( 13 ) two-factor solution, and found that our data exhibited acceptable fit to this solution ( Table 3 ).

(A) Simplified path diagram for six-factor solution. Rectangles represent items (names have been shortened). Circles represent factors. Standardized loadings are shown. (Simplified path diagram for bifactor solution. Solid rectangles represent items (names have been shortened). Circles represent factors. Standardized factor loadings are shown; negative loadings have been highlighted by a dotted rectangle.

Table 3.

Standardized CFA loadings of selected solutions

Note: Some items have been shortened. Items are standardized loadings from confirmatory factor analysis.

Measurement invariance.

Configural, threshold, metric, and scalar measurement invariance of both the six-factor and bifactor solutions were evaluated sequentially across age (n = 3491), sex (n = 3510), and enrollment date (n = 1870). Owing to the smaller sample size for the enrollment date analyses, too few non-zero responses to the items “Hearing things…” and “…hear your thoughts” meant that they had to be excluded from the solution. For both models, full measurement invariance was supported across age, sex, and enrollment date ( Table 4 ).

Table 4.

Results of measurement invariance analyses

Note: CFI = comparative fit index (higher is better); TLI = Tucker-Lewis fit index (higher is better); RMSEA = root mean square error of approximation (lower is better)

4. Discussion

In this secondary analysis, we explored the factor structure of the DSM-XC using a large data set of individuals with varying degrees of psychological health. While the previously published solution of internalizing and externalizing factors ( 13 ) had a mediocre fit to our data, we evaluated the evidence for the factorial validity and measurement invariance of two other multidimensional solutions. Given the lack of literature on the DSM-XC and its latent dimensionality, we believe these findings will help future researchers score and interpret DSM-XC responses.

While the existing 13-domain structure of the DSM-XC would have been an obvious candidate for evaluation, such an analysis was not possible because several domains include only one or two items. As with any factor analysis, alternative structures may provide similar fit to the data. However, both candidate solutions we propose herein suggest that these items are better conceptualized as fewer, broader constructs. Both the six-factor and bifactor solutions exhibited strong factorial validity, demonstrated through a confirmatory analysis in a withheld subset of our data and a series of measurement invariance analyses.

We argue that the six-factor solution holds considerable face validity and appears to be clinically coherent based on clinician consensus review of symptom item clusters. The items on each factor suggest that they represent the following constructs (respectively): mood, worry, activation, somatic, thoughts, and substance use ( Figure 1 ). In most cases, items which shared a domain on the DSM-XC also shared a factor in our proposed solution (e.g., the two depression items load onto factor 1, three anxiety items load onto factor 2).

The results of our bifactor analysis are theoretically consistent with findings from Caspi and colleagues ( 15 ). The general psychopathology factor is a satisfying solution given the known overlap of symptoms across psychiatric disorders and their comorbidity. While the general psychopathology factor explained most of the variance, two residual factors were identified. The first included the three items from the DSM-XC anxiety domain, which we termed “internalizing.” The second included items that index psychosis, mania and depression symptoms which we labeled “thought disorder.” Both labels were chosen to be consistent with the Caspi model.

Our findings are particularly relevant for use of the DSM-XC since it has been promoted as a dimensional measure of psychopathology ( 5 ). Our findings first confirm that the DSM-XC is a multidimensional instrument. While researchers may decide to use the DSM-XC as an indicator of overall symptom burden and simply tally the total score of all items, all available factor analyses of the instrument, including ours, indicate that the scale is not unidimensional. Even considering the general factor in the multidimensional bifactor solution, we note that some items load much more strongly on the general factor than others, a fact obscured by the simple sum total of scores. Further, the use of factor scores, rather than subscale totals, enable researchers to measure with more precision the dimensional mental health constructs reflected by the scale. Second, our findings provide a scoring solution for future researchers who wish to measure distinct mental health constructs. While thirteen clinically derived domains are provided, our findings suggest that there are fewer, broader constructs, and fitting with our conceptualization of mental health, that many items are not unique to a particular construct. This solution therefore accounts for cross-cutting symptoms that relate to multiple constructs of interest. Future research should evaluate the structures proposed in this paper to confirm its consistency across research populations and settings. An additional important future direction is the evaluation of the validity of the proposed solutions with convergent measures; such an analysis was outside the scope of this secondary analysis.

Several limitations stem from the fact that this study was conducted online during a global pandemic. First, given constraints on recruitment methods during the COVID-19 pandemic, we enrolled a convenience sample with demographic characteristics not representative of the general population (i.e., 90% white, 83% female, 52% advanced degree). It is also important to note that our population was likely subject to sampling bias due to the study’s focus on mental health: over 50% of our sample endorsed a history of psychiatric medication use, which is substantially higher than rates of use in the general population ( 26 ). Similarly, factor scores are likely elevated due to the COVID-19 pandemic ( 27 , 28 ). Finally, this study used a modified version of the DSM-XC, as we chose to omit the self-harm item.

Despite these limitations, this study contributes to the growing body of literature on the DSM-XC and can help with the interpretation of scores for future research. The measure has been recommended for clinical research by the NIMH Extramural Research Program ( 7 ) and for further evaluation by the APA ( 29 ). Continued evaluation of its psychometric properties is supported by a need to better identify mental health in the general population; while many domain-specific mental health surveys have been validated, the DSM-XC captures symptoms that cross-cut psychiatric diagnostic categories which accounts for the dimensionality and frequent co-occurrence of mental illnesses.