Assessment of ADHD using ratings and tests

Most diagnostic guidelines (e.g., [15–17]) require that ADHD be assessed and diagnosed by relying on information provided via a variety of methods (e.g., clinical interviews, observations and ratings) and collected from multiple sources (e.g., parents and teachers). However, using subjective measures always incorporates the risk of informant biases [18] and clinicians are often confronted with great inconsistencies between ratings obtained from different sources [19, 20]. Although the discrepancies between different informants can be of clinical relevance [21], the use of objective measures in addition to subjective ratings might yield valuable information facilitating the diagnosis of ADHD. In the present study, we therefore aimed to investigate the role of objective measures when trying to distinguish between individuals with ADHD and controls. We also aimed to investigate how objective measures are related to subjective measures by investigating how well we could discriminate between ADHD and controls when using the combination of these two types of measures. The combination of objective and subjective measures may provide additional information than objective measures alone as it has been argued that tests and ratings may capture at least partly different constructs [22, 23] and should not be used interchangeably. Toplak and colleagues [23] argue that one important difference between ratings and tests is that the former measure typical performance (i.e., how an individual normally performs), whereas tests usually capture optimal performance (i.e., how well an individual performs under relatively optimal conditions). Thus, objective measures assess performance free of influences of the different situations. However, this study primarily investigated whether only objective measures would be sufficient to develop a statistical model as the bias and inter-operator error inherent in subjective measures are not well suited to developing a robust and objective classifier. Whilst the value of including subjective measures in a classifier alongside objective measures has been explored, developing an objective statistical method using only subjective data would not be expected to produce a robust classifier that would generalize to corresponding data acquired by other operators.

The relative importance of individual variables towards a diagnosis of ADHD is an issue that has not been empirically examined, at least not in studies employing statistical methods that can handle numerous variables to make an objective prediction. Similarly, few studies focus on objective measurements of ADHD symptom levels rather than constructs (such as executive functioning deficits) that are known to be associated with ADHD.

Objective measures

Aim of the present study

In summary, there are several studies reporting promising results regarding the ability of the QbTest 6–12 and the Qb+^© to differentiate between patients with and without ADHD. Nevertheless, findings are inconsistent, and often suggest using neuropsychological tests only as an additional resource within a comprehensive assessment strategy incorporating a variety of methods [27]. In the clinical community there is a high controversy about the usefulness of objective measures for diagnostic purposes as problems regarding sensitivity, specificity and ecological validity have been reported [40]. In light of the evidence that ratings and tests seem to assess partly different constructs, objective and subjective measures could be seen as complementing each other. The diagnostic value of objective tests becomes all the more important when the potential risks of subjective measures are taken into account, informant bias being the most important thereof. There is lack of studies evaluating the differential contributions of objective and subjective measures for correctly classifying ADHD. Whilst the study investigates the relative contribution of subjective measures in a classifier, the diagnostic accuracy using objective measurements only is considered more generalisable due to the inherent inter-operator variability in subjective measures. In contrast to most previous studies using neuropsychological measures and ratings to differentiate between patients with ADHD and healthy controls, we used machine learning rather than discriminant function analysis or logistic regression analysis. The advantage of the former is that it is data-driven and less sensitive to outliers [41]. Furthermore, it is a multivariate approach, as it does not rely on summary scores, but considers every single item. The risk of losing information is therefore reduced [41]. More specifically, the present study used support vector machine (SVM). This machine-learning approach is known to be very robust and capable of translating well in studies using imaging data [42]. However, it has been predominantly implemented in studies using neuroimaging data to diagnostically classify clinical populations [43, 44] and not in studies using standard clinical assessments as recommended by the various ADHD diagnostic guidelines outlined above.

The first aim of this study was thus to investigate the accuracy of employing only variables from the objective measures to reveal their specific potential contribution free from the potential confound of subjective measures. We further aimed to investigate how objective measures are related to subjective measures by investigating how well we could discriminate between ADHD and controls when using the combination of these two types of measures. In contrast to previous research, we used a machine-learning technique (SVM) to analyze the data.

Participants and procedure

Thirty children with ADHD and thirty controls matched at group level according to age and gender were enrolled in the childhood ADHD prediction. Thirty-eight adults with ADHD and thirty-eight age and gender-matched controls were enrolled in the adulthood ADHD prediction resulting in a total sample of N = 136. All children and adults with ADHD were recruited through an ADHD outpatient clinic within the university. Control children were recruited through local schools and children who participated in the study were given a movie voucher. Control adults were recruited at the university and via advertisements; they also received movie vouchers for study participation. No established or suspected ADHD diagnosis, or family history of ADHD were allowed for the individuals in the control groups.

Clinically-referred children were included in the study if they met the DSM-IV [45] criteria for ADHD (either combined, predominantly inattentive or predominantly hyperactive/impulsive subtype) and had an IQ-score ≥ 80 (short version of the Wechsler Intelligence Scale for Children IV [46]: block-design, similarities, digit span, information and picture arrangement; [47]). The exclusion criteria were symptoms of inattention, hyperactivity or impulsivity due to other medical conditions such as hyperthyroidism, autism, epilepsy, brain disorders and any genetic or medical disorder associated with externalizing behavior. Comorbid disorders like oppositional defiance disorder (ODD) or conduct disorder (CD) did not constitute an exclusion criterion as they are prevalent in about 30–50% of the population [48]. Other comorbid disorders (e.g. learning disorders, anxiety or depression) also did not result in exclusion as long as ADHD was the primary diagnosis. Participants were allowed to take medication but were asked to stop taking it 2 days before the objective tests were applied. Similar inclusion and exclusion criteria applied to adults with the exception that IQ was not assessed. Adult patients were recruited from a specialized outpatient clinic whose standard diagnostic procedure does not include extensive testing of cognitive abilities. Instead, achieved schooling and current job position are gathered. According to that, none of the patients were estimated to score below IQ of 80 as all of them have at least completed middle school. ADHD diagnoses were based on a DSM-IV-oriented clinical interview conducted by an experienced clinician, as this is known to be a highly reliable method for making an ADHD diagnosis [49–51]. For the children, we conducted the Schedule for Affective Disorders and Schizophrenia for School-Age Children-Present and Lifetime Version interview (K-SADS-PL; [52]) with a parent. Its inter-rater reliability ranges from .93 to 1.0 [53]. The adult patients completed the Wender Reimherr Interview (WRI; [54]), which has inter-rater reliability ranging from .45–.95 [55]. Rating scales were completed at home and the objective tests were instructed by clinicians or research assistants.

In the child sample, the male-to-female ratio was 21/9 (ADHD) and 19/11 (controls). The ADHD group’s mean age was 8.9 years (SD = 1.4, range 7.0–11.0 years) and the controls’ 8.7 years (SD = 1.2, range 6.9–10.8 years). A percentage of 26.7 of the children diagnosed with ADHD were predominantly inattentive, 3.3% were predominantly hyperactive-impulsive and 60% fulfilled the diagnostic criteria for the combined subtype. Unfortunately, for 10% of the children the subtype information was not available. The adult ADHD group’s mean age was 35.1 years (SD = 11.7, range 19–63 years); and the controls’ 32.2 years (SD = 9.6, range 21–56 years). Both adult groups had a male-to-female ratio of 25/13. A percentage of 10.5 of the adults diagnosed with ADHD were predominantly inattentive, 2.6% were predominantly hyperactive-impulsive and 81.6% fulfilled the diagnostic criteria for the combined subtype. Unfortunately, for 5.3% of the adults the subtype information was not available.

Measures

The standard diagnostic procedure for ADHD at the outpatient clinic from which our participants were recruited incorporates a variety of measures like clinical interviews, self- and observer ratings and neuropsychological tests. Instruments used for our SVM analyses are described in greater detail below.

Analyses

Analyses apply feature selection to identify those variables most relevant to the diagnosis, and we took a popular pattern recognition approach (SVM) to make the diagnosis prediction. To ensure each prediction was made based on data that was novel to the classifier, as would occur in routine clinical practice, we used cross-validation. Due to the differences in measures for childhood and adulthood ADHD diagnoses, we carried out two separate but identical analyses for each age group.

Prediction of ADHD diagnosis in adults

Using a linear SVM and feature selection on the adult dataset, we were able to predict an ADHD diagnosis with 89.5% accuracy (sensitivity = .90, specificity = .90, χ² = 44.26, p < .0001). The majority of the variables relevant to the classification were selected from the CAARS scores. Many of the scores from the self-ratings were especially predictive, with each of the 51 variables selected in 36.20% of the predictions on average. Similarly, the observer-rated scores contained many predictive variables, with each variable selected in 20% of the cross-validated classifiers on average. This was expected, as symptom severity scores are likely to readily distinguish patients from controls.

We did not find the test battery of attention scores to be particularly predictive of ADHD diagnosis. Of the 28 variables included in the calculation, each variable was selected in 2.96% of the predictions on average. The 16 QbTest+© variables entered into the classifier were selected in 8.96% of the predictions on average.

The variables selected in all of the predictions were questions from the self- and observer-rated CAARS. From the self-ratings, we found the following items to be relevant to all predictions: “I have trouble keeping my attention focused when working”, “I feel restless inside even if I am sitting still”, “Things I hear or see distract me from what I’m doing”, and “I am restless or overactive”. Similarly, three items from the observer-rated questionnaire were used in all predictions: “is easily frustrated”, “is distracted by sights or sounds when trying to concentrate”, and “can’t keep his/her mind on something unless it’s really interesting”.

In addition, although not used in all the predictions, a number of self-rated CAARS items were selected in over 75% of predictions: “Many things set me off easily”, “My moods are unpredictable”, “Sometimes my attention narrows so much that I’m oblivious to everything else; other times it’s so broad that everything distracts me”, “I can’t keep my mind on something unless it’s really interesting”, “I am distracted when things are going on around me”.

When including only objective measures (the output from the QbTest+© and TAP tasks), we predicted an ADHD diagnosis with 79% accuracy (sensitivity = .82, specificity = .76, χ² = 23.28, p < .0001). The following variables were used in over 85% of predictions: overall omission errors made at the subtest sustained attention in the TAP; QbTest+©: omission errors, error rate, normalized reaction time variance, normalized reaction time variance without outliers, and the ability of the patient to distinguish between target and non-target.

Prediction of ADHD diagnosis in children

By applying the same technique used to predict diagnosis in the adult population to 30 children with ADHD and 30 controls, we were able to predict diagnosis with 86.7% accuracy (sensitivity = .83, specificity = .90, χ² = 29.53, p < .0001). As in the adult study, the Conners’ scores proved to be the most predictive of an ADHD diagnosis. Only 12 parent-rated scores were entered into the classification procedure, and those variables were selected in 36.53% of the predictions on average. The Conners’ parent subscores relating to executive function, inattention DSM-IV ratings and the ADHD index were selected in all predictions, and the general inattention score was selected in 75% of the predictions. None of the variables taken from the KiTAP were selected in any of the predictions, while the 15 QbTest 6–12 variables entered into the classifier were selected in only 1.11% of the predictions on average.

ADHD assessment in adults

Our results demonstrate that using both subjective and objective measures in adults, it was possible to predict an ADHD diagnosis with an accuracy of 89.5%, with self-rated scores being the most predictive ones (selected in an average of 36.20% predictions) followed by the observer-rated scores (selected on average in 20%). Variables of both the Qb+^© and TAP were less relevant to predicting an ADHD diagnosis than were the self- and observer-ratings. Nevertheless, prediction was still possible using only these objective measures with an accuracy of 79%.

Regarding the TAP, the variable omission errors from the subtest “sustained attention” was selected in more than 85% of the predictions; the same accounts for the Qb+^© variables omission errors, error rate, reaction time variance, and the participant’s ability to distinguish between target and non-target. This finding is in line with previous findings, as a review of 33 studies on the neuropsychology of adults with ADHD identified omission errors and reaction time variance as variables well able to discriminate between adults with ADHD and controls [73]. This is further supported by our analysis that revealed some significant correlations between the number of omission errors in the TAP, the overall ADHD and inattention symptom scales of the CAARS. These two variables are generally considered to be indicative of inattention, which has been supported in previous factor analyses [38]. Our findings thus reveal those variables indicative of inattention as the most predictive ones for diagnosing ADHD in adults. This is in line with research suggesting that inattention is the ADHD symptom domain most likely to persist through lifetime, while hyperactivity and impulsivity seem to decline to a greater extent [74]. It is unlikely that these results are due to our sample having a higher proportion of the inattentive subtype as those were only 10.5%, but 81.6% of the participants had the combined subtype. Furthermore, our findings demonstrate high accuracy rates when the prediction was based on objective measures only, thus supporting their diagnostic value independently of subjective measures, removing a significant source of variability and thus increase the likelihood that the classification accuracy obtained would be able to be independently replicated. This is important, as in clinical practice we do not just want to know whether patients are inattentive in daily life (due to several potential reasons besides a primary attention deficit), but also whether they are inattentive in structured situations such as when completing a neuropsychological task. Thus, as emphasized by Toplak et al. [23], ratings and tests capture at least partly different constructs and should therefore be seen as complementing one another. This is further supported by our findings that showed rather low and non-significant correlations between subjective and objective measures. Furthermore, an objective assessment of a patient’s impairment might be especially relevant in cases where observer-ratings are unavailable [75], when self-ratings might be questionable due to the potential faking of results (e.g., [76, 77]), or when answers are considered biased [18]. Recent studies have provided additional support for the value of objective assessments. For example, Hirsch and Christiansen [78] showed that inattention as measured with the Qb+^© was indicative of overall impairment, adding key supplemental information. Another study demonstrated that the Qb+^© is sensitive to medication effects, showing an improvement in 54% of patients who reported no changes in the subjective measure [79].

ADHD assessment in children

We were able to predict an ADHD diagnosis in children based on all of the variables with 86.7% accuracy. Here, no self-ratings were used, but the sub-scores “executive functioning”, “DSM IV inattention” and the “ADHD index” of the parent-ratings were selected in all predictions demonstrating their diagnostic value. The prediction using the objective measures did not produce as accurate a classifier as the combined objective and subjective measures in our child sample. However, predicting an ADHD diagnosis using only the objective measures was still significant with 78% accuracy.

Similar to the adult sample, the variable omission errors of the subtest “sustained attention” of the KiTAP was selected in more than 98% of the predictions, as was the median of the reaction time and number of errors in the “Go/NoGo” subtest. This is in line with previous research showing that children with ADHD commit both more omission errors and commission errors than healthy controls [80–82]. Looking at the QbTest 6–12, the most important variables for the prediction differ from those of the adult sample, as the variables assessing hyperactivity were predominantly selected, namely the distance moved and the area covered by the participant, micro-movements, complexity of movement patterns, as well as multiple pressing of the button, reaction time variance, and reaction times < 150 ms. This reflects the findings on the development of ADHD symptoms from childhood to adulthood, demonstrating that hyperactivity is more observable in children than in adults [14, 83, 84]. Adults, in contrast, report more feelings of restlessness or being driven by an internal motor than overtly exhibiting hyperactive behavior. Furthermore, it is interesting that for the QbTest, the variables assessing hyperactivity were most often selected for the prediction, whereas for the Conners-3 parent ratings the sub-score assessing inattention seemed to be more important. Hyperactive behavior thus appears to be evaluated better applying objective measures rather than subjective reports. A likely reason for this finding is that the QbTest can capture tiny movements that the patient alone might not even notice. Including objective measures thus resulted in high accuracy for the children. Regarding hyperactivity, the objective measures even outperformed the ratings. These findings support the inclusion of objective measures in ADHD diagnostics.

We found significant negative correlations between IQ, the Conners’ symptoms scale of inattention and the ADHD index, two of the KiTAP variables indicative for inattention (reaction time variance and omission errors) and the Qb factors. This is not surprising as the children with ADHD had lower IQ, which is known to be typical for this population [56, 57]. The fact that we found these correlations not only for the objective measures but also for some of the subjective variables underlines that the performance in the objective tasks was not only due to the lower IQ, but also influenced by the deficits these children present.

Limitations and future research

According to the DSM-criteria [1] for ADHD, impairment should be established in multiple settings. It would therefore have been valuable to also include teacher ratings in the present study. Unfortunately, there was too much missing data for the teacher ratings to be absorbed in our analyses. This is a problem we confront constantly during our daily diagnostic routine. Here, objective measures are given additional weight, as they have the potential to add valuable information not otherwise obtainable.

Another limitation of our study is the relatively small number of participants in each group (adults ADHD/controls: 38; children ADHD/controls: 30). As we aimed to be able to make reliable claims regarding the diagnostic accuracy of the instruments assessed, we focused on having gender- and age-matched groups. This resulted in relatively small group sizes, but the total number of participants (N = 136) can be regarded as satisfactory.

Future research should preferably include a clinical control sample, as it is highly relevant for clinicians to be able to distinguish between patients with ADHD and those exhibiting similar symptoms due to another underlying disorder (e.g., inattention is also common in patients with depression).