In clinical and epidemiological studies, biomarkers are associated with disease diagnosis and prognosis. Using biomarkers to classify subjects into groups, such as high-risk or low-risk, may help with the application of the most appropriate care or procedure within each group. In the case of a continuous biomarker, a cutoff value to define the groups should be determined. A widespread and straightforward method is to select a cutoff value that minimizes the P-value when comparing the outcomes between the two groups. However, a problem that arises with this procedure is that of multiple testing, which leads to an increase in false positive error rate, and thus the significance of the obtained cutoff value tends to be overestimated. In this article, we introduce several methods to correct the P-value for determining the statistical significance of an optimal cutoff value for a quantitatively measured biomarker with applications to clinical data.
In clinical and epidemiological studies, biomarkers are associated with disease diagnosis and prognosis, the two major determining factors for managing a patient’s care. For this purpose, biomarkers can be used to divide patients into two groups, such as high-risk or low-risk groups, which have statistical and clinical advantages. A continuous variable may have clinical significance concerning the outcome, but its effects may be non-linear or non-monotonic [
There are two statistical approaches for determining a cutoff value: biomarker-oriented and outcome-oriented. The biomarker-oriented approach splits a continuous marker about some percentile or the mean of biomarker values. In contrast, the outcome-oriented approach selects the biomarker cutoff that takes into account the association between outcome and biomarker. The outcome-oriented approach is expected to provide a better cutoff value than the biomarker-oriented approach [
A widespread, straightforward method is to select a cutoff value that corresponds to the most significant difference in the prognosis of an outcome between the two groups and one that provides the minimum P-value among all potential cutoffs. Cutoff values are determined from the entire range (minimum, maximum) for a biomarker or from a selected range, which excludes clinically non-relevant cutoff values. For example, in a data set with a size (n=10), a binary outcome (Y=0, 1), and a continuous biomarker (X), the data sets are examined to determine what cutoff value best separates the two groups. For each potential cutoff, c, the investigator draws a 2×2 table and then selects a cutoff value that maximizes the chi-square statistic, while equivalently minimizing the P-value (P_{min}) (
To protect from the increase of false positive error rate, a new P-value is obtained by multiplying the minimum P-value by the number of tests performed.
In the
The Miller and Siegmund method [
where z is the
This approach [
This method has the advantage that it can be applied to data with ordinal outcome, continuous outcome and time to event outcome as well as binary outcome, but it has the same disadvantage as the Miller and Siegmund method concerning small data sets.
Lausen et al. [
where l_{i} is the number of observations which is less than or equal to i^{th} candidate cutoff value (i=1,…,k-1). The value of the function of D is calculated with the following formula:
This formula can be also applied to binary outcome, ordinal outcome, continuous outcome and time to event outcome. We recommend to use the smaller of P-values of P^{(2)}and P^{(3)} [
This method provides a P-value that is derived from an asymptotic distribution of an adjusted statistic for the log-rank test to compare survival curves [
where L_{k} is the log-rank statistic computed at the k^{th} candidate cutoff value, and s is calculated with
Competing risk is an event that either hinders the observation of the event of interest or modifies the chance that this event occurs. For example, after a bone marrow transplantation, when studying leukemia relapse, non-relapse death can be considered as a competing risk. With these types of data, a popular method to compare survival curves is Gray’s test [
G_{k} is the Gray’s statistic calculated at the k^{th} cutoff value and s is calculated with
Software packages for the Miller and Siegmund method, the Lausen and Schumacher methods 1 and 2, and the Hothorn and Lausen method are available in the R package (R Foundation for Statistical Computing, Vienna, Austria) “maxstat”. Also, package “survMisc” in the R can be used for the Contal and O’Quigley method. Woo et al. method was implemented in a SAS macro (SAS Institute Inc., Cary, NC, USA), and code will be made available upon request.
The data from 141 subjects were collected to investigate the clinical usefulness of leukocyte elastase in the diagnosis of coronary artery disease (CAD) [
In data from 38 lymphoma patients (
Because the outcome of this data is death, which is a time to event, we could determine an MGE cutoff for the prognosis of death in lymphoma patients using the Contal and O’Quigley method as well as the Bonferroni method, and the Lausen & Schmacher methods 1 and 2. From these methods, the estimated MGE cutoff value was the same at 0.186 and the P-values corresponding to this cutoff were all significant except for the Bonferroni method result (
The data from 758 lung cancer patients undergoing tumor removal from 1991 to 2005 were collected. The data consisted of tumor size (cm) as a biomarker for recurrence. In this data, death after surgery was treated as a competing risk event. Among 758 patients, 580 patients had recurrences, 65 patients died without a recurrence, and 113 patients were censored. The distribution of tumor size was skewed to the right, with a median, mean and range of 3.0, 3.3, and 0.0 to 19.0, respectively. A Martingale residual plot showed a non- linear pattern of association between tumor size and time to recur and demonstrated a possible cutoff for tumor size around 3.0 cm. The rightmost plot in
Biomarkers, which can be measured as continuous or categorical data types, play a role as risk factors or predictors of clinical outcomes. In clinical practice, biomarker data, which is usually a continuous measurement, is often divided into two categories based on an optimal cutoff value. This kind of categorization makes it easier to interpret the effects of a biomarker on the outcome via an odds ratio or risk ratio. It also helps clinicians by providing objective criteria in the selection of treatment options. In this article, we focused on the dichotomization of continuous biomarkers and presented several methods to resolve the problem associated with the increase in false positive error rate, which occurs during multiple testing. We summarized the methods according to the type of outcome and presented some of their applications.
Miller and Siegmund method [
There are possibilities associated with the loss of information from the data, including a decrease in the power for detecting statistical significance and the biased estimation in the process of categorizing a continuous biomarker. Therefore, the cutoff value should be estimated after fully examining the feasibility of categorization using both clinical and data-driven factors [
No potential conflict of interest relevant to this article was reported.
Conception or design: SYW, SK. Acquisition, analysis, or interpretation of data: SYW, SK. Drafting the work or revising: SYW, SK. Final approval of the manuscript: SK.
This work was supported by grant of National Research Foundation of Korea (No. NTX1170701), Republic of Korea
Histogram of leukocyte elastase (A), values showing the log odds ratio against the mid-point values for each quartile of elastase (B), and a plot of a standardized two-sample statistic against elastase values with the vertical line representing an elastase cutoff value of 36 (C).
Histogram of mean gene expressions (MGE) (A), martingale residuals against MGE and their smoothed curve (B), and plot of the standardized log rank statistics against the MGE values with the vertical line representing an MGE cutoff value of 0.186 (C).
Box plot of tumor size (A), martingale residuals against tumor sizes and their smoothed curve (B), and a plot of the standardized Gray’s statistic against tumor sizes with the vertical line indicating a 2.7 cm tumor (C).
2×2 Table for the data set divided by a cutoff
X≤c | X>c | |
---|---|---|
Y=1 | ||
Y=0 |
X, continuous biomarker; c, a candidate cutoff value; Y, binary outcome.
Approaches for optimal cutoff value determination by outcome type
Outcome type | Adjustment methods |
|||||
---|---|---|---|---|---|---|
Bonferroni method | Miller & Siegmund method | Lausen & Schumacher method 1 | Lausen & Schumacher method 2 | Contal & O’Quigley method | Woo et al. method | |
Continuous | V | V | V | |||
Binary | V | V | V | V | ||
Ordinal | V | V | V | |||
Time to event (without CR) | V | V | V | V | ||
Time to event (with CR) | V | V |
CR, competing risk.
P-values corresponding to a leukocyte elastase cutoff of 36 for the diagnosis of coronary artery disease in each method
P_{min} | P_{bon} | P^{a)} | P^{b)} | P^{c)} | |
---|---|---|---|---|---|
P-value | 0.0014 | 0.1445 | 0.0026 | 0.0012 | 0.0007 |
P_{min}, minimum P-value approach; P_{bon}, Bonferroni method.
P was calculated using Miller & Siegmund method;
P was calculated using Lausen & Schumacher method 1;
P was calculated using Lausen & Schumacher method 2.
P-values corresponding to a mean gene expression cutoff of 0.186 for the risk of death in each method
P_{min} | P_{bon} | P^{a)} | P^{b)} | P^{c)} | |
---|---|---|---|---|---|
P-value | 0.0263 | 0.9731 | 0.036 | 0.024 | 0.0344 |
P_{min}, minimum P-value approach; P_{bon}, Bonferroni method.
P was calculated using the Lausen & Schumacher method 1;
P was calculated using the Lausen & Schumacher method 2;
P was calculated using the Contal and O’Quigley method.