Miracle Drug - Part 2

In a previous post, we discussed the dangers of subgroup analyses. Specifically, performing increasing numbers of post-hoc subgroup analyses increases the risk of a type 1 error while the inevitable reduction in numbers tested in subgroup analyses decreases power and increases the risk of a type 2 error.

This is a particular problem in nephrology where there is a lack of randomized trials and a lot of our information is gleaned from subgroup analyses of trials that were never intended to specifically test patients with CKD. A paper was recently published in CJASN which examined the issue of subgroup analysis in the nephrology literature and suggested some potential guidelines for reporting:

1. Plan subgroup analyses during the study's design period

2. Limit the analyses to variables where a plausible hypothesis exists for why there might be a difference between groups

3. In the methods section, list the pre-specified analyses and explain the rationale for each analysis

4. Do not report subgroup analyses in the abstract

5. Use statistical tests for interaction and report effect estimates (confidence intervals rather than p-values)

6. Avoid over-interpretation of subgroup analysis results, explain that this is hypothesis-generating rather than definitive and emphasize the overall study results

The idea is to prevent fishing - performing multiple post-hoc analyses until one is found that is positive. Technically, if one is going to do this, there should be a correction for multiple statistical testing but this is rarely done. The authors of the review examined published randomized trials in 4 major journals and found that about 1/3 of nephrology papers published subgroup analyses (which was actually below average). However, more than 3/4 of these had issues with the analysis including the lack of pre-specified subgroups and inappropriate statistical testing. 

The take home from this for the general audience who will not necessarily be performing these analyses is to carefully examine any trial that makes far-reaching claims based on a subgroup analysis. Remember that in the majority of cases, the trials were never designed to look at these subgroups and that these results should not be regarded the same way as the primary results of the papers. Similarly, a negative result in a subgroup analysis should be examined carefully to determine if the study had sufficient power to come to this conclusion. This is certainly not to say that they should never be done. Some extremely valuable information has come from these analyses and one should never assume that all groups will respond in the same way to the same treatments.