Statistics and its Limitations

LIES, DAMN LIES AND NO MORE

Randomised controlled trials have been in vogue for over a hundred years and are well known to anyone involved in research. It is the ultimate gold standard quality check for interventions.

It is able to perform this by using an invaluable ingredient embedded within the structure of the design itself. This is its cold ruthlessness in mitigating human folly, emotions, and heuristic errors. It leaves the assignment essentially to the toss of a coin and then carefully observes what happens when the chips fall where they may.

A crucial detail is that the group should be quite homogeneous, lest the results no longer be random. And this is where the rubber finally hits the road. In order to have such a population, meticulous planning and impeccable execution are critical. This requires extreme effort, not to mention the not-so-tiny issue of cost!

The other major category of studies is ‘observational’, and these have the benefit of somewhat quickly assembling a group with relative ease. The pendulum of the problem then swings to the other end and leads to the issue of heterogeneity. Every researcher knows that this means new variables, both known and unknown, being unevenly distributed amongst the study arms.

In 1983, Paul R. Rosenbaum and Donald Rubin, two American statisticians, came up with a novel approach to partially mitigate this problem. They called it Propensity Score Matching (PSM), which is based on each member of the study being assigned a probability of receiving the intervention. For instance, in an RCT the probability would be 0.5 (remember the coin toss!).

However, for an observational study, the probability would rather vary based on numerous variables ranging from age and sex to co-morbidities, etc. calculated by regression.

The next step is matching. Individuals from each arm of the study are linked to their counterparts if they have similar probability scores. There are many ways to do this mathematically (think proverbial cat-skinning). At the end, unmatched individuals could be excluded from the PSM analysis.

Finally, the age ol’ statistical trick that no one really understands (except fellows and chairs of various such departments worldwide). This is usually some form of analysis such as Cox/logistic/linear models, based on the nature of the data.

There are numerous studies in vascular surgery that have used this method for analysis in the past decade or so. These include the VALOR trial, which looked into outcomes following stent graft repairs in thoracic aneurysms; the TCAR study based on the VQI database; and the UK-COMPASS, which looked into complex EVARs in juxta-renal aneurysms amongst others.

One limitation of the PSM technique is that it can only account for known variables. The advantage of an RCT in part is that you could safely bet the unknowns would be equally distributed across the aisle.

Now that you’re a master boss with PSM, you would be able to make sense of the following.