Naomi Aronson has made technology assessment a science.
Medicine might be an art, but technology assessment is anything but. For Naomi Aronson and the staff at the Technology Evaluation Center of the Blue Cross and Blue Shield Association in Chicago, technology assessment means meticulous review of clinical evidence, and face-to-face meetings and teleconferences among clinicians and TEC staff members. In short, it’s a science.
Founded in 1985, TEC operates as part of the Blue Cross and Blue Shield Association’s Office of Clinical Affairs. Although it is part of Blue Cross, its assessments do not necessarily mean that individual Blue Cross and Blue Shield plans will reimburse providers who use the technology in question. Nor does TEC advocate, recommend, encourage or discourage any particular treatment, procedure or service.
With a staff of physicians and doctorate-level scientists with experience in academic and primary research – and the assistance of a medical advisory panel of outside experts – TEC publishes somewhere between 20 and 25 assessments a year. Recent assessments discussed artificial lumbar disc replacement, carotid artery angioplasty and stenting, and metal-on-metal total hip resurfacing. For 14 years, TEC has maintained a collaborative relationship with Kaiser Permanente, which gives TEC access to Kaiser’s clinical experts on a wide range of topics.
Although it’s unlikely that the typical IDN can duplicate TEC’s work, Journal of Healthcare Contracting readers can insist that new technologies be subjected to cold, scientific scrutiny before a purchasing decision is made. For direction, they can access TEC assessments online (at www.bcbs.com/betterknowledge/tec/) as well as assessments by other organizations, including the Federal Agency for Healthcare Research and Quality (www.ahrq.gov).
JHC recently spoke to TEC Executive Director Naomi Aronson, Ph.D., about TEC and the science of technology assessment. Aronson has directed more than 300 assessments as well as seven evidence reports for AHRQ. (TEC is one of AHRQ’s 13 Evidence-based Practice Centers, which serve as the foundation for organizations to develop clinical practice guidelines as well as tools and strategies for improving the quality of healthcare services they provide.)
Journal of Healthcare Contracting: Why did the Blue Cross and Blue Shield Association create TEC in 1985? Has its mission or the nature of its technology assessments changed since then?
Naomi Aronson: The mission is still the same – to provide the best possible, evidence-based information to help decision-makers understand whether a diagnostic, therapeutic or biological intervention improves outcomes. That’s what people care about – length of life, quality of life, the ability to function.
Regarding the nature of our assessments, [I would say that] technology has become more complicated. There’s definitely more technology out there. Sometimes it can be more difficult to understand its utility. Maybe it has an interesting technical twist, but does it make a difference to the patient?
We have maintained a medical advisory panel with overall clinical and scientific accountability. They have the authority to review our assessments, approve them or, if they think necessary, rewrite them. The panel comprises 17 individuals. Only four – three physicians and one pharmacist – are affiliated with [Blue Cross] plans. The remaining members are appointed by a number of specialty societies, representing surgeons, pediatricians, family physicians and others. The panel also has representation from two areas we feel are an important part of our range of clinical concerns: the American College of Cardiology and the American College of Medical Genetics.
We have always tried to play a leadership role. Our work has always been independent and subject to review. Now we’re truly transparent. Our assessments are available on our Web site. Anyone can see our work – our assumptions, data and analysis. We hope decision-makers will consult us and find it useful.
JHC: TEC is said to be a leader in producing evidence-based technology assessments. What are evidence-based assessments, and how do they differ from others?
Aronson: The question you have to ask is, “What is evidence, and how do you appraise its quality?” [The fact is], not all evidence is created equal.
Evidence is what’s published in medical journals every day. There are clinical trials, in which patients are given an intervention or diagnostic test and the results are recorded. The best trials are those that are controlled [in which one group gets the intervention, and another – the control – does not]. The gold standard is the randomized controlled trial, so we truly know there is not some kind of bias, or some way in which those who got the treatment are different than those who did not.
In the case of diagnostic technology, we’re interested in comparing the new technology to a reference standard – some call it a gold standard – that is, a diagnostic modality that has a known ability to accurately diagnose a condition. We want that same individual to have the reference test as well as the test that is being investigated.
There are other types of evidence. But the two main ones are randomized controlled trials for therapeutic devices and procedures; and benchmarking against reference technology for diagnostics.
All trials are not of equal quality, validity or reliability. They’re not equally able to tell us accurately about what is really happening in the world. There are a number of standards for appraising the quality of trials. There are rules about how you collect data, how you classify it; rules about what goes in the numerator and the denominator.
When you don’t have a well-conducted trial, you don’t have good information. As a result, you make poor decisions. Much of TEC’s expertise is in critically reading the literature and being able to evaluate its strength and weaknesses, to make an assessment about how compelling it is.
There’s a tendency for positive results to be reported more often than negative ones, because everybody wants to find something new, something they think is true. But before we truly believe we have something new, we examine the quality of the studies and make some kind of judgment about how compelling they are. There are set and expected rules about how trials should be conducted, and you can make assessments against them.
JHC: What’s the process you use to review literature?
Aronson: The process that we carry out – a systematic review – is really quite formalized. It is structured so we will get an unbiased view of literature. TEC is not the only organization conducting such reviews. The Cochrane Collaboration [www.cochrane.com] and AHRQ are two others.
Here’s an example. AHRQ has 13 Evidence-based Practice Centers [including TEC], which conduct systematic reviews and research aimed at answering a set of clinically important questions. About five years ago, we were involved on a project to support the National Heart, Lung, and Blood Institute [of the National Institutes of Health] as it updated guidelines for managing asthma. [NHLBI] and other stakeholders – including family practitioners and pediatricians – had questions they wanted to ask. As part of the project, we were given a set of six or seven detailed questions, like “If you’re going to start inhaled corticosteroids, should you start them as early as possible, or should you wait until the symptoms are not tolerable to the patient?” Another question concerned very young children and corticosteroids.
In order to answer questions such as these, we can’t just go out and read some articles for the answers. We have to make sure the questions are carefully constructed and objective. So we analyze their component parts. We ask, “Which patients should we be looking at?” “What interventions are we looking at – just a corticosteroid inhaler, or combination therapy?” If the latter, does it have to be a separate analysis? “What are the outcomes we are examining?” Lung function, of course. But how do we measure it, and at what length of time? A month? A year? Five years?
So we have to make sure the study is relevant to the population of interest, the intervention of interest, and the outcomes of interest. In this case, the population of interest was asthmatics. But we have to define, “Are these people with mild asthma or serious asthma? Are they adults or children?” Regarding interventions of interest, we ask, “Is it a corticosteroid inhaler, or is it an inhaler in combination with rescue medication?” and “What will we compare it to – a placebo?” The outcomes of interest are lung function. But as I said before, there are different measures of lung function?
The point I want to make is this: This review is precise and well-organized, which is why we call it a systematic review. We try to find all the relevant information and an acceptable study design for answering the questions. Then we research the literature, extract data, analyze it and draw conclusions.
I want to contrast this with a narrative review. That’s where an expert or two get together and write about what they think is most important in their field. This type of review is interesting, but not as specific, focused and insulated against bias as a systematic review. It does have its place, in that it tells you how experts in the field see things going. But one of the flaws we see in many studies is that the authors make conclusions that don’t really derive from their data. They see their data more broadly or optimistically than what is merited in their tables,
JHC: TEC completes 20 to 25 assessments a year. How do you decide what technologies to assess?
Aronson: The topic must be of national importance; it must be of clinical importance to patients; and it must have evidence attached to it with enough complexity to need analysis. TEC takes on more complicated analyses, or those that are public or controversial. Sometimes we don’t consider the evidence issues to be that difficult, but we know the technology has been played up in the media, or there’s advocacy around it, or there’s just a lot of curiosity about it. We review the journals, see what’s going on in the FDA, attend specialty meetings. Then we pool all that information, make a preliminary list, and – with the input of our advisory group – draw up an agenda.
JHC: Can you discuss the challenges of determining whether a technology improves net health outcomes, that is, whether its beneficial effects outweigh any harmful effects? How about the challenges of determining whether a new technology is as beneficial as established alternatives?
Aronson: Assuming the quality of the trials is reasonably good, we can set up a balance table [and ask] what are the potential harms and benefits? Can we quantify the likelihood of each outcome, good and bad? For example, in the recent FDA discussions of erythropoietic stimulants [for treatment of anemia in patients undergoing cancer treatment], there was discussion about how much does treatment reduce the risk of transfusion? How much does it increase the risk of a serious thromboembolic event? How does that compare with the risk of a serious reaction to transfusion or to a transfusion-acquired disease?
A major challenge to healthcare is that there is limited evidence on adverse effects and long-term effects. For a new technology, we often don’t have data on its use in large numbers of people or over many years. The Institute of Medicine and other experts have recently recommended that the FDA establish a better system of post-marketing follow-up. Another challenge is that there is limited information on comparative effectiveness. For example, when is surgery better than medical management, or vice versa?
So even when reasonably good evidence is available, and trials show that a technology is beneficial, we should appreciate that uncertainties remain. Our healthcare system needs to do a better job of studying the long-term outcomes and relative effectiveness of medical interventions.
JHC: Short of assembling a team such as TEC’s, how can a hospital or multihospital system evaluate technologies?
Aronson: Our position is that clinical research shouldn’t be done multiple times by multiple organizations. We’ve made our evidence available. And other organizations have done the same. I can’t imagine how even the strongest multihospital system could maintain the kind of research units that we and the AHRQ’s Evidence-based Practice Centers have.
And if you rely on manufacturers for all your information, you might not be getting a full, complete and balanced story. It’s great when the manufacturer puts a packet together. But until an experienced research team investigates that packet, you won’t know if it’s selective or complete.
JHC: What is your opinion about the way medical technologies are developed and marketed today? What can healthcare providers do to see that the process is improved?
Aronson: Part of the issue revolves around clinical leadership. Some specialty societies are steeped in evidence-based medicine, others are advocates of procedures. Ask yourself, “What is your physician leadership like?”
My message to industry is this: The questions we’re asking about outcomes and quality of evidence should be important to you, not only because it’s important to payers, but because it is important to physicians and patients. Ultimately, to be sustainable, technology needs to give us good interventions, good outcomes, and a good balance between benefits and harm.
We don’t explicitly look at cost, but affordability is one of the most important policy questions facing us today. There are 45 million uninsured individuals today. That’s about half the total population insured by Blue Cross and Blue Shield, and about the same number of people insured by Medicare. What’s more, there are employers who are having a difficult time affording healthcare benefits. So this is a collective problem, a broad public problem. If you’re in industry, it’s not enough to capture market share. You need to figure out how to bring value to the entire system.
JHC: Finally, can you comment on what you believe will be the most promising technologies of the next five years?
Aronson: I never make pronouncements; I’m a scientist, and that is speculation. That said, I can tell you areas in which there’s a lot of development – genomics; imaging and biological therapies.