Advances in communication and information technologies have led to an exponential increase in the collection of real-world data. Data in the health sector are not only generated during clinical research but also during many instances of the patient-clinician relationship. Such data are then processed to administer and manage health services and stored by a greater number of health registries and medical devices. This data serves as the basis for the growing use of real world evidence (RWE) in medical decision-making. However, data itself is not evidence. A core element of producing RWE includes the use of designs and analytical methods that are both valid and appropriate for such data. This book is about the analytical methods used to turn real world data into valid and meaningful real world evidence.

In 2010, we produced a book, Analysis of Observational HealthCare Data Using SAS (Faries et al. 2010), to bring together in a single place many of the best practices for real-world and observational data research. A focus of that effort was to make the implementation of best practice analyses feasible by providing SAS Code with example applications. However, since that time there have been several improvements in analytic methods, coalescing of thoughts on best practices, and significant upgrades in SAS procedures targeted for real world research, such as the PSMATCH and CAUSALTRT procedures. In addition, the growing demand for real world evidence and interest in improving the quality of real world evidence to the level required for regulatory decision making has necessitated updating the prior work.

This book has the same general objective as the 2010 text: to bring together best practices in a single location and to provide SAS code and examples to make the analyses relatively easy and efficient. In addition, we use newer SAS procedures for efficient coding that allow for the implementation of previously challenging methods (such as optimal matching). We will also present several emerging topics of interest, including algorithms for personalized medicine, methods that address the complexities of time varying confounding, extensions of propensity scoring to comparisons between more than two interventions, sensitivity analyses for unmeasured confounding, use of real-world data to generalize RCT evidence, and implementation of model averaging. As before, implementation of foundational methods such as propensity score matching and stratification and weighting methods are still included in detail.

The main focus of this book is causal inference methods – or the challenge of producing valid comparisons of outcomes between intervention groups using non-randomized data sources. The remainder of this introductory chapter provides a brief overview of real world data, uses of real world data, designs and guidance for real world data research, and some general best practices. This serves as a reference and introductory reading prior to the detailed applications using SAS in later chapters.

1.2 Definition and Types of Real World Data (RWD)

Real world data has been defined by the International Society for Pharmacoeconomics and Outcome Research (ISPOR) as everything that goes beyond what is normally collected in the phase III clinical trials programs (RCTs) (Garrison et al. 2007). Similarly, the Duke-Margolis Center for Health Policy and the Food and Drug Administration define RWD as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources.” These definitions include many different types and sources of data which are not limited to data from observational studies conducted in clinical setting but also electronic health records (EHRs), claims and billing data, product and disease registries, and data gathered through personal devices and health applications (NEHI 2015). RWD can comprise data from patients, clinicians, hospitals, payers and many other sources. There is some debate regarding the limits of RWD, since some institutions also consider pragmatic clinical trials to be RWD (Makady et al. 2015). Others describe pragmatic trials on a continuum between purely observational and clinical trial like based on a set of factors (Tosh et al. 2011). Note, in this book we use the terms “real world” and “observational” interchangeably.

1.3 Experimental Versus Observational Research

One of the main, if not the most, important objective of medicine is discovering the best treatment for each disease. To achieve this objective, medical researchers usually compare the effects of different treatments on the course of a disease with the randomized clinical trial (RCT) as the gold-standard design for such research. In an RCT, the investigator compares the outcomes of patients assigned to different treatments. To ensure a high degree of internal validity of the results, treatment assignment is usually random, which is expected to produce treatment groups that are similar at baseline regarding the factors that may determine the outcomes, such as disease severity, co-morbidities, or other prognostic factors. With this design, we assume that outcome differences among the groups are caused by differences in the efficacy of treatments. (See Chapter 2 for a technical discussion of causal inference.) Given that the research protocol decides who will receive a treatment, RCTs are considered experimental research. However, in observational research in which the investigators collect information without changing clinical practice, medications are not assigned to the patients randomly, but are prescribed by clinicians following their own criteria. This means that similarities between groups of patients receiving different treatments cannot be assumed. For example, assume that there are two treatments for a disease, one of which is known to be more effective but might produce more frequent and severe adverse events, and the other, which is much better tolerated but it is known to be less effective. Typically, physicians will prescribe the more effective treatment to the more severe patients and may prefer to start treatment of the milder patients with the better tolerated treatment. The simple comparison of outcomes of patients receiving the two treatments, which is the usual strategy in RCTs, can produce biased results since more severe patients may be prone to worse outcomes. This book will describe strategies to produce valid results taking into account the differences between treatment groups.

RCTs have other design features that improve internal validity, such as standardized treatment protocols; strict patient and investigator selection criteria; common data collection forms; and blinding of patients, treatment providers, and evaluators (Wells 1999, Rothwell 1995). However, these design features almost certainly compromise external validity or generalizability, posing important limitations on translating findings to common practice and informing clinical practice and policy decisions about treatments (Gilboby et al. 2002). Patients with co-morbidities, those who might be less compliant with treatments, and those who are difficult to treat are many times excluded from clinical trials. Accordingly, it is not clear if the findings from clinical trials can be generalized to the overall population of patients. Real world data by definition includes a more representative sample of patients, and therefore can produce more generalizable results.

The traditional view is that RWD, data from observational studies that is collected during usual clinical work, can complement the results of RCTs by assessing the outcomes of treatments in more representative samples of patients and in circumstances much nearer to the day-to-day clinical practice. However, real world data research is quickly expanding to a broader set of clinical questions for drug development and health policy as discussed in the next sections.

1.4 Types of Real World Studies

There are two large types of studies: descriptive and analytical. Descriptive studies simply describe a health situation such as a prevalence study that conducts a survey to determine the frequency or prevalence of a disorder or an incidence study in which we follow a group of individuals to determine the incidence of a given disease. In analytical studies, we analyze the influence of an intervention (exposure) on an outcome. Analytical studies can be divided, as we have seen above, into experimental and observational. In experimental studies, the investigator is able to select the interventions and then compare the outcomes (that is, cure from disease) of individuals exposed to the different interventions. The RCT is the typical example of a clinical experimental study. Conversely, in analytical observational studies, which are the ones that are conducted using RWD, the investigator only observes and records what happens, but does not modify the interventions the subjects receive. The rest of this section is a very brief and high-level look at the different types of analytical observational studies given in Table 1.1. For a thorough presentation of study designs, see the following references (Rothman et al. 2012, Fletcher et al. 2014).

Table 1.1: Types of Analytical Epidemiological and Clinical Studies

Experimental	Observational
Randomized clinical trial	Cross-sectional
Randomized community intervention	Retrospective or case-control
	Prospective or cohort

1.4.1 Cross-sectional Studies

The classification of analytical observational studies is based on the time frame that we observe the subjects. In cross-sectional studies, we simultaneously study intervention/exposure and disease in a well-defined population at a given time. This simultaneous measurement does not allow us to know the temporal sequence of the events, and it is therefore not possible to determine whether the exposure preceded the disease or vice versa.

An example of a cross-sectional study is the assessment of individuals who are treated for a disease in a health care center. This information is very useful to assess the health status of a community and determine its needs, but cannot inform on the causes of a disorder or the outcomes of a treatment. Cross-sectional studies often serve as descriptive studies and help formulate etiological hypotheses.

1.4.2 Retrospective or Case-control Studies

Retrospective or case-control studies identify individuals who have already experienced the outcome of interest, for example, comparing individuals with a disease with an appropriate control group that does not have the disease. The relationship between one or several factors related to the disease are examined by comparing the frequency of exposure to risk or protective factors between cases and controls. These studies are named “retrospective” because they start from the effect and retrospectively evaluate the exposure of interest in the individuals who have and do not have the disease to ascertain the factors that may be related to that disease. If the frequency of exposure to the cause is greater in the group of cases of the disease than in the controls, we can say that there is an association between the exposure and the outcome.

1.4.3 Prospective or Cohort Studies

Finally, in cohort studies, individuals are identified based on the presence or absence of an intervention (for example, a treatment of interest). At this time, the participants have not experienced the outcome and are followed for a period of time to observe the frequency of the outcome of interests. At the end of the observation period, the outcomes from each of the cohorts (intervention groups) are compared. If the outcomes are different, we can conclude that there is a statistical association between the intervention and outcome. In this type of study, since the participants have not experienced the outcome at the start of the follow-up, the temporal sequence between exposure and disease can be established more clearly. In turn, this type of study allows the examination of multiple effects before a given intervention.

Cohort studies can be prospective and historical depending on the temporal relationship between the start of the study and the outcome of interest. In the retrospective, both the intervention and the outcome have already happened when the study was started. In the prospective, the exposure could have occurred or not, but the outcome has not been observed. Therefore, a follow-up period is required to determine the frequency of the outcome. Cohort studies are the observational studies most appropriate to analyze the effects of treatments and are the source for the data sets described in Chapter 3 that are used across the remainder of this book.

1.5 Questions Addressed by Real World Studies

Common objectives of health research include:

1. characterizing diseases and describing their natural course

2. assessing the frequency, impact and correlates of the diseases at the population level

3. finding the causes of diseases

4. discovering the best treatments

5. analyzing the best way to provide treatment

6. understanding the health systems and the costs associated with diseases

All these questions can be addressed with RWD and produce RWE. Real world research is actually the only way of addressing some of these questions, given feasibility and/or ethical challenges.

In drug development, there are a growing number of uses of RWE across the entire life cycle of a product. (See Figure 1.1.) Examples range from epidemiologic and treatment pattern studies to support early phase clinical development to comparative effectiveness, access and commercialization studies, and safety monitoring using claims and EMR data after launch. Recently, RWE has expanded to additional uses such as (1) forming control arms for single arm studies in rare or severe diseases for regulatory evaluation, and (2) used as the basis for evaluating value-based agreements between drug manufacturers and health care payers.

Figure 1.1: Use of RWE Across the Drug Development Life Cycle

1.6 The Issues: Bias and Confounding

Regardless of the type of design, any study should aim to produce results that are valid. Biases are the main threat to the validity of research studies. A bias is a systematic error in the design, implementation, or analysis of a study. While there are multiple classifications of the various types of biases, we follow the taxonomy used by Grimes et al. (2002) and discuss selection bias, information bias, and confounding.

1.6.1 Selection Bias

Selection biases can occur when there are differences – other than the intervention itself – between the intervention/control groups being compared. It is common in observational health care research that there will be systematic differences in the types of patients in each intervention group. When these differences are in variables that are prognostic (and thus confounding exists), bias can result and must be addressed. Selection bias can also appear in other forms. Bias can result when the sample from which the results are obtained are not representative of the population, not because of chance, but because of an error in the inclusion or exclusion criteria, or in the recruitment process.

A second source of bias is loss to follow up, when data that are not obtained are systematically different from data that is available. A third reason for selection bias is the absence of response. This is typical of many studies because many times those who do not answer differ in something from those who do. Fourth, selective survival occurs when prevalent cases are selected instead of incidents. This type of bias is typical of case-control studies, in which the more severe or milder cases are under-represented by exitus or cure. Finally, self-selection bias can occur due to volunteer participation. In general, there is a risk that these individuals have different characteristics than non-volunteers.

1.6.2 Information Bias

Information or classification bias occurs when there is error in the measurement of the study variables in all or some of the study subjects. This can occur due to the use of non-sensitive or unspecific tests, use of incorrect or variable diagnostic criteria, and inaccuracy in the collection of data. When the error is similar in both intervention groups of interest, this is termed non-differential information bias. On the contrary, if errors are preferentially or exclusively in one group, the bias is differential. The non-differential bias skews the results in favor of the null hypothesis (tends to decrease the magnitude of the differences between groups), so in cases where significant differences are still observed, the result can still have value. However, the impact of differential bias is difficult to predict and seriously compromises the validity of the study.

There are two common information biases in case-control studies (also those with retrospective cohorts):

● memory bias – for example, those with a health problem remember their antecedents in a different way than those who do not

● interviewer bias – the information is requested or interpreted differently according to the group to which the subject belongs

However, prospective studies are also subject to information biases because, for example, a patient may try to answer to please the investigator (social desirability bias) or the investigator might voluntarily or involuntarily modify the assessment in the direction of the hypothesis that she or he wants to prove.

1.6.3 Confounding

Confounding occurs when the association between the study factor (intervention or treatment) and the response variable can be explained by a third variable, the confounding variable, or, on the contrary, when a real association is masked by this factor. For a variable to act as a confounder, it must be a prognostic factor of the outcome and be associated with exposure to the intervention, but it must not be included in the pathway between exposure and outcome. For example, assume that we studied the association between smoking and coronary heart disease and that the group of patients who smoke most often is the youngest. If we do not take into account age, the measure of global association will not be valid because the “beneficial” effect of being younger could dilute the harmful effect of tobacco on the occurrence of heart disease. In this case, the confounding variable would underestimate the effect of the exposure, but in other cases, it can result in overestimation. If a confounding factor exists but is not measured or available for analysis in a particular study, it is referred to as an unmeasured confounder.

It is confounding that raises the greatest challenge with causal inference analyses based on RWD. Even if one appropriately adjusts for measured confounders (the topic of much of this book), there is no guarantee that unmeasured confounders do not exist. This is an unprovable assumption that is necessary for most causal inference methods. Thus, comparative observational research sits lower on the hierarchy of evidence than randomized controlled trials. Chapter 2 provides a full discussion of causal inference and the assumptions necessary for causal inference analyses from non-randomized data.

1.7 Guidance for Real World Research

The growing use of real world evidence research and the growing recognition of the challenges to validity of such evidence has sparked multiple groups to propose guidance documents for the design, conduct, and reporting of observational research. The specific aims of each effort varies, but the general goal is to improve the quality and reliability in real world data research. Table 1.2 provides a summary and references to key guidance documents.

Table 1.2: Summary of Guidance Documents for Real World Evidence Research

Year	Guidance or Sponsor	Reference	Summary
2004	TREND - CDC	Des Jarlais DC, Lyles C, Crepaz N, and the TREND Group (2004). Improving the Reporting Quality of Nonrandomized Evaluations of Behavioral and Public Health Interventions: The TREND Statement. Am J Public Health 94:361-366.https://www.cdc.gov/trendstatement	22-Item Checklist – designed to be a non-randomized research complement to the CONSORT guidelines for reporting randomized trials.
2007	STROBE	von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP, and the STROBE Initiative (2007). The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Epidemiology 18(6):800-4.Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, Poole C, Schlesselman JJ, Egger M, and the STROBE Initiative (2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE):explanation and elaboration. Epidemiology 18(6):805-35.https://strobe-statement.org	Checklist focused on improving the reporting of observational studies.
2009	ISPOR Good Practices	Berger ML, Mamdani M, Atkins D, Johnson ML (2009). Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: The ISPOR good research practices for retrospective database analysis task force report—Part I. Value in Health 12:1044-52.Cox E, Martin BC, Van Staa T, Garbe E, Siebert U, Johnson ML (2009). Good Research Practices for Comparative Effectiveness Research: Approaches To Mitigate Bias And Confounding In The Design Of Non-randomized Studies of Treatment Effects Using Secondary Databases: Part II. Value in Health 12(8):1053-61.Johnson ML, Crown W, Martin BC, Dormuth CR, Siebert U (2009). Good Research Practices for Comparative Effectiveness Research: Analytic Methods to Improve Causal Inference from Nonrandomized Studies of Treatment Effects using Secondary Data Sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—Part III. Value in Health. 2009;12(8):1062-1073.https://www.ispor.org/heor-resources/good-practices-for-outcomes-reserarch/report	ISPOR sponsored effort to provide guidance on quality observational research at a more detailed level than previous checklists (three-part manuscript series).
2010	GRACE	Dreyer NA, Schneeweiss S, McNeil B, et al (2010). GRACE Principles: Recognizing high-quality observational studies of comparative effectiveness. American Journal of Managed Care 16(6):467-471.Dreyer NA, Velentgas P, Westrich K et al (2014). The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution. Journal of Managed Care Pharmacy 20(3):301-08.Dreyer NA, Bryant A, Velentgas P (2016). The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness. Journal of Managed Care and Specialty Pharmacy 22(10):1107-13.https://www.graceprinciples.org/publications.html	Collaboration with ISPE to develop principles to allow assessment of the quality of observational research for comparative effectiveness: principles document and a validated checklist.
2014	Joint Effort from ISPOR – AMPC & NPC	Berger M, Martin B, Husereau D Worley K, Allen D, Yang W, Mullins CD, Kahler K, Quon NC, Devine S, Graham J, Cannon E, Crown W (2014). A Questionnaire to assess the relevance and credibility of observational studies to inform healthcare decision making: an ISPOR-AMCP- NPC Good Practice Task Force. Value in Health 2014; 17(2):143-156.https://www.isport.org/heor-resources/good-practices-for-outcomes-research	Joint effort between 3 professional societies to produce a questionnaire in flowchart format to assess the credibility of observational studies.
2017	Joint ISPOR-ISPE Task Force	Berger ML, Sox H, Willke RJ, Brixner DL, Eichler HG, Goettsch W, Madigan D, Makady A, Schneeweiss S, Tarricone R, Wang SV, Watkins J, Mullins CD (2017). Good Practices for Real‐World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR‐ISPE Special Task Force on Real‐World Evidence in Health Care Decision Making. Pharmacoepidemiology and Drug Safety 26(9): 1033-1039.https://www.ispor.org/heor-resources/good-practices-for-outcomes-research	Joint effort between ISPOR and ISPE building upon previous work within each society focusing improving the transparency and replicability of observational research.
2017	PCORI	Patient Centered Outcomes Research Institute (PCORI) Methodology Committee. (2017). Chapter 8. Available at https://www.PCORI.org/sites/defaultfiles/PCORI-Methodology-Report.pdf.	One section of the PCORI Methodology report focused on good practice principles for causal inference from observational research.
2017	FDA (CDRH)Device Guidance	Use of Real World Evidence to Support Regulatory Decision-Making for Medical Devices.https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM513027.pdf	FDA (CDRH) Guidance for Industry on the use of real world evidence for regulatory decision making for medical devices.
2017	Duke-Margolis White Paper	A Framework for Regulatory Use of Real World Evidence.https://healthpolicy.duke.edu/sites/default/files/atoms/files/rwe_white_paper_2017.09.06.pdf.	Duke-Margolis center led effort with multiple stakeholders to guide development of what constitutes RWE that is fit for regulatory purposes
2018	FDA	Framework for FDA’s Real World Evidence Program.https://www.fda.gov/media/120060/download	FDA guidance (for medicines, not devices) to assist developers on the use of RWE to support regulatory decision making

Early efforts on guidance documents produced checklists focused on quality reporting of observational research with items ranging from study background to bias control methods to funding sources (Table 1.2). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) was a collaboration of epidemiologists, journal editors, and other researchers involved in the conduct and reporting of observational research. The TREND group checklist was designed to mimic the CONSORT checklist for randomized controlled trials. Both of these efforts produced 22-item checklists and reminded those disclosing observational research of the core issues that were both common to randomized research reporting and the unique reporting issues for observational research.

The next set of guidance documents was largely led by key professional societies involved in the conduct and reporting of real world evidence. The Good Research for Comparative Effectiveness (GRACE) principles was a collaboration between experienced academic and private researchers and the International Society of Pharmacoepidemiology (ISPE). This began with a set of quality principles published in 2010 that could be used to assess the quality of comparative observational research and provided a set of good practice principles regarding the design conduct, analysis, and reporting of observational research. These principles were further developed into a checklist, which was validated as a tool through multiple research studies.

The International Society of Pharmacoeconomics and Outcomes Research (ISPOR) commissioned a task force to develop its own guidance with a goal of providing more detail than a checklist as well as covering more of the research process. Specifically, they began with guidance on developing the research question and concluded with much more detail regarding methods for control of confounding. The end result was a three-paper series concluding with a focused discussion of analytic methods.

More recently, joint efforts have produced further quality guidance for researchers developing and disclosing observational studies. A joint ISPOR-ISPE task force was created to produce good procedural practices that would increase decision maker’s confidence in real world evidence. The intent here was to build on the earlier separate work from ISPE and ISPOR on the basic principles and address the transparency of observational research. Specifically, this covered seven topics including study registration, replicability, and stakeholder involvement. For instance, these guidelines recommend a priori registration of hypothesis evaluating treatment effectiveness (HETE) studies for greater credibility.

ISPOR, the Academy of Managed Care Pharmacy (AMPC), and the National Pharmaceutical Council (NPC) jointly produced a document to guide reviewers on the degree of confidence one can place on a specific piece of observational research as well as further educate the field on the subtleties of observational research issues. The format used was a questionnaire in flowchart format that focused on issues of credibility and relevance.

Recently, the debate has focused on the potential regulatory use of RWE. This has been hastened by the 21st Century Cures Act, which mandates the FDA to produce a guidance document regarding regulatory decision making with RWE. The FDA had previously released guidance for industry on the use of RWE for regulatory decision making for medical devices. A main focus of this document was on ensuring the quality of the data – as much real world data is not captured in a research setting and inaccurate recordings of diagnoses and outcome ascertainment can seriously bias analyses. The Duke-Margolis Center for Health Policy has taken up leadership in the debate on regulatory use of RWE and organized multiple stakeholders to develop a framework for the regulatory use of RWE. They released a white paper (Duke Margolis Center for Health Policy, 2017) that discusses what quality steps are necessary for the development and conduct of real world evidence that could be fit for regulatory purposes. Most recently (December 2018), the FDA released a framework for the use of RWE for regulatory decision making. This outlines how the FDA will evaluate the potential use of RWE to support new indications for approved drugs or satisfy post-approval commitments.

Also of note is the Get Real Innovative Medicine Initiative (IMI), a European consortium of pharmaceutical companies, academia, HTA agencies, and regulators. The goals are to speed the development and adoption of new RWE-related methods into the drug development process. A series of reports or publications on topic such as assessing the validity or RWE designs and analysis methods and innovative approaches to generalizability have been or are under development (http://www.imi-getreal.eu).

Common themes among all of the guidance documents include pre-specification of analysis plans, ensuring appropriate and valid outcome measurement (data source), adjustment for biases, and transparency in reporting.

1.8 Best Practices for Real World Research

Regarding the process for performing a comparative analysis from real world data, we follow the proposals of Rubin (2007) and Bind and Rubin (2017), which are in alignment with the guidance documents in Table 1.2. Specifically, they propose four stages for a research project:

1. Conceptual

2. Design

3. Statistical Analysis

4. Conclusions

In the initial conceptual stage, researchers conceptualize how they would conduct the experiment as a randomized controlled trial. This allows the development of a clear and specific causal question. At this stage we also recommend following the International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) E9 guidance of carefully defining your estimand after the objectives of the study are developed. The estimand consists of the population that you want to draw inference to, the outcome to be measured on each patient, intercurrent events (for example, post initiation events such as switching of medications, non-adherence), and the population level summary of the outcome (https://www.ema.europa.eu/documents/scientific-guideline/draft-ich-e9-r1-addendum-estimands-sensitivity-analysis-clinical-trials-guideline-statistical_en.pdf). At the end of Stage 1 you have a clear goal allowing for development of an analysis plan.

Stage 2 is the design stage. The goal here is to approximate the conditions of the conceptualized randomized trial and ensure balance in covariates between treatment groups. This design stage will include a quantitative assessment of the feasibility of the study and confirmation that the bias adjustment methods (such as propensity matching) bring balance similar to a randomized study. Creating directed acyclic graphs (DAGs) are very useful here as this process will inform the feasibility (do we even have the right covariates?) and selection of the variables for the bias adjustment models. A key issue here is that the design stage is conducted “outcome free.” That is, one conducts the feasibility assessment, finalizes, and documents the statistical analysis methods prior to accessing the outcome data. One can use the baseline (pre-index) data – this will allow confirmation of the feasibility of the data to achieve the research objectives – but should have no outcomes data in sight. For a detailed practical discussion of the design phase planning for causal inference studies, we recommend following the concepts described by Hernan and Robins (2016) in their target trial approach.

Stage 3 is the analysis stage. Too often this is the first step in an analysis that can lead to “cherry-picking” of methods that give the desired results or analyses not tied to the estimand of interest. In this stage, the researcher conducts the pre-planned analyses for the estimand, sensitivity analyses to assess the robustness of the results, analyses of secondary objectives (different estimands), and any ad hoc analyses driven by the results (such should be denoted as ad hoc). Note that while some sensitivity analyses should cover study specific analytic issues, in general researchers should include assessment of the core assumptions needed for causal inference using real world data (unmeasured confounding, appropriate modeling, positivity; see Chapter 2).

Lastly, stage 4 studies the causal conclusions from the findings. Because this text is focused on the analytic portions of real world research, we will focus primarily on stages 2 and 3 of this process in the chapters moving forward.

1.9 Contents of This Book

The book is organized as follows. This chapter and Chapter 2 provide foundational information about real world data research with a focus on causal inference in Chapter 2. Chapter 3 introduces the data sets that are used in the example analyses throughout the remainder of the book as well as a brief discussion on how to simulate real world data. Chapters 4–10 contain specific methods demonstrating comparative (causal) analyses of outcomes between two or more interventions that adjust for baseline confounding using propensity matching, stratification, weighting methods, and model averaging. Chapters 11 and 12 demonstrate the use of more complex methods that can adjust for both baseline and time-varying confounders and are applicable for longitudinal data such as to account for changes in the interventions over time. Lastly, Chapters 13–15 present analyses regarding the emerging topics of unmeasured confounding sensitivity analyses, quantitative generalizability analyses, and personalized medicine.

Each chapter (beginning with Chapter 3) contains: (1) an introduction to the topic and methods discussion at a sufficient level to understand the implementation of and the pros and cons of each approach, (2) a brief discussion of best practices and guidance on the use of the methods, (3) SAS code to implement the methods, and (4) an example analysis using the SAS code applied to one of the data sets discussed in Chapter 3.

References

Berger ML, Mamdani M, Atkins D, Johnson ML (2009). Good research practices for comparative effectiveness research: defining, reporting and interpreting nonrandomized studies of treatment effects using secondary data sources: The ISPOR good research practices for retrospective database analysis task force report—Part I. Value in Health 12:1044-52.

Berger M, Martin B, Husereau D Worley K, Allen D, Yang W, Mullins CD, Kahler K, Quon NC, Devine S, Graham J, Cannon E, Crown W (2014). A Questionnaire to assess the relevance and credibility of observational studies to inform healthcare decision making: an ISPOR-AMCP- NPC Good Practice Task Force. Value in Health 17(2):143-156.

Berger ML, Sox H, Willke RJ Brixner DL, Eichler HG, Goettsch W, Madigan D, Makady A, Schneeweiss S, Tarricone R, Wang SV, Watkins J, Mullins CD (2017). Good Practices for Real‐World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR‐ISPE Special Task Force on Real‐World Evidence in Health Care Decision Making. Pharmacoepidemiology and Drug Safety 26(9): 1033-1039.

Bind MAC, Rubin DB (2017). Bridging Observational Studies and Randomized Experiments by Embedding the Former in the Latter. Statistical Methods in Medical Research 28(7):1958-1978.

Cox E, Martin BC, Van Staa T, Garbe E, Siebert U, Johnson ML (2009). Good Research Practices for Comparative Effectiveness Research: Approaches To Mitigate Bias And Confounding In The Design Of Non-randomized Studies of Treatment Effects Using Secondary Databases: Part II. Value in Health 12(8):1053-61.

Des Jarlais DC, Lyles C, Crepaz N, TREND Group (2004). Improving the Reporting Quality of Nonrandomized Evaluations of Behavioral and Public Health Interventions: The TREND Statement. Am J Public Health. 94:361-366.

Dreyer NA, Bryant A, Velentgas P. (2016). The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness. Journal of Managed Care and Specialty Pharmacy 22(10):1107-13.

Dreyer NA, Schneeweiss S, McNeil B, et al. (2010). GRACE Principles: Recognizing high-quality observational studies of comparative effectiveness. American Journal of Managed Care 16(6):467-471.

Dreyer NA, Velentgas P, Westrich K et al. (2014). The GRACE Checklist for Rating the Quality of Observational Studies of Comparative Effectiveness: A Tale of Hope and Caution. Journal of Managed Care Pharmacy 20(3):301-08.

Duke Margolis Center for Health Policy - White Paper (2017). A Framework for Regulatory Use of Real World Evidence. Accessed on Jan 12, 2019 at https://healthpolicy.duke.edu/sites/default/files/atoms/files/rwe_white_paper_2017.09.06.pdf.

Faries D, Leon AC, Haro JM, Obenchain RL. (2010). Analysis of Observational Health Care Data Using SAS. Cary, NC: SAS Institute Inc.

Fletcher RH, Fletcher SW, Fletcher GS (2014). Clinical Epidemiology, 5ª Edition, Baltimore, MD: Wolters Kluwer.

Food and Drug Administration (FDA). Use of Real World Evidence to Support Regulatory Decision-Making for Medical Devices. https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuemnts/UCM513027.pdf. Accessed 10/3/2019.

Food and Drug Administration (FDA). Framework for FDA’s Real World Evidence Program. https://www.fda.gov/media/120060/download. Accessed 10/3/2019.

Garrison LP, Neumann PJ, Erickson P, Marshall D, Mullins CD (2007). Using Real-World Data for Coverage and Payment Decisions: The ISPOR Real-World Data Task Force Report. Value in Health 10(5): 326-335.

Gilbody S, Wahlbeck K, Adams C (2002). Randomized controlled trials in schizophrenia: a critical perspective on the literature. Acta Psychiatr Scand 105:243–51.

Guidance for Industry and FDA Staff: Best Practices for Conducting and Reporting Epidemiologic Safety Studies Using Electronic Healthcare Data (2013). Accessed January 2019 at: https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM243537.pdf.

Hernan MA, Robins JM (2016). Using Big Data to Emulate a Target Trial When a Randomized Trial is Not Available. Am J Epi 183(8):758-764.

Johnson ML, Crown W, Martin BC, Dormuth CR, Siebert U (2009). Good Research Practices for Comparative Effectiveness Research: Analytic Methods to Improve Causal Inference from Nonrandomized Studies of Treatment Effects using Secondary Data Sources: the ISPOR Good Research Practices for Retrospective Database Analysis Task Force Report—Part III. Value in Health 12(8):1062-1073.

Makady MS A, de Boer A, Hillege H, Klungel O, Goettsch W on behalf of GetReal Work Package 1 (2017). What Is Real-World Data? A Review of Definitions Based on Literature and Stakeholder Interviews. Value in Health 20(7):858-865

Network for Excellence in Health Innovation (NEHI) (2015). Real World Evidence: A New Era for Health Care Innovation. https://www.nehi.net/publications/66-real-world-evidence-a-new-era-for-health-care-innovation/view. Posted September 22, 2015, Accessed October 2, 2019.

Patient Centered Outcomes Research Institute (PCORI) Methodology Committee. (2017). Chapter 8. Available at https://www.PCORI.org/sites/defaultfiles/PCORI-Methodology-Report.pdf.

Rothman KJ, Lash TL, Greenland S (2012). Modern Epidemiology, 3rd Edition. Baltimore, MD: Wolters Kluwer.

Rothwell PM (1995). Can overall results of clinical trials be applied to all patients? Lancet 345:1616–1619.

Rubin DB (2007). The Design versus the Analysis of Observational Studies for Causal Effects: Parallels with the Design of Randomized Trials. Statistics in Medicine 26(1): 20-36.

Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, Poole C, Schlesselman JJ, Egger M and the STROBE Initiative (2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE):explanation and elaboration. Epidemiology 18(6):805-35.

von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP and the STROBE Initiative (2007).

The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Epidemiology 18(6):800-4.

Wells KB (1999). Treatment research at the crossroads: the scientific interface of clinical trials and effectiveness research. Am J Psychiatry 156:5–10.

Woodward M (2019). Epidemiology: Study Design and Data Analysis, Third Edition. Boca Raton, FL: CRC Press.

Подняться наверх