# Research Synthesis and Decision Making # ## April 15 ## --- name: outline ## Outline - New Material - Aggregating results (meta-analysis) - Decision making from experiments - Review of course - Discuss exam --- ## Meta-analysis - We rarely care about the results of individual studies 1. Measurement error 2. Under-powered designs 3. Limited external validity (SUTO) - Meta-analysis is the quantitative synthesis (aggregation) of experimental findings --- ## Meta-analysis Conducting a meta-analysis involves the following: 1. Identifying a research question 2. Establishing procedures for identifying studies on that question 3. Finding studies (published and unpublished) - This is to reduce "file drawer" problems 4. Convert published results into a standardized effect measure 5. Code studies for features (SUTO) 6. Regression study-specific effect sizes on study features, weighted by sample size --- ## Meta-analysis - Common effect size measures: 1. Standardized mean difference 2. `\( R^2 \)` 3. Standardized `\( \beta \)` - All depend on variation in the sample outcome --- ## Meta-analysis - Fully randomized experiments can only tell us about the ATE - Blocked designs can additionally inform us about effect heterogeneity - Meta-analysis lets us make inferences about heterogeneity from fully randomized designs --- ## Meta-analysis - Making decisions from single studies is tenuous - Meta-analysis should be the basis for decision making - Weighted based on study power (sample size) - Less sensitive to external validity concerns (SUTO) - Use meta-analytic effect estimates to design future studies --- ## Decision making - We do experiments to find out **what works** - Typical focus is on direction/significance of effects - We ultimately care about effect sizes - We rarely consider trade-offs in effects --- ## Decision making - Trade-offs in outcomes - Trade-offs in costs - Treatment propensities - Combine `\( Pr(Treated|Assigned) \)` with ATT - Effect uncertainty - Measurement error - External validity (SUTO) ??? Medical literature is really good about dealing with trade-offs between outcomes They also have standardized measures (quality-adjusted life years). Political science doesn't have these kind of measures, except maybe in terms of voter turnout numbers or vote shares. Think about situations like mobilizing citizens to vote for a particular candidate: - What is the effect on turnout? - What is the effect on vote share? - What is the effect on opponent-party turnout and vote share? - What is the cost? - What are the effects on other outcomes (political cynicism, trust in government, etc.)? --- template: outline --- ## Review 1. Fundamental problem of causal inference ??? Why do we do experiments? How do they help us? -- 2. Validity ??? Construct validity Internal validity External validity Statistical conclusion validity -- 3. Experimental Design ??? Experiments are complicated and involve many choices. All of those choices have to follow from a causal question and hypothesis. --- template: outline --- ## Exam 1. Cover everything in the protocol ??? 1. What is the research question? What constructs are be used? 2. What is the theoretical perspective? What are the precisely stated hypotheses? (Hypotheses must be testable and falsifiable via a comparison of two or more treatment groups.) 3. What variables are used? Manipulated? Measured? Outcomes? Are they valid operationalizations? 4. What is the experimental design? What conditions are involved? How are the hypothesized factors manipulated? 5. When, where, and how are data collected? Lab, field, or survey? How are units recruited and assigned? What costs are involved? Are there ethical concerns, compliance or attrition issues, or other practical considerations? What challenges do you anticipate in the implementation? 6. How will the experiment be analyzed and what will you do with the results? -- 2. Design and justify a budget ??? 1. What is your ideal experiment? How much does it cost? How many units are in the study? What is its statistical power? If you had less money, what would the design look like? -- 3. Focus on anticipating challenges and your strategies for addressing them ??? 1. You both want to address challenges before they happen and have strategies (practical or statistical for addressing them). How will you reallocate resources to address compliance or attrition? Will you use intention-to-treat, as-treated, or some other kind of analysis? --- ## Rest of semester - Work independently on exams - Setup a meeting with me (via email) to check on the status