# Concepts, Questions, and Hypotheses # # February 11 # --- name: outline1 ## Outline - What is an experiment? -- name: outline2 - Concepts -- name: outline3 - Theory/hypotheses -- name: outline4 - Druckman and Nelson exercise -- name: outline5 - Next week --- .left-column[ ## Definition ] .right-column[ > The observation of units after, and possibly before, a randomly assigned intervention in a controlled setting, which tests one or more precise causal expectations. ] --- .left-column[ ## Definition ## Elements ] .right-column[ 1. Causal theory or expectation 2. Physical intervention 3. Control 4. Randomization 5. Measurement ] --- ## "Just do an experiment?" - We never *just do an experiment* - Experiments have to be: - theoretically motivated - test interesting questions --- <<<<<<< HEAD ## What can we text experimentally? ======= ## What can we test experimentally? >>>>>>> 0a43252cb965e83390b62bff70e59c9a4bd7db86 - Forward causal questions? - Backward causal questions? --- <<<<<<< HEAD ## What can we text experimentally? ======= ## What can we test experimentally? >>>>>>> 0a43252cb965e83390b62bff70e59c9a4bd7db86 - Forward causal questions? - Does X cause Y? - What effects does X have? - "What if?" questions - Backward causal questions? ??? Experiments are good at answering "what if" questions - What if we gave these units some treatment - What if we changed the way we did things We need to think about whether these units would ever actually experience this treatment in the real world E.g., Giving poor people in India microfinance loans can help us understand the causal effect of loans on some outcome (fertility, health, income) - i.e., *what* would happen *if* poor people had access to microcredit? But that doesn't tell us anything about whether microfinance loands *do* cause effects on those outcomes in the real world. Our treatments may never happen in the real world --- <<<<<<< HEAD ## What can we text experimentally? ======= ## What can we test experimentally? >>>>>>> 0a43252cb965e83390b62bff70e59c9a4bd7db86 - Forward causal questions? - Does X cause Y? - What effects does X have? - Backward causal questions? - What causes Y? - How much of Y is attributable to X? --- ## Why not backward causal questions? - The set of potential X's is infinite - We can only test a few at a time - Some X's might be unobservable or unknown - Showing that X1 causes Y doesn't tell us anything about whether X2 causes Y --- name: questions class: middle, center Questions? --- ## Protocol Protocol is the complete planning document for how to design, implement, and analyze an experiment It contains details of: 1. Theory/hypotheses 2. Instrumentation 3. Sampling 4. Implementation 5. Analysis 6. Procedures for recording deviations from protocol ??? This is what you should use in designing your own study. What questions did people have about protocol? What was surprising or unexpected in the document? --- ## Protocol Today we'll focus on the first of these: 1. Theory/hypotheses --- template: outline2 --- ## Concepts - Experimenters often focus a lot on procedure, but not on concepts - Defining concepts of interest has to be a first step - Concepts are what connect the experiment to extant literature and the real world * .footnote[* Shadish, Cook, Campbell p.65] --- > "The empiricist perspective seems reasonable on the face of things. And yet we are unable to talk about questions of fact without getting caught up in the language that we use to describe these facts. To be sure, things exist the world separate from the language we use to describe them. However, we cannot talk about them unless and until we introduce linguistic symbols." .footnote[Gerring p.114] --- ## Concept definition - Term/label - Attributes (i.e., definition) - Indicators (i.e., operationalization) --- ## Example **Democracy** - What attributes does democracy have? ??? Free and fair elections? Competitiveness? Individual liberty? Market economy? -- - How can we measure it? ??? -- - How do we distinguish it from other concepts? ??? Polyarchy Representative democracy vs. direct democracy **Start thinking about concepts that interest you** --- ## Pattern matching - How well do particular cases fit your definition of the construct? - Is Ukraine a democracy? - Is abortion a contentious political issue? - Is 1 Euro/day a poverty wage? - When is an immigrant considered "Danish"? --- ## How do we develops concepts? - Look to existing literature - Clear definition - Disagreement about definition and measurement - Generate our own ??? Concepts are not mutually exclusive and the same real-world phenomenon might have different names in different contexts; or different concepts might share a name in different contexts E.g., "Individual rights" means something different here and now than it means in other places today or in this place a long time ago Generating our own concepts is often harder, but sometimes necessary when existing concepts don't fit our phenomenon of interest This happens all the time in psychological research --- ## Conceptualization - Resonance - Domain - Consistency - Fecundity - Differentiation - Utility - Operationalization .footnote[ * Gerring Table 5.1 (p.117) ] --- ## Resonance What does this mean? ??? Is the concept intuitive? How faithful is the concept to extant definitions *and* established usage? Avoid neologism (Gerring p.118); we don't need new terms for the sake of having new terms --- ## Domain What does this mean? ??? The contexts in which this concept resonates and applies Example: "vouchers". What does this mean? - This has a clear meaning in the United States, in debates about education policy, but that is a narrow domain - Contrast this with democracy or terrorism, which are concepts with broad domains --- ## Consistency What does this mean? ??? We can change the definition of a concept by adding, removing, or modifying attributes Contrast with "slippage" or "stretching" --- ## Intension versus Extension - Intension: Number of attributes - Extension: Number of referants .footnote[Gerring Figure 5.1 (p.123)] ??? Necessary conditions as a limiting force: more necessary attributes reduces extension Sufficient conditions as an expanding force: more sufficient conditions increases extension E.g., "planet" does not cover all celestial bodies, only those of a particular size and age We could change the definition of planet OR we could create a new concept to describe objects incongruent with the existing definition --- ## Fecundity What does this mean? - Dictionary definition: > Fertility or fruitfulness ??? Is this concept useful? What does it do for us theoretically? - Neologisms are prone to lacking fecundity - But a project that suggest two existing concepts do not create a useful distinction is focused on the lack of fecundity in existing concepts --- ## Differentiation What does this mean? ??? How does this concept contrast or help create distinctions between existing sets of phenomena? --- ## An example In psychological work, we have concepts of "value" and "opinion": - *Opinion* is a summary evaluation of a particular object - *Value* is a belief about a desired end-state of the world Are these different concepts? Why? --- ## Utility What does this mean? ??? In an experimental context, we need to be able to manipulate putatively causal constructs It is therefore not particularly useful to have a study about non-manipulable causes (e.g., race, gender, democracy, age) Similarly, if we are interested in a relationship between, e.g., the concepts of "media coverage" and "opinions", we need to build theory about those things and not other concepts. The concept of "political knowledge" or "democracy" is not useful in that study. --- template: questions --- ## Construct validity - Once we know our concept(s) of interest, how do we *operationalize* them? - How do we know something when we see it? ??? These are the measurable attributes in the concept definition It is possible that there are attributes that are not measurable - Rethink our concept definition? --- ## An example Definition: *Opinion* is a summary evaluation of a particular object Operationalization? ??? Agree/disagree Oppose/support Degree of favorability Warm/cool Positive/negative Implicit/explicit How many scale points? --- ## Construct validity - What are possible threats to construct validity? .footnote[* Shadish, Cook, and Campbell Table 3.1 (p.73)] ??? Highlights: - Bad concept definition - Mono-operation and mono-method bias: when the operationalization mismeasures the concept or when the operationalization itself becomes part of the concept (e.g., self-reported income versus actual income) - Experimenter expectancies: does the implmeentation of the experiment actually expose units to things other than the concept of interest (e.g., by encouraging particular types of behavior aside from the behavior of interest) - Compensatory equalization and rivalry: Knowledge of treatment status actually changes behavior - Treatment diffusion: Stable unit treatment value assumption from Holland -- - Which of these threats do you have questions about? --- template: outline3 --- .left-column[ ## Research Question ] .right-column[ - Once we know what we want to study, we need a research question - What makes a good research question? ] ??? - KKV's two criteria - 1: Politically important - 2: Contribute to scientific understanding/literature - Add: - 3: Personally interesting - 4: Unresolved --- .left-column[ ## Research Question ## Theory ] .right-column[ - We're not going to talk about this during the course - It's your job to find or develop theory relevant to your RQ - Theory must be testable (i.e., falsifiable) ] ??? We may have a "big" theory (like the effect of democratization on economic growth). This isn't particularly well-suited to experimental intervention for various reasons (cost, ethics, finite population of countries, etc.), but there are possibly causal mechanisms or empirical implications that could be tested. E.g., we can design an experiment where some citizens are invited to deliberate with authoritarian rulers on a local level (while a control group is not invited to participate) and then compare their economic productivity --- .left-column[ ## Research Question ## Theory ## Hypotheses ] .right-column[ - From theory, we derive testable hypotheses - Hypotheses are expectations about differences in outcomes across levels of a putatively causal variable ] ??? - Use theory (and extant evidence, if any) to derive a specific causal expectation (i.e., a difference in one or more outcomes between units experiencing one level of a putatively causal variable and other units receiving a different level (possibly the absence) of a that variable - (For purposes of exam, think about feasibility...we may want to test something that in practice is difficult, expensive, or unethical) --- .left-column[ ## Research Question ## Theory ## Hypotheses ## Design ] .right-column[ - Derive experimental design from hypotheses - In observational research, we look for data to test theories - In experimental research, we have to intervene to generate data - We can only test hypotheses by comparing two (or more) experimental conditions ] ??? Thus an experimental design is a manipulation of the putatively causal variable(s) --- .left-column[ ## Research Question ## Theory ## Hypotheses ## Design ] .right-column[ - Experimental "factors" are expressions of hypotheses as randomized groups - What intervention each group receives depends on hypotheses - presence/absence - levels/doses - qualitative variations ] --- template: questions --- template: outline4 --- ## Example: Druckman and Nelson - Research question - Theory/hypotheses - Variables - Design - Data collection/protocol - Analysis - Results/findings --- ## For next week .left-column[ ## Readings ] .right-column[ - Shadish, Cook, and Campbell (Ch. 1,2,8) - Internal validity: How do we know experiments work? - Freedman - Start thinking about the ethics of experiments ] --- ## For next week .left-column[ ## Readings ## Exercise ] .right-column[ - Get a sense of what can be studied experimentally - Visit Time-Sharing Experiments for the Social Sciences - http://tessexperiments.org/ - Pick two studies from TESS and write-up the details from the worksheet - We will share them in class next week ]