In practice, it is in the data acquisition and selection, experimental design, and selection of an appropriate inferential test (each with its own assumption of data distribution function) that bedevil otherwise good researchers or reporters. Errors somewhere along the line can creep into even the most careful of studies.
Not by any means is it all lost to the laypersons. A few simple rules can be used to help you decide which study to believe or rely upon. Look for them and you will be richer for the effort. You don’t even have to know how to accumulate sums; all you need to know is how to think in forming your judgment of the solidity underlying a finding or conclusion.
A valid plan of inquiry will answer the following questions:
- Were the samples or subjects selected randomly, where every sample or subject has the same likelihood of being selected as all others in the block or universe of possibilities?
- Were the samples representative? The perspective here involves the universe of data. One example, extreme enough to be remembered should fix the issue in your mind. “A truckload of gold ore is worth $45,000. It is one thing if every pebble and rock contains its fair share of gold. It is quite another if all the gold is concentrated in a single nugget! In that case, only as sample comprised of the entire truck load is representative, no matter how random the sampling plan.”
- In medicine, the sampling issue is equally acute. It is rarely possible to ignore genetic, gender, or other similar differences. To what group or sub-group do the results apply specifically?
- Were there enough samples taken to define an error of measurement, the standard deviation, SD, (square root of the average sum of squares of the quantity being measured) that when divided into the mean (average) and multiplied by the square root of the number of samples, yields a value of “t”, that defines the probability that the quantity being measured has a mean value that is not real? That was a long sentence, but use it a few times and it becomes second-nature.
- Was a baseline or control group employed with which to compare the data?
- What is the probability that the reported conclusions were right? What is their probability of being wrong—though the former is more generally useful?
Treating trends in data, regression (also known as least squares) is a common method.
- How well was it defined?
- What distribution function was assumed?
- Was data transformation applied?
- Was the residual error normal?
Differences in level of response between groups, “t” test for two groups, ANOVA for two or more groups are most common:
- Was the distribution function or any transformations of data reported?
- Dry and arcane it is—known to only a few.
Making research more competitive, powerful and efficient by using balanced experimental designs that yield data bases that provide trends if effects and their corresponding significance levels from;
- Two or more main variable.
- Interaction among main variables.
- Residual test errors against which all main effects and interactions can be tested against.
Consider the following questions.
- Young parents: is our child normal, gifted, retarded or harboring a genetic defect? And what then? Should our child be vaccinated? What are the individual and society tradeoffs?
- Homeowner: How much insurance do I really need? What are the real risks? What about all the other opportunities to buy insurance? Are any cost effective to me? Where can I find out?
- Commuter from the countryside: Which route to work is safest? How can I be sure?
- Older parent: Should my child go to college and if so which one? And what then?
- Sally citizen: Is the earth getting warmer? And so what if it is?
- Young engineer designing a flying car: How can I avoid letting a drunk kill him/herself and others? How can I answer that?
- Police Chief: Are my policies making any difference in view of social trends that naturally occur? If so in what direction and by how "sure" can I be? How sure can I be?
- Voter deciding on candidate: Are those people across the water really as bad as the candidates make them out to be? If so, what can I do? If not, same question.
- Concerned politician: How can I forecast what is to come and be ready with policy or response in time? What does history really have to say?
- Budding scientist: Is this effect I see real? How sure do I want to be? How do I interpret the data? Is any of it false?
- Child realizing death is inevitable: How long will I live?
- Economic Policy: Does economic history support or deny claims made by politicians?
- Data Mining: Theses studies are fraught with two kinds of error. Are both reported? How many null hypotheses were considered?
The crucial issue is that no one knows or can ever know the outcomes exactly. But you can know the odds. Knowing the odds is how insurance companies and casino operators can be sure of their profits—to very, very, high degrees of probability.
Probability reigns supreme in this universe, and the best each of us can do is understand it, hopefully on both the intuitive or mathematical levels akin to those above, which should be enough to get you started.
For example visit Clinical Trials for what the government is making available, though not all concerned scientists are in fact reporting.
Posted by RoadToPeace on Monday, April 30, 2012.