niccol:
Designing Scientific Experiments for Dummies
Critical phrase is:
Experiments can have only one independent variable
Yes, I know that this is not definitive. But if any one can find me a reputable reference that says 'you can have as many independent variables as you choose and it doesn't effect the validity of the experiment' then I will buy them a beer.
DivaVocals:
eyeroll :rolleyes: Oh yes.. OBVIOUSLY I did..:shocking: Apparently I need to be "schooled" by those in the KNOW.. (yeah right eyeroll :rolleyes:)
<MASSIVE SIGH> You two really believe what you are saying don't you?
I thought you were just trying to yank my chain until now, but obviously that isn't the case. :(
Back to school time....
I'll start with the "Critical phrase" Experiments can have only one independent variable
These are true words. 100% accurate, no dispute whatsoever, but this is just another example of text being taken in isolation without considering the bigger picture.... So lets look at the bigger picture. Oh, 1st up, they call these things 'books for Dummies' with good reason. I digress.
Does this book, or any other book, state that there can only be one test to prove/disprove a given theory? No of course not. That would be silly.
**This one simple thing is all that is needed to show how short sighted you are being. **
Since we all seem to like Niccol's car example so much, I'll stay with it......
So, we have a red sports car and a grey Fiesta having a race. The red sports car wins (assuming no mechanical failure).
One question we could ask is WHY did the sports win?
The ill-considered response is "It won because it is faster". This response, although factual, doesn't answer the question. It just poses another question, which is WHY is the sports car faster?
I have a theory about this... The sports car is faster because it is painted red. :)
To** test** my theory, I propose an experiment. I'll paint the Fiesta the same colour red as the sports car (therefore changing ** only one independent variable**) and race them again. If my theory is correct then I'd expect both car to finish the race at the same time. If the theory is incorrect, the sports car will still win, therefore dis-proving the theory.
We run the race, my theory is proven wrong. The sports car still wins. **This is empirical evidence that the color of the car has nothing to do with the speed of the car.
**
I still don't know why the red sports car is faster though, so now I now need to propose another theory. How about, the red sports car is faster because it is being driven by a more experienced driver. (The drivers being yet another variable).
To test this theory I'll get the two drivers to swap cars, if the theory is correct, the Fiesta will win the next race because that'll be the car driven by the more experienced driver. If the theory is incorrect, the sports car will still win. Agreed? wrong This test will be invalid because I've changed more than one variable... The color of the car and the driver. No matter which car wins this time I won't be able to draw any definitive conclusions because I won't know for sure if it was just the change of drivers that made a difference, or a combination of the red colour and the driver.
This is why Experiments can have only one independent variable in order to be able to draw any valid conclusion. In short, the theory itself is fine, it is the test that is invalid.
No problem though.... we just need to repaint the Fiesta back to its original grey and run this test again. The only variable being tested now is the driver(s). This time, if the car being driven by the more experienced driver wins (the fiesta) we can positively conclude that it is the driver that makes the car go faster. We all know that the race won't end this way though.. The sports car will still win (it is inherently faster), which will prove the theory to be false, it isn't the driver that makes the car faster. We need to propose yet another theory and another test.
My next theory is that it is the size of the engine that makes the sports car go faster.
To test this, I'll swap engines (assuming it were physically possible). If the theory is correct, the car with the bigger engine will win the race. If incorrect we can conclude the engine itself isn't a factor. Needles to say, we have to place the drivers back in their original cars, and both cars need to be put back to their original colours (else we are trying to test more than one variable). The test would be invalid otherwise. **The only variable being tested now is the engine. **
Lets assume the car with the bigger engine does win. This is empirical (not anecdotal) evidence that the bigger engine is what has made the car go faster. At this point, it is important to note that although there are still dozens of other variables (differences) between the two cars, any of which can also make a difference to the speed (eg, the gearbox, the size of the wheels, the type of fuel being used, etc,etc) these variables **do not **invalidate the fact that the engine itself has been proven to be a significant factor in the faster speed .
However, if this test shows the car with the smaller engine wins (or they both finish at the same time), it will disprove the engine theory and we'll be able to conclude the engine isn't the reason for the difference in speed. We'll need to propose another theory, and another test.
At this point, and assuming the car with the bigger engine wins, we can end the experiments, having empirically proven that a bigger engine makes the car go faster.
We don't have to stop there though. They are still lots of other differences/variables between the two cars, and indeed, any one (or more) of them may also have an effect on speed, well, that is the theory, so to test these variables we need to set up another test.
My next theory is that it is the gearbox that has made the red sports car faster.
My test for this will be to swap gearboxes (assuming it were physically possible). All else (including the engine) being set back to the original state, **the only variable being tested now is the gearbox. **
The cars race. The car with the faster engine still wins (the red sports car), but this time it has been observed that the other car wasn't quite as far behind. This is empirical evidence that the gearbox has made the fiesta a little faster (or the sports car a little slower). At this point we don't know which, so we'll need another experiment to determine this.
If however, the fiesta happens to win, this is also empirical proof that it was the gearbox that has made the car faster. It is the only variable that has changed. At this point we can end the experiments, having empirically proven that a better/different gearbox makes the car go faster.
Just like the engine experiment though, this doesn't prove the gearbox is the only thing that makes the red sports car faster, but it does prove, like the engine, that the gearbox is a significant factor that affects the cars speed.
The position now is that we have empirically proven that BOTH the engine AND the gearbox are two of the factors that have made sports car faster. (there may still be other things that could make the sports car faster, but like the engine/gearbox, these need to be tested in the same manner. Both cars back to their original state, and we test the next theory, say, different wheels.
By performing this repeated theory/test/theory/test with each of the variables of interest being tested **independently ** (as the Dummys book states) we can build up a clear picture based on empirical evidence that will eventually show, with no shadow of doubt, that the red sports car is faster than the fiesta because:
a) It has a bigger engine
b) it has a better gear box
c) it has bigger wheels.
These same tests can also empirically show that:
a) The color of the car has no affect on the speed of the car
b) The experience of the driver has no effect on the speed of the car (assuming both drivers race with pedal to the metal)
c) The time or day the race is run has no effect on the speed of the car
Now that we have proven by Experimenting with only one independent variable at a time (and using multiple tests) which of the many variables affect the speed, and which ones don't.
If we want, we could end the experiments here. We now know without a doubt what does, and just as importantly, doesn't make the Sports car faster than the fiesta. The 'unknown' and/or 'untested' variables are irrelevant... we can test each and all of them, and we'll still end up with two lists, one showing what does improve speed, and the other showing what doesn't.
Without this testing, we could take any group of people, and ask about their own personal observations, many will have observed that cars with bigger engines are faster than those with smaller engines, others will have observed that cars with bigger wheels are faster than cars with smaller wheels. Whether their observations are correct or not will be open to debate. They only have anecdotal evidence.
Lets not end the experiments here though. We now know that there are three proven factors that will (not may) increase the speed of** a **car.
We now wish to find the order of significance.
More experiments are needed
I propose to run four tests. I'll use my own car for these experiments (A Ford Falcon).
The 1st test is to drive my car around the racetrack at full throttle and time how long it takes. This is my 'control' . Assume it takes 4minutes exactly. The purpose of this 'control' is to help maintain the integrity of the tests to follow.
My next test is to replace my engine with a bigger one, and drive around the track again. lets say it only takes 2 minutes this time.
My 3rd test, is to put the original engine back in, and replace the wheels, and once more around the track. This time it takes 3minutes 30 seconds.
The final test is to put the original wheels back on, and replace the gearbox. This time, lets assume it takes 3 minutes exactly.
I have now tested all three** independent variables, one at a time. ** Each of them showing a speed increase compared to the control. The experiment is valid (if any of them caused my car to run slower, the test would be invalid. Something has gone amiss, that will need further investigation).
Looking at the results of these tests, we can now conclude** without any doubt **that the engine size is more significant than the gearbox(it is also moresignificant than the wheels). We can also conclude that the wheels have the least effect.
So, to sum up, in spite of having two wildly different cars (The red sports car, and the grey fiesta) with more variables than you can poke a stick at, we can use the scientific methods to empirically and conclusively prove that the sports car is faster because it has a bigger engine, a better gearbox, and bigger wheels (in that order of significance). We have also been able to empirically and conclusively prove the the colour of the car(s), and the driver(s) are** not **a factor in regards to the speed the cars are capable of.
All of this can easily be translated across to the 'SEO experiments' to prove (or disprove) the effect of any given/single change we care to make, and even give a level of significance in regards to the change.
This isn't 'anecdotal' evidence, it is empirically/scientifically demonstrated evidence.
DivaVocals, you are wrong when you claim "Because of this there is no **reliable **conclusion that can be gleaned from the "results" of this "test". The only thing anyone could walk away with is MORE anecdotal data".
Go back and take another look at the test I proposed. There was only ** one independent variable** being tested, with* almost*** every other variable being eliminated from the equation** (Although not strictly necessary, this means with less variables to consider, the less testing we need to do in order to come to a valid conclusion, else we'd end up with the engine/gearbox/wheel scenario where each 'variable' produces its own result that would then need further testing to determine which is the most significant).
In isolation I agree, the proposed test doesn't tell us much, but it was not to be performed in isolation, the result of this test would have given us a result that would/could then be used for further tests, and so on, until we ended up with a clear picture as to what variable has what effect and to what degree.
Niccol and DivaVocals, you have both ignored the big picture.
Niccol, you have demonstrated this by ignoring the possibly of multiple tests as being a valid and vital part of the ''scientific method".
DivaVocals, you have demonstrated this by implying that this test is of little to no use because (I can only assume) you thought I had intended to use the results of this test alone to prove some kind of point. You've ignored the fact that this is/was to be one of the many tests I had planned/hoped to make so that we can come to reliable/provable conclusions.
To simply toss your hands in the air and proclaim 'that there are too many variables' and as such we can't come to any 'reliable conclusion' is probably one of the most ridiculous things I've read in a long time.
Where would science be if the scientists had this attitude? "Put a man on the moon? Are you crazy, there are far to many variables, Might as well not even try" or... "OK, so what if Diver Joe has demonstrated that we can eliminate the 'lack of air variable' by giving the astronauts an oxygen tank", there's still too many other variables to consider. Joe's oxygen tank idea isn't going to be of any use in our endeavours at all".
Thank god, our scientists aren't as short sighted as you two.
Rod.
)
