Randomized Test

[data_digger]

This is support thread for my new module - Randomized Tests.
Please report here all bugs, installation problems and feature requests

This module allows You to perform experiments on Your website. You can test for example how Your new 'Add to Cart' button performs compared to old one.
Sample image:http://www.data-diggers.com/contribs...stats_demo.jpg

Download it here. - The link points to my website. I'll change it as soon as this module will be added to download area.
This module requires also my UTI contribution to work. Download it here.

More info (probably screencast) will be later posted on my website: http://www.data-diggers.com

Installation instructions are in .zip file.

To create experimnent follow these steps:

1. Go to Admin->Tools->[RT] Groupsets.
2. Create groupset (it's set of groups). Call it 'Basic Test Groupset'.
3. Click on Basic Test Groupset - You will be forwarded to page where You can edit it.
4. Each groupset must have EXACTLY one control group. I usually call it 'Control Group'. Create two groups (you can create as many groups in groupset as You want, but for simplicity We use 2 now):
   - Control Group - with id 'cg' (as ControlGroup). Check 'control group' checkbox
   - Test Group - with id 'tg'. DO NOT check 'control group' checkbox.
5. Go to Admin->Tools->[RT] Experiments.
6. Create Experiment 'Exp #1' using groupset 'Basic Test Groupset'.
7. Click on Exp #1 - You will be forwarded to its stats:
   - Impressions - in this version of Randomized Tests its simply number of visitors in each group
   - Customers Registered - (visitor to customer conversion rate) / (change in regard to control group) / (total count of newly registered customers in this group)
   - Orders Placed - (visitor to order conversion rate) / as above / (total number of orders visitors in this group placed)
   - Items Bought - (avg. size of order) / as above / (total number of items bought)
   - Orders Value - (avg. order value) / as above / (total orders value)

Ok, You've created experiment, now You have to create two versions of Your page. Let's assume that You want to change 'Add to Cart' button on product_info page. Edit includes/templates/your_template/templates/tpl_product_info_display.php.
Find code responsible for displaying 'Add to Cart' button - in my case it's:

echo $display_button; 

Change it to:

// $exp is global variable (it's instance of ExperimentManager defined in includes/classes/randomized_tests.php) and holds basic information about experiment and group to which has been assigned current visitor
// $exp->groupID is 'id' of group that You assigned in step 4. You can use it to detect in which group is current visitor and perform custom action.
if($exp->groupID == 'tg') {
// visitor has been assigned to Test Group
echo $display_my_new_button;
} else {
// user is in Control Group - We don't change anything
echo $display_button;
} 

You're done. Wait until experiment becomes statistically significant ( >2000 visitors in each group, many orders ) and check results.

[data_digger]

This module requires PHP >= 5

[swamyg1]

Wow, this looks very impressive. Excellent concept and great to have it built into zencart itself. Gonna have to give it a shot.

[yellow1912]

Nice one, a/b split testing inside zen is pretty awesome.

[data_digger]

Thanks!

I already work on new version - current has one big flaw. It's impossible to treat outliers in some special way (ignore them/assign to special group/...). By outliers I mean:

• super customers - every site has them ( customers that place - possibly multiple - orders every day and each order is for many items ). Group containing such customer is probably biased.
• super orders - very large orders that happen rarely / as above

[swamyg1]

For the UTI module, should there be a completely new "includes" directory, inside the "includes directory? I'm assuming this directory is duplicated for testing purposes, otherwise it wouldn't be there.

Not completely sure though!

[data_digger]

For the UTI module, should there be a completely new "includes" directory, inside the "includes directory? I'm assuming this directory is duplicated for testing purposes, otherwise it wouldn't be there.

Not completely sure though!

Files from includes/includes are needed but they should be in includes/ . I've packed uti module again and You can download corrected version from:
http://www.data-diggers.com/contribs...ti-current.zip
I'm also uploading it to zen cart addons now.

First remove 'includes' directory in 'includes' directory (includes/includes) and then just upload new UTI.

[swamyg1]

Hi, I'm wondering if anybody has this running yet... would like to get feedback on how the A/B testing functionality works.

Anybody?

[data_digger]

What You want to know?

[yellow1912]

Hi, I'm wondering if anybody has this running yet... would like to get feedback on how the A/B testing functionality works.

Anybody?

I do plan to use this on some stores in the next few weeks. I will report back results.