Funnel testing in webpage optimisation: representation, design and analysis

Figures & data

Figure 1. Directed graph representation of the toy example.

Table 1. Design matrix, toy example.

A	B	C	D	E	F
1	1	1	1	1	1
1	1	1	−1	1	−1
1	1	−1	1	−1	−1
1	1	−1	−1	−1	1
1	−1	1	1	−1	−1
1	−1	1	−1	−1	1
1	−1	−1	1	1	1
1	−1	−1	−1	1	−1
−1	1	1	1	−1	1
−1	1	1	−1	−1	−1
−1	1	−1	1	1	−1
−1	1	−1	−1	1	1
−1	−1	1	1	1	−1
−1	−1	1	−1	1	1
−1	−1	−1	1	−1	1
−1	−1	−1	−1	−1	−1

Table 2. Functions for decision probabilities.

t1 =  0.25 − 0.1A

t2 =  0.42  +  0.21B

t3 =  0.1  +  0.08A − 0.3D + 0.1AD

c1 =  0.38 − 0.05B + 0.12C + 0.08BD

c2 =  0.07  +  0.2E + 0.07AE

c3 =  0.15  +  0.06D − 0.1F + 0.02AD + 0.01CF

Table 3. Conversion rates for different funnels, toy example.

CF1	CF2	CF3
0.00000000	0.16349206	0.08730159
0.00960000	0.16960000	0.05440000
0.00000000	0.00000000	0.05546218
0.00000000	0.00000000	0.02073365
0.00000000	0.00000000	0.06833333
0.00000000	0.00000000	0.10194175
0.00000000	0.06718750	0.03593750
0.01093750	0.06406250	0.07812500
0.00000000	0.00000000	0.14803150
0.02044025	0.00000000	0.12421384
0.00000000	0.07833333	0.09166667
0.00000000	0.08258258	0.05255255
0.00000000	0.02508361	0.16722408
0.00000000	0.03475513	0.21169036
0.00000000	0.00000000	0.07954545
0.01716069	0.00000000	0.14820593

Table 4. Design matrix and response data, conversion rates for CF₁.

A	B	C	D	E	F	CR1
1	1	1	1	1	1	0.00000000
1	1	1	−1	1	−1	0.00960000
1	1	−1	1	−1	−1	0.00000000
1	1	−1	−1	-1	1	0.00000000
1	−1	1	1	−1	−1	0.00000000
1	−1	1	−1	−1	1	0.00000000
1	−1	−1	1	1	1	0.00000000
1	−1	−1	−1	1	−1	0.01093750
−1	1	1	1	−1	1	0.00000000
−1	1	1	−1	−1	−1	0.02044025
−1	1	−1	1	1	−1	0.00000000
−1	1	−1	−1	1	1	0.00000000
−1	−1	1	1	1	−1	0.00000000
−1	−1	1	−1	1	1	0.00000000
−1	−−1	−1	1	−1	1	0.00000000
−1	−1	−1	−1	−1	−1	0.01716069

Figure 2. Half-normal plot, conversion rate for CF₁.

Table 5. Design matrix and response data, conversion rates for CF₂.

A	B	E	F	CR2
1	1	1	1	0.16349206
1	1	1	−1	0.16960000
1	1	−1	−1	0.00000000
1	1	−1	1	0.00000000
1	−1	−1	−1	0.00000000
1	−1	−1	1	0.00000000
1	−1	1	1	0.06718750
1	−1	1	−1	0.06406250
−1	1	−1	1	0.00000000
−1	1	−1	−1	0.00000000
−1	1	1	−1	0.07833333
−1	1	1	1	0.08258258
−1	−1	1	−1	0.02508361
−1	−1	1	1	0.03475513
−1	−1	−1	1	0.00000000
−1	−1	−1	−1	0.00000000

Figure 3. Half-normal plot, conversion rate for CF₂.

Table 6. Design matrix and response data, conversion rates for CF₃.

A	B	C	D	CR2
1	1	1	1	0.08730159
1	1	1	−1	0.05440000
1	1	−1	1	0.05546218
1	1	−1	−1	0.02073365
1	−1	1	1	0.06833333
1	−1	1	−1	0.10194175
1	−1	−1	1	0.03593750
1	−1	−1	−1	0.07812500
−1	1	1	1	0.14803150
−1	1	1	−1	0.12421384
−1	1	−1	1	0.09166667
−1	1	−1	−1	0.05255255
−1	−1	1	1	0.16722408
−1	−1	1	−1	0.21169036
−1	−1	−1	1	0.07954545
−1	−1	−1	−1	0.14820593

Figure 4. Half-normal plot, conversion rate for CF₃.

Figure 5. Directed graph representation of the simulated example.

Table 7. Design matrix, simulated example.

Run	A	B	C	D	E	F	G	H	I	J	K	L
1	1	1	1	1	1	1	1	1	1	1	1	1
2	1	1	1	1	1	−1	1	1	1	−1	−1	−1
3	1	1	1	1	−1	1	1	1	−1	1	−1	−1
4	1	1	1	1	−1	−1	1	1	−1	−1	1	1
5	1	1	1	−1	1	1	1	−1	−1	−1	1	−1
6	1	1	1	−1	1	−1	1	−1	−1	1	−1	1
7	1	1	1	−1	-1	1	1	−1	1	−1	−1	1
8	1	1	1	−1	−1	−1	1	−1	1	1	1	−1
9	1	1	−1	1	1	1	−1	1	−1	−1	1	−1
10	1	1	−1	1	1	−1	−1	1	−1	1	−1	1
11	1	1	−1	1	−1	1	−1	1	1	−1	−1	1
12	1	1	−1	1	−1	−1	−1	1	1	1	1	−1
13	1	1	−1	−1	1	1	−1	−1	1	1	1	1
14	1	1	−1	−1	1	−1	−1	−1	1	−1	−1	−1
15	1	1	−1	−1	−1	1	−1	−1	−1	1	−1	−1
16	1	1	−1	−1	−1	−1	−1	−1	−1	−1	1	1
17	1	−1	1	1	1	1	−1	−1	1	1	−1	−1
18	1	−1	1	1	1	−1	−1	−1	1	−1	1	1
19	1	−1	1	1	−1	1	−1	−1	−1	1	1	1
20	1	−1	1	1	−1	−1	−1	−1	−1	−1	−1	−1
21	1	−1	1	−1	1	1	−1	1	−1	−1	−1	1
22	1	−1	1	−1	1	−1	−1	1	−1	1	1	−1
23	1	−1	1	−1	−1	1	−1	1	1	−1	1	−1
24	1	−1	1	−1	−1	−1	−1	1	1	1	−1	1
25	1	−1	−1	1	1	1	1	−1	−1	−1	−1	1
26	1	−1	−1	1	1	−1	1	−1	−1	1	1	−1
27	1	−1	−1	1	−1	1	1	−1	1	−1	1	−1
28	1	−1	−1	1	−1	−1	1	−1	1	1	−1	1
29	1	−1	−1	−1	1	1	1	1	1	1	−1	−1
30	1	−1	−1	−1	1	−1	1	1	1	−1	1	1
31	1	−1	−1	−1	−1	1	1	1	−1	1	1	1
32	1	−1	−1	−1	−1	−1	1	1	−1	−1	−1	−1
33	−1	1	1	1	1	1	−1	−1	−1	−1	−1	1
34	−1	1	1	1	1	−1	−1	−1	−1	1	1	−1
35	−1	1	1	1	−1	1	−1	−1	1	−1	1	−1
36	−1	1	1	1	−1	−1	−1	−1	1	1	−1	1
37	−1	1	1	−1	1	1	−1	1	1	1	−1	−1
38	−1	1	1	−1	1	−1	−1	1	1	−1	1	1
39	−1	1	1	−1	−1	1	−1	1	−1	1	1	1
40	−1	1	1	−1	−1	−1	−1	1	−1	−1	−1	−1
41	−1	1	−1	1	1	1	1	−1	1	1	−1	−1
42	−1	1	−1	1	1	−1	1	−1	1	−1	1	1
43	−1	1	−1	1	−1	1	1	−1	−1	1	1	1
44	−1	1	−1	1	−1	−1	1	−1	−1	−1	−1	−1
45	−1	1	−1	−1	1	1	1	1	−1	−1	−1	1
46	−1	1	−1	−1	1	−1	1	1	−1	1	1	−1
47	−1	1	−1	−1	−1	1	1	1	1	−1	1	−1
48	−1	1	−1	−1	−1	−1	1	1	1	1	−1	1
49	−1	−1	1	1	1	1	1	1	−1	−1	1	−1
50	−1	−1	1	1	1	−1	1	1	−1	1	−1	1
51	−1	−1	1	1	−1	1	1	1	1	−1	−1	1
52	−1	−1	1	1	−1	−1	1	1	1	1	1	−1
53	−1	−1	1	−1	1	1	1	−1	1	1	1	1
54	−1	−1	1	−1	1	−1	1	−1	1	−1	−1	−1
55	−1	−1	1	−1	−1	1	1	−1	−1	1	−1	−1
56	−1	−1	1	−1	−1	−1	1	−1	−1	−1	1	1
57	−1	−1	−1	1	1	1	−1	1	1	1	1	1
58	−1	−1	−1	1	1	−1	−1	1	1	−1	−1	−1
59	−1	−1	−1	1	−1	1	−1	1	−1	1	−1	−1
60	−1	−1	−1	1	−1	−1	−1	1	−1	−1	1	1
61	−1	−1	−1	−1	1	1	−1	−1	−1	−1	1	−1
62	−1	−1	−1	−1	1	−1	−1	−1	−1	1	−1	1
63	−1	−1	−1	−1	−1	1	−1	−1	1	−1	−1	1
64	−1	−1	−1	−1	−1	−1	−1	−1	1	1	1	−1

Table 8. Functions for decision probabilities.

tLI = 0.5 + 0.1A

tLB = 0.2 − 0.1B

tI1 = 0.1 + 0.1A − 0.1C + 0.1BC

tI2 = 0.05D − 0.05AD

tB3 = 0.1 + 0.1E − 0.1F + 0.1C − 0.1EF

tB2 = 0.1A − 0.1AE

t12 = 0.05 − 0.05DH + 0.05C − 0.05AG

t32 = 0.05K − 0.05EL

c1 = 0.1 + 0.1G − 0.1CG

c3 = 0.1 + 0.1K − 0.1FL + 0.05BK − 0.05AF

c2I = 0.1 + 0.1I − 0.1J − 0.1CJ + 0.1BI

c2B = 0.1 + 0.1J − 0.1IJ + 0.05E − 0.05AJ

c21 = 0.05I − 0.05DJ + 0.05H + 0.05CI

c23 = 0.05J − 0.05E − 0.05FK + 0.1IJ

Table 9. Conversion rates for different funnels, simulated example.

Run	CR1	CR3	CR2I	CR2B	CR21	CR23
1	0.011559	0.00073	0	0	0	0
2	0.011608	0	0	0	0	0
3	0.010784	0	0	0.002698	0	0
4	0.011201	0.00253	0	0	0	0
5	0.011635	0.002146	0	0	0	0
6	0.011559	0.003635	0	0	0	0
7	0.011191	0	0	0.001426	0	0
8	0.012039	0.001333	0	0	0	0
9	0	0	0	0	0	0
10	0	0.002268	0	0	0	0
11	0	0	0	0.001311	0	0
12	0	0	0	0	0	0
13	0	0	0	0	0	0
14	0	0	0	0	0	0
15	0	0	0	0.002778	0	0
16	0	0	0	0	0	0
17	0	0.001971	0	0	0	0
18	0	0.030319	0	0	0	0
19	0	0	0	0.008195	0	0
20	0	0	0	0	0	0
21	0	0	0	0	0	0
22	0	0.010542	0	0	0	0
23	0	0.003757	0	0.003858	0	0
24	0	0.00412	0	0	0	0
25	0.059645	0	0	0	0	0
26	0.062159	0.006101	0	0	0	0
27	0.0621	0	0	0.003884	0	0
28	0.060896	0	0	0	0	0
29	0.061177	0	0	0	0	0
30	0.061716	0.018862	0	0	0	0
31	0.06134	0	0	0.008334	0	0
32	0.061483	0	0	0	0	0
33	0	0	0.004004	0	0	0
34	0	0.003807	0	0	0	0
35	0	0.003262	0.018645	0	0	0
36	0	0	0.003678	0	0	0
37	0	0.000699	0	0	0	0
38	0	0.012743	0	0	0	0
39	0	0.001638	0	0	0	0
40	0	0	0	0	0	0
41	0	0	0.010877	0	0	0
42	0	0.00712	0.011838	0	0	0
43	0	0	0	0	0	0
44	0	0	0	0	0	0
45	0	0	0	0	0	0
46	0	0.002403	0	0	0	0
47	0	0	0	0	0	0
48	0	0	0	0	0	0
49	0	0.007058	0.010615	0	0	0
50	0	0.012452	0	0	0	0
51	0	0	0.010327	0	0	0
52	0	0	0	0	0	0
53	0	0.002347	0	0	0	0
54	0	0	0	0	0	0
55	0	0.005054	0	0	0	0
56	0	0.005172	0	0	0	0
57	0	0	0.002712	0	0	0
58	0	0	0.002895	0	0	0
59	0	0	0.00266	0	0	0
60	0	0	0.002505	0	0	0
61	0	0	0	0	0	0
62	0	0.007343	0	0	0	0
63	0	0	0	0	0	0
64	0	0	0	0	0	0

Table 10. Conversion rates and related factors, simulated example.

CR1	A B C D G H
CR3	A B E F K L
CR2I	A B C D I J
CR2B	A B E F I J
CR21	A B C D G H I J
CR23	A B E F I J K L

Table 11. Fitted models for each conversion rate, simulated example.

CR1 = 0.0091 + 0.0091A − 0.0062B − 0.0062C − 0.0091G − 0.0062AB − 0.0062AC + 0.0091AG

CR2 = 0.0025 + 0.0017E − 0.0016F + 0.0013K − 0.0016EF − 0.0016FL

CR2I = 0.0013 − 0.0013A + 0.0013D − 0.0013AD

CR2B = 0.0005 + 0.0005A − 0.0005E + 0.0005F − 0.0005AE + 0.0005AF − 0.0005EF

CR21 = 0

CR23 = 0