The first application of an experimental design to website optimization was done by Moskowitz Jacobs Inc. in 1998 in a simulation demo-project for Lego website (Denmark). MVLPO did not become a mainstream approach until 2003-2004.

Execution modes

MVLPO can be executed in a live (production) environment (e.g. Google website optimizer, Optimost.com, etc.) or through a Market Research Survey / Simulation (e.g., StyleMap.NET).

Live environment MVLPO execution

In live environment MVLPO execution, a special tool makes dynamic changes to the web site, so the visitors are directed to different executions of landing pages created according to an [experimental design]. The system keeps track of the visitors and their behavior (including their conversion rate, time spent on the page, etc.) and with sufficient data accumulated, estimates the impact of individual components on the target measurement (e.g., conversion rate).

Pros of live environment MVLPO execution

• This approach is very reliable because it tests the effect of variations as a real life experience, generally transparent to the visitors.
• It has evolved to a relatively simple and inexpensive to execute approach (e.g., Google Optimizer).

Cons of live environment MVLPO execution (applicable mostly to the tools prior to Google Optimizer)

• High cost
• Complexity involved in modifying a production-level website
• Long time it may take to achieve statistically reliable data caused by variations in the amount of traffic, which generates the data necessary for the decision
• This approach may not be appropriate for low traffic / high importance websites when the site administrators do not want to lose any potential customers.

Many of these drawbacks are reduced or eliminated with the introduction of the Google Website Optimizer – a free DIY MVLPO tool that made the process more democratic and available to the website administrators directly.

Simulation (survey) based MVLPO

A simulation (survey) based MVLPO is built on advanced market research techniques. In the research phase, the respondents are directed to a survey, which presents them with a set of experimentally designed combinations of the landing page executions. The respondents rate each execution (screen) on a rating question (e.g., purchase intent). At the end of the study, regression model(s) are created (either individual or for the total panel). The outcome relates the presence/absence of the elements in the different landing page executions to the respondents’ ratings and can be used to synthesize new pages as combinations of the top-scored elements optimized for subgroups, segments, with or without interactions.

Pros of the Simulation approach

• Much faster and easier to prepare and execute (in many cases) compared to the live environment optimization
• It works for low traffic websites
• Usually produces more robust and rich data because of a higher control of the design.

Cons of the Simulation approach

• Possible bias of a simulated environment as opposed to a live one
• A necessity to recruit and optionally incentivise the respondents.

MVLPO paradigm is based on an experimental design (e.g., conjoint analysis, Taguchi methods, etc.) which tests structured combination of elements. Some vendors use full factorial approach (e.g., Google Optimizer that tests all possible combinations of elements). This approach requires very large sample sizes (typically, many thousands) to achieve statistical importance. Fractional designs typically used in simulation environments require the testing of small subsets of possible combinations. Some critics of the approach raise the question of possible interactions between the elements of the web pages and the inability of most fractional designs to address the issue.

To resolve these limitations, an advanced simulation method based on the Rule Developing Experimentation paradigm (RDE) has been introduced. RDE creates individual models for each respondent, discovers any and all synergies and suppressions between the elements, uncovers attitudinal segmentation, and allows for databasing across tests and over time.