➩ Two or more versions of the same page can be displayed to the visitor randomly to compare which one is better.

➩ The comparison is often based on click-through rates (CTR), which is computed as the number of clicks on a specific link on the page divided by the number of times the page is displayed to the visitor.

➩ CTR is also used for advertisement, computed as the number of clicks on the ad divided by the number of times the ad is shown to the visitor.

➩ Instead of clicks, scroll, subscribe, purchase, hover, shares, likes, comments can be measured to compare the effectiveness of the two versions of the same page.

➩ The two versions are called control (version A) and treatment (version B) in statistics, and the two versions can differ in wording, speed, font, size, color, icons, graphic design of the two pages. These differences are called treatment factors.

➩ Treatment factors can be introduced one at a time or displayed at the same time, i.e. the two versions of the page can differ in one or more factors.

➩ The choice of which page to display can be random, for example, if random.random() < 0.5: return "Version A" else return "Version B".

➩ It can also be displayed in a fixed ordering, for example, suppose count is a global variable to keep track of the number of visitors, then if count % 2 == 0: return "Version A" else return "Version B" count = count + 1 would alternate between the two versions.

➩ This can be done for a fixed number of times, and after that, only the one with higher click-through rate would be displayed to all visitors.

➩ IP address or user agent information can be used to figure out which version of the page is displayed to which visitors.

➩ Cookies can be used too, use flask.request.cookies.get(key, default) to get a cookie (and returns default if no such cookies exist), and flask.Response.set_cookie(key, value) to set a cookie. Cookies are stored on the visitors' computer as text files.

➩ Query strings can be used to figure out which version of a page the visitor came from, it is the string after "?" at the end of a URL: Link.

➩ "index?x=1&y=2" is a URL specifying the path "index" with the query string x=1 and y=2.

➩ Use flask.request.args to get a dictionary of key-value pairs of the query string.

➩ To perform AB testing of a page with two versions, both contain a link to index: on version A, the URL can be <a href="index?from=A">Link<\a>, and on version B, the same URL can be <a href="index?from=B">Link<\a>.

➩ If version A URL is used, request.args["from"] would be "A" and if version B URL is used request.args["from"] would be "B".

➩ The previous design has experiments with alternatives for a fixed number of time then exploit the best alternatives.

➩ This design can lead to a large "regret", for example, if displaying bad version is costly. In other settings such as drug or vaccine trials and stock selection, experimenting with bad alternatives can be costly.

➩ The problem of optimal experimentation vs exploitation is studied in reinforcement learning as the multi-armed bandit problem: Link.

➩ Upper Confidence Bound (UCB) is a no-regret algorithm that minimizes the loss from experimentation.

➩ After every version is displayed once, the algorithm keeps track of the average value (for example click-through rate) from each version, say \(\hat{\mu}_{A}, \hat{\mu}_{B}\) and computes the UCBs \(\hat{\mu}_{A} + c \sqrt{\dfrac{2 \log\left(n\right)}{n_{A}}}\) and \(\hat{\mu}_{B} + c \sqrt{\dfrac{2 \log\left(n\right)}{n_{B}}}\), where \(n\) is the total number of visitors, \(n_{A}\) is the number of visitors of version A, \(n_{B}\) is the number of visitors of version B, and \(c\) is a constant, and always picks the version with a higher UCB.

➩ The UCB algorithm uses the principle of optimism under uncertainty and the UCBs are optimistic guesses: with high probability (the probability can be determined by \(c\)), the actual average is less than UCB.