Last time we discussed how data-driven engineering becomes a key skill for software developers. Here is an approach to data-driven engineering that does not require deep math skills.

Consider a hypothetical startup that just launched a web site that has a landing page, a sign up page and a welcome page after sign up.

This is a new product so growing the number of signed up users is the top priority. Unfortunately only few users has signed up so far. Can we do something better than just assuming that the startup idea is not appealing? Yes we can.

1. Instrument Page Views

The first step is to record an event when each page is displayed to a user. There are several commercial and open-source solutions available to instrument and store this data. Writing a row to a database from the page rendering code might be good enough to get started. The goal is to create a table that has a page name, a timestamp and a user ID. There are many clever ways to uniquely identify users. Storing a GUID as a cookie works in most cases.

2. Prioritize with Funnels

Marketing funnel is a well-known technique to drive users from prospects to clients. The idea is to identify an entry stage, a desired stage and stages between them. The number of users dropping at each stage can be measured to focus marketing resources on a problematic stage. The same technique can be applied to users navigating web pages.

This can be done by filtering the timestamp to the time period of interest, grouping by the page name and finding a distinct count of users.

The result shows the number of unique users per page. The lowest percent of users going to the next page indicates the focus area.

It appears that 90% of users who saw the landing page clicked the sign up button. Not bad for a startup. This hints that customers are interested in the product. However only 10% of users who got to the sign up page were able to sign up successfully.

3. Debug with Segments

Something must be wrong with the sign up page but this high level picture is not actionable. Analyzing data user by user is too expensive and not necessarily actionable as well. We need to find something between these two extremes. The goal is to narrow a problem by finding a subset of users (aka segment) who have challenges signing up. Let’s apply creativity and domain knowledge to brainstorm segments that can impact ability to sign up. Here are some examples: a user age, a country, a day of the week and a browser type. The segments should be prioritized by their potential impact and ability to get data for them. For example, people tend not to share age online so the age segment goes to the bottom of the list.

A browser type can be retrieved from a user agent string. This is easy enough to get started. So far the telemetry table has a page name, a timestamp and a user ID. Let’s add the browser column for each page view, filter by users who got to the sign up page or the welcome page and group by the browser type and the page name.

Are there browsers with low sign up rates? Indeed it looks like the sign up page has a problem in Firefox:

This aha moment is called an insight. It’s now possible to debug the sign up page in Firefox to pinpoint and fix a browser-specific issue.

Caution: real life examples will not be so black and white. Difference in values for a segment can be caused by noise in data. For example we may have only one Firefox user who decided not to sign up. Math has been developed to calculate so called statistical significance to measure a chance of the delta for a segment be a result of noise. Learn and use this math or just investigate largest segments that are most different from other segments.

Empty Theory

There must be another issue in this example because percentage of users who were able to sign up even for IE and Chrome is still quite low. Let’s use the same technique to segment telemetry data by a day of the week.

And this data shows us (drum-roll…) nothing. The ability to sign up does not depend on the day of the week. It’s OK. It’s common to try several empty theories before finding an insight. Do not give up when this happens. Continue brainstorming segments and involve new people with domain knowledge when running out of ideas.

Another Insight

Next let’s segment telemetry data by country. It will be a bit more work because it requires to reverse geocode a client IP address.

This segment brings an interesting insight. Users outside of USA have very low sign up rates. It is not a language issue because Great Britain also speaks English. Looking at this data side-by-side with the sign up page hints that the required State field allows to only pick from 50 USA states. Additional check on users who did sign up from Great Britain and China show that most of them have picked the Alabama state that is the first one in the list. The sign up page should be redesigned to support customers from all countries.

Takeaway

1. Instrument page views
2. Prioritize with funnels
3. Debug with segments

Instrumentation and funnels are straightforward software development work. Segmentation is an art. It requires domain knowledge and creativity to come up with a prioritized list of segments and then code knowledge to debug interesting segments.

This technique can be applied to any web site or an application that has a flow of user actions. It is a good way to start on data-driven engineering without spending a lot of time learning math upfront.