The Boxxet Blog

Metrics of a Closed Beta: Losing 3/4 of Potential Users

February 12th, 2007 You Mon Tsang

This is an installment of the ongoing “Metrics Series.”

Boxxet spent a long time in a closed beta, longer than we expected. Some reasons were normal: stamping out bugs, making sure the application scales, and determining weak areas that need improvement or protection. More importantly, after our first wave of users of the closed beta, we felt that we needed to add more automation to the system.

There are, of course, pluses to an open beta (get quick feedback from the real world on what works or does not) and a closed beta (you make sure your first impression to the masses is a good one). We felt we did the right thing by staying in a closed beta. It allowed us to make large changes without worry about massive disruption, shore up our infrastructure and experiment with new algorithms at will.

However, the downsides are significant and I’d like to share some metrics for others who need to make such a decision. Perhaps the numbers will help folks make a more educated determination about how to proceed with a beta.

When we announced Boxxet was going into beta, we got slashdotted. We estimate that approximately 40% of the people who came to our front page signed up for the beta. Frankly, this is more than I expected. However, upon letting people know that they have been given access to the beta (within a few weeks), 60% returned to take a look. In some cases, we waited a few months before we gave people access and the drop off was even greater than 60%.

Of the people who came to our front door, 40% signed up and 60% returned when we emailed them. That means 76% of the people who stop by to a closed beta did not immediately return to see Boxxet when we let them in.

Of course, if we did not effectively serve these 76% (slow pages, crashed servers), we may have done something worse: lose them forever.

But this is a choice and I wanted to make sure that others understood the numbers on both sides of the equation. As we all know, the cost of user acquisition is quite high and not giving 76% a first impression hurts.

Entry Filed under: Boxxet, Metrics Series