This is why we invite you to join & contribute to the Big Venture Studio Research 2024

The majority of the sources measure the performance of studios by comparing their metrics to industrial averages (not to mention our research (Max Pog, 2023), which has the same sin), taking the “one-size-fits-all” approach. Of course, we understand this logic: it is faster and less expensive to use available and thought-to-be confident sources of data than to build own sample of research subjects.

Yet, this method makes it hard to control (or at least, get knowledge of) the built-in terms and data processing, which are essential when studying such vague phenomena as “a new venture”, “startup”, and “a venture studio”. For instance, many sources use average numbers of startups’ time to various funding rounds and exit events (e.g., Burris, Mohammadi and Maiocco, 2023; Kannan and Peterman, 2022; Mohammadi and Maiocco, 2022; Zasowski, 2022), stating that startups supported by studios evolve faster than “traditional” ones (whatever that means). Consider, for instance, the difference in raising a Seed round, which is from 2 to 3 times faster for studios’ startups than for “traditional” ones (Zasowski, 2022; Max Pog, 2023).

However, there could be a huge variance of timing variables due to the different factors like the influence of startup supporting organizations (maybe accelerators and VC funds also comparably increase the growth speed?), startup industry (for some industries - e.g., bio, pharma, hardware - it may take longer to get external funding), and external factors (COVID, economic crisis, wars, etc.).

We believe that conclusions made on averaged data may lessen the value of the results since researchers may (and probably do) miss the influence of some critical factors and, because of that, get false-positive/negative outcomes.

By metrics to compare, the majority of sources often employ IRR, portfolio success/exit/death/etc. rates, TVPI, MOIC, etc. Although these measures are well-known, they’re all given per se; in rare cases, we might get an understanding of how they were calculated or taken.

As an illustration, the following phrase from this blog post (Lawrence, 2017) was heavily used as a reference in the report (Zasowski, 2022) to compare startups’ success rates: “Idealab – 100+ companies in 20 years. >70% still active. About 35% IPO or Acquired. About 5% unicorn. Pretty damn incredible”. But does “still active” mean “success” in terms of developing a startup as proposed? Not sure. From our perspective, that only means that 30% are “dead”, while the rest remain in an uncertain “alive” state.

Another example from the report (Zasowski, 2022) should also be mentioned. Authors compare the TVPI of studios to the TVPI of VC funds, claiming a 3.7x difference between them. While the studios’ related measure is given without any reference at all (perhaps an internal questionnaire, but we never know), the VC-related TVPI is taken from this source (PitchBook, 2017) and has nothing to do with VC since reflects the median TVPI of “2008 Vintage US Buyout Funds with Materials and Resources Investments”. For sure, such shortcomings are rare but quite painful and decrease the validity of the results.

Another important metric, which is often used to assess studios’ performance, is IRR. Although it is a common measure, so far, we haven’t seen any study that would have provided the exact IRR built-in logic. For example, was it calculated for realized assets or not, and how the unrealized assets were evaluated if not? Simply saying that the “Average IRR of startups created by studios is 53% while for traditional startups is 21.3%” (Zasowski, 2022) does not sound confident albeit looks sexy.

Called after the first-time discovered statistical phenomenon of studying the subjects, which got into the sample because they have survived the influence of some factor. Although econometric and social studies often face this type of bias (Thomas, 2023), it may be overcome or at least controlled. In terms of the topic, this bias may arise 1) when a studio reports only those companies, which were able “to survive” during the early development process; 2) when a study considers only “survived” studios hence ignoring those that are not.

So far, we haven’t met the direct evidence of this bias (only the indirect ones mentioned in this MSc thesis (Abreu, 2021) and this book (Kannan and Peterman, 2022)); however, bearing in mind the metrics-related drawback, we cannot be fully assured that, e.g., the IRRs or MOICs were not calculated accounting only the “successful” cases and studios.

Some critical numbers are given without any direct reference. For instance, taken from this report (Mohammadi and Maiocco, 2022) statement claims that “the startups produced by a studio have an average value of 74 million compared to 50 million of traditional startups”. For a reader, it may be interesting to know more on this 1.5x difference but the source is somewhere in the general list of references.

Another example is the timing of the fundraising events presented in this report (Zasowski, 2022), which are heavily cited in other sources (Abreu, 2021; Mohammadi and Maiocco, 2022; Burris, Mohammadi and Maiocco, 2023). According to the presented data, “traditional startups” are 2x slower in getting from 0 to Series A, 3x (sic!) slower in moving from Seed to Series A, and 1.4x slower in doing so from Seed to Series A. Where did the numbers for “traditional startups” come from? What were the procedures for getting them? Of course, one can dig into the list of provided sources and spend hours finding the exact reference, but will one really do that?

If a reader is not willing to go deep into the topic (we do), they will just decide to believe this data or not based on his own experience and the trustworthiness of a source. And considering venture studios and their outstanding performance, in many cases this will be “not”. “Extraordinary claims require extraordinary evidence,” Carl Sagan.

While, in general, the data and logic provided in the reviewed sources look valid, sometimes we faced errors in methods, which again may decrease the confidence of the provided conclusions. For instance, this report (Zasowski, 2022) states that “the average startup is roughly three years old by the time it raises a seed round” referring to this source (Dillon, 2018). If to check it, one can see that it is the median while the average was not presented and can be higher or lower than the median.

Similarly, in this whitepaper (Mohammadi and Maiocco, 2022) the IPO exit time of “traditional startups” is compared to the average of the studio ones for all types of exits. Such unintentional flaws again make conclusions about venture studios’ performance weaker and less believable.

Another example is when a report describes “best startups,” which come from a studio. For instance, in this whitepaper (Alhokail, Celen and Tilani, 2019), there are four startups presented as “Select Portfolio” of Science Inc. venture studio: Dollar Shave Club, DogVacay (Rover), Pillow, and Mammoth Media. From this list and according to public data, only the famous Dollar Shave Club can be called a “genuine” Science Inc. startup. Two other startups were invested/acquired later in their development (DogVacay - invested at Seed (Mills, 2012), Mammoth Media - created on the technology of the acquired startup PlayHaven (Jones, 2019)), and one (Pillow) is not connected to Science Inc. by any means.

Of course, it is a normal situation when a studio acquires or invests in a third-party startup (Kannan and Peterman, 2022), but we believe that such startups should not be presented as the “select portfolio” of a studio nor accounted for in a studio’s performance measures since in this case, a studio behaves as a common VC fund and should be treated accordingly.

From our perspective, some presented measures reflect no more than the vanity of venture studios’ founders and the startups they build. For example, think of fundraising events’ speed presented as “success” measures in various sources (Zasowski, 2022; Max Pog, 2023). Question yourself - can we actually conclude “success” by only Series A/B/C happening?

In many cases, the answer will be no. We think that speaking of funding rounds one should mention the related valuations or shares sold in each particular series of investments. It may happen that a startup raised a round with a huge discount due to running out of cash or failing to grow as expected. We never know without context. The only conclusion we can confidently make here is that studio founders have better-than-average fundraising opportunities.

Another metric of this kind (which is, by the way, often used even in scientific studies) is the number of startup employees (Patel and Chan, 2023). Yes, it demonstrates the growth of a particular company, but can we use it to compare the growth of different startups? Maybe when analyzed startups develop very similar products and serve the same markets - yes. In other cases, it basically shows nothing.

Various products require various amounts of employees and they may cost differently. For example, if we compare a deep tech startup with 10 employees doing hard scientific stuff to some common marketplace venture, the amount and value of employees will differ significantly, while both companies may have similar valuations or sales.

We believe that to get reliable results and make strong conclusions, other, more transparent and straightforward, measures should be taken.