How fast can we learn?
Or: Why experiments are often a bad idea (bring out the pitchforks).
I’m building an app, as a solo founder. And here’s how I’ve used social media to launch features:
I write a post, it contains a link to my website.
People go to the website.
On social media, they tell me things they want
I listen to them and find out if they like it and what they want added.
Seems sensible, but I’d like to convince you that this is wrong. Just extremely slow and noisy, and very much distorted by loud users over the many non-loud users.
If you were in a job interview at a big tech firm, you’d say: Duh, Chris: You’re a Data Scientist! That way of working is stupid, obviously you’d run an experiment here and you track important stuff like engagement or conversions. And that would … also be a horrible answer. For me! And for anyone else who’s trying to bootstrap an app with limited users: There’s just not enough data!
Now all I need to convince you of is that there is a third way: You do not need to build a feature just because a single user demanded it. You also don’t need to run experiments and wait two months until you got an answer.
You can just: Launch, post, and measure. It’s really fast, and it works well with an agentic world where changing stuff is also fast.
Technical notes: I haven’t done full research here, I’m not pretending I’m the first to come up with this approach. Maybe there’s a name for it already! I do, however, have decent experience in web tracking and app building to claim that these ideas are underused.
Let’s take an example. I build a site with NYC real estate data at https://citytracker.ai. People really care about architects, so let’s add a nice bar chart:
And the quiz to you is: Is this a good feature? Does it add value?
Let’s do some math here, this is a technical blog after all. Let’s say we have 100 users per day, and conversion rate for a visit is 1%. Let’s assume this feature is phenomenal, and doubles conversion. Even in that case, detecting this via experiment would take about 46 days. If you’re building some startup, you don’t have that type of time!
And I’m about to claim that don’t need 46 days. 1 day is enough.
Why? You’re allowed to bring in outside information and simplify! Let’s say we know this: On our architects page, roughly 10% of users engage with most features. No standard errors required here. We can take the full history. 3,000 visits over the last month is enough to estimate this 10% well.
Then all we need is one metric: Engagement % with the new feature. If the feature is twice as good, we see 20% of users engaging with it. What’s the confidence interval on this? 12% to 27%, after one day. In other words: We’ve reached a stat sig improvement in our page, in one day!
And yes, I hear you saying: Why not take this outcome metric “engagement with feature” and measure this experimentally? Now we’re talking: 4 days. But it’s not so obvious that this is actually helpful: Are we replacing some other feature in the same location? And if not, do we have two features we can easily compare?
Let’s summarize this:
Experiment for conversion: 46 days — best but oh so long
Experiment for engagement: 4 days — maybe good
Engagement: 1 day — easy to understand, super fast
Listen to user feedback: 1 day — super biased
Of course, experimental data is the only way to get causal answers. And of course I’d run (1) if I have thousands of users. But I don’t! And if you’re starting up something, you also don’t. And sadly, you won’t get these users until you know what they want.
When you’re in that situation, it feels clear to me that (3) is best: Within a day, I have a rough idea on a feature, and then, on the next day, I can just build another feature, observe outcomes and be done. Based on objective data!
You could run that experiment, and even if you feel like it answers your question, Claude Code has already built 3 new features for me. And if instead you listen to qualitative feedback by users, there’s a decent change you might just build something that the majority doesn’t really want.
Let me give you one more example. This is how I show a filing to my users:
Is this a good way? I have no idea! I don’t really know what my users want, I need to find that out. But what’s easy is to ask Claude to create new versions for me. And then I could switch out the version, and again … within a day I’ll have a rough answer. Maybe people really care about architects. So it’ll say more about them. And then, tomorrow, I’ll tell you even more about architects.
In theory, an experiment could work even better here, but I think that’s the wrong way to think about this. An experiment takes setup and time and consideration, so what really happens is that … you just don’t run it. And in this case, fast and messy beats clean.
Or so I think! Wait for one of my next updates, where I’ll tell you how this model is working.