Ilya Koliev
Senior Data Scientist at inDrive · Limassol, Cyprus
I use data to find where a product can grow
Then I test whether it did, with A/B tests, holdbacks and causal methods. AI agents do much of the routine; I pick the comparison and stand behind the number. Since 2021 at T-Bank, VK Mail and now inDrive.
Ship it? #1Made-up numbers, real trap
The test says win. Would you ship it?
New product page. 50/50 split, two weeks.
| Control | Treatment | |
|---|---|---|
| Users6,000 apart | 503,000 | 497,000 |
| Buyers | 15,090 | 15,283 |
| Conversion | 3.000% | 3.075% |
- Lift
- +2.5%
- p-value
- 0.029significant
The call is D: don't ship yet.
Look at the users, not the buyers. A 50/50 split of 1,000,000 users puts 500,000 in each group, give or take. In 95% of fair tests control lands within ±980 of that. Here it is 3,000 off: 6 standard deviations, a 1 in 507 million event.
This test is 6 standard deviations from fair. Move across the curve to try any other split.
That is a sample ratio mismatch (χ² = 36). Somewhere between randomization and the dashboard, one group lost more users than the other. The groups are no longer comparable, so the +2.5% measures the leak as much as the page.
One way it happens: the new page loads slower, and 6,000 treatment visitors leave before the tracking event fires. None of them buy. Count them back in and the lift is +1.28% with p = 0.26: too small to tell from zero.
Why not the others
- AShip it. p < 0.05, done. The p-value assumes randomization and logging worked. Here they did not.
- BShip with a 5% holdback. A holdback on the same pipeline inherits the same bias.
- CRun two more weeks. More data shrinks variance, not bias.
Find the leak, fix it, rerun. It is not rare: about 6% of A/B tests at Microsoft had a sample ratio mismatch (Fabijan et al., KDD 2019).
Writing
Articles are on the way. For now, the posts are on LinkedIn.
On LinkedIn
In Russian. 2K+ followers.
- Odd one out: a final-round task for Senior Data Analysts
86K views · 255 comments
- How to tell a middle analyst from a junior with one question
342 reactions · 127 comments
- How to answer “name the key metrics of a product”
40 reactions · 5 comments
Work
inDrive
Senior Data Scientist · Super App team
We improve personalization with ML. My part is research, A/B tests and causal studies.
Earlier
VK Mail · 2025–26
Metric audit
Audited partner-monetization metrics with product and engineering. Priorities were reset.
30%revenue overstatement found
VK Mail · 2024–25
Holdback system
Automated holdback experiments across Mail, Cloud and the Mail.ru portal to separate product impact from market effects.
60M+monthly users
2releases canceled
T-Bank · 2021–24
Voice-bot business cases
Found and justified voice-bot features, and replaced the key A/B metric with more precise measures of impact.
5M+ RUBmonthly savings
Before inDrive: Analytics Team Lead and Lead Product Analyst at VK Mail (2024–26), Analytics Team Lead and Product Analyst at T-Bank (2021–24). BA in Economics, New Economic School & HSE University, 2023.
Say hi
A question about experiments or just hello.