The review prompt I botched spectacularly
· · Not by Prompt Alone · episode 12 · Czytaj po polsku
"Hopeless! I really messed up those assumptions. So we'll wait another month for the first review prompt 😢"
That's how I summed up my own idea when Claude checked how a feature I had approved shortly before really worked. It wasn't about complicated sync or a new app mode. It was about a small window asking for a rating in the store.
Zakupomat could already ask for one. The code was in the app, the version was in the stores, the conditions were sensible. Except that on 23 October, a week after releasing 1.7.0, there still wasn't a single recorded prompt.
And the more closely we checked the conditions, the less happy I was with my own decisions.
I didn't want to be annoying
At the end of September Zakupomat had one rating in the App Store, and Google Play showed none. The app didn't ask for stars at all. I decided it was time to change that, but without a popup on first launch. I don't like those popups myself, so why would I inflict them on others?
We picked the moment after a successful shopping trip. Someone finishes using the list, sees the congratulations, closes them - only then can the app ask for a rating. With the system window from Apple or Google, no custom survey on the way.
On top of that, a few safeguards. At least three finished shopping trips on this phone. At least a week since the first launch of the version that can ask. At most one prompt per version and no more often than every 90 days.
I could justify every condition. Three trips, so a person has something to rate. A week, so they get used to it. Rare reminders, so we don't annoy anyone. Counting on the phone, so a new household member doesn't get asked just because the rest of the family has been using the app for months.
Read one by one, they sound pretty good, right? I thought so too.
Forty-one households. And zero prompts
When we came back to it, the admin panel showed 41 households with at least three shopping trips. So there's someone to ask! Except that, according to the check at the time, none of those households was on the new version yet.
Updates spread sooooo slowly. I have a new build, I test it right away, and in my head everyone already has it. But people have their own lives. The app works, shopping gets done, why think about updating?
That was the first problem. The second one turned out even better.
I remembered that we check the household's shopping history. I wrote something like: on a given phone we already know whether the household has done its shopping before. Claude looked into the code and it turned out that... that's not what I had approved 😂
The panel counted the household's trips on the server. But the app had a separate counter on the phone. It only went up when that particular phone finished a shopping trip. History from before the feature didn't count.
So those 41 households did not mean 41 phones ready to show the question. I was comparing two different things, and only reading the code made me realise it. One week was enough for me to forget a detail of my own decision.
Users first had to start earning it
The real path looked like this: first the update, then the first launch of the new version, then three shopping trips finished on that same phone. Plus at least a week since the first launch.
If someone finishes shopping once a week, collecting three new trips can take about three weeks. The week of waiting runs in parallel, but before all that they still need to get the update. And if one person adds products and another closes the shopping trip, the first person's phone doesn't collect any points towards my wonderful condition at all.
A household could have been using Zakupomat for months. To the local counter it was a beginner.
I didn't break the window itself. I broke the road that was supposed to lead to it. A test of a single condition could pass with no problem. What I needed was to walk through the whole story of an ordinary user beforehand and ask: when will they actually reach the end?
The fix has to reach the phone too
On 24 October we changed the decision. Instead of three new trips on this phone, what counts now is two of the household's trips from its whole history, excluding trial ones. No extra week of waiting. The prompt can still appear after a successful trip on the phone where someone just finished it.
I consciously accepted a trade-off I had rejected before: a new member of a household with history can get asked after their very first shopping trip. I decided that was a smaller problem than waiting weeks for any effect.
The threshold now comes from the server. It can be changed without releasing another app version. We also added the option to trigger another prompt for a chosen household, still at the end of a successful shopping trip.
The catch is that you can't push a new way of control to old code with a server change alone. Support for it arrived in the app in 1.9.0. Anyone on an older version still counted the old way. While fixing the problem of slow updates, I had to... wait for another update.
And that's exactly what I should have foreseen earlier.
What I'd ask the agent for today
Not just to implement the conditions. I'd ask it to walk through the whole path on a few ordinary examples: a household shopping once a week, two people taking turns finishing the trips, someone who updates the app only after a month. How much time passes from release to the first possible prompt?
And I'd prepare server-side activation earlier. The code can already be reaching phones, and I switch the feature on when I'm ready. Thresholds that will probably need tuning don't have to be hardcoded in the app either.
That doesn't mean every little thing needs an elaborate feature-flag system. Here a threshold in the server response and handling on the phone were enough. The point is to decide before release what I want to be able to control later.
The agent helped build this feature. But I'm the one who approved the conditions. I won't pretend now that Claude is to blame, because I could have asked it earlier to check the effect of the whole combination.
Did the ratings finally move?
I checked the database on 15 November. The first prompt went out on 24 October, the day of the change, from version 1.9.0. Since then the app has triggered it 81 times in 63 households: 63 times on iPhones and 18 times on Android. That's without the demo accounts for store reviewers. In the week before, with the old conditions: zero.
The ratings are more modest. At the end of September the Polish App Store showed one rating. On 15 November it shows four, all five stars. Google Play is still at zero, but the chances there are much smaller, because few people use Zakupomat on Android.
So from 81 prompts I got roughly three new ratings in the App Store. And I don't even know for sure that all of them came from that window.
There's one more limit to the measurement. Our counter records that the prompt was triggered. It doesn't confirm that a person saw the window, or that they left a rating. Whether it's actually shown is up to the store, within its own limits. Apple and Google describe this. So even correctly working code doesn't give me certainty that there was one more chance for stars.
But whatever the result, the lesson is already here. In a mobile app, "released" is the beginning of the road to the user. And a few sensible conditions can stretch that road so far that even the author is surprised.
What this means for your team: before a release, work out the time to the first real effect, including the update and all the conditions together. Also decide which decisions you'll be able to change from the server, and which will send you back to the stores. At ANSLAN we help prepare systems and ways of working with AI agents so that these questions get asked before implementation.
Zakupomat is free and has no ads. You can rate it, even though I made getting to that offer a bit harder 😉