Draft vs. Live Questions: Why Editing a Question After Responses Exist Is a Trap

There's a specific moment that catches almost everyone the first time they run a survey: responses start coming in, someone on the team reads question three and realizes it's ambiguously worded, and the instinct is to just... fix it. Go edit the question, tighten the wording, republish. Simple, right?

It's not, and it's worth understanding exactly why before you're the one staring at that edit button mid-survey.

What changing a live question actually breaks

A survey isn't really the questions — it's the questions plus the responses collected against them. If you edit question three's wording after ten people have already answered the old version, you now have ten responses to a question that no longer exists in the form anyone else will see, sitting next to however many new responses come in against the new wording. Averages, breakdowns, and NPS calculations all assume every response was made against the same question. Mix versions and the numbers stop meaning anything specific — they're an unlabeled blend of two different questions pretending to be one.

This is worse than it sounds because it's invisible. Nothing errors out. The survey detail page keeps showing a single average rating or a single choice breakdown, and there's no flag anywhere saying "by the way, this number is actually two different questions stitched together." You'd have to know to go check.

Why SparkyLoop locks questions at publish, not before

This is the actual design in SparkyLoop, and it's deliberate rather than an oversight: while a survey is still a draft, you can add, edit, remove, and reorder questions freely — as many times as you want, in whatever order you want. There's no penalty for indecision at the draft stage. The moment you publish, though, the question set (and the distribution settings alongside it) locks. From that point on, every response collected is guaranteed to be answered against the exact same set of questions as every other response.

That guarantee is the entire value of the lock. It's what makes an NPS score you're tracking quarter over quarter actually comparable, and what makes a 360-degree competency report make sense at all — a combined report grouping self, manager, peer, and direct-report responses only works because every rater answered the identical set of competency rows.

What to do instead when a question genuinely needs to change

The lock isn't there to trap you into shipping a flawed survey forever — it's there to keep you from silently corrupting the data you've already collected. If a live question really does need to change, the honest path is usually one of two things.

First: close the current survey and start a fresh one with the corrected wording. You lose nothing — the old survey's responses are still there, intact and internally consistent, and the new survey's responses will be too. You just have two clean datasets instead of one contaminated one, and you can decide for yourself whether it makes sense to compare them or treat the new survey as a clean restart.

Second, and often better if you catch the problem early: close the survey before it's collected many responses, copy the intent into a new draft with the fix, and republish. If only two or three people had answered, the cost of restarting is close to nothing — much less than the cost of a metric nobody can fully trust for the rest of its life.

Either way, the fix is never "edit the live question and hope the old responses don't matter much." They matter exactly as much as the new ones do, which is the whole problem.

Draft as much as you want — that part really is free

Because drafts are unrestricted and locking only happens at publish, the right habit is to draft generously and publish deliberately. Write the survey, sit on it for a day, reread every question as if you were the person answering it cold, reorder anything that reads awkwardly, and only publish once you're genuinely comfortable that this is the version you want live for the survey's whole run. That upfront review is cheaper than any fix available to you after the fact.

How to do this in SparkyLoop

Build and refine your questions freely while a survey is still a draft — add, edit, remove, and reorder as needed. Once you publish, questions and distribution settings lock for the life of that survey; if something needs to change, close it and start a new draft rather than editing live. See the full question builder on SparkyLoop's features page.