Repeated response patterns: 6–7% of respondents that standard checks missed
Author: prof. PhDr. Zoltán Rózsa, PhD.
Online panels are a common source of data for both academic and commercial research. When checking their quality, we usually ask whether a respondent completed the questionnaire attentively. But what if their answers look plausible, yet match those of other people exactly?
We came across this problem in an online panel survey conducted in Czechia and Slovakia in 2026. In each country 600 respondents took part, 1,200 in total. The questionnaire contained 31 questions in four thematic blocks, answered on a scale from 1 to 5.
What standard checks found: straightlining and careless responding
Data quality checks usually look for signs of careless responding. They check, for example, whether a respondent chooses the same answer for many questions in a row (so-called straightlining), whether their answers vary too little or unusually much, whether they contradict themselves on similarly worded questions, and whether their overall profile differs markedly from everyone else's.
Based on these checks, we excluded 71 of the 1,200 completed questionnaires: 40 in the Czech and 31 in the Slovak sample – roughly 6% of questionnaires. In every case at least two indicators had to agree. A suspicious result on a single indicator was not enough for exclusion.
What standard checks missed
Another problem emerged when we compared answers across respondents. We found groups of people who answered all 31 questions in exactly the same way.
In the Czech sample we found 365 pairs of respondents with an identical response profile, in the Slovak sample 246. This is the number of pairs, not of people: each respondent can form an identical pair with several others.
For comparison, we randomly reshuffled the answers between respondents within each question. The number of each answer to every question stayed the same, but the combinations within questionnaires changed. After this reshuffling we did not find a single fully identical pair.
Table of values
| Item | Pattern A – Czechia, 24 respondents | Pattern A shifted one point lower – Czechia, 13 respondents | Pattern C – Slovakia, 17 respondents |
|---|---|---|---|
| JA1 | 4 | 3 | 4 |
| JA2 | 3 | 2 | 4 |
| JA3 | 5 | 4 | 3 |
| JA4 | 4 | 3 | 4 |
| JA5 | 3 | 2 | 4 |
| JA6 | 5 | 4 | 5 |
| JA7 | 4 | 3 | 4 |
| JA8 | 3 | 2 | 4 |
| JA9 | 5 | 4 | 3 |
| TP1 | 4 | 3 | 4 |
| TP2 | 3 | 2 | 4 |
| TP3 | 4 | 3 | 5 |
| TP4 | 4 | 3 | 4 |
| TP5 | 5 | 4 | 4 |
| AIT1 | 4 | 3 | 4 |
| AIT2 | 3 | 2 | 4 |
| AIT3 | 4 | 3 | 4 |
| AIT4 | 4 | 3 | 3 |
| AIT5 | 4 | 3 | 4 |
| AIT6 | 4 | 3 | 4 |
| AIT7 | 4 | 3 | 5 |
| AIT8 | 4 | 3 | 4 |
| AIT9 | 4 | 3 | 4 |
| AIT10 | 4 | 3 | 4 |
| AIT11 | 5 | 4 | 4 |
| AIC1 | 4 | 3 | 4 |
| AIC2 | 3 | 2 | 4 |
| AIC3 | 4 | 3 | 3 |
| AIC4 | 4 | 3 | 4 |
| AIC5 | 4 | 3 | 4 |
| AIC6 | 5 | 4 | 4 |
The figure shows three of the identified response patterns. In pattern A, which we found in 24 respondents, the sequence “4 – 3 – 5” repeats three times over the first nine questions. A similar pattern continues in the other blocks regardless of what the questions are about – autonomy at work, time pressure or attitudes towards AI. The next group shown has exactly the same profile, but every answer is one point lower.
In total we identified five such patterns among 80 respondents: 44 in Czechia (7.3%) and 36 in Slovakia (6.0%). None of them met the exclusion criteria based on the standard indicators. These answers do not look suspicious in a routine check: the respondent does not always pick the same option, the answers vary, and they stay mostly around the middle of the scale.
Data cleaning in numbers
| Step | Czechia | Slovakia | Total |
|---|---|---|---|
| Completed questionnaires | 600 | 600 | 1,200 |
| Repeated response patterns (excluded) | 44 | 36 | 80 |
| Careless responding – at least 2 indicators (excluded) | 40 | 31 | 71 |
| Final sample | 516 | 533 | 1,049 |
Why it matters
Repeated response patterns can inflate relationships between variables. The associations found then partly reflect a shared response pattern. In the Slovak sample, after excluding respondents with these patterns, the correlation between time pressure and trust in AI fell from 0.33 to 0.25, between time pressure and AI-crafting from 0.36 to 0.30, and between autonomy and AI-crafting from 0.40 to 0.35. In the Czech sample we saw a similar but smaller decrease.
Table of values
| Relationship | SK including patterns | SK after exclusion | CZ including patterns | CZ after exclusion |
|---|---|---|---|---|
| Autonomy – AI-crafting | 0.40 | 0.35 | 0.37 | 0.35 |
| Time pressure – AI-crafting | 0.36 | 0.30 | 0.24 | 0.21 |
| Time pressure – trust in AI | 0.33 | 0.25 | 0.28 | 0.25 |
| Autonomy – trust in AI | 0.40 | 0.36 | 0.33 | 0.31 |
| Trust in AI – AI-crafting | 0.80 | 0.78 | 0.74 | 0.72 |
So if you keep respondents with repeated response patterns in your data, relationships between variables may look stronger than they really are. Means alone may not reveal the problem.
How we found it – and how to check your own data
We came across the repeated response patterns during a routine check for questionnaires completed more than once. Several respondents with different IDs had identical answers to all 31 questions. When we joined each respondent's answers into a single string – for example 435 435 435 | 43445 | 43444444445 | 434445 – the regularity was obvious at first glance. In the first block the sequence “4 – 3 – 5” repeated three times, and similarly regular sequences appeared in the other blocks despite the different content of the questions.
We then compared all pairs of respondents. We found 365 pairs in Czechia and 246 in Slovakia with identical answers to all 31 questions. For comparison, we randomly reshuffled the answers between respondents within each question. The number of each answer to every question stayed the same, but their combinations within questionnaires changed. After reshuffling we did not find a single fully identical pair.
Finally, we also looked for the same patterns shifted along the scale. From each answer we subtracted the value the respondent used most often. This let us compare the shape of the answers regardless of their overall level. It turned out, for example, that the sequence “3 – 2 – 4” copies the pattern “4 – 3 – 5”, only one point lower on every question.
In your own data, you can proceed in three steps:
- Look for identical response profiles. Join each respondent's answers to the scale questions into a single text string and count how many times it occurs. In Excel you can use the formula
=TEXTJOIN("",TRUE,D2:AH2)followed byCOUNTIF. In SPSS you can use the CONCAT function and the Identify Duplicate Cases tool. - Compare patterns shifted along the scale as well. Subtract the respondent's most frequent value from each answer and compare the resulting strings. In our analysis we treated identical patterns as suspicious if at least four answers differed from the most frequent value. For comparison, randomly reshuffle the answers separately within each column and see how many matches arise after reshuffling.
- Look at the suspicious patterns in a chart. Plot the answers as lines across the individual questions, as in the figure above. This shows where the same sequences repeat and which profiles are shifted versions of each other.
What to watch out for:
- Questionnaire length. With several dozen questions, fully identical profiles are more suspicious than with a short scale of four or five items, where they can easily arise naturally. That is why we compare answers to the whole questionnaire and complement this with a comparison against randomly reshuffled data. Reshuffling, however, also removes natural associations between answers, so on its own it is not proof of poor-quality responses.
- Low-variability answers. Two people may naturally answer 4 almost everywhere and 5 just once. That is why, when looking for shifted patterns, we required at least four answers to differ from the respondent's most frequent value. This is a working threshold whose choice should be justified in the methods section.
- Small changes in the pattern. The procedure only catches exactly identical profiles or profiles shifted by the same number of points. If someone changes a few answers at random, such a pattern may go undetected. Completion times and metadata from the provider can help with further checks.
- Describing the procedure in your work. In a thesis, dissertation or article, explain precisely which rule you used to exclude respondents and report their number. Compare the results before and after exclusion.
What else helps
- Ask for completion times, ideally including times for individual pages. Combined with repeated response patterns, they can help when assessing suspicious records.
- Include attention-check items, for example “For this question, please select ‘disagree’”. Reverse-worded statements can help reveal inconsistencies in answers, but they can also cause misunderstandings. Do not use either of these methods as the only reason for exclusion.
- Agree on access to metadata, such as a completion flag or a device identifier. Also agree in advance with the provider on how suspicious records will be reviewed and claimed.
What follows from this
- Checks for careless responding should be complemented by comparing respondents with one another. In our survey, the standard indicators caught 71 respondents. Another 80 with repeated response patterns, however, did not meet the exclusion criteria. The two approaches thus caught different groups.
- Even a small part of the sample can affect the results. In our survey, excluding roughly 6–7% of respondents lowered several correlations. Means alone may not reveal such a problem.
- Ask how the provider checks data quality. When commissioning a survey, agree on receiving the original data including completion times, and an explanation of the rules by which the provider screens and removes responses.
Limitations. The findings come from a single survey. The share of respondents with repeated response patterns may differ in other panels and projects. The procedure only catches exactly identical or uniformly shifted profiles; patterns with random changes may remain undetected. From the available data we also cannot determine how these matches arose – whether through automated completion, a coordinated group of respondents, or one person with several accounts. Completion times and other metadata could help examine these possibilities, but on their own they might not reliably distinguish between them.
The research was supported by the project VEGA 1/0316/26 – Workplace of the Future.
Sources
- Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. https://doi.org/10.1016/j.jesp.2015.07.006
- Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085