Observing another person performing an action can lead to a false memory of having performed the action oneself – the observation-inflation effect. In the experimental paradigm, participants first perform or do not perform simple actions, and then observe another person perform some of these actions. The observation-inflation effect is found when participants later remember performing actions that they have merely observed. In this case, self and other are confused in action memory. We examined social conditions of this self-other confusion when remembering actions, specifically whether the effect depends on the observed actor’s group membership. In our experiment, we manipulated group membership based on physical appearance, specifically complexion of the hands. Fair-skinned participants observed either an in-group (i.e., fair-skinned) or an out-group (i.e., dark-skinned) actor. Our results revealed that the observed actor’s group membership moderated the observation-inflation effect: False memories were significantly reduced when the actor was from the out-group (vs. in-group). We found no difference to a control condition in which the actor wore black gloves, suggesting that distinctiveness of perceptual or sensory features alone (due to the out-group member’s dark skin) is not critical. We discuss these findings in light of social-neuroscience studies demonstrating the impact of an observed person’s group membership on motor simulation. Overall, our findings suggest that action memory can be affected by a ubiquitous feature of people’s social perception, that is, group-based social categorization of others.