The Tokyo Olympics organising committee said in their Thursday update that two athletes staying in the Games Village were among those, who tested positive.
Organisers said two more people from the Olympic Village are affected, but they are not athletes.
All four must now go into quarantine.
The latest number brings the total of positive tests since July 1 to 87.
The 2020 Olympics, which was postponed from last year to 2021 due to Coronavirus, has been marred with controversy.
Before the kickoff of the event, protesters had stood against the continuation of the games amid the spike in COVID-19 cases in the country.
On Thursday morning, it was reported that the Tokyo Games organisers fired the director of the opening ceremony, Kentaro Kobayashi, over a joke he made about the Holocaust as part of his comedy act in the 1990s that recently resurfaced in domestic media.
Earlier this year, the head of the Tokyo 2020 organising committee resigned after making sexist remarks and Tokyo Olympics creative head followed after he made derogatory comments about a popular Japanese female entertainer.