|`Sōri Kōtei` The Official Residence just after completion in 1929|
The conservative leader took office in December but has yet to move into the 11-room brick home in central Tokyo.
According to local media, it is the longest holdout among any of his predecessors.
Several former prime ministers have reported experiencing unusual phenomena at the mansion, which was centre-stage for two failed but bloody coups in the 1930s.
Former prime minister Junichiro Koizumi once told reporters he had never encountered any ghosts, despite wanting to see them.
Some first ladies have refused to live in the mansion over fears its was inhabited by spirits.
An opposition politician asked about the presence of ghosts in the prime minister's residence in a letter to Mr Abe's cabinet.
"There are rumours that the official residence is haunted by ghosts. Is it true?" the politician wrote.
"Does prime minister Abe refuse to move to the official residence because of the rumours?"
He said the decision could delay the prime minister's response time in emergencies because the residence is next door to his executive office.
|Prime Minister Shinzo Abe|
Mr Abe's cabinet issued a terse written statement in response, saying: "We do not assent to what was asked."
In May 1932, a revolt by naval officers ended in the murder of then-prime minister Tsuyoshi Inukai and the plotters' surrender to military police.
Several years later in 1936, about 1,400 rebel troops killed several political leaders and seized the heart of Tokyo's government district including the official residence for four days.