Social media photographs showed the men, aged 25 to 29, clutched together, their arms in the air amid the crowd at 5pm on Sunday.
Several videos posted to Instagram showed two of the men chanting “Aussie Aussie Aussie, oi oi oi” while standing in the middle of a crowd wearing socks and budgie smuggler-style swimmers.
They later poured beer into their shoes and drank from them.
Officials and some onlookers at the track at Sepang, near Kuala Lumpur’s international airport, took offence.
“Shame on those Aussies doing this. So inappropriate,” tweeted Nik Asyraaf.
The government-controlled New Straits Times newspaper reported that opinion was divided on the behaviour, saying some people thought it humorous while others saw it as insulting and disrespectful to local culture.
Sepang district police chief Abdul Aziz Ali said the men would be held in jail for at least four days while further action against them is considered.
He told reporters they are being investigated for breaching the peace, public indecency and disrespecting the national flag.
The flag appears to be printed on tight-fitting swimmers made by a Sydney-based company.
If charges are not laid the men will almost certainly be deported from the country after spending days in jail.
Malaysia is a Muslim-majority country of 30 million where authorities take a dim view of acts of nudity by foreigners.
Last year 10 foreigners were accused of desecrating a holy site after they posted a photo of themselves posing nude atop Malaysia’s Mount Kinabalu.
Passions were inflamed further over the act when days later a 5.9 earthquake struck the mountain, killing at least 16 people and trapping scores of climbers.
Superstitious local officials accused the nudists of causing the quake.
A spokeswoman for the Department of Foreign Affairs and Trade (DFAT) said assistance is being provided to the men.
“The Department of Foreign Affairs and Trade is providing consular assistance, in accordance with the Consular Services Charter, to a group of Australians who were arrested in Malaysia,” the spokeswoman said. “Due to privacy obligations, we are unable to provide further information.”