A police spokeswoman said the two were found in an atrium of an apartment building near the corner of Bourke and Spencer streets about 10.30am on Thursday.
Police have confirmed that a broken balcony or railing were not to blame for the deaths.
A spokeswoman also said police were not looking for anyone else in connection with the incident.
It is understood that the woman was holding the baby when they fell. The pair died at the scene.
Police said it was believed they may have fallen from an internal balcony.
Some time after the deaths, a large group of family and friends of the woman were speaking to police and social workers in the foyer of the building.
One spoke briefly to the media as he left the building.
“It’s my friend’s wife,” he said. “We don’t know what happened.”
Not long after that, two police officers were seen in the building’s foyer holding several brown paper bags of what appeared to be evidence.
The coroner was at the scene and appeared to be removing the bodies from the laneway about 2.30pm.
Bystander Christine Harms, from Geelong, said she heard screams soon after the fall.
“A lady went in to the alleyway and then there was some screaming,” Ms Harms said.
“I just kept hearing ‘Oh no. Oh no’. It was gut-wrenching to hear.”
Details remain sketchy but three distressed construction workers were sent home from a neighbouring building, 200 Spencer Street.
A work colleague said one of the workers also heard the woman scream and was upset at the aftermath.
Police have cordoned off parts of level three at 200 Spencer Street.
An Ambulance Victoria spokeswoman said paramedics were called to the area but could not assist the woman or the child.
They later returned to attend to a member of the group of family and friends.
It is understood that the woman and child fell into a courtyard behind the Chocolate Frog cafe.
An area around the Chocolate Frog Cafe and the Whitehouse Institute of Design was cordoned off, while olice gathered statements from witnesses, including a visibly shaken manager of the Chocolate Frog.
Residents from the 29-storey apartment tower nearby said they heard nothing unusual leading up to the fall.
Musician Alex Champ, who lives in the City Point building, was shocked at the news.
“That’s crazy to think it just happened a few floors from me,” he said.
Mr Champ, 26, said a mixture of people lived in the building, including families.
He said apartments in the building were also used as holiday or short stay rentals.
Mr Champ described an internal area of the building with a “huge drop” facing tennis courts, a pool and gym.
A senior police officer told the media they would not be providing further details on the circumstances of the deaths at the scene.
Part of Bourke Street heading away from Southern Cross Station was blocked to traffic, but trams were still able to pass through the area.
Police at the scene have also been spotted going into the cafe.