Initial reports suggested the man fell more than 10 metres from Magic Point, a popular whale watching vantage point.
Police officers, surf life savers and paramedics found the man alive, believed to be in his 30s, at the base of the cliff.
He was treated by paramedics but died a short time later.
Initial investigations suggest his death was not suspicious, police said. The man has not formally been identified.
Officers established a crime scene and will prepare a report for the coroner. Anyone with information has been urged to contact Crime Stoppers on 1800 333 000