Bill Shorten has called for Indigenous affairs minister, Nigel Scullion, to be stripped of his portfolio as criticism mounts over the minister’s response to the Don Dale abuses.

The opposition leader told Guardian Australia Scullion was “either asleep at the wheel or he just doesn’t get it”, and that he wouldn’t have Scullion in his own Cabinet.

“Mr Turnbull should consider a different job for Senator Scullion because he’s not doing this one very well,” said Shorten.

Scullion had been under pressure since the broadcast of Four Corners last week showed young boys inside Darwin’s Don Dale juvenile detention centre being stripped and thrown around by guards, hit, and in the case of 17-year-old Dylan Voller, covered with a spithood and strapped in a restraint chair.

On Wednesday the co-chair of Reconciliation Australia, Tom Calma, called for Scullion to resign.

“We rely on him … [to be] pushing government to make sure Aboriginal and Torres Strait Islanders get fair and equitable treatment,” Calma told ABC Radio.

“Any minister that doesn’t listen to their constituency, doesn’t stand up for the portfolio they’re representing should really consider their position.”

Scullion had been criticised after the Four Corners broadcast for claiming he hadn’t seen the footage, and that it hadn’t “piqued” his interest. He later said he had thought the Northern Territory government was taking care of matters, butapologised that he wasn’t better informed.

On Thursday Scullion admitted he had received a briefing on media reports on the use of teargas against detainees in 2014, but not a full briefing on the contents of a children’s commissioner’s inquiry into the incident or the NT government’s amendments expanding the use of restraints.

He said the failure showed his department “can’t make an assumption that another jurisdiction is dealing with it just because substantially it’s their responsibility”.

“If it’s an Indigenous person, it doesn’t matter whose jurisdiction it is, it is my responsibility as well,” he said.

The prime minister, Malcolm Turnbull, gave his support to Scullion, and said criticisms were the nature of politics.

“Nigel Scullion is a very committed minister for Indigenous affairs and he certainly has my confidence and the confidence of his colleagues,” said Turnbull.

“He understands the community – it is his job to support the Indigenous Australians. He understands that area very well and of course he understands the territory very well.”