Prince Harry had a private audience with Queen Elizabeth TWICE before flying back to LA