A court has denied Brad Pitt's request for ex-wife Angelina Jolie to publicly disclose private messages in their dispute over the sale of shares in their French estate.