Blackstone Group LP, the world’s largest private-equity firm, ended talks to buy Brocade Communications Systems Inc. because the price was too high for a leveraged buyout, said a person with knowledge of the situation.