Coinstar, the self-service retail business, prepares for the decline of movie rentals by experimenting with kiosks for products like coffee and electronics.