To answer your question, a drug starts to be used off-label when they find that it does more than just what it was being used for. For example, Finasteride (known as Propecia) was actually a medication for men with prostate problems. Suddenly, men on the drug started reporting that their hair was growing back. Studies were done and it was confirmed that lower doses of Finasteride could help a lot of men keep/grow hair.

Sometimes they find out these things by observation, sometimes they do studies to see how things correlate...

It would be illegal for a drug company to sell a drug for an off-label condition that the drug was not approved for officially, but it is NOT illegal for a doctor to prescribe a drug for off-label conditions.