Yup, that is the downside. Wall street, however you might view it, does have rules and is supposed to abide by laws of the USA, protecting you from things like fraud and insider trading, and does, for the most part work quite well. One exception is the penny stocks, which for whatever reason, are like the wild west.
So, (not to rain on the parade) the down side of the proposed plan is that there would be no standards in place, things like required audits and clean accounting practices. A market like that would leave room for all kinds of fraud and predators who could claim anything just to drive there "stock" price up. Also who sets the price? Pure supply and demand? Against what benchmark? P/E ratios? Claims by company? All this is new.... and will take years to sort out, probably after scores of fraud cases and other types of crimes.