Early data and the pattern of the virus spreading supports your theory. We can use the Washington outbreak as an example: a high school student in the area was diagnosed with the corona virus, but he had no known contact with anyone in the nursing homes (where most of the cases and deaths have originated from) and no contact with any other person with a confirmed case. His confirmed case indicates that the virus is spreading to others and is not being recognized or reported because the symptoms are mild. The death rate in Washington is so high because the outbreak is centered around two nursing homes.
This case also indicates that the actual number of cases is higher than confirmed cases because many patients aren’t going to the clinic to get evaluated. Mild cases present like a common cold and most with a common cold just take OTC medications until they’re symptoms resolve. I imagine that many of the confirmed cases in the U.S. are from individuals who have been in contact with a confirmed case and were instructed to go to the clinic and get tested for even the most minimal of symptoms.