In short, a Stonewielder is an individual with the ability to use Stones. To an ordinary human, Stones are nothing more than colored rocks. In fact, many humans have, indeed, kept powerful Stones in private collections for decades, not understanding their true value.
Being a Stonewielder is not simply a magical condition, as was thought for many centuries. Instead, the Stonewielder body is infused with millions of microscopic Stones. For reasons still unknown, when a Stonewielder baby is born, all the Stones in that individual's body will be the same color, despite there being so many. Additionally, there is no genetic tendencies towards certain colors. For example, a red father and a blue mother could quite easily have a green child, even if there were no other greens in their entire family tree.
A quick word of clarification. The color of a Stonewielder does not indicate the literal color of their skin. Instead, when a Stonewielder is simply going about everyday business (or relaxing at home), they will appear like any average, everyday humans. However, when they draw upon their internal Stones, an external change takes place. Robes, the color of their internal Stones, will replace whatever clothes they were wearing. This change is known as "going on edge," and is necessary in order to use a Stonewielder's inherent powers or interact with macroscopic Stones.
While there is not a genetic component to the color of the Stonewielder, there most certainly is a genetic component to the Stonewielders themselves. In recorded history, there has never once been a Stonewielder born to non-Stonewielder parents. Every single time this has supposedly happened, it has come to light that one of the parents either was a Stonewielder that ran away from society, or that the mother had a secret lover. In addition, there has never been a child born to a Stonewielder parent that didn't have Stones. There are several diseases and genetic conditions that can prevent Stones from appearing until a later age, but the child will always have those inherent Stones.