Depending on how you are doing your sampling, your ensemble of possible conformations will be a thermodynamic ensemble and you could look at them using Boltzmann statistics – X many units of energy means Y times less likely a conformation.
Less rigorously, look for a natural cutoff in your models. You may find that there is a steep change in energy after the top 3% of models, so only consider the top 3%. Or, just take any models that don’t have an identifiable flaw and are within your resources to spend time considering.
This question is best answered empirically; the right way to do it is whatever works.