Ever been to one of those buffets that are just a random assortment of mismatched food items? Like, it's fried chicken, mongolian beef, scrambled eggs, sushi, mashed potatoes, and mini tacos? This is kind of like our version of that buffet, only it's not all-you-can-eat.
If you've got something dark in your past that you don't want people to know about, that means you've got skeletons in your closet, figuratively speaking. If you've got actual skeletons in your closet, that means you have skeletons in your closet, in both the literal and the figurative sense.