In 1858, when a wealthy Louisiana cotton broker named Colonel W.R. Stuart married Elizabeth McCauley, who came from a long line of North Carolina plantation owners, her family gave the couple a slave named Tempy Burton as a wedding gift. Elizabeth was sickly, unable to have children. But Tempy could and did have several with her new master, the colonel. Their youngest child, Josephine, was my great-grandmother.

I stumbled onto this tangled legacy back in 1981, when I was 12, by asking my fair-skinned paternal grandfather, Martin Ford, if he was white. In his liquid Louisiana drawl, he said he wasn’t and told me the story of his grandparents, the colonel and Tempy, and his mother, Josephine. I filed this history away for decades until my daughter one day declared that she was white, like her father (who is of Irish-Finnish ancestry), and not at all black like me.

My daughter has butterscotch skin; I’m cocoa colored, like my mom, my dad’s mom and Tempy. I wanted my daughter, then five, to embrace all her roots, but at almost 40, I wasn’t sure I had ever done that myself. Perhaps learning more about Josephine, my mixed-race great-grandmother, would help my daughter identify with my side as well as her father’s. So I set out to unearth the story of my own interracial, Confederate-era kin.

As Zora Neale Hurston put it in Their Eyes Were Watching God, “Us colored folk is branches without roots and that makes things come round in queer ways.” Investigating my family’s branches uncovered enough history to fill a dozen binders; they now line my office shelves, filled with everything from newspaper editorials by Josephine—in eloquent prose, she laid out the dangers of being too prideful—to her great-grandfather’s ruminations on the Revolutionary War as he headed off to battle. And it led me to my living history, which included new friends like Monique, my third cousin once removed, who is also descended from Tempy and the colonel. Like me, Monique lives in New Jersey; like me, she is a black woman married to a white man and raising two biracial girls. I was happy to meet her—what I’d never bargained for was also meeting descendants of the people who had enslaved Tempy. But it happened. And what brought us together was a 120-year-old photograph of our ancestors, black and white.

Tempy is in the center of the picture. Elizabeth and the colonel are sitting behind her. On either side of Tempy are two mixed-race-looking girls, probably her daughters with the colonel. The one on the left, curly haired and creamy skinned like my daughters, might even be my great-grandmother, Josephine. Ever since I found this photo on a genealogy website on my 38th birthday, it has become like a brand on me, a searing reminder of the people, and the pain, from which I came. The photograph is now the screen saver on my computer, the face of a homemade clock in my office, the sticker attached to a jar full of sand taken from the beach of my ancestral home, Ocean Springs, Mississippi. Tempy’s expression is haunting, as if she’s trying to solve a puzzle. The puzzle I need to solve: Why would she pose for what amounted to a family portrait, a good 25 years after slavery ended, with people who stole her freedom?