You might assume that a scion of one of America’s most established families would live grandly, perhaps on Louisburg Square. But Oakes Plimpton, 83, resides in a modest house in Arlington Heights, high in the hills adjacent to Robbins Farm Park, where he started a communal garden. He also helps tend the family gardens at home. His uniform? A T-shirt with birds on it, supplemented by a turtleneck in colder weather.

Plimpton is the quirky, lanky brother of the late “Paris Review’’ founder George Plimpton. He grew up on Fifth Avenue and Long Island, the son of Francis T.P. Plimpton, a diplomat and partner at Debevoise & Plimpton, the white-shoe Manhattan law firm, and Pauline Ames Plimpton. His grandmother, Blanche Ames, was a prominent suffragette and artist. Known for orchid illustrations, her works hang throughout the house. Plimpton’s grandfather, Oakes Ames, was a famed botanist and head of Harvard’s Botanical Museum.

And so this Plimpton is more comfortable gardening than cocktailing. An anti-poverty lawyer, community organizer, and environmental advocate who once worked for The Nature Conservancy, he became a leader of the gleaning movement in the 1990s. Plimpton made a career out of picking leftover crops from local farms — which might otherwise go to waste — and distributing the bounty to food pantries. He founded and managed the Arlington Farmers’ Market and oversaw the Waltham Fields Community Farm, taking leftover goods to Menotomy Manor public housing in Arlington and selling bags for $2 ($1 during the recession).

“I was born with a trust fund and didn’t have to work for a living. I felt I should give back to society,’’ Plimpton said, relaxing on a long wooden bench in the dining room. Bird figurines dangle from a chandelier. His wife, Pat Magee, offered up fresh muffins and kept him on track if the conversation meandered.

He also spearheaded the Friends of Robbins Farm Park, steps from his house, and supported its gardens, playground, and even kite festivals. He never stops, Magee said, noting that he is now working on another book on Arlington’s history. His book “Farms and Farmers of the Arlington Farmers’ Market 1997–2005’’ is on view at the Arlington Public Library.

“I’m something of a town historian,’’ he said modestly.

But he does take time to relax, usually by bird-watching.

“It’s been my hobby since childhood,’’ he said. Back then, he birded in The Ramble in Central Park. Now he teaches beginning birder classes.

Fittingly, his adult son is named Robin Plimpton-Magee. The walls of Robin’s childhood bedroom are still decorated with painted birds: an owl, peregrine falcon, and a pileated woodpecker. The backyard, however, is Oakes Plimpton’s bird-watching haven. The home is perched high enough that on a clear day, it’s possible to glimpse twinkles of the Atlantic. Here, he spies cardinals, robins, chickadees, warblers, and more.

“I once kept a ‘backyard bird list’ on my porch until the writing faded out. We have over 80 species,’’ he said.

Magee tends a garden of azaleas, blueberry and raspberry bushes, myriad flowers, and vegetables.

“We try to grow a lot of plants to attract and feed butterflies, especially monarch butterflies and swallowtails,’’ she said.

It’s a long way from Plimpton’s youth, when he once squired socialites like family friend Edie Sedgwick to swanky galas. “She took me to the Peppermint Lounge, and I learned how to twist,’’ he recalled with a laugh, ducking back into the dining room.

“You’ve had quite a life, Oakes,’’ Magee said.

Kara Baskin can be reached at kcbaskin@gmail.com. Follow her on Twitter @kcbaskin.