The connected home is inevitable. Like the plumbing, electricity, and refrigeration that came before it, the smart-home system will soon be an integral part of our domestic lives. Today, more than a third of Americans with high-speed Internet own smart-home devices, the most popular being smart speakers and connected security cameras. According to research firm Berg Insight, almost half of all households in America will somehow qualify as “smart” in just three years.

The lifeblood of the smart home is data: sensors sending information to linked devices, artificial intelligence learning our habits to improve service, infrastructure and appliances talking directly to each other. But with all that tech keeping track of how we live and what we do, skepticism and fear about smart devices are as widespread as the gadgets themselves.

Is somebody eavesdropping on our most intimate conversations through that smart speaker? Could an intruder get into our house by hacking a connected door? Might a smart oven one day go rogue and burn down the house? Yes and no.

“I’m not as concerned about the risk of someone hacking a smart home in a way that’s destructive,” says host of Internet of Things podcast Stacey Higginbotham. “I would rank that at the bottom. Above that is malware targeting popular devices, and above that is the collection and storage of data for later parsing. Basically, people can do anything with that data. There are no protections.”

Recent revelations that humans have been listening to what we say to Google’s Nest, Amazon’s Echo, and Apple’s HomePod present a case for reining in the power of these products. “The only way to create a smart home that doesn’t compromise your privacy is to have strong regulations in place to make sure people can’t use that data maliciously or without being completely transparent about their intentions,” says Higginbotham.

A natural response would be to demand that our elected officials draft policies to protect us from such invasive violations, but some see government involvement as potentially an even bigger problem. Higginbotham cites the growing controversy over smart-security company Ring’s partnerships with more than 400 police departments.

These alliances can provide officers with access to homeowners’ video footage from their doorbell cameras, with their permission. “Targeting neighborhoods and sending the camera feeds that have been collected to ICE, that’s totally something that could be done today. And that’s very scary,” she says. More than 30 civil rights groups have written an open letter to state and local governments to end these partnerships.

But these devices are not spying on you in the traditional sense, argues Jeff Jarvis, professor and author of the book Public Parts: How Sharing in the Digital Age Improves the Way We Work and Live. In most cases, he says, “humans listening to voice commands are simply checking samples to see how good the technologies are.” It’s a case of extrapolating the dangers of what a technology can do before any harm has actually happened, he says. “If we focus on everything bad that could happen, we’ll rule out a lot of good.”

As we learn more about how Facebook, Equifax, and other large tech companies struggle to protect vast collections of consumer and private data—while fighting off government regulation—the smart-home companies are not standing idly by.

“You have a lot of sensitive information in the home, and today it can go to third parties that you may not know, or you may not understand how they use that data,” says Rishi Chandra, who oversees Google Nest, Google’s smart-home division. The company’s newest device, the Google Nest Hub Max smart display, is a mix of every conceivable privacy concern—a speaker that is always listening, a camera that can identify you, a hub to control all your Google-connected lights, locks, and thermostats—but it also has some of the clearest opt-out instructions on the market.

Other new technologies allow you to control your home without your data’s ever seeing the cloud. Apple’s HomeKit is locally controlled and end-to-end encrypted, meaning your voice and video data are entirely private to you. Not even Apple knows how and when you use your home accessories. The person-specific, room-by-room sensor system, RoomMe, sits on the ceiling and connects to each family member’s smartphone over Bluetooth, adjusting things like lighting, temperature, and music to individual preferences. There’s no need to send voice commands to the Internet just to turn on a light.

The ultimate question for us each to ask ourselves as we hurtle toward the connected home is, “Is the exchange of my personal data worth the convenience the smart home offers?” If it’s not, proceed with caution.

Related Reading:

To See or Not to See? The Evolution of Whether and How to Hide Television Screens Comes Full Circle

We Charge Our Phones and Our Cars. Is it Time to Charge Our Homes?