My answer is: it depends.
When people are on public streets they inherently relinquish a large portion of privacy, which means that if I'm photographing a street scene, people really shouldn't complain. That doesn't mean I just walk up to a stranger and shove a lens in his face; if I'm keen on a particular individual the right thing to do is ask permission, and if it isn't given the solution is to move on, not sneak around the corner and whip out the telephoto lens, paparazzi style, because when permission is denied then making the image anyway is an invasion of privacy.
Then again the goal of many street photographers is to capture life as is (not posed or artificial), in which case asking first removes spontaneity. It's a fine line to walk and ultimately depends on the photographer's intent. And that's where digital photography shines, giving us the ability to interact and demonstrate that we are not trying to harm people; folks often respond positively when shown how great they look. Many countries also have laws that state if I plan to sell the image, the person in the photograph must sign a model release.
See this guy? I asked and was given permission to photograph, and yet the image doesn't feel contrived because that's pretty much how he works:
Heck, I've had my own image taken without consent numerous times while giving photography lessons in a coffee shop. I don't like it, but it's a public space and so I have to accept the situation. For all I know images of my students and I are floating around the Internet, and there's little I can do about it.
But when people are at home, taking their photograph through open windows is a breach of privacy, and that's the current controversy surrounding Michael Wolf's latest effort, "Window Watching", in which Hong Kongers are unknowingly displayed in their flats via telephoto lens. In some images the subject's face isn't shown, but in many others people are clearly identifiable.
And that's crossing the line. Some will argue that it's an uneasy mix of voyeurism and artistic freedom, and that if those people had wanted full privacy they'd have drawn their curtains (living in a building surrounded by other high-rises means it's possible that at any given moment someone may be peeping, and the only way to prevent it is to block the view, so the onus is on you). Wolf's attitude is that a strong photograph "can make me think", and certainly these images evoke interest in their depiction of how people live and make use of limited space. And if no one's face had been visible I wouldn't have felt he'd gone too far.
Contrast that with Wolf's previous collection, "Tokyo Compression", which shows the difference between beautiful art and invasiveness, but again, these were made in a public space, where the expectation of privacy is greatly diminished.
I like some of Wolf's work, but my initial response to this latest series was mostly negative, the exceptions being where people's faces are obscured. There is a little voyeur in all of us, and gently surreptitious photography can give us a glimpse into others' lives as a means of evaluating ourselves. But I wonder if this woman would be thrilled to have her image sold in a hardbound book without her approval. I know I wouldn't, and that's often the litmus test I use when deciding whether to include people in my images.
A person's home should remain off limits (unless access is explicitly granted); these days it's one of the last refuges of privacy people have left.
Update: the Window Watching series no longer appears on Wolf's web site.