(Not the large-scale process, from editorial letter to line-edit to copyedit; but what happens in your brain, letter by letter, word by word, if you too have the editorial bug.)
1. Notice that something feels off to you. It may be a dangling modifier; it may be a mistake in the chronology; it may be as big as the fact that the main character is turning out to be a smarmy jerk or you're bored at a point where the action ought to make you excited; it may be as tiny as an "an" where the author should really have a "the." In any case, like Miss Clavel, you sense that SOMETHING IS NOT RIGHT.
3. Identify the problem and the principle it's violating. "A problem well stated is a problem half solved," said Charles Kettering, and indeed zooming in on the problem is half the battle. If it's a spelling, grammatical, punctuation, or style error, the answer is often pretty obvious; you've known those rules for years and you have Merriam-Webster's 14th New Collegiate Dictionary and the Chicago Manual of AWESOME* to back you up. If it's a plot or character problem, you can measure it against Freytag's triangle and other editorial principles inculcated in us over decades, often even without our knowing it: Protagonists should be interesting people, generate energy, and take action; we should see a change in both the character and his or her circumstances from beginning to end; the child character must solve the problem; the climax needs to be the culmination of all that came before it, and so on and so forth.
But sometimes a sentence just sounds wrong. Why? Uh . . . Hrmm. Is the thought coming out of nowhere? Coming in at the wrong time in a paragraph? Are all the words used correctly? Would it be better in active as opposed to passive voice? Is it repeating a word or thought or phrase or sentence rhythm you read (heard, really) in the last two pages or so? Is it just your taste vs. the author's style? Is the sentence actually a violation of the author's style in some way and so you should push him on it? Sometimes it's not until I change the sentence or paragraph to what sounds right to me that I can figure out why something sounds wrong. Until that point, I just stare at it, which is one of the reasons my personal editorial process is extremely slow.
(You can run this whole process on illustrations too, by the way; you just then have to know your visual principles as well as your verbal and narrative ones.)
* This is an in-joke with one of my authors, who prefers to call it the Chicago Manual of Boring.
4. Weigh the problem. Is this worth bringing up with the author? Well, that depends on the nature and severity of the problem, the importance of the principle it's violating, the work that would be required to fix it, where you all are in the editorial process, how much the reader would be likely to notice the problem and care, the other things you're already asking the author to do in this round (and those things might be a higher priority for now, so you could pick this one up in a later draft if it's still an issue then), your understanding of the author's revision capabilities (can she do both small revisions and large-scale ones at the same time, or is it better to save the small ones for later with her? Can he fix plot problems but is utterly hopeless at deepening his characters?), the strength of your authority here (Does the author appreciate your comments or resent them? What if Chicago and Words into Type conflict?), what the author's vision of this book is and whether correcting this would serve that, your knowledge of your personal editorial irritants (because every editor has that one thing that drives them crazy and nobody else, which is then often not worth asking about). . . . Editors truly consider all of these things -- many of them subconsciously in about 2.5 seconds -- in choosing what to query with an author.
5. If it is of sufficient weight: Articulate the problem in a manner tailored to the author and manuscript. Having half solved the problem by stating it clearly for yourself, you now have to state it for the author in a manner that he can appreciate and which will inspire him to take action. Name the problem and the principle it's violating clearly and nonjudgmentally; it's not a personal failure of the author, it's a simple mistake in the manuscript, and mistakes can be corrected. Just as in disagreements in relationships, it's often useful to put things in terms of your own emotional reaction ("Because of [X factor in the manuscript], I felt [Y negative feeling]"), which can again be changed if X factor in the manuscript is changed. Remember that just as the author needs to show-not-tell the story to you, you have to show-not-tell the problems to her, and thus it's useful to back up your assertions with solid examples from the manuscript. Sometimes a series of questions is the best way to show that there's a problem, even if you fear you'll sound stupid; I sometimes call myself the Designated Dumb Lady within a ms. if I'm not getting what's going on, and that frees me up to ask the dumb but necessary questions. Suggest a strategy for a fix (or multiple options for a fix) if you think the author will be open to it and find it useful, but remember it's always the author's choice whether and how to fix it, not yours. (If the author is repeatedly making bad fix choices, from your point of view, then you may not be a good editorial match. Or you may just be too persnickety or egotistical; that's always a possibility worth staying aware of.)
Plot and character stuff usually belongs in an editorial letter; it's extremely useful to know which one is the author's greatest strength or primary interest, if one or the other, so you can couch your argument for making the change in terms of that strength, which might make her feel more excited and capable of doing it. Ditto for the usefulness of knowing what the author's goal for and/or vision of this manuscript is, whether to explore the idea of death or make a reader fall in love with the character or write a really breakneck adventure; you can then phrase your argument for this particular change in service of that (if it truly is; authors are smart and can see when you're going back to the same well too often, so you shouldn't overuse any of these strategies). With mechanical stuff, which you should be saving for the copyediting and proofreading stages anyway, you can usually just say something like, "Hey, Chicago says we should capitalize 'Princess' here--OK?", or the even briefer "Cap as per CMOS 6.24."** Paragraph- and sentence-level stuff is always basically the effort to explain why "an" vs. "the" is so very important (one is a new or random reference; the other refers to something we've already seen in the text) or the equivalent, or what you as a reader WANT to be feeling at this point in the text and why you aren't and how if we can just cut this sentence, please oh please, you will be.
If you are an editor who does multiple passes through a line-edit, like I do, then it's often wise to save your argument for a change for the second or third pass through, so you can reread your suggested change outside the heat of the moment and see whether it's really a problem or if you were just in a weird editorial mood. That happens.
** This reference number not verified in the Chicago Manual.
6. Make sure the author knows you're open to conversation to help them better understand the problem or brainstorm solutions.
7. Hope for a response that fixes or removes the problem in the next draft. If that doesn't happen, then repeat steps #1-6, perhaps making your argument in #5 from a different angle. If it's a small thing, or a thing that is mostly a matter of your taste vs. the author's taste, then consider just letting the issue go. But that depends on what you weighed in #3 (and also how careful you know the author is; some authors get distracted easily and might just have missed a query they'll gratefully address later).
8. Read the next line. If necessary: Repeat.