References: Reputable editors will give you a list of the projects they've worked on. Look at these critically: Has the work been published anywhere? self-published or chosen for publication by a third party? Read a sample of the books on Amazon (or elsewhere online). Is this the general level of quality you want for your own work? (Personal references are great, but we all know from our own experience that we don't offer references who won't speak well of us. Don't rely on personal references alone.)
Genre: Not every editor is comfortable editing every genre. A good editor can tell you with what genre(s) he or she is most fluent. For example, if you're writing Y/A, an editor who reads and edits mostly horror may not be the best fit for your work. Genre (in the sense of style and target audience) is important, even if you consider your work outside of the usual genre descriptions.
Samples: Reputable editors will do a sample edit of a few pages of your work, either free or for a nominal fee, so that you can see whether their style and approach is compatible with yours. Look at the sample and ask yourself whether the questions and comments are helpful to you.
Know what you need: Developmental editors look at the big picture of your work and help you shape the story, refine the characters, and improve the plot. They can also give you advice about how your work fits genre trends, which can be important when it comes to correctly marketing your work. Copy editors (many of whom also do proofreading) read for errors in grammar, spelling, and word choice. Some may also give you advice about plotting and language, but this is not their primary specialty. Keep in mind, the same editor can sometimes do both types of editing, but never at the same time. Hire the right kind of editor for your work. Confident in your ability to string words together, or have a fantastic friend or beta reader who spots all the typos? Then a developmental editor may be what you need at the beginning of your project, to make sure the structure of your work is sound. Have a number of books already published and feel confident in your ability to wrangle a plot? Then what you need may be a copy editor at the end of your project, to clean up any stray threads.
You get what you pay for: Cheaper isn't always better, in anything, and an editor who promises a lot for very little may not have the level of experience you need. Not all editors are equal, and just having paid for an editor doesn't mean you have a better book. Check out galleycat's guidelines for freelance editors' rates for a general idea of what professionals charge. However, different people work at different speeds and price themselves accordingly. Also, there are many professional editors who will give you a special deal or barter their services for something you have to offer them. You don't have to feel that you're priced out of editing services.
Bottom line: caveat emptor It's your money, and you should get the kind of editing that will make your work better. Look at what else an editor has worked on, ask about genre specialties, get a sample of your work edited, and think about what kind of editing you need. And, remember: not every great editor will be right for your work; saying they're an editor doesn't make a person one; and just being likeable online isn't a good guide for finding a good editor. Do your research first, and find someone fantastic.
Most of all, keep writing!