First of all, I wouldn’t be a fan of being called a “B List Author” either. It seems like an outrageous thing for a publisher to tell anyone. But keep in mind, it’s just one obviously jaded editor on one day.
Remember that publishers only make money when they purchase manuscripts that they think they can sell. Period. They aren’t going to take on a project they personally like but they don’t think can make money. So, their feedback is helpful for you.
That said, I would follow up with a letter to the editor thanking them for the time and input and asking what would have made the book more marketable? Tell them you are looking to rework the manuscript and would appreciate any feedback they can offer. Some won’t reply, but some likely will. Take their comments to heart and as a gift to you to help improve your story.
Years ago, I had a book proposal that was rejected by about 20 publishers. I reworked it, taking their comments and reasons for rejection into account to make the book better, and then finally sold it.
As far as “hanging up your hat,” I really can’t say. However, remember that if the book isn’t marketable to a traditional publisher who has a marketing team that knows your market, it won’t likely be any more marketable if it’s self-published.
And, it’ll only be more difficult to sell to consumers since (1) you don’t have the marketing experience that a publisher has, (2) you don’t have the contacts they have in bookstores, (3) you may not know the market as well as they do.
For all authors—traditionally published or self-published—the key is producing stories that entertain readers, and then connecting with those readers so that they discover you. If you’re convinced you have done the first and that you can do the second all on your own, then perhaps self-publishing will work. Otherwise, rework your story so that it’s marketable, or move on to another project or career.