Jon,
The most serious treatment of your subject matter would take you (and the rest of us who follow along) into two areas that 99.9 percent of Protestant evangelicals have no training at all, not even the passive training that comes with reading quality work in either area. These subjects are musical aesthetics and literary aesthetics. Sheesh, aesthetics today is about as alien to modern minds as multi-dimensional string theory. In its place is usually the notion that “I can’t explain why it’s good, but I know what I like!”
Aesthetics is the way one explains why good music is good, why good poetry is good, why good architecture is good, and so on. It’s the general theory of goodness and badness as those concepts apply to specific subject areas (food, drink, preaching, garden design, etc. etc.).
I had a good introductory training in music performance as a boy, less so in music theory, almost none in musical aesthetics. Similarly, I had good, maybe very good, training in writing and appreciation of literature as a boy, less so in formal literary aesthetics, almost no training in poetry.
I think the fullest and most satisfactory answer to your question is going to be in musical and poetic aesthetics.
Absent that, what may you (or the rest of us too!) look to for examples of good, non-gay songs - taking as our basic definition that song is the union of music and words, or words sung to music/musical-accompaniment?
These suggestions are, obviously, not exhaustive. I look forward to what others will suggest. But, be sure to check out:
1. The OT Psalms: This collection of songs is the polar opposite of Fanny Crosby hymns (or, worse, the treacly sentimental, effeminate hymnody that our greatgrandmothers reared our grandmothers on and our grandfathers meekly imagined themselves to be spiritual whenever deigning to sing them).
My parish is not one of those psalms-only organizations, but we do sing the English-text (NKJV) Psalms every Sunday in our worship, along with three or four canticles (ancient Christian hymns lifted directly from Scripture (Song of Zecheriah, Nunc Dimmittis, Song of Moses, etc.) or built out of Scripture/doctrinal texts (Te Deum, St. Ambrose’s Fourth Century hymn).
Routinely singing these hymns from Scripture or derived from Scripture does a lot over time to develop a sense for what authentically Christian singing is like as to its themes, imagery, and poetical toolbox. By this touchstone, those truly wretched things in our hymnbook (thankfully these are few in the 1940 Episcopal hymnal) stand out in all their poetical and musical poverty.
2. Tony Esolen’s “Illuminations” in Touchstone. This is Esolen’s regular column in Touchstone and you may need to be a subscriber to access these online. Esolen is a masterful scholar of poetics (recently published a new translation of Dante’s Divine Comedy). Esolen is Roman, but he treats favorably Protestant hymnody (when it’s good), and any hymnody (Roman, Protestant, modern) very unfavorably when it is bad, explaining in often hilarious ways just why it is so very, very bad.
And, he’s no mean poet himself! Check out Esolen’s satirical Dies Irae lampooning (mostly Roman) modern hymnody for all the same vices which infect modern Protestant hymnody. It’s a riot!