You're right - it's pretty awkward. It's just about acceptable though - if rather mannered - in some formal contexts - possibly in public speaking. This sort of conditional is sometimes fossilized in idioms: "Had I but known" is a pretty common alternative to 'If I'd only known".
Here are some Google results:
Results 1 - 10 of about 9,250 English pages for "Had I but known".
Results 1 - 10 of about 28,800 English pages for "If I'd only known".
So, about a third of the worldwide hits. But in UK pages, the ratio flips from 1:3 to 3:1:
Results 1 - 10 of about 378 English pages for "If I'd only known" :.uk.
Results 1 - 10 of about 1,040 English pages for "Had I but known" :.uk.
b