tl;dr—We don't know (via science, at least).
Letting alone ethanol, which triggers innumerable biochemical pathways in our bodies, hops itself may affect potency and sexual desire, in males at least, but there appears to be few clinical studies supporting this claim directly. Many articles online appear to have chained together several studies (or played news-telephone) to publish indirect conclusions, many of which oversimplify the effects of estrogen. For example, contrary to popular belief, it isn't testosterone (alone) which masculinizes the male brain and behavior, but rather its interaction with estrogen, produced from testosterone through a process known as aromatization.1 So it's not as simple as more-estrogen-less-sex-drive.
I'm certainly not qualified to deliver conclusions myself, but I can at least point to some of the important studies from which current (mis)conceptions have derived.
1: Wu, M. V. et al. Estrogen Masculinizes Neural Pathways and Sex-Specific Behaviors. 2009. [PDF]
Why some claim that hops decreases sex drive...
Hops contains phytoestrogens,
substances that promote estrogenic actions in mammals and structurally are similar to mammalian estrogen 17β-estradiol (E2) [...]
Ososki, A. L. and Kennelly, E. J. Phytoestrogens: a Review of the Present State of Research. 2003. [PDF]
a recurring suggestion has been that hops have a powerful estrogenic activity and that beer may also be estrogenic. [...] We have identified a potent phytoestrogen in hops, 8-prenylnaringenin, which has an activity greater than other established plant estrogens.
Milligan, S. R. et al. Identification of a potent phytoestrogen in hops (Humulus lupulus L.) and beer. 1999. [PDF]
However, it's worth noting that the same paper concludes,
[...] despite the high estrogenic activity of 8-prenylnarigenin, the total estrogenic activity of beer made using whole hops is still low [...] and no detrimental health effects due to "estrogens in beer" are to be expected.
Anyway, another study led by the same researcher delved into the mechanism of action:
8-Prenylnaringenin alone competed strongly with 17β-estradiol for binding to both the α- and β-estrogen receptors.
Milligan, S. R. et al. The endocrine activities of 8-prenylnargingenin and related hop (Humulus lupulus L.) flavonoids. 2000. [PDF]
(In case one is unfamiliar with how receptors work, receptors "catch" freestanding compounds in a medium such as the bloodstream, thereby reducing their effects. The phytoestrogen in hops appears to get itself "caught" by estrogen receptors, thereby blocking those receptors from catching estrogen it'd normally catch, thereby leaving higher levels of estrogen in the bloodstream.)
...but it's not that simple!
Again, from these findings alone, conclusions about male potency and sex drive can't be so easily drawn. Hormone interactions are sufficiently complex that researchers are still trying to untangle, isolate, and explain ever-more-specific biochemical pathways. Certainly the average Joe (or journalist) wouldn't be capable of appreciating the magnitude (or triviality) of the effects of biochemical pathways described in researchers' findings, much less draw definitive conclusions about physiological effects like "potency," "sex drive," and how it all affects, say, muscle-building. Not to mention all the other chemicals in play!
As a very example of the latter, one site [link] (which, unfortunately, ranks high among Google search results) is clearly biased, grasping for evidence to "prove" that beer has detrimental effects on testosterone activity. I'm not saying beer doesn't, but their article states that Xanthohumol, another compound found in hops, "blocks testosterone," when in fact the very paper it cites says (in its abstract nonetheless),
Although hops is commonly linked with phytoestrogenic effects, we identified XN [Xanthohumol] as a pure estrogen antogonist. Interestingly, XN may also reduce the generation of estrogens by inhibition of the enzymatic activity of aromatase, which converts testosterone to estrogen. Anti-estrogenic effects of XN [...] were confirmed in vivo in an uterotrophy assay with prepubertal rats.
Strathmann, J. et al. Xanthohumol from Hops Prevents Hormone-Dependent Tumourigenesis In Vitro and In Vivo. 2008. [PDF]
On top of all this, ethanol affects so many parts of the brain via so many chemical pathways—what's even the significance of 8-prenylnarigenin or Xanthohumol when stacked against ethanol? Do you know?—I sure don't—and probably no one does—else it'd have been stated by researchers themselves, without the "help" of attention-seeking bloggers and headline-hunting journalists.
I'm not saying these blogs, articles, and threads are necessarily wrong—they might be right, whether right-by-chance, or right-by-empiricism (observation). What I am saying is that they can't be right-by-science, as currently available research seems extremely domain-specific, incomplete in larger perspectives, and therefore not generalizable.
The study by Ososki and Kennelly also states
As potential endocrine disrupters, phytoestrogens may act as antiestrogens and harm the reproductive health of males (Sharpe and Skakkebaek, 1993; Santti et al., 1998). Reduced sperm quality, undescended testes and urogenital tract abnormalities were increased in the sons of mothers taking DES compared with those who did not take the miscarriage preventative drug [...]
This statement is initially misleading as it says "males" instead of "developing males"—a fact not made clear until the following sentence. From the studies it cites (emphasis mine),
We argue that the increasing incidence of reproductive abnormalities in the human male may be related to increased oestrogen exposure in utero, and identify mechanisms by which this exposure could occur.
Sharpe, R. M. and Skakkebaek, N. E. Are oestrogens involved in falling sperm counts and disorders of the male reproductive tract? 1993. [URL]
Exposure to diethylstilbestrol (DES) induces persistent structural and functional alterations in the developing reproductive tract of males.
Santti R. et al. Phytoestrogens: Potential Endocrine Disruptors in Males. [URL]
So, more potential sources of misunderstanding and misreporting.