False discovery rate methods don't get around the conundrum, as you state it. In general, methods for family-wise error rate control are aimed at controlling the probability of any type I error occurring across a family of hypotheses, while methods for false discovery rate control are instead designed to control the proportion of positive results that are type I errors across a family of hypotheses. Both methods rely on some a priori definition of what that "family" of hypotheses is, but have somewhat different theoretical approaches for conceptualizing "error control".
EDIT: Alexis comments below that: "False discover rate methods do not require a definition of family, and they scale rejection probabilities to different numbers of comparisons so that the FDR rate is conserved regardless of number of comparisons". It is true that FDR methods are invariant with respect to the number of comparisons, but I don't think it is strictly true that they do not require a definition of a family, but rather that they relax the stringency of the assumptions regarding that family, analagous to some of the methods I discuss later. Regardless, FDR methods conceptualize error differently than FWER methods, but still require careful consideration of what the "set" of hypotheses are.
In terms of what constitutes a family, this isn't clear cut. The most general definition would be the one HEITZ provides, which is "all of the tests that are performed." However, this definition isn't particularly clear, either; does it refer only to the set of a priori hypotheses related to the outcome of the study? Or does it refer to all hypothesis tests conducted at any stage of the study? Do studies analyzing data from publicly-available datasets have to adjust their findings for ever hypothesis tested on that dataset from other studies? The scope of these questions veer quickly out of the realm of statistical theory and into the realm of philosophy of science. This is one of the many motivations for some paradigms, for example Bayesian inference, being favored over traditional frequentist approaches, since they avoid the question entirely by not focusing on the null hypothesis testing framework (not that Bayesians aren't forced to answer such questions in different ways, for example in the assumptions made about priors, but the point is that they don't need to worry about family-wise error in the frequentist sense).
In addition, there are more rigorous and flexible ways of handling type I error than simple FWER or FDR methods like the ones you mention. For example, generalized procedures based on the closed testing principle (see ) or one of a variety of "gatekeeping" and graphical procedures (see ). Note that these do not solve the problem of how you define a "family" of hypotheses, but address the problem in a different way by allowing for a more nuanced consideration of how different combinations of null hypothesis tests may relate to one another.
: Kevin S.S. Henning & Peter H. Westfall. "Closed Testing in Pharmaceutical Research: Historical and Recent Developments." Stat Biopharm Res. 2015; 7(2): 126-147. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4564263/
: Mohamed Alosh, Frank Bretz, & Mohammad Huque. "Advanced multiplicity adjustment methods in clinical trials." Statistics in Medicine. 2014; 33(4): 693-713. https://onlinelibrary.wiley.com/doi/abs/10.1002/sim.5974