The answer definitely depends on:
What are actually trying to use the $Q$ test for?
The common reason is: to be more or less confident about joint statistical significance of the null hypothesis of no autocorrelation up to lag $h$ (alternatively assuming that you have something close to a weak white noise) and to build a parsimonious model, having as little number of parameters as possible.
Usually time series data has natural seasonal pattern, so the practical rule-of-thumb would be to set $h$ to twice this value. Another one is the forecasting horizon, if you use the model for forecasting needs. Finally if you find some significant departures at latter lags try to think about the corrections (could this be due to some seasonal effects, or the data was not corrected for outliers).
Rather than using a single value for h, suppose that I do the Ljung-Box test for all h<50, and then pick the h which gives the minimum p value.
It's a joint significance test, so if the choice of $h$ is data-driven, then why should I care about some small (occasional?) departures at any lag less than $h$, supposing that it is much less than $n$ of course (the power of the test you mentioned). Seeking to find a simple yet relevant model I suggest the information criteria as described below.
My question concerns how to interpret the test if $p<0.05$ for some values of $h$ and not for other values.
So it will depend on how far from the present it happens. Disadvantages of far departures: more parameters to estimate, less degrees of freedom, worse predictive power of the model.
Try to estimate the model including the MA and\or AR parts at the lag where the departure occurs AND additionally look at one of information criteria (either AIC or BIC depending on the sample size) this would bring you more insights on what model is more parsimonious. Any out-of-sample prediction exercises are also welcome here.