Selecting variables based on a p-value significance is generally a bad idea. You can see that by looking at the formulas of the test statistic in a simple linear regression setting (for illustrative purposes):
the $t$ statistic for testing wether $\beta_i$ is 0:
$$ \frac{\hat{\beta_i}}{\hat{\textrm{se}}(\hat{\beta_i})}$$
And, $$\hat{\textrm{se}}(\hat{\beta_i}) = \sqrt{\frac{\hat{\sigma^2}}{n}(\frac{1}{n}\mathbf{x}^T\mathbf{x})^{-1}}$$
Now, given that in a single dataset $n$ and $\sigma$ are the same, the things that will make the test statistic larger, and as a result more significant are:
- Magnitude of $|\beta_i|$ (the larger the more significant)
- The sample variance of $X_i$ (when larger it makes the standard error smaller thus increasing the test statistic).
This is assuming that there is no coliniarity between predictors. If there is, the denominator of the test statistic formula will get an extra addition and when the correlation is large, all else being equal, this will tend to decrease the test statistic and thus the significance.
So we can see that the most significant variables are not necessarily the most "important" ones, they may be just measured more precisely.
Furthermore, you can see that significance also depends on the design of your predictor matrix , so you can have two people with slightly different designs for $\mathbf{x}$ dealing with the same dataset that will get different sets of significant coefficients! This is because adding or removing variables affects the significance through both changing $\beta_i$ and the standard error.
One additional point is that if you could increase the number of observations in your dataset (everything else staying the same), this thing alone can increase the significance of your variables. If $\beta_i$ is not exactly zero, the t-statistic will tend to grow towards infinity as the number of observations increases.
You can state several models that you think, based on theory and other domain-specific considerations, will best describe your dependent variable and then use methods like the AIC to select the best model among them. Note that if you do that you should first split your data into two parts. One of them you are going to use to select the best model specification, and the other will be used to estimate the coefficients. That is the proper way.