I was going through the definition of parametric and non-parametric models. So the parametric are the ones which have a fixed number of parameters that you are trying to learn and this number is invariant of the data. Non-parametric models also do have parameters but the number of such parameters can be potentially infinite and grows with the dataset. Is my understanding correct?
Assuming it's correct, I was thinking of decision trees. Especially ID3 algorithm for making decision trees. DT is in general considered non-parametric. And one can think that it's true since, for a simpler dataset, probably fewer nodes (and thus "parameters") may do the job. Hence, the number of parameters are dataset dependent. So far so good. But if one thinks about it, algorithms such as ID3 use one attribute only once. Thus, at each level in the tree, they are picking up one feature and never using it again. Thus, in this way, the maximum depth of the tree is k, where k is the number of features. Thus, one can say that there is an upper bound on the number of parameters for a DT for a given feature set. Then the question arises since there's a bound on the number of parameters, why are DT (especially ID3 algorithm), not considered a parametric model?
Sure, then the number of parameters is growing with the dataset. But it can only grow to a finite extent. Is the distinction between parametric vs non-parametric that ambiguous or am I missing a point?