What exactly is building a statistical model?
These days as I am applying for research jobs or consulting jobs, the term "building a model" or "modelling" often comes up. The term sounds cool, but what are they referring to exactly? How do you build your model?
I looked up predictive modelling, which includes k-nn and logistic regression.
I'll take a crack at this although I'm not a statistician by any means but land up doing a lot of 'modeling' - statistical and non-statistical.
First let's start with the basics:
What IS a model exactly?
A model is a representation of reality albeit highly simplified. Think of a wax/wood 'model' for a house. You can touch/feel/smell it. Now a mathematical model is a representation of reality using numbers.
What is this 'reality' I hear you ask? Okay. So think of this simple situation: The governor of your state implements a policy saying that the price of a pack of cigarettes would now cost $100 for the next year. The 'aim' is to deter the people from purchasing cigarettes thereby decreasing smoking thereby making the smokers healthier (because they'd quit).
After 1 year the governor asks you - was this a success? How can you say that? Well you capture data like number of packets sold/day or per year, survey responses, any measurable data you can get your hands on that is relevant to the problem. You've just begun to 'model' the problem. Now you want to analyze what this 'model' says. That's where statistical modeling comes in handy. You could run a simple correlation/scatter plot to see what the model 'looks like'. You could get fancy to determine causality i.e., if increasing price did lead to decrease in smoking or were there other confounding factors at play (i.e., maybe it's something else altogether and your model missed it perhaps?).
Now, constructing this model is done by a 'set of rules' (more like guidelines) i.e., what is/isn't legal or what does/doesn't make sense. You should know what you are doing and how to interpret the results of this model. Building/Executing/Interpreting this model requires basic knowledge of statistics. In the example above you need to know about correlation/scatter plots, regression (uni and multivariate) and other stuff. I suggest reading the absolute fun/informative read on understanding statistics intuitively: What is a p-value anyway It's a humorous intro to statistics and will teach you 'modeling' along the way from simple to advanced (i.e., linear regression). Then you can go on and read other stuff.
So, remember a model is a representation of reality and that "All models are wrong but some are more useful than others". A model is a simplified representation of reality and you can't possibly consider everything but you must know what to and what not to consider to have a good model that can give you meaningful results.
It doesn't stop here. You can create models to simulate reality too! That is how a bunch of numbers will change over time (say). These numbers map to some meaningful interpretation in your domain. You can also create these models to mine your data to see how the various measures relate to each other (the application of statistics here maybe questionable, but don't worry for now). Example: You look at grocery sales for a store per month and realize that whenever beer is bought so is a pack of diapers (you build a model that runs through the data set and shows you this association). It may be weird but it may imply that mostly fathers buy this over the weekend when baby sitting their kids? Put diapers near beers and you may increase your sales! Aaah! Modeling :)
These are just examples and by no means a reference for professional work. You basically build models to understand/estimate how reality will/did function and to take better decisions based on the outputs. Statistics or not, you've probably doing modeling all your life without realizing it. Best of luck :)
Building a statistical model involves constructing a mathematical description of some real-world phenomena that accounts for the uncertainty and/or randomness involved in that system. Depending on the field of application, this could range from something as simple as linear regression, or basic hypothesis testing, through complicated multivariate factor analysis or data mining.
Modeling to me involves specifying a probabilistic framework for observed data with estimable parameters that can be used to discern valuable differences in observable data when they exist. This is called power. Probabilistic models can be used for either prediction or inference. They can be used to calibrate machinery, to demonstrate deficiency in return on investment, to forecast weather or stocks, or simplify medical decision making.
A model does not necessarily need to be built. In an isolated experiment, one can use a non-parametric modeling approach, such as the t-test to determine whether there is a significant difference in means between two groups. However, for many forecasting purposes, models can be built so as to detect changes in time. For instance, transition based Markov models can be used to predict up and down swings in market value for investments, but to what extent can a "dip" be considered worse than expected? Using historical evidence and observed predictors, one can build a sophisticated model to calibrate whether observed dips are significantly different from those which have historically been sustained. Using tools like control charts, cumulative incidence charts, survival curves, and other "time based" charts, it's possible to examine the difference between observed and expected events according to a model based simulation and call in judgement when necessary.
Alternately, some models are "built" by having the flexibility to adapt as data grow. Twitter's detection of trending and Netflix's recommendation system are prime examples of such models. They have a general specification (Bayesian Model Averaging, for the latter) that allows a flexible model to accommodate historical shifts and trends and recalibrate to maintain best prediction, such as the introduction of high impact films, a large uptake of new users, or a dramatic shift in film preference due to seasonality.
Some of the data mining approaches are introduced because they are highly adept at achieving certain types of prediction approaches (again, the issue of obtaining "expected" trends or values in data). K-NN is a way of incorporating high dimensional data and inferring whether subjects can receive reliable predictions simply due to proximity (whether from age, musical taste, sexual history, or some other measurable trait). Logistic regression on the other hand can obtain a binary classifier, but is much more commonly used to infer about the association between a binary outcome and one or more exposures and conditions through a parameter called the odds ratio. Because of limit theorems and its relationship to the generalized linear models, odds ratios are highly regular parameters that have a "highly conserved" type I error (i.e. the p-value means what you think it means).
Modelling is the process of identifying a suitable model.
Frequently a modeller will have a good idea of important variables, and perhaps even have a theoretical basis for a particular model. They will also know some facts about the response and the general kind of relationships with the predictors, but may still not be certain that their general idea of a model is completely adequate - even with an excellent theoretical idea of how the mean should work, they might not, for example, be confident that the variance isn't related to the mean, or they might suspect some serial dependence could be possible.
So there may be a cycle of several stages of model identification that makes reference to (at least some of) the data. The alternative is to regularly risk having quite unsuitable models.
(Of course, if they're being responsible, they must take account of how using data in this way impacts their inferences.)
The actual process varies somewhat from area to area and from person to person, but it's possible to find some people explicitly listing steps in their process (e.g. Box and Jenkins outline one such approach in their book on time series). Ideas about how to do model identification alter over time.
I don't think there's a common definition of what constitutes a statistical model. From my experience in the industry it seems to be a synonym to what in econometrics is called a reduced form model. I'll explain.
Suppose, that in your field there are established relationships or "laws," e.g. in Physics this would be $F=m\frac {d^2x}{dt^2}$ stating that force is proportional to the acceleration (aka "2nd law of mechanics"). So, knowing this law you could build a mathematical model of a cannon ball trajectory.
This model will have what Physicists call "constants" or "coeffiecients", e.g. an air density at a given temperature and elevation. You'll have to find out what are these coefficients experimentally. In our case we'll have ask the artillery to fire the cannons under many different, tightly controlled conditions, such as angles, temperature etc.
We collect all the data, and fit the model using statistical techniques. It could be as simple as linear regression or averages. Once got all the coefficients, we now run our mathematical model to produce the firing tables. This is neatly described in the unclassified document here, called "THE PRODUCTION OF FIRING TABLES FOR CANNON ARTILLERY."
What I just described is not a statistical model. Yes, it does use statistics, but this model uses establishes laws of Physics, which are the essence of the model. Here, statistics is a mere tool to determine the values of a few important parameters. The dynamics of the system are described and pre-determined by the field.
Suppose, that we did not know or did not care for the laws of Physics, and simply tried to establish the relationships between cannon flying distance and the parameters such as firing angle and temperature using a "statistical model." We'd create a big data set with a bunch of candidate variables, or features, and transformations of variables, maybe polynomial series of temperature etc. Then we'd run a regression of sorts, and identified coefficients. These coefficients would not necessarily have established interpretations in the field. We'd call them sensitivities to square of temperature etc. This model may actually be quite good at predicting the end points of cannon balls, because the underlying process is quite stable.
The simplest possible approximation, depiction and abstraction of real world situation, through the nearest possible counterpart in the theoretical world, is called a model.
The data is turned into random variable, and which has an associated theory to find its various estimates, associations and hypothesis.
