Correct statistical test to answer: "did my intervention help my patients?" I am doing something to a set of $N$ patients (keeping this vague for generality).
In order to assess if my intervention/treatment is making any difference I did the following:
Before my intervention I measured something about the patients (call that measurement $X_i$ for patient $i$), and repeated the same measurement after my intervention ($Y_i$). So I have $N$ pairs $(X_i, Y_i)$
Let's say my measurements/test spits out a number from 0 to 5 for each patient. Assume I only know high numbers are good and low numbers are bad. (e.g. the difference between a 5 and a 4, might not be necessarily the same as the difference between the 2 and a 1). (I think this is what is called an ordinal measurement). Furthermore assume that when looking at the results of my test for the whole set of $N$ patients we observe that they don't look normally distributed.. (e.g. their most common result could be a 1 while the least common could be a 5)
What is the correct statistical test to see if we could say that my intervention is improving my patients score on those measurements?
I'm also interested in understanding how slight changes to my problem would affect the choice of a statistical test
Does this depend on $N$?
What if actually my measurements are a continuous variable that actually looks normally distributed?
I'm also curious to know if I had first divided the group into 2 groups and only did my intervention to one of the groups, (while for the other group I also measured twice but without any intervention), if this would have been better somehow, and in this case if the test would be different
Thank you
 A: Fix your experimental design first; then use an ordinal regression model
Firstly, before you get to the inference phase, you need to alter your experimental design.  Since you are trying to infer the causal effect of a medical treatment, you should use randomised control and treatment groups (e.g., using block randomisation) to allow you to infer the causal effect from a comparison of these groups.  Without a control group, you have no way of knowing if a statsitical change in the outcome occurred because of your treatment or because of some contemporaneous change in the patient cohort from another source.  Depending on the natue of the medical treatment, you should also consider a "blinding" protocol to prevent the patients from knowing which group they are in, and possibly even a "double blinding" protocol to also prevent the treating doctor from knowing which group each patient is in.  Randomisation and blinding are protocols used to sever the statistical connection between the treatment variable and any possible confounding variables in the analysis.
Once you have fixed your experimental design, you will then have data that can be used to infer a causal effect.  It sounds like you are using a five-point ordinal scale for your outcome variable, so you would model this using some kind of ordinal regression model.  Typically, you would form a regression model that includes the treatment variable, relevant demographic variables, and any other useful variables relating to the medical state of the patients.  (Make sure to exclude any mediator variables --- i.e., variables that are intermediate medical outcomes from the treatment to a change in the outcome.)  You would then use this model to infer the effect of the treatment variable on the ordinal outcome variable when other (non-mediating) variables are held constant.  In answer to your question, yes, the strength of the inference will depend on the number of data points in your analysis --- roughly speaking, the more patients you have in your sample the more accurate your inferences about the effect of the treatment.
If you will permit me to speculate a little, it appears from your question that you are probably a novice in the field of medical experimental design.  In almost all institutions that perform medical research (universities, hospitals, etc.) there are consulting statisticians available to give assistance on experimental design, modelling and inference.  Even if you are outside these institutions (e.g., if you are a GP in private practice) you will probably find that consulting statisticians in these institutions will be happy to help you with some basic advice.  I strongly recommend going to talk to a consulting statistician working in the medical/population health sector for assistance with planning and executing your medical experiment.  Medical experimentation generally also requires ethical clearance from relevant bodies and so you should ensure you have gone through an appropriate ethics process for your study prior to executing it.
A: This question is conflating two distinct issues, which both are at the core of statistics: 1) Is this difference just due to sampling error and 2) what does this difference mean?
You seem to be mostly focused on the first one, but the second one is the important one if your real question is "how can I help patients?"
In general, a statistical test is a procedure that is designed to tell you how confident you can be that a difference (e.g. the difference between two groups on some outcome, or the difference between a measurement before and after a treatment)
observed within a random *sample actually reflects an underlying difference in the population from which the sample is drawn, as opposed to being just due to "random chance." That is, a statistical test is a way to guard against sampling error that occurs when you try to use random samples to generalize about populations. And the answer almost always depends on "N" the number of people in your sample. The more people in your sample the less likely it is that a given difference you see is just due to sampling error. In your case a paired sample t test - which tests whether the averages of two measurements from the same sample are different or not, is probably fine. Even though the underlying measurement is ordinal, the average of those measurements is continuous.
However, none of what I just said tells you - by itself - whether the treatment actually helped patients. Whether a treatment works or not is a question of causal inference. And while statistical tests can tell you if a difference is real or not, they can't (by themselves) tell you anything about why that difference existed. To answer that question you have to think about research design. In your case, you are describing what's called a pre-post, no-control group design. You did a measurement of a single group, gave them a treatment, and then measured them again. The problem with this approach is that, even if the group got "significantly" better when you measured them the section time (according to whatever test you ran), that doesn't mean the treatment helped. It might be that they just got better on their own - and would have done just as well if you didn't give them the treatment! No statistical test can solve this problem. You need a different research design.
And indeed what you proposed at the end of your answer is one way to (maybe) solve this problem: "First divided the group into 2 groups and only did my intervention to one of the groups, (while for the other group I also measured twice but without any intervention)." This is what is called a difference-in-differences design. The assumption here (which might be wrong!) is that the chance you observe among the group that didn't get the treatment is what would have happened to the treatment group if they didn't get the treatment. So to estimate the effect of the treatment, you compare the change observed among the control group to the change observed among the treatment groups, the "difference between those differences" is the estimated effect of the treatment. Of course you still have to test if that difference is big enough that it's unlikely to be due to sampling error. In practice people often analyze "Diff-in-Diff" designs with regression models and interaction terms (which you should read up on, since these are critical to the question of causal inference), but the actual "test" that is used at the end of the process is often just a different form of the "t test" used above.
That probably wasn't a very satisfying answer, but your question really gets at the two most fundamental challenges in statistics. Picking the right statistical test for the job is usually not the hardest part of research. It often depends on the measurement level of the variable and a few other things, but you can always figure it out with some googling, and often it doesn't even matter so much if you choose the wrong one. But causal inference is the core of using statistics to actually figure stuff out in the world - and doing it well or poorly is what separates good research from bad.
