what would be the best way to find out if a student's marks is increasing with the number of attempts he or she takes for a test? Please bear with me. I am quite a beginner in statistics.
I want to analyse what would be the best way to find out if a student's marks is increasing with the number of attempts he or she takes for a test?
So as to answer that if more practice means better performance?
for example for a student in the first attempt he or she scores less marks then as he or she keeps taking more and more attempts his or her performance is increasing/decreasing.
Say there is a data set of 40 students.
For all the students we have a data set of number of attempts by each student.
Now I have to analyse whether more number of attempts by a student means better performance than a student making lesser attempts
 A: 
So as to answer that if more practice means better performance?

This question talks about the relationship between practice and performance. Regression analysis can be used to answer the posed question. But, regression is not the only tool to get an idea of the relationship. Most commonly used Pearson correlation can also be used to get an idea about how do these random variables move together. A potential question that might come into the OP's mind is:

So, why did I point out regression and not correlation?

It doesn't matter which variable you call "independent" and which one "dependent" when calculating correlation. But, it really makes a difference in interpretation from a regression analysis. 
With regression you might have to think about the  directional relationship that would be there among the variables and model the potential cause as independent variable and the effect as dependent variable, if you want meaningful interpretation about the relationship among the variables.
(stolen from this link):
Correlation is almost always used when you measure both variables. It rarely is appropriate when one variable is something you experimentally manipulate. With linear regression, the X variable is usually something you experimentally manipulate (time, concentration...) and the Y variable is something you measure.
coming back to the question:
If you have a positive correlation among practice and performance then one might also claim that better performance means better practice, but essentially the relationship that you want to examine incorporates the "potential cause" as practice and performance as effect. So, correlation is not going to help you with your analysis.
For purposes of doing regression have a look at this thread. For a basic understanding of regression have a look at this answer of mine in the very popular thread.

Now I have to analyse whether more number of attempts by a student means better performance than a student making lesser attempts

To answer this you can perform a paired t-test for two different experimental conditions, i.e. you can compare the scores of the students from the $1^{st}$ test to $k^{th}$ test as long as the assumption is satisfied.
The assumption is : The differences $w_i = x_i−y_i$  where $x_i$ and $y_i$ are the performance scores of the $i^{th}$ student in the $1^{st}$ test and the $k^{th}$ test, between the paired samples are independent draws from a normal distribution $N(µ, σ^2)$, where µ and σ are unknown.
A: For me, two simple first analysis will calculate correlation between marks and number of attempts. If it is positive and high, it might by true (but correlation does not imply causation)
A second, more sophisticated approach will be using regression. Take into account that because number of attemps is a discrete one, you need to take care of that (see Multiple regression with categorical and numeric predictors and https://stackoverflow.com/questions/22192934/linear-model-lm-with-dependent-variable-being-a-factor-categorical-variable)
Best!
