However, both the training and validation accuracy are increasing, am I overfitting ?
I'm assuming you are performing nested cross validation often called m-by-n fold validation where m is say 5 and n is 10.
There are two important points here that I want you to note before answering your question:
Your validation set is the one used for parameter tuning (inner loop in nested CV). Your test set (actual unseen data) is the one that is not used/seen in anyway during training/tuning your model (here the outer loop of CV). So you test your best model m times and get a m-component vector of performance (each value is from one round of test) and then say average it to get the final performance.
You need to make sure you have enough samples in your validation and test set. If you don't have enough samples in your validation, you will get optimistic performance results and that would affect your model selection (tuning). If you have small number of test samples, that's bad, because then you are conclusion is not that reliable.
Okay, now that you know these, the figure looks fine because all it is showing is that the more samples the model see, the better its performance is on unseen data. I'm assuming validation here is the actual unseen data (test set).
But there is a catch, on x-axis you are showing the training set size, what is your validation set size? And again is this the test set you call validation or the parameter tuning set (the actual validation set)? That's very important. Is it getting smaller and smaller? I assume so, then yeah, that's not ideal, particularly if you have an unbalanced dataset.
So be very careful with your performance evaluation metric. Read about different methods available and choose one that makes most sense, I'm afraid accuracy is often not the best choice.