This depends on the goal of clustering.
t-SNE as well as various clustering methods (like hierarchical clustering) can work on distance matrices. And it's your job to construct a distance measure that captures what you wish to achieve. A few examples below.
If you want to group the students based on their ability to get good grades the simplest solution would be to ignore the missing classes and simply compare the average grades they achieved. So the distance between two students might simply be the difference of their average grades.
A good additional idea here would be to weight each class according to how hard it was (based for example on the average grades students get in that class)
If students are free to choose their classes you might want to group them by their interests. In this case the interests would be reflected by the type of classes they actually chose. In this scenario you would ignore all of their marks and simply code the missing classes as 0 and attended classes as 1. Then compute a distance measure between students based on how many classes they overlap at.
Another possible scenario is if you wish to group students based on their ability on various subjects. Here you would have to incorporate both the grades as well as the selection of subjects. A simple (read dumb) solution would be to replace all missing entries for each student with their average ability. Or with the average ability for every student on that subject.
The idea is that when the student didn't take the class - your best guess that he is average at that class.
But you could possibly construct better metrics after some reflection. Just need to think about what the similarity should be between students when none of their classes overlap.
t-SNE and clustering
The examples above show some ways how one could construct a distance matrix between students. After that you can use that matrix for both t-SNE and clustering.