How to make sure that the random sample is representative for the whole sample? I have 14k tweets and I want to code these tweets (categorize them based on their topics), but since it is difficult to do the coding for the whole dataset, I decided to take a sample from it.
What I am thinking about is to take a randomly selected 20% of the whole sample (although I am not sure why I decided 20%) and then do the coding just for this sample (20%). My question here is how to check if the random sample that I picked is representative?
 A: So long as you have no wish to incorporate covariate information into your sampling scheme (e.g., balancing tweets from males/females), the usual method is to take a simple random sample without replacement.  This can be implemented in R using the sample.int function.  In the code below I show you how to generate a simple random sample from $N$ population values.  For convenience, the sample is sorted into ascending order, so it is a list of numbers of the tweets to include in the sample.  (Remember to set your seed for reproducible randomisation.)
#Generate simple random sample of tweets
set.seed(1)
N <- 14000
p <- 0.2
n <- ceiling(p*N)
SAMPLE <- sort(sample.int(N, size = n, replace = FALSE))

#Show the sample
SAMPLE

   [1]     8    13    17    18    21    25    27    42    59    64  ...
  [24]   126   128   129   149   152   155   157   172   173   179  ...
  [47]   237   241   244   262   267   274   277   289   308   311  ...
  ...
  ...
  ...
[2761] 13775 13777 13779 13780 13784 13785 13787 13788 13796 13798  ...
[2784] 13879 13880 13886 13896 13908 13918 13923 13927 13942 13944  ...

If you are looking for a randomiser that gives a "representative" sample with respect to some variables of interest (e.g., men and women, etc.) then you can use block randomisation instead of simple-random-sampling.  Block randomisaton allows you to ensure that known variables in your data are distributed in a representative fashion across your sample.  It is a bit more complicated than the above coding but it can also be implemented in a reproducible way using scripted coding.
You should note that with any sampling method, it is possible to make post hoc checks of the distributions of known variables in the sampled and non-sampled parts.  However, rejection of a random sample based on post-hoc analysis is highly discouraged and can lead to serious problems in your analysis.
A: Here I prefer the technique of Systematic Sampling where one selects every kth individual from the population. Thus, from a list of n arrived tweets, every kth tweet is chosen to construct a sample set of 's' tweets, such that k*s is close to n.
Advantages:

*

*Simple statistically valid procedure


*Accurate


*Easier to implement and verify the correct tweets have been selected


*Unbiased and representative, even more likely so than a Simple Random Sampling scheme, in the current context as this also sorts by time of arrival, where the latter criteria is likely material, as it spreads the sample over the day. As such, it can, for example, likely isolate workers, largely inactive 9 AM to 5 PM day, versus non-workers including students active 3 PM - 8 PM (after school), and older adults active latter in the evening.
Thus, the application of simple, easy to implement, unbiased and representative Systematic Sampling here likely also results in a spread of the sample over important age demographics and income classes.
Note: How one arrives at the best sample size 's' is an important topic, best discussed separately.
[EDIT] An important point that is duly noted by this educational reference, to quote:

You don't have a complete list, so simple random sampling doesn't apply...

So, technically the employment of a simple random sampling scheme, to assess characteristics of the parent population, is valid when one has a complete list of the population over which to subsample. This is NOT the case with a continuous occurring series of generated tweets constituting a subset of the tweeting universe. So, inferences on the parent population and, in particular, the very question as to whether it is representative of the 'whole sample', implying the parent population, only arguably can be answered here by a simple random sampling scheme. However, the same source does affirm the validity of systematic sampling in such a context, to quote:

Since we don't have access to the complete list, just stand at a corner and pick every 10th* person walking by.


*Of course, choosing 10 here is just an example. It would depend on the number of students typically passing by that spot and what sample size was needed.

A: What you want is a sample that is representative in terms of the topics you are going to manually code.
First of all, you want to be sure that your coding procedure is not biased. This is really important because a representative sample is useless if your coding procedure is biased. Thus you need at least two independent coders to code the tweets (usually just a part of the tweets you are going to code), and a test to evaluate the coherence between the coding results of the independent coders (such as the Krippendorff’s alpha coefficient).
Having said that, in your case the universe is composed by 14,000 tweets and a random sample would avoid per definition systematic biases in the selection of tweets. However, you might consider a more systematic sampling to be sure that every day of the week and every hours of the day is properly represented. For instance, you could sample a certain number of tweets per hours, for every hours of the day, for all the days in your dataset. In media studies there is also a procedure consisting in created a constructed week, where the data for each day are sampled for the same day across many weeks. With regards to tweets, this method has been compared to simple random sampling finding that the latter performs better.
In general, you can find a lot of examples in literature based on media data and also twitter data. If you want to be really sure of the appropriateness of your sample strategy, you might consider a sort of cross-validation approach. Instead of pick up just a sample, you pick up two samples. Without forgetting to code the tweets with independent coders and verify the validity of the coding, you first code one sample and then the other sample, and finally compare the proportions of codes in the two samples. You could also use a statistical test to be sure that the code proportions in the samples do not differ too much. However, a so detailed approach could be unusual. You should take into account the best practice in your field.
You might also want to try some supervised classification methods that seems to work fine also with a limited quantity of manually coded data.
