Does effect size mean anything when when the t-test is not significant? I have two sets of data and I would like to have a metric that says anything about HOW different the sets are. I use Cohen's d measure. Let's say d=0.5 but p=0.32. What does it mean?
 A: My answer assumes unpaired two-sample t-tests (assuming equal variance). The settings are as follows;


*

*The sample size, mean, and standard deviation of group A are $n_a$, $m_a$, $sd_a$, and

*The sample size, mean, and standard deviation of group B are  $n_b$, $m_b$, $sd_b$, respectively.


Where the standard deviation is calculated using the n-1 method.
Given these data, you can calculate t and d and p values.
【Visual understanding using Excel】
You can feel the relationship between p-value and d-value by entering various $n_a$, $m_a$, $sd_a$, $n_b$, $m_b$, $sd_b$ into the following Excel file;
Download EffectSizeCalculator.xls from 
here
The following example will give a visual answer to your question;

Here, the p-value is displayed in column K, and the d-value is displayed in column N. 
【Mathematical explanation; the relationship between the t and d values】 
Use only elementary algebra to define t and d values. If you compare the definitions of t-value and d-value, you will understand why and how.
[Preparation]
Now, define $degf$ and $sd_p$ as follows.These help to define d and t.
$$degf := n_a + n_b -2 \tag{1}$$
$$sd_p:=\sqrt{\frac{({n_a}-1)*{sd_a} + ({n_b}-1)*{sd_b}}{degf}} \tag{2}$$
[Cohen's d] 
Many people use "Cohen's d," as defined in the following equation in this problem.  Therefore, we adopt the following definition.
$$d:=\frac{|m_a - m_b|}{sd_p} \tag{3}$$
Unfortunately, here are confusing on the definition of Cohen's d; different effect-sizes are called by the same name, "Cohen's d." 
See this link for more information about this unfortunate confusion.
[Student's t] 
On the other hand, there is essentially only one definition of appropriate t-value for this problem. A typical textbook would have the following definition:
$$t := \frac{|{m_a} - {m_b}|}{{sd_p} \cdot \sqrt{\frac{1}{n_a}+\frac{1}{n_b}}} \tag{4}$$
[Relation between the t and d values] 
The t-value of formula (4) can be transformed as follows;
$$t = \frac{|{m_a} - {m_b}|}{sd_p}\sqrt{\frac{{n_a}*{n_b}}{{n_a}+{n_b}}}=d\sqrt{N_{tot}} \tag{5}$$
Here, 
$${N_{tot}} :=\frac{{n_a}*{n_b}}{{n_a}+{n_b }} \tag{6}$$
Thus, the t-value was separated into terms of "effect size" and "terms that depend only on sample size."
【Conclusion】
Look at Equation 5, and all of your questions will be clear, at least for this problem. Again, 
$$t = d\sqrt{N_{tot}} . \tag{5'}$$
As you know, the larger the t value, the smaller the p-value. Therefore, you can increase the p-value by increasing the effect size or by increasing $N_{tot} $.
Conversely, if ${N_{tot}}$ becomes smaller, t becomes smaller, and the p-value becomes larger.
Note:
The mathematical explanation for converting a t value to a p value is a bit difficult. But it's easy with Excel.
The formula for converting the t value to the p value is described in the above Excel file. If there is $degf$ and the t value, p-value can be obtained by TDIST function of the Excel.
P.S. I'm not very good at English, so I'm sorry if I have some impolite or unclear expressions. I welcome any corrections and English review. (You can edit my question and description to improve them).
