• P-values are more informative than the reject-or-not result of a level α test. Beware of placing too much weight on traditional values of α, such as α = 0.05.
• Very small effects can be highly significant (small P), especially when a test is based on a large sample. A statistically significant effect need not be practically important. Plot the data to display the effect you are seeking, and use confidence intervals to estimate the actual values of parameters.
• On the other hand, lack of significance does not imply that H0 is true, especially when the test has a low probability of detecting an effect.
• Significance tests are not always valid. Faulty data collection, outliers in the data, and testing a hypothesis on the same data that suggested the hypothesis can invalidate a test. Many tests run at once will probably produce some significant results by chance alone, even if all the null hypotheses are true.