Validity

Validity is whether a test actually measures the thing it claims to measure, rather than something else entirely.

This sounds obvious until you notice how often it's violated. A job interview that's supposed to predict who will perform well often ends up measuring who interviews confidently, which is a related but distinct thing. A personality quiz that claims to reveal your "true self" might really just be measuring how you want to be seen. Both can be reliable — consistent — without being valid.

Validity isn't one property you either have or don't; researchers break it into several related questions. Does the test correlate with other established measures of the same thing (convergent validity)? Does it not correlate with things it shouldn't (discriminant validity)? Does it predict outcomes it should logically predict, like job performance or relationship satisfaction (criterion validity)? A test's overall validity is really the accumulated evidence across many of these checks, not a single score.

This is also why face validity — whether a test looks like it's measuring what it claims — is the weakest form of evidence. Plenty of valid tests don't look impressive on the surface, and plenty of invalid ones, like horoscopes, look compelling anyway.