Word embeddings typically represent differ- ent meanings of a word in a single conflated vector. Empirical analysis of embeddings of ambiguous words is currently limited by the small size of manually annotated resources and by the fact that word senses are treated as unrelated individual concepts. We present a large dataset based on manual Wikipedia an- notations and word senses, where word senses from different words are related by semantic classes. This is the basis for novel diagnos- tic tests for an embedding’s content: we probe word embeddings for semantic classes and an- alyze the embedding space by classifying em- beddings into semantic classes. Our main find- ings are: (i) Information about a sense is gen- erally represented well in a single-vector em- bedding – if the sense is frequent. (ii) A clas- sifier can accurately predict whether a word is single-sense or multi-sense, based only on its embedding. (iii) Although rare senses are not well represented in single-vector embed- dings, this does not have negative impact on an NLP application whose performance depends on frequent senses.