Jan Freeman has a good article explaining that adverbs (e.g. "Think different") aren't turning into adjectives as a result of losing their -ly suffix; many aren't even really losing their -ly endings. They're simply reverting to (or continuing) their prior -lyless life as "flat" (or uninflected) adverbs.
In explaining that adverbs come in many guises, Freeman gives a few examples: soon, indubitably, almost, down, Sunday. It's only here where I would disagree. As the Cambridge Grammar of the English Language explains, there are good reasons for saying that two of these words are not like the others; two of these words just don't belong. Instead down should be in the preposition camp, while Sunday should go with its fellow nouns.
The CGEL sets out a framework in which words generally belong to a given word class but may have a variety of functions in phrase and clause structure (an approach that is widely used in linguistics). Under such a framework, nouns do not become adjectives simply because they are modifying other nouns (e.g., world war, interest rates, subject area), nor do they become adverbs simply by virtue of the fact that they tell us when something happened (e.g., We have a soccer game Sunday at 2:00 [a common usage in North American English]). In the first case, we have a noun functioning as a modifier. In the second, it is functioning as a temporal adjunct. In other words, Sunday is never an adverb, even in the above example.
The argument for down as a preposition, never an adverb, is somewhat different. In English, words in most classes come in flavours that require a complement and those that don't. Transitive verbs take an object (internal complement), intransitives don't. Adjectives may license prepositional complements (e.g., He's interested in...), or not (e.g., He's tall). Among all the word classes, prepositions alone have traditionally been bumped into another class simply based on the complements that they allow. That is, prepositions licensing noun phrase complements (e.g., She stood before the door) are held to be the one true race of prepositions, while those licensing clausal complements (e.g., She stood before she walked) sublimate into "conjunctions" and the poor buggers with no complement at all (e.g., She had stood before) get assigned to the adverb ghetto.
This is quite unfair and there's no real justification for it; the definition of preposition that has it always followed by a noun is simply begging the question. Far from being the monogamists they've been protrayed as, prepositions are more like grammatical sluts, going out with everything from other prepositions (e.g., out from under...) and adjectives (e.g., on high, for free), to adverbs (e.g., until recently) and interrogative clauses (e.g., we can't agree on whether to have children or not).
In the end, regardless of how classily it conducts itself or who it's hanging with, down is a preposition is a preposition is a preposition.