So von Neumann invented the Singularity after all! Toward the bottom of the Life obit (25 February 1957, pg89-104):

Machines creating new machines

After the last visitor had departed Von Neumann would retire to his second-floor study to work on the paper which he knew would be his last contribution to science. It was an attempt to formulate a concept shedding new light on the workings of the human brain. He believed that if such a concept could be stated with certainty, it would also be applicable to electronic computers and would permit man to make a major step forward in using these “automata.” In principle, he reasoned, there was no reason why some day a machine might not be built which not only could perform most of the functions of the human brain but could actually reproduce itself, i.e., create more supermachines like it. He proposed to present this paper at Yale, where he had been invited to give the 1956 Silliman Lectures.

This is interesting. As described, it can't be a von Neumann machine because it's almost a decade later, the 'paper' could only have been the short work which was published as The Computer and the Brain which doesn't cover replicators but how the brain works & how a Turing machine can emulate it & how much computation is necessary*, and it sounds all wrong for a simple self-replicator anyway. There's a quote from von Neumann about a 'singularity in the future of humanity', Vinge in 1993 says:

...In the 1950s very few saw it: Stan Ulam [1958] paraphrased John von Neumann as saying:

One conversation centered on the ever-accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.

Von Neumann even uses the term singularity, though it appears he is thinking of normal progress, not the creation of superhuman intellect. (For me, the superhumanity is the essence of the Singularity. Without that we would get a glut of technical riches, never properly absorbed.)

* Incidentally, Kurzweil wrote the preface to the third edition of The Computer and the Brain, and specifically says that von Neumann saw the Singularity coming and that was the motivation for the book and notes the Ulam quote use of 'singularity'! I've skimmed the book, since it's so short; unfortunately, it is radically incomplete, just sort of stopping in the middle of a digression about the arbitrariness of human languages and the parallelism of thought, and lacks any kind of conclusion or speculation section, and the previous material doesn't provide any smoking gun restating the Life quote.

As Vinge points out, as worded it's not clear if he's thinking of a Good 1959/1965-like argument, about intelligent machines making more intelligent machines ie the Singularity-with-capital-S. It could just as easily refer to something like he discussed in his 1955 essay "Can We Survive Technology?" about the need for a world-government due to nukes, which are 'technology' as well, and the timing ambiguous as both nukes & computing were major focuses at the time. Ulam provides no particular context (other than, perhaps, previously noting von Neumann's intense interest in history & long-term trends, epitomized by a near-photographic memory of Gibbon's Decline and Fall of the Roman Empire) and it's in a rather miscellaneous section. But this Life obit quote, 8 years before Good, makes that interpretation untenable - he's clearly thinking about brain-like computers making 'more supermachines', and is discussing the concept with enough people that they'll tell it to a journalist writing an obit.

lukeprog points out that in his Singularity history anthology, von Neumann in 1948 & 1949 clearly believes automatons could construct more powerful automatons than themselves, which shows he did believe in the missing piece at that point, so it seems peculiar to suggest that he had abandoned or forgotten it despite all the successes of computers & his research program by 1957:

...“complication" on its lower levels is probably degenerative, that is, that every automaton that can produce other automata will only be able to produce less complicated ones. There is, however, a certain minimum level where this degenerative characteristic ceases to be universal. At this point automata which can reproduce themselves, or even construct higher entities, become possible.

1949 ("Theory of self-reproducing automata", pg80):

There is thus this completely decisive property of complexity, that there exists a critical size below which the process of synthesis is degenerative, but above which the phenomenon of synthesis, if properly arranged, can become explosive, in other words, where syntheses of automata can proceed in such a manner that each automaton will produce other automata which are more complex and of higher potentialities than itself.

You can try to argue that maybe Good still has priority because Good explicitly wrote about a recursive loop of improvements and the Life summary, strictly read, has a missing piece of recursive improvement (merely more supermachines 'like it'), but given von Neumann's attitude on 'ever-accelerating progress' to Ulam and elsewhere and the breakdown of 'human' affairs, confident extrapolations in The Computer and the Brain, belief that automation will be a big part of boosting science/tech ("Can We Survive Technology?" etc), the remarkable echo of von Neumann's 'explosive' in Good's 'intelligence explosion' (on top of Vinge's echo of 'singularity'), that seems like a very big stretch to me.

I wonder if somewhere in his unpublished papers or letters there is a more detailed description of what he thought? It seems improbable that some weekly magazine journalist would be the only person to hear about his speculations - posthumously at that. It wouldn't be surprising if there's something there but no one has noticed: 'Singularity' as an idea or phrase only began taking off around 2000 (I noticed this strongly doing some searches in LexisNexis to see if I could find any other old media articles about von Neumann mentioning either "singularity" or "supermachines"), and of course, I only just noticed this Life obituary while reading it for amusement value.