The Unfinished Search for Wave-Particle and Classical-Quantum Harmony

The main purpose of this paper is to review the progress that has taken place so far in the search for a single unifying principle that harmonizes (i) the wave and particle natures of matter and radiation, both at the quantum and the classical levels, on the one hand and (ii) the classical and quantum theories of matter and radiation on the other hand. In the author's opinion, the Koopman-von Neumann-Sudarshan (KvNS) Hilbert space theory based on complex wave functions underlying particle trajectories in classical phase space, is an important step forward in that direction. To appreciate the similarities and differences between classical and quantum wave functions, it is important first to review the famous paradoxes of quantum theory arising mainly from the dual character of matter and radiation and the mysterious nature of measurements in quantum theory. They have given rise to the suspicion that quantum mechanics is an incomplete theory and to multiple interpretations of quantum mechanics as well as no-go theorems ruling out certain types of hidden variable theories introduced to 'complete' quantum mechanics in some way. It is also important to point out that experimental verifications of the predictions of the double-prism experiment and the observation of average single photon trajectories in weak measurements appear to favour the spacetime picture of particles favoured by Einstein, de Broglie and Bohm over the complementarity approach favoured by Bohr. The KvN theory of classical mechanics provides a clear and beautiful harmony of classical waves and particles. Sudarshan has given an alternative but equivalent formulation that shows that classical mechanics can be regarded as a quantum theory with essentially hidden noncommuting variables. An extension of KvNS theory to classical electrodynamics provides a sound Hilbert space foundation to it and satisfactorily accounts for entanglement and Bell-CHSH-like violations already observed in classical polarization optics. An important new insight that has been obtained through these developments is that entanglement and Bell-like inequality violations are neither unique signatures of quantumness nor of non-locality—they are rather signatures of non-separability. Another new insight is the interpretation of the Wigner function as a KvNS wave function, i.e., a probability amplitude which need not be positive everywhere. This has important implications for simulating certain types of quantum information processes using classical polarization optics. Finally, Sudarshan's proposed solution to the measurement problem using KvNS theory for the measuring apparatus is sketched to show to what extent wave and particles can be harmonized in quantum theory.