Abstract
In classical Whitham modulation theory, the transition of the dispersionless Whitham equations from hyperbolic to elliptic is associated with a pair of nonzero purely imaginary eigenvalues coalescing and becoming a complex quartet, suggesting that a Krein signature is operational. However, there is no natural symplectic structure. Instead, we find that the operational signature is the “sign characteristic” of real eigenvalues of Hermitian matrix pencils. Its role in classical Whitham single‐phase theory is elaborated for illustration. However, the main setting where the sign characteristic becomes important is in multiphase modulation. It is shown that a necessary condition for two coalescing characteristics to become unstable (the generalization of the hyperbolic to elliptic transition) is that the characteristics have opposite sign characteristic. For example the theory is applied to multiphase modulation of the two‐phase traveling wave solutions of coupled nonlinear Schrödinger equation.