Abstract. A canonical differential equation is a system $y'=zJHy$ with a real, nonnegative and locally integrable $2\times2$-matrix valued function $H$. The theory of a canonical system is closely related to the spectral theory of a symmetric operator $T_{\min }(H)$ which acts in a Hilbert space $L^2(H)$, and, moreover, is closely related to the theory of positive definite Nevanlinna functions by means of the Titchmarsh--Weyl coefficient associated to it. In the present paper we define an indefinite analogue of canonical systems, construct an operator model which now acts in a Pontryagin space, and show that the spectral theory of the indefinite model is the perfect analogue of the classical theory of $T_{\min }(H)$.

AMS Subject Classification (1991): 47E05, 46C20; 47B25, 34L05

Keyword(s): canonical system, indefinite inner product, operator model

Received November 18, 2005, and in revised form September 6, 2006. (Registered under 5946/2009.)