In recent years the sample covariance matrix of high-dimensional vectors with iid entries has attracted a lot of attention. A deep theory exists if the entries of the vectors are iid light-tailed; the Tracy-Widom distribution typically appears as weak limit of the largest eigenvalue of the sample covariance matrix. In the heavy-tailed case (assuming infinite 4th moments) the situation changes dramatically. Work by Soshnikov, Auffinger, Ben Arous and Peche shows that the largest eigenvalues are approximated by the points of a suitable non-homogeneous Poisson process. We follows this line of research. First, we consider a p-dimensional time series with iid heavy-tailed entries where p is any power of the sample size n. The point process of the scaled eigenvalues of the sample covariance matrix converges weakly to a Poisson process. Next, we consider p-dimensional heavy-tailed time series with dependence through time and across the rows. In particular, we consider entries with a linear dependence or a stochastic volatility structure. In this case, the limiting point process is typically a Poisson cluster process. We discuss the suitability of the aforementioned models for large portfolios of return series.