This paper presents an ensemble-SVM method that features a data selection mechanism with stochastic and deterministic properties, the use of extreme value theory for classifier calibration, and the introduction of random forest for classifier combination. We applied the proposed algorithm to 2 event recognition datasets and the PASCAL2007 object detection dataset and compared it to single SVM and common computer vision ensemble-SVM methods. Our algorithm outperforms its competitors and shows a considerable boost on datasets with a limited amount of outliers.