If you want to generate cascade with OpenCV training tools, you should be ready for waiting plenty of time. For example, on training set: 3000 positive / 5000 negative, it takes about 6 days! to get cascade for face detection.  I wanted to generate many cascades with different training sets, also I added my own features to standart OpenCV’s ones  and refactor algorithms a little bit.  So waiting for 6 days to understand, that your cascade does nothing good =) was really anoying.  To reduce time, I chose paralleling methods.

OpenMP.

In OpenCV code supports OpenMP.  OpenMP is library, which allows to run program in several threads.  All this makes sence, if you have appropriate processor like Intel Core Duo or with Hyper Threading support at least.

The advantage of this method is that, it’s already implemented and, I really believe, debugged in OpenCV.  OpenMP will speed up cascade generation – 4 days instead of 6 on my Intel Core2  1.8GHZ  2GB.

MPI.

We constructed Linux-based cluster from 11 machines with  configuration:  2.7GHZ processor with 2GB RAM.    Computers are linked via 100 Ethernet LAN.

OpenCV internal data structures are matrix and vectors – really good for paralleling.  So we decided to add MPI API calls to places with OpenMP defines – so just clone OpenMP schemas, who knows why we did so =) – hurry I suppose.   In this way we commonly paralleled  loops. But MPI does not have shared memory, unlike OpenMP,  so data(MBs of traffic) synchronization time over Ethernet LAN brought computation speed-ups to nothing.  We understood, that  for MPI we needed parallel schema, in which data synchronization would be small.

First, I wanted to investigate, what functions take most of all time –  printf  profiling in cvCreateTreeCascadeClassifier helped me.  And what do you think?  Function icvGetHaarTrainingDataFromBG is hero of the occasion – computation time was 9 hours on 11th cascade stage! Unlike it, icvGetHaarTrainingDataFromVec  took about 10 minutes. The matter is that, positive samples are resized to 20×20, when make training vec file and each picture is just  run through  cascade. Negative samples have original resolution and it’s various, that’s why each picture is scanned with scalling 20×20 window to find false-positive, like haardetect does.  The process is stopped, when we have required number of false-positive pictures.  To reduce time,  we needed to parallel icvGetHaarTrainingDataFromBG, but avoiding large data synchronization.

icvGetHaarTrainingDataFromBG works in such way:

• it gets negative samples
• found false positive until required number is reached
• return false-positives

If we shuffle negative samples and then call icvGetHaarTrainingDataFromBG, what will happen?  Anything bad? In output we will have another false-positive pictures, but algorithm in whole will work correctly and generate right cascade.  So we decided to split negative samples into 11 parts(11 machines in cluster) and each cluster calls cvGetHaarTrainingDataFromBG on it’s own negative set, then clusters outputs are joined together.

Computation time was accelerated much, instead of 6 days, cascade was generated within 21 hours!  With perfomance tool we compare our cascade with one, generated on single machine with the same training set.  Results are very similar.