So it took approximately four weekends a bit of a brain jogging and I got some serious and stable marker tracking. The last big thing was actually to cut down processing. Thanks to some minor tweaks I squeezed out a 10fold increase.
Now the stats, measure with a 1280x1024 iSight on a 2.4GHz MacBook Pro:
- Speed: 119156.4 pixel/ms
- Normalized on 640x480 this are 2.56ms or a theoretical 390fps
The only bootleneck as of now is the video capture.
Results are available here