Most of the problem was sorting stuff.
While a few optimizations could be made, YouTube playback is almost as seamless as local playback.