I would set your cache level either really high or at zero :)) I have mine at 5gb which is huge and then manually clear it every few months.

But Fencer is right, it seems to only happen with IE, although it does not happen with me since I set it so high :)