Nine times out of ten when aluminum chips start causing premature tool failure even when the specs look good, you aren't fighting speed, you're fighting chip evacuation. Aluminum has this tendency to gummy up, and when a chip piles up and locks in the cutting zone, it acts like a sudden impact rather than a controlled cut. That trapped metal causes enormous localized stress right at the cutting edge, leading to those microfractures you're seeing.
What I saw happen on a setup like yours was that even with plenty of coolant flow, the chips were just wrapping around the tool instead of being pushed out and flushed. This usally means you need a specific chip breaker designed for gummy materials. If you're using generic positive geometry inserts, they are likely failing because they aren't managing the chip formation. Switching to a negative rake tool with a specific chip breaker profile can dramatically change how the metal breaks and slides off. Another thing to check is your feed rates in relation to the depth of cut. If you are taking too shallow a pass with too light a feed, you are creating long, continuous chips that are difficult to manage. On raw aluminum, you really need to feed aggressively enough to shear the material cleanly rather than tearing it, which produces those long, wire-like strands that weld to the tool. Try increasing the feed rate slightly and reducing the depth of cut until you see consistent, manageable chip breakage. If you alredy have aggressive feeds and the chipping continues, then look at the setup for runout. You mentioned the vibration level isn't high, but there could still be a subtle wobble in the spindle or a runout on the insert itself that only manifests under the specific pressure of a gummy cut. It’s a lot about controlling the chip, not just the spin.