Yeah, drill it out. That crumbly core will keep compressing forever no matter how much epoxy you think you soaked into it, and you'll end up with a handle that feels loose a year later. I drill the bulk of it with a regular bit, then switch to a rasp or a half-round file to clean up the walls so you've got solid white bone all the way around. Here's the trick that saved me a bunch of ruined handles: grind the tip off your drill bit so the flutes cut flat instead of grabbing. Antler pith grabs a sharp point and yanks the piece right out of your hands, especally near the end of the cut.
Leave a rough interior, don't sand it smooth. Epoxy keys into those little ridges way better. I mix a slow-cure epoxy, thicken it slightly, and add a pinch of brown coffee grounds so the fill doesn't glow white against the bone if any squeezes out at the pins. Pack it in with a dowel, trowel style, working from the bottom up so you don't trap air pockets.
Then just walk away from it for a couple days before you shape anything. Rushing the cure is how you get voids.