You don't necessarily need to do anything beyond exploit the structure of your problem.
In particular, don't compute, or store or in any other way chew up time and space, dealing with the parts of the calculation that will just be zero. Figure out where the things you need to change will change and only store and update those. Also, some people over-compress the steps of the calculation ("look, I wrote it in one less line of algebra"), but in the process, can mask some of the opportunities to exploit redundant calculation. Some algorithms don't explicitly compute the Kalman gain as a step, for example, but there might be a speedup there (I've seen some algorithms implicitly computing it three times).
In addition there are some other obvious things to avoid
don't multiply by 1.
don't add zero.
I don't mean "check for this calculation in code and skip it" (that's almost useless) - I mean work out where you would have done this calculation using your knowledge of the structure of the problem and never go there in code at all.
don't compute the same thing twice. (At worst, you could consider memoization, but much better to avoid even considering calculating what you know you'll already know; don't do in software what your brain can easily anticipate and you can simply avoid altogether)
There's often a variety of symmetries to exploit as well. e.g. If you're already computing one thing, don't then turn around and compute its transpose; you already have it (this is a subset of computing the same thing twice).
Often there can be some kind of block structure (or similar structures), sometimes with some efficiencies between (such as one block being closely related to another) or within blocks (e.g. symmetries or skew-symmetries)
Some time thinking about the avoidable calculations you're doing can make a huge difference. Do the biggest things first (I mean the things that will avoid a lot of calculation, not the things that take the most time to implement); you may find you don't need to get every ounce of gain when you can get 85% of it for less than half the work.
But by the same token, don't overoptimize to the extent that you're getting in the way of the things the hardware and numerical libraries can do well (this depends on the precise nature of your hardware and libraries). Some of this can be subtle, but don't knock yourself out over tiny gains (or even negative gains if you're not careful)