Keep dropping chain (maybe once per ride) on GRX 11speed front derailleur

[cyclingnolycra]

Bike is a Ridley Kanzo A with GRX810 front derailleur. I've adjusted it so the limit screws are less than 1mm away from the chain in the smallest gear (small ring + largest cog, no trim) and biggest gear (big ring + smallest cog.) Crankset is the stock FORZA 46-30.

Pretty much every ride there'll be a dropped chain off the inside of the small ring. Easy to recover from - just shift into big ring and the chain comes back on. But annoying nonetheless. I'm wondering if it's somehow to do with trim - it looks like after I shift from big ring to small ring, the position of the front derailleur is basically the same as when I press to shift into the small ring again (which normally makes the derailleur move closer to the frame again). This would increase the chance of the chain dropping obviously. How can I fix this?

I am usually competent in setting up derailleurs and know the basics about limit screws and tension, but I feel like I'm missing something here. Any help appreciated, thanks!