You probably want to fight the samurai when you have the age advantage, especially with that speed.

The Burgrave does still provide these units with additional charge damage, but against deflective armor, it just doesn't matter.

Getting Meinwerk Palace means having access to very cheap and very quickly researched blacksmith upgrades, which can be very important when up against a potential Yumi Ashigaru transition from the Japanese. Having very early blacksmith upgrades against Yumi Ashigaru, when they're already doing less damage, makes it quite tough.