So why exactly was Metal Sonic made to be a longer-than usual Act 2 boss in Sonic Mania? Because, once you've beaten him, you've gone through half of the entire game! There are (AT LEAST) 12 Zones, but after Stardust Speedway, you've only gone through the first 6!
I can see why the Phantom Ruby just lies there after you beat him. It was meant to have your character feel like your mission was done! But then a Badnik knocks the Ruby over to Eggman and shows that you have at least 6 more zones to finish!
----------
Also, with the new version of the entire Metal Sonic fight, namely in its final phase, this isn't the first time Knuckles gets to fight a gigantic monstrous version of Metal Sonic (in the Classic era, that is). He's previously done so in his own game, Knuckles Chaotix, where the giant Metal Sonic Kai was red instead of blue like he is, here.
Besides that, Knuckles never really got to show up in Sonic CD, since the game came out a year before Sonic 3, where Knuckles came from. Make there can be a hack that adds him into CD's smartphone remake...