One such issue at launch was the lack of an option to not combine taskbar icons (also called ungrouping) and show labels for each window. Thankfully, starting with Windows 11 version 23H2 ...

Windows has changed a lot in the last few years, for good, but some features were better in the previous versions of Windows. For example, the grouped icons in our taskbar. With the latest Windows ...

By default, the never combine option enables labels only for the open programs, leaving pins in the icon-only mode. To change that, place a checkmark next to the Show labels on taskbar pins option.