I think a lot of the problem is how proprietary some of the hardware is. For instance, the Raspberry pi only runs the raspberry pi kernel which has a lot of proprietary blobs.

Meanwhile boards from Pine64 don’t need proprietary software to boot. The achieve this by being selective with the hardware and hardware vendors.