On the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with the added benefit that I can also pair the strix halo with an egpu to drastically speed things up.
From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension.
The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
So when I said "a couple of nvidia engineers", I indeed meant Jeff.
VK_KHR_cooperative_matrix - embrace?
VK_NV_cooperative_matrix2 - extend?
I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
> I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s
> Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable
__VK_ENABLE_DEVICE_GROUPS=1.
They were maintained by Codeplay - paid for my Intel. Nvidia can make contributions anytime they want, and here is the problem: Nvidia does not want to. Until each vendor starts pitching in with contributing their backend to an open standard, you will have to rely on others doing it for them.
Yeah talking about the (vendor-preferred) compute part here
Vulkan's SPIR-V dialect is substantially different from the OpenCL one, notably with the former having structured control flow. They're incompatible between each other.
This just means that Apple is a big enough player for ARM to retroactively amend the standard. Does not mean that what Apple did was not a violation of then-standard tho.
> Decoupling the User/Kernel boundary in Windows is a monumental task and highly non-trivial, however, we have been working hard to stabilize this boundary across all of Windows to provide our customers the flexibility to run down-level containers. Starting with Windows 11 and Windows Server 2022 we are enabling the ability to run process-isolated WS2022 containers on Windows 11 hosts.
One thing in the story is that the SPU - the secure enclave on Qualcomm chips that is separate from just running on TrustZone on the main processor - is only available on Snapdragon 8 and X tier products. It's market segmented away from 7 series and below.
Also, I think Qualcomm has only started supporting MTE on Snapdragon 8 Elite Gen 5. Older and cheaper SoCs do not support it yet. Outside that, I think only Google Tensor, recent Exynos generations, and Apple A-series support MTE (though IIRC Apple has an improved extension of it).
Snapdragon 8 Elite Gen 5 has at support for MTE. It doesn't yet support running the whole kernel and userspace with it in practice as GrapheneOS requires but we're going to be working on it with Motorola and Qualcomm.
reply