Two billion parameters are now placed not in the server rack and not even in the smartphone, but right in the frame of the glasses. PrismML showed the Bonsai multimodal model, which locally works on smart glasses with the Qualcomm Snapdragon AR1 Gen 1 platform and is able to analyze in real time what the owner sees.
The model combines the language part by 1.7 billion parameters and a visual encoder by another 0.3 billion. The camera transmits the image to local AI, after which the user can ask about objects in front of the eyes and receive a response without necessarily sending a request to the cloud. Such a scenario brings smart glasses closer to the role of a permanent personal assistant, not just a camera or headset.
The main feature of Bonsai is not related to the number of parameters, but to how the weights of the model are stored. The language part uses a 1-bit representation, while the visual encoder remains 4-bit. Therefore, it would be inaccurate to call the entire two-billion-dollar system completely 1-bit.
According to Qualcomm, the weight of the language part occupies about 0.43 GB instead of 1.66 GB in a comparable 4-bit version. The reduction almost four times allowed to launch a noticeably larger model in the same memory restrictions. On a test platform with 4GB of memory and performance up to 6, TOPS generation speeds reached 15.36 tokens per second against 7.44 at 4-bit variant. A similar race to transfer local AI from the cloud to compact iron is already going on in other classes of devices.
For points, savings are especially valuable. In a thin frame, there is little room for memory and battery, and excess heat quickly turns into a problem. PrismML, together with Qualcomm, has optimized the architecture for the Hexagon neuroprocessor to reduce data volume and speed up processing without constant communication with the data center.
The stated figures should be taken as the results of the controlled tests. The PrismML and Qualcomm tests were conducted on special configurations, including an internal version of the QNN software kit with 1-bit computation support. Developers warn that speed depends on model configuration, software and load.
The technology has not yet reached the serial product. Neither PrismML nor partners presented consumer glasses with Bonsai, so the current demonstration shows a technical capability, not a ready-made gadget. Serial glasses with such model have not yet been announced.
Just two days earlier, Qualcomm showed a different scale of local models: the new mobile processor is able to launch AI for 30 billion parameters directly on the smartphone. In glasses, the available amount of memory and power consumption is much tougher, so the PrismML rate is made not on the absolute size, but on the maximum compression.
Another experiment shows how far miniaturization has gone. In August, the developer launched a neural network on a microcontroller worth less than $10. Against this background, 2 billion parameters inside the glasses look like another step towards AI, which is increasingly working next to the user, rather than on a remote server.
The model combines the language part by 1.7 billion parameters and a visual encoder by another 0.3 billion. The camera transmits the image to local AI, after which the user can ask about objects in front of the eyes and receive a response without necessarily sending a request to the cloud. Such a scenario brings smart glasses closer to the role of a permanent personal assistant, not just a camera or headset.
The main feature of Bonsai is not related to the number of parameters, but to how the weights of the model are stored. The language part uses a 1-bit representation, while the visual encoder remains 4-bit. Therefore, it would be inaccurate to call the entire two-billion-dollar system completely 1-bit.
According to Qualcomm, the weight of the language part occupies about 0.43 GB instead of 1.66 GB in a comparable 4-bit version. The reduction almost four times allowed to launch a noticeably larger model in the same memory restrictions. On a test platform with 4GB of memory and performance up to 6, TOPS generation speeds reached 15.36 tokens per second against 7.44 at 4-bit variant. A similar race to transfer local AI from the cloud to compact iron is already going on in other classes of devices.
For points, savings are especially valuable. In a thin frame, there is little room for memory and battery, and excess heat quickly turns into a problem. PrismML, together with Qualcomm, has optimized the architecture for the Hexagon neuroprocessor to reduce data volume and speed up processing without constant communication with the data center.
The stated figures should be taken as the results of the controlled tests. The PrismML and Qualcomm tests were conducted on special configurations, including an internal version of the QNN software kit with 1-bit computation support. Developers warn that speed depends on model configuration, software and load.
The technology has not yet reached the serial product. Neither PrismML nor partners presented consumer glasses with Bonsai, so the current demonstration shows a technical capability, not a ready-made gadget. Serial glasses with such model have not yet been announced.
Just two days earlier, Qualcomm showed a different scale of local models: the new mobile processor is able to launch AI for 30 billion parameters directly on the smartphone. In glasses, the available amount of memory and power consumption is much tougher, so the PrismML rate is made not on the absolute size, but on the maximum compression.
Another experiment shows how far miniaturization has gone. In August, the developer launched a neural network on a microcontroller worth less than $10. Against this background, 2 billion parameters inside the glasses look like another step towards AI, which is increasingly working next to the user, rather than on a remote server.
