Showing 1–4 of 4 results for author: Benazir, A

Profiling Large Language Model Inference on Apple Silicon: A Quantization Perspective

Authors: Afsara Benazir, Felix Xiaozhu Lin

Abstract: A systematic understanding of Apple Silicon is lacking in the current landscape of hardware efficiency; research focus is largely centered on accelerating GPUs for large-scale training or inference on CUDA devices. This paper investigates Apple Silicon's unique memory architecture that offers a unified memory integrating CPU and GPU memory and its implications for on-device LLM inference. We dec… ▽ More A systematic understanding of Apple Silicon is lacking in the current landscape of hardware efficiency; research focus is largely centered on accelerating GPUs for large-scale training or inference on CUDA devices. This paper investigates Apple Silicon's unique memory architecture that offers a unified memory integrating CPU and GPU memory and its implications for on-device LLM inference. We decipher myths about whether Apple Silicon is efficient for on-device inference compared to competitors such as NVIDIA GPUs by directly conducting latency and throughput comparison benchmarks. We explain the performance gap between them through profiling low level hardware metrics - ALU utilization, memory bandwidth, buffer usage, cache residency etc. at runtime. We draw several insights regarding performance bottlenecks such as dequantization overhead, compute throughput and memory bandwidth. We debunk existing false claims regarding large language model inference such as compressing models to lower bit precision is a defacto promise for faster inference across all hardware platforms. We find that the large unified memory enables Apple Silicon to be both cost effective and efficient against NVIDIA GPUs for ultra large language models. Our large scale evaluation on 5 hardware testbeds incorporating three Apple M-series devices: M2 Ultra, M2 Max and M4 Pro and two NVIDIA GPUs: NVIDIA RTX A6000, a multi GPU setup with 2xNVIDIA RTX A6000, 5 model scales ranging from 8B to 405B parameters and 14 quantization schemes gives an understanding of how Apple Silicon fits within the paradigm of on-device LLM inference. Our analysis reveals multiple resource interdependencies and unexpected findings, while also quantifying established insights. To the best of our knowledge, this study makes the first attempt to present a thorough characterization and analysis of Apple Silicon for on-device inference. △ Less

Submitted 11 August, 2025; originally announced August 2025.

A Journey of Modern OS Construction From boot to DOOM

Authors: Wonkyo Choe, Rongxiang Wang, Afsara Benazir, Felix Xiaozhu Lin

Abstract: VOS is a first-of-its-kind instructional OS that: (1) Runs on commodity, portable hardware. (2) Showcases modern features, including per-app address spaces, threading, commodity filesystems, USB, DMA, multicore, self-hosted debugging, and a window manager. (3) Supports rich applications such as 2D/3D games, music and video players, and a blockchain miner. Unlike traditional instructional systems,… ▽ More VOS is a first-of-its-kind instructional OS that: (1) Runs on commodity, portable hardware. (2) Showcases modern features, including per-app address spaces, threading, commodity filesystems, USB, DMA, multicore, self-hosted debugging, and a window manager. (3) Supports rich applications such as 2D/3D games, music and video players, and a blockchain miner. Unlike traditional instructional systems, VOS emphasizes strong motivation for building systems-supporting engaging, media-rich apps that go beyond basic terminal programs. To achieve this, we design VOS to strike a careful balance between essential OS complexity and overall simplicity. Our method, which we call inverse engineering, breaks down a full-featured OS into a set of incremental, self-contained prototypes. Each prototype introduces a minimal set of OS mechanisms, driven by the needs of specific apps. The construction process (i.e., forward engineering) then progressively enables these apps by bringing up one mechanism at a time. VOS makes it accessible for a wider audience to experience building a software system that is self-contained and usable in everyday scenarios. △ Less

Submitted 24 April, 2025; originally announced April 2025.

Privacy-Preserving Edge Speech Understanding with Tiny Foundation Models

Authors: Afsara Benazir, Felix Xiaozhu Lin

Abstract: Robust speech recognition systems rely on cloud service providers for inference. It needs to ensure that an untrustworthy provider cannot deduce the sensitive content in speech. Sanitization can be done on speech content keeping in mind that it has to avoid compromising transcription accuracy. Realizing the under utilized capabilities of tiny speech foundation models (FMs), for the first time, we… ▽ More Robust speech recognition systems rely on cloud service providers for inference. It needs to ensure that an untrustworthy provider cannot deduce the sensitive content in speech. Sanitization can be done on speech content keeping in mind that it has to avoid compromising transcription accuracy. Realizing the under utilized capabilities of tiny speech foundation models (FMs), for the first time, we propose a novel use: enhancing speech privacy on resource-constrained devices. We introduce XYZ, an edge/cloud privacy preserving speech inference engine that can filter sensitive entities without compromising transcript accuracy. We utilize a timestamp based on-device masking approach that utilizes a token to entity prediction model to filter sensitive entities. Our choice of mask strategically conceals parts of the input and hides sensitive data. The masked input is sent to a trusted cloud service or to a local hub to generate the masked output. The effectiveness of XYZ hinges on how well the entity time segments are masked. Our recovery is a confidence score based approach that chooses the best prediction between cloud and on-device model. We implement XYZ on a 64 bit Raspberry Pi 4B. Experiments show that our solution leads to robust speech recognition without forsaking privacy. XYZ with < 100 MB memory, achieves state-of-the-art (SOTA) speech transcription performance while filtering about 83% of private entities directly on-device. XYZ is 16x smaller in memory and 17x more compute efficient than prior privacy preserving speech frameworks and has a relative reduction in word error rate (WER) by 38.8-77.5% when compared to existing offline transcription services. △ Less

Submitted 29 January, 2025; originally announced February 2025.

Speech Understanding on Tiny Devices with A Learning Cache

Authors: Afsara Benazir, Zhiming Xu, Felix Xiaozhu Lin

Abstract: This paper addresses spoken language understanding (SLU) on microcontroller-like embedded devices, integrating on-device execution with cloud offloading in a novel fashion. We leverage temporal locality in the speech inputs to a device and reuse recent SLU inferences accordingly. Our idea is simple: let the device match incoming inputs against cached results, and only offload inputs not matched to… ▽ More This paper addresses spoken language understanding (SLU) on microcontroller-like embedded devices, integrating on-device execution with cloud offloading in a novel fashion. We leverage temporal locality in the speech inputs to a device and reuse recent SLU inferences accordingly. Our idea is simple: let the device match incoming inputs against cached results, and only offload inputs not matched to any cached ones to the cloud for full inference. Realization of this idea, however, is non-trivial: the device needs to compare acoustic features in a robust yet low-cost way. To this end, we present SpeechCache (or SC), a speech cache for tiny devices. It matches speech inputs at two levels of representations: first by sequences of clustered raw sound units, then as sequences of phonemes. Working in tandem, the two representations offer complementary tradeoffs between cost and efficiency. To boost accuracy even further, our cache learns to personalize: with the mismatched and then offloaded inputs, it continuously finetunes the device's feature extractors with the assistance of the cloud. We implement SC on an off-the-shelf STM32 microcontroller. The complete implementation has a small memory footprint of 2MB. Evaluated on challenging speech benchmarks, our system resolves 45%-90% of inputs on device, reducing the average latency by up to 80% compared to offloading to popular cloud speech recognition services. The benefit brought by our proposed SC is notable even in adversarial settings - noisy environments, cold cache, or one device shared by a number of users. △ Less

Submitted 8 May, 2024; v1 submitted 29 November, 2023; originally announced November 2023.

Comments: accepted at MobiSys'24