-
Measuring and Guiding Monosemanticity
Authors:
Ruben Härle,
Felix Friedrich,
Manuel Brack,
Stephan Wäldchen,
Björn Deiseroth,
Patrick Schramowski,
Kristian Kersting
Abstract:
There is growing interest in leveraging mechanistic interpretability and controllability to better understand and influence the internal dynamics of large language models (LLMs). However, current methods face fundamental challenges in reliably localizing and manipulating feature representations. Sparse Autoencoders (SAEs) have recently emerged as a promising direction for feature extraction at sca…
▽ More
There is growing interest in leveraging mechanistic interpretability and controllability to better understand and influence the internal dynamics of large language models (LLMs). However, current methods face fundamental challenges in reliably localizing and manipulating feature representations. Sparse Autoencoders (SAEs) have recently emerged as a promising direction for feature extraction at scale, yet they, too, are limited by incomplete feature isolation and unreliable monosemanticity. To systematically quantify these limitations, we introduce Feature Monosemanticity Score (FMS), a novel metric to quantify feature monosemanticity in latent representation. Building on these insights, we propose Guided Sparse Autoencoders (G-SAE), a method that conditions latent representations on labeled concepts during training. We demonstrate that reliable localization and disentanglement of target concepts within the latent space improve interpretability, detection of behavior, and control. Specifically, our evaluations on toxicity detection, writing style identification, and privacy attribute recognition show that G-SAE not only enhances monosemanticity but also enables more effective and fine-grained steering with less quality degradation. Our findings provide actionable guidelines for measuring and advancing mechanistic interpretability and control of LLMs.
△ Less
Submitted 24 June, 2025;
originally announced June 2025.
-
Scaling laws in wearable human activity recognition
Authors:
Tom Hoddes,
Alex Bijamov,
Saket Joshi,
Daniel Roggen,
Ali Etemad,
Robert Harle,
David Racz
Abstract:
Many deep architectures and self-supervised pre-training techniques have been proposed for human activity recognition (HAR) from wearable multimodal sensors. Scaling laws have the potential to help move towards more principled design by linking model capacity with pre-training data volume. Yet, scaling laws have not been established for HAR to the same extent as in language and vision. By conducti…
▽ More
Many deep architectures and self-supervised pre-training techniques have been proposed for human activity recognition (HAR) from wearable multimodal sensors. Scaling laws have the potential to help move towards more principled design by linking model capacity with pre-training data volume. Yet, scaling laws have not been established for HAR to the same extent as in language and vision. By conducting an exhaustive grid search on both amount of pre-training data and Transformer architectures, we establish the first known scaling laws for HAR. We show that pre-training loss scales with a power law relationship to amount of data and parameter count and that increasing the number of users in a dataset results in a steeper improvement in performance than increasing data per user, indicating that diversity of pre-training data is important, which contrasts to some previously reported findings in self-supervised HAR. We show that these scaling laws translate to downstream performance improvements on three HAR benchmark datasets of postures, modes of locomotion and activities of daily living: UCI HAR and WISDM Phone and WISDM Watch. Finally, we suggest some previously published works should be revisited in light of these scaling laws with more adequate model capacities.
△ Less
Submitted 20 May, 2025; v1 submitted 5 February, 2025;
originally announced February 2025.
-
SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMs
Authors:
Ruben Härle,
Felix Friedrich,
Manuel Brack,
Björn Deiseroth,
Patrick Schramowski,
Kristian Kersting
Abstract:
Large Language Models (LLMs) have demonstrated remarkable capabilities in generating human-like text, but their output may not be aligned with the user or even produce harmful content. This paper presents a novel approach to detect and steer concepts such as toxicity before generation. We introduce the Sparse Conditioned Autoencoder (SCAR), a single trained module that extends the otherwise untouc…
▽ More
Large Language Models (LLMs) have demonstrated remarkable capabilities in generating human-like text, but their output may not be aligned with the user or even produce harmful content. This paper presents a novel approach to detect and steer concepts such as toxicity before generation. We introduce the Sparse Conditioned Autoencoder (SCAR), a single trained module that extends the otherwise untouched LLM. SCAR ensures full steerability, towards and away from concepts (e.g., toxic content), without compromising the quality of the model's text generation on standard evaluation benchmarks. We demonstrate the effective application of our approach through a variety of concepts, including toxicity, safety, and writing style alignment. As such, this work establishes a robust framework for controlling LLM generations, ensuring their ethical and safe deployment in real-world applications.
△ Less
Submitted 5 December, 2024; v1 submitted 11 November, 2024;
originally announced November 2024.
-
EarGate: Gait-based User Identification with In-ear Microphones
Authors:
Andrea Ferlini,
Dong Ma,
Robert Harle,
Cecilia Mascolo
Abstract:
Human gait is a widely used biometric trait for user identification and recognition. Given the wide-spreading, steady diffusion of ear-worn wearables (Earables) as the new frontier of wearable devices, we investigate the feasibility of earable-based gait identification. Specifically, we look at gait-based identification from the sounds induced by walking and propagated through the musculoskeletal…
▽ More
Human gait is a widely used biometric trait for user identification and recognition. Given the wide-spreading, steady diffusion of ear-worn wearables (Earables) as the new frontier of wearable devices, we investigate the feasibility of earable-based gait identification. Specifically, we look at gait-based identification from the sounds induced by walking and propagated through the musculoskeletal system in the body. Our system, EarGate, leverages an in-ear facing microphone which exploits the earable's occlusion effect to reliably detect the user's gait from inside the ear canal, without impairing the general usage of earphones. With data collected from 31 subjects, we show that EarGate achieves up to 97.26% Balanced Accuracy (BAC) with very low False Acceptance Rate (FAR) and False Rejection Rate (FRR) of 3.23% and 2.25%, respectively. Further, our measurement of power consumption and latency investigates how this gait identification model could live both as a stand-alone or cloud-coupled earable system.
△ Less
Submitted 27 August, 2021;
originally announced August 2021.
-
Semi-automated Signal Surveying Using Smartphones and Floorplans
Authors:
Chao Gao,
Robert Harle
Abstract:
Location fingerprinting locates devices based on pattern matching signal observations to a pre-defined signal map. This paper introduces a technique to enable fast signal map creation given a dedicated surveyor with a smartphone and floorplan. Our technique (PFSurvey) uses accelerometer, gyroscope and magnetometer data to estimate the surveyor's trajectory post-hoc using Simultaneous Localisation…
▽ More
Location fingerprinting locates devices based on pattern matching signal observations to a pre-defined signal map. This paper introduces a technique to enable fast signal map creation given a dedicated surveyor with a smartphone and floorplan. Our technique (PFSurvey) uses accelerometer, gyroscope and magnetometer data to estimate the surveyor's trajectory post-hoc using Simultaneous Localisation and Mapping and particle filtering to incorporate a building floorplan. We demonstrate conventional methods can fail to recover the survey path robustly and determine the room unambiguously. To counter this we use a novel loop closure detection method based on magnetic field signals and propose to incorporate the magnetic loop closures and straight-line constraints into the filtering process to ensure robust trajectory recovery. We show this allows room ambiguities to be resolved.
An entire building can be surveyed by the proposed system in minutes rather than days. We evaluate in a large office space and compare to state-of-the-art approaches. We achieve trajectories within 1.1 m of the ground truth 90% of the time. Output signal maps well approximate those built from conventional, laborious manual survey. We also demonstrate that the signal maps built by PFSurvey provide similar or even better positioning performance than the manual signal maps.
△ Less
Submitted 17 November, 2017;
originally announced November 2017.
-
A Contextual Investigation of Location in the Home Using Bluetooth Low Energy Beacons
Authors:
Charith Perera,
Saeed Aghaee,
Ramsey Faragher,
Robert Harle,
Alan Blackwell
Abstract:
Location sensing is a key enabling technology for Ubicomp to support contextual interaction. However, the laboratories where calibrated testing of location technologies is done are very different to the domestic situations where `context' is a problematic social construct. This study reports measurements of Bluetooth beacons, informed by laboratory studies, but done in diverse domestic settings. T…
▽ More
Location sensing is a key enabling technology for Ubicomp to support contextual interaction. However, the laboratories where calibrated testing of location technologies is done are very different to the domestic situations where `context' is a problematic social construct. This study reports measurements of Bluetooth beacons, informed by laboratory studies, but done in diverse domestic settings. The design of these surveys has been motivated by the natural environment implied in the Bluetooth beacon standards - relating the technical environment of the beacon to the function of spaces within the home. This research method can be considered as a situated, `ethnographic' technical response to the study of physical infrastructure that arises through social processes. The results offer insights for the future design of `seamful' approaches to indoor location sensing, and to the ways that context might be constructed and interpreted in a seamful manner.
△ Less
Submitted 12 March, 2017;
originally announced March 2017.
