Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds

arXiv:2606.25234v1 Announce Type: new Abstract: What is the geometry of a visual percept? The most widely used protocols for decomposing neural network representations into interpretable parts treat concepts as isolated directions, yet recent work shows that concepts are often realized as geometric structures in low dimensional regions of activation space. We turn to the literature of Structured sparsity to close this gap, and show that block sparsity, which groups directions into blocks, is the...