Abstract: Vision Transformers (ViTs) are increasingly deployed in IoT applications such as smart surveillance and vehicular monitoring for their strong representational power. Yet their heavy architectures require lightweight adaptation on resource-constrained devices, where accuracy often drops under data drift caused by dynamic conditions like illumination or SNR changes. The problem worsens as multiple evolving tasks compete for limited memory and compute: some models remain stuck in low-accuracy states, while redundant deployment wastes resources and increases latency. Existing multitask learning reduces redundancy but suffers from negative transfer and fails under asynchronous evolution, yielding unstable accuracy. A crowd-intelligence-driven collaborative evolution framework for lightweight ViTs is proposed. It introduces a multi-task expert sparse-sharing model guided by task correlation and importance, and a joint parameter–structure refinement mechanism for dynamic adaptation. Experiments show our approach improves accuracy and efficiency over single-task ViTs and multitask baselines, enabling scalable deployment in dynamic IoT environments.