StyleTailor: Towards Personalized Fashion Styling via Hierarchical Negative Feedback

1 Tsinghua University 2 National University of Singapore 3 ByteDance Seed
4 Guangming Laboratory 5 Hangzhou Dianzi University
6 University of Hong Kong

^*Corresponding Author

Abstract

The advancement of intelligent agents has revolutionized problem-solving across diverse domains, yet solutions for personalized fashion styling remain underexplored, which holds immense promise for promoting shopping experiences. In this work, we present StyleTailor, the first collaborative agent framework that seamlessly unifies personalized apparel design, shopping recommendation, virtual try-on, and systematic evaluation into a cohesive workflow. To this end, StyleTailor pioneers an iterative visual refinement paradigm driven by multi-level negative feedback, enabling adaptive and precise user alignment. Specifically, our framework features two core agents, i.e., Designer for personalized garment selection and Consultant for virtual try-on, whose outputs are progressively refined via hierarchical vision-language model feedback spanning individual items, complete outfits, and try-on efficacy. Counterexamples are aggregated into negative prompts, forming a closed-loop mechanism that enhances recommendation quality. To assess the performance, we introduce a comprehensive evaluation suite encompassing style consistency, visual quality, face similarity, and artistic appraisal. Extensive experiments demonstrate StyleTailor’s superior performance in delivering personalized designs and recommendations, outperforming strong baselines without negative feedback and establishing a new benchmark for intelligent fashion systems.

Model Overview

The Designer module analyzes the user-provided image and style preferences, generates garment specifications, and retrieves suitable clothing images. Two hierarchical negative feedback mechanisms within the Designer refine retrieval at the item and outfit levels. The Consultant module generates virtual try-on results and applies higher-level feedback to further improve alignment with user requirements. The Critic quantitatively evaluates the final outputs. This multi-stage feedback ensures the system progressively optimizes recommendations.

Comparison with Baseline

Qualitative comparison between our method and the baseline under two conditions: (1) The same user image with different descriptions; (2) The same description with different user images. The visualization results demonstrate both the personalized design capacity of our approach and its superior performance in comparison with the baseline method. The red text indicates the apparently inappropriate garment retrieval by the baseline.

Personalized Presentation

Visualizations of diverse user images and style descriptions, along with the corresponding outputs produced by our StyleTailor. These examples demonstrate StyleTailor’s ability to effectively handle various user appearances and styling preferences, highlighting its robustness and adaptability to complex, real-world input scenarios.

BibTeX

@misc{ma2025styletailorpersonalizedfashionstyling, title={StyleTailor: Towards Personalized Fashion Styling via Hierarchical Negative Feedback}, author={Hongbo Ma and Fei Shen and Hongbin Xu and Xiaoce Wang and Gang Xu and Jinkai Zheng and Liangqiong Qu and Ming Li}, year={2025}, eprint={2508.06555}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2508.06555}, }