Skin beauty adviser assistant based on large language model and computer vision

Abstract

In recent years, personalized skincare has gained popularity for its effectiveness. However, current solutions often rely on expensive, inconvenient offline treatments or professional medical devices, which is difficult to meet the needs of the public. With advancements in machine learning and deep learning, AI-driven techniques offer new, accessible solutions for personalized skincare. Based on this, this dissertation designed and implemented a personalized skin beauty adviser assistant that combines computer vision and large language models (LLMs). This dissertation details the system's design, technical architecture, implementation process and test results. Through a self-built dataset, the system uses the ResNet50 pre-trained model and combines SENet and CBAM attention mechanisms to classify skin types, including oily, dry, and normal skin. Additionally, the ChatGLM3-6B model is fine-tuned with dialogue dataset using methods such as LoRA, QLoRA and Freeze to generate personalized skin care advice. And their performance are analyzed and compared. The experimental results show that after adding the attention mechanism, the accuracy of skin type classification increased by 2.21% (SENet) and 3.80% (CBAM). And the fine-tuned large language model has improved in multiple dimensions in generating personalized skin care advice compared to the baseline. In summary, the system can provide users with high-quality personalized skin care advice, support multiple rounds of interaction, and significantly improve user experience. In the future, the system is expected to be further applied in areas such as remote diagnosis of skin diseases and personalized skin care products.