Image-based social relation recognition using graph neural network

Abstract

Social relation, which indicates how people are connected in society, is an essential part of our social life. With the boom of social media, data like pictures, videos, and texts, become available and can be used for social relation recognition (SRR). Meantime, the advancement of computing infrastructure and computer vision research in recent years has made it possible for computers to process these kinds of data to recognize social relations in our life. SRR is a complex topic as social relations among humans diverse a lot. Because of Convolutional Neural Network (CNN) and Graph Neural Network (GNN), it has become possible for a machine to recognize social relations in an acceptable condition. SRR problems were mostly solved by feature extraction and graph reasoning process which depend on CNN and GNN respectively. In this work, the proposed SRR was based on the Interpersonal Relation benchmark dataset [1]. Also, following existing work, we extracted features from multi-scale views from images and reasoned by two-directional graphs with Gated Recurrent Unit (GRU) attention mechanism. The results showed that the proposed work surpasses the state-of-the-art work on the Interpersonal Relation dataset by about 10% in balanced accuracy.