Multimodal Deep Learning based Framework for Detecting Depression and Suicidal Behaviour by Affective Analysis of Social Media Posts

INTRODUCTION AND OBJECTIVES: Currently, no social media platform has deployed a real time system which can analyse users’ state of mind based on day to day posts on continual basis and detect the onset of depression, suicidal or self harming behaviour etc. Platforms majorly rely on manual reporting of suicidal and self harming behavior. In this paper, we propose a real time, deep learning based system for affective analysis of a user’s online social media posts of multimodal nature, with the objective of detecting onset of depression and suicidal or self harming behaviour; as depression often drives people to commit suicide or harm themselves physically.

METHODS: Joint representations are obtained by fusing the individual vector representations of multiple modalities from user’s social media feed: text, image and videos. These vector representations are in turn obtained through state of the art approaches for each modality e.g. VGG-16 for feature extraction from images, word2vec for text and Faster R-CNN on video frames. These joint representations are used to obtain weighted average score which can be used for making the final classification using the Softmax prediction layer.

SIGNIFICANCE AND IMPACT: To the best of our knowledge, this is the first research where the use of deep learning techniques has been proposed for real time detection of onset of depression and suicidal behaviour by analysing multimodal user generated content.