{"componentChunkName":"component---src-templates-post-template-js","path":"/posts/dyadic-relational-graph-convolutional-networks-for-skeleton-based-human-interaction-recognition","result":{"data":{"markdownRemark":{"id":"53a6091c-39cf-564d-b679-d5cdad22ba6d","html":"\n

Here, I briefly introduce our work. Some contents are extracted from the accepted version of our paper. For more information please see our paper. Code is available at github.

\n
\n \n
\n
\n\tOverall architecture of DR-GCN.\n\t
\n
\n

Highlights

\n\n

Abstract

\n

Skeleton-based human interaction recognition is a challenging task requiring all abilities to recognize spatial, temporal, and interactive features. These abilities rarely co-exist in existing methods. Graph convolutional network (GCN) based methods fail to extract interactive features. Traditional interaction recognition methods cannot effectively capture spatial features from skeletons. Toward this end, we propose a novel Dyadic Relational Graph Convolutional Network (DR-GCN) for interaction recognition. Specifically, we make four contributions: (i) we design a Relational Adjacency Matrix (RAM) that represents dynamic relational graphs. These graphs are constructed combining both geometric features and relative attention from the two skeleton sequences; (ii) we propose a Dyadic Relational Graph Convolution Block (DR-GCB) that extracts spatial-temporal interactive features; (iii) we stack the proposed DR-GCBs to build DR-GCN and integrate our methods with an advanced model. (iv) Our models achieve state-of-the-art results on SBU and significant improvements on the mutual action sub-datasets of NTU-RGB+D and NTU-RGB+D 120.

\n

Method

\n
\n \n
\n
An illustration of the relational graph. Green dots in this image represents\nbody joints. The orange links represent relational links denoting the strong relation between joints of the two actors.
\n
\n

Above image shows one relational graph at a single frame, which is represented by proposed Relational Adjacency Matrix. It is generated separately for each frame of a sequence of frames in the skeleton sequence.

\n
\n \n
\n
\n\tAn illustration of the Relational Adjacency Matrix (RAM) generation procedure.\n\t
\n
\n

The generation and utilization are the key components of our paper. Briefly speaking, we generate the relational links, or the RAM, which represents the relational links by considering two components. They are the geometric component and the relative attention component. The geometric component is straitforward. If two joints each from one actor are close, then we consider them to be correlated. This simple assumption turns out to be very effective. For the relative attention component, we hope it can capture semantic information and connect joints that are semantically similar. We do this by first encode each joint with spatial-temporal graph convolutional layers and then calculate similarity between each joint pairs. Basing on above two component, we combine them using network-learned param and then we have the RAM.

\n
\n \n
\n
\n\tAn illustration of Dyadic Relational Graph Convolution Block (DR-GCB). DR-GC refers to dyadic relational graph convolution.\n\t
\n
\n

With the RAM, we propose Dyadic Relational Graph Convolution Block (DR-GCB) that apply dyadic relational graph convolution on the two skeletons to learn relational features. DR-GCB is highly extensible and can be plugged to other networks to improve their performance.

\n

Results

\n

We have done extensive experiments. Results show our network and methods achieve significantly better results comparing with other state-of-the-art methods. They also prove the extensibility of our methods. To review the data, please read our paper.

\n

Below we show some generated relational graphs.\n\"demo-1\"\n\"demo-2\"\n\"demo-3\"\n\"demo-4\"

\n
\n\t
\n\tSome demos of the generated relational graphs.\n\t
\n
\n

Conclusion

\n

This article is only meant for a brief introduction, if interested please read our paper.

\n

Our paper presents a novel Dyadic Relational Graph Convolutional\nNetwork (DR-GCN) for skeleton-based interaction recognition. We devise\nRelational Adjacency Matrix (RAM) denoting relational graph. It combines\nboth the geometric features and relative attention of the two skeletons in\ninteraction. Dyadic Relational Graph Convolution Block (DR-GCB) is fur-\nther proposed to extract spatial-temporal interactive features with RAM.\nWe stack multiple layers of DR-GCBs to build the backbone of our network.\nWe further propose Two-Stream Dyadic Relational AGCN (2S-DRAGCN)\nthat demonstrates our methods’ compatibility with ST-GCN based mod-\nels. Our proposed models show superior abilities in interaction recognition.\nThey achieve the highest accuracy on the mutual action sub-dataset of NTU-\nRGB+D, that of NTU-RGB+D 120, and the interaction dataset, SBU.

\n

© 2021. Contents from the accepted version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/.

","fields":{"slug":"/posts/dyadic-relational-graph-convolutional-networks-for-skeleton-based-human-interaction-recognition","tagSlugs":["/tag/computer-vision/","/tag/action-recognition/","/tag/skeleton-based-action-recognition/","/tag/graph-convolutional-networks/","/tag/deep-learning/","/tag/research/"]},"frontmatter":{"date":"2021-02-19T14:13:40.121Z","description":"

We apply Graph Convolutional Networks on skeleton-based human-human interaction recognitions. We designed a Relational Adjacency Matrix (RAM) to represent dynamic relational graphs on the two actor's skeletons.

Liping Zhu*, Bohua Wan*, Chengyang Li, Gangyi Tian, Yi Hou, Kun Yuan

Pattern Recognition 115 (2021): 107920.

","tags":["Computer Vision","Action Recognition","Skeleton-based Action Recognition","Graph Convolutional Networks","Deep Learning","Research"],"title":"Dyadic Relational Graph Convolutional Networks for Skeleton-based Human Interaction Recognition","socialImage":{"publicURL":"/static/99d2a147d44057e4dd6664a84a9d8f20/structure.jpg"}}}},"pageContext":{"slug":"/posts/dyadic-relational-graph-convolutional-networks-for-skeleton-based-human-interaction-recognition"}},"staticQueryHashes":["251939775","401334301","41472230"]}