Mediapipe手部檢測框架結合Inflated 3DCNN網路之臺灣手語單字辨識

手語辨識的研究一直都許多研究學者經常關注的問題，但在以往的研究中其方法大多需要在指定的背景及服裝，或者使用複雜的設備來完成，本篇論文使用容易取得的 webcam 做為資料收集及測試的設備，並以近年來辨識動態動作具有良好準確性的 Inflated 3DCNN 網路為基底進而改良，加上 Mediapipe 框架提取的手部骨架資訊整合至訓練樣本的資訊中，解決了雙流 Inflated 3DCNN 網路訓練方法中較無法提取細部手指訊息的缺點，進而讓此系統能夠更廣泛的使用在各種手語單字的辨識上。
為使其訓練及測試資料集有一定的公信力，本篇論文也開發了一個結合手語教學及資料收集的網頁，並邀請 15 名研究生在不同的背景及裝扮下，使用此教學網頁進行資料的收集，接著使用改良後的 Inflated 3DCNN 網路進行訓練及驗證，並收集與原訓練及驗證資料集完全不同背景及裝束資料以供測試，其測試結果在 98個單字下達到 97.4%準確度，最後將訓練完的模型應用至即時的手語單字辨識系統中。

The research of sign language recognition has always been a problem that many
researchers pay attention to, but most of the previous research needs to be completed in the specified background and clothing, or using complex equipment.This paper uses the readily available webcam as the equipment for data collection and testing, and uses the Inflated 3DCNN network, which has good accuracy in identifying dynamic actions in recent years, as the base and improves it, and the hand skeleton features extracted by the Mediapipe framework are combined to Among the feature of the training samples,Resolved the lack of detailed finger information that cannot be extracted in the Inflated 3DCNN network training method, and this system can be more widely used in the recognition of different sign language words.
In order to make the training and testing data set have certain credibility, this paper also developed a web page that combines sign language teaching and data collection, and invited 15 master students in different backgrounds and clothing to use this teaching web page to collect data, and then use the improved Inflated 3DCNN network for training and verification, and collect background and clothing data completely different from the original training and verification data sets for testing. The test results reach 97.4% accuracy in 98 words. The trained model is applied to a real-time sign language word recognition system.