Advantages
- Achieves a high validation accuracy of 99.05% with a Vision Transformer (ViT) neural network model, demonstrating superior performance compared to other common deep learning models.
- Requires no external reference devices or special writing surfaces, allowing it to be used anywhere, including construction sites and underwater.
- Can be used as a digital pen and also has potential applications such as a learning aid for children to practice writing characters.
Technology Overview & Background
In our increasingly digitized world, the efficient digitization of handwritten notes has become a critical challenge in many fields. Conventional digital pens have been limited by portability, cost, and usage environments, as they typically require dedicated smart pads or special paper. Additionally, handwritten content was saved as images, requiring a separate process for text conversion. Studies so far using only an Inertial Measurement Unit (IMU) sensor have struggled to achieve sufficient recognition accuracy for small-scale movements, such as note-taking, due to the sensor’s susceptibility to noise and drift errors.
To address these challenges, the researcher developed a new AI-powered smart digital pen. It compactly integrates three miniature force sensors, a 6-channel inertial sensor, and a microcomputer in addition to a standard ballpoint ink chamber. This design allows for high-precision capture of both pen-tip movement and writing pressure. The collected data is then analyzed by a deep learning model to output characters in real-time.
The most significant feature of this technology is its ability to achieve high-precision recognition of small alphanumeric characters (36 types: digits 0-9 and lowercase Latin letters a-z), which was difficult to do with an IMU alone. This is accomplished by combining inertial data with information from the force sensors. As a result, even in locations where paper cannot be used, such as construction sites or underwater, users can trace on a hard surface to record and save handwritten text as digital text in real-time. In addition to its use as a digital pen, the technology can also be applied to learning aids for children to practice writing characters.
 |
Data
- Handwritten data for 36 alphanumeric characters (10 digits and 26 lowercase Latin letters) was collected from six test subjects. After the dataset was appropriately trimmed and reconstructed, a neural network model (Vision Transformer; ViT) was trained and validated using deep learning methods, yielding an accuracy of 99.05%.
Publication(s)
Alemayoh T.T., et al., “Deep-Learning-Based Character Recognition from Handwriting Motion Data Captured Using IMU and Force Sensors.” Sensors (2022), 22, 7840.
DOI: https://doi.org/10.3390/s22207840
Patent(s)
JP 7644963 (B2)
Principal Investigator & Academic Institution
Dr. Jae Hoon LEE (Professor, Graduate School of Science and Engineering, Ehime University).
Development Stage & Future Research Plans
Current Stage: Recognition of 36 alphanumeric characters (10 digits and 26 lowercase letters).
Next Stages:
- Recognition of uppercase letters, hiragana, katakana, and special characters.
- Recognition of whole words written continuously, rather than character by character.
- Integration of a small camera for leveraging image information.
- Development of learning aids for writing practice.
Expectations
Currently, the researcher has progressed to the point of recognizing numbers and the alphabet, and he hopes to tackle more complex characters and symbols like hiragana, katakana, and special characters in the future. We are seeking stationery and learning device companies interested in this invention to partner with us in its further development.
Confidential data can be disclosed after signing a non-disclosure agreement with Ehime University, and meetings with the researchers are also possible. Please feel free to contact us for more information.
Project ID:JT-04922