Dallas tech company Continual Engine, which offers artificial intelligence-driven accessibility products, is patenting a method for translating image-based math content into more accessible formats like audio or Braille.
Fig. 1. Sample mathematical equations and their corresponding LaTeX representations and text descriptions.
The technology involves a server system that receives images of math content from users. The system takes an image and determines its set of characteristics. This set is then converted into text, ready to be relayed back to the user in a suitable format.
A machine learning equation model correlates portions of the image to tokens representing the math content. The model predicts the sequence of the tokens and converts it to text by replacing each token with a corresponding term.
Continual Engine states that the tokens would be identified according to LaTeX representation. LaTeX documents consist of plain text and do not contain hidden formatting codes or binary instructions. This simplicity makes LaTeX a popular software system in academia, where scientific and multilingual materials must be relayed accurately. It is the LaTeX representation of mathematical equations that the server system relies on to generate accessible content.
The patent application also covers AI-driven conversions of graphs. The server system would employ an ML graph model that works in a similar manner to the ML equation model: correlating characteristics to image portions before predicting the content’s characteristic set. The approach differs in that the set of characteristics includes the graph’s type, line slopes, vertex, roots, and parabola direction.
Fig. 2. Sample graph and their corresponding text description.
Up to 20% of the population has a language-based learning disability. Education is becoming increasingly computer-based, more so due to the COVID-19 lockdowns. Accessibility is crucial to ensuring a fifth of the student population would not be left behind.
Continual Engine’s proposed system is an example of technology being used to raise the quality of education for people with disabilities, even when advanced learning materials are involved. The company says AI can be leveraged to provide accurate and cost-effective assistive services in STEM fields.
The featured patent application, “Providing Math Content For Visually Impaired”, was filed with the USPTO on December 24, 2020 and published thereafter on September 23, 2021. The listed applicant is Continual Engine IP, LLP. The listed inventors are Gautam Kapoor, Vikram Prasanna, Mousumi Kapoor, Pranshu Agarwal, Ravi Vemagiri, and Ranjeet Kumar.
