This smart TV displays health information based on viewers’ facial features

FaceHeart, a Taiwanese AI company, is patenting a health monitoring function for smart televisions, one that bases its data on a viewer’s facial features.

The average person will watch over 78,705 hours of television over their lifetime, according to a British poll commissioned by LG Electronics. Three in five respondents admit they would be “lost” without their TV set. These devices play a significant part in many people’s lives, a fact further held up by streaming sites, whose on-demand offerings edged over cable and broadcast TV. FaceHeart’s proposed system aimed to add a health-conscious dimension to an otherwise unhealthy, yet ubiquitous endeavor.

Excessive television consumption is linked to greater adverse health outcomes. This sedentary activity is particularly harmful: watching TV leads to worse health than other forms of sitting. A large-scale, long-term Spanish study directly associated TV watching with all-cause mortality, more than comparable activities such as using a computer or driving.

By providing a convenient way for television viewers to monitor their vitals, FaceHeart may inspire better health decisions in people who stand to benefit more from them. It starts with integrating an image-based physiological signal (IBPS) measuring device into the TV.

Fig. 1. A first stereo diagram of a home infotainment system with a non-contact IBPS measurement device [11], comprising an image capturing module [111] and a core processor module [112].

The “home infotainment system” described in the patent application comprises an IBPS device coupled to a television’s display processor. The IBPS device itself comprises a core processor module coupled to an image capturing module. The filing claimed it will be able to detect a person’s heart rate, respiratory rate, blood volume, arterial oxygen saturation, and blood pressure, among others.

To do this, a camera is configured to capture at least one image frame of a person, which is analyzed for specific portions of the face or chest. The IBPS device extracts a user’s physiological indices from this information, which can be displayed in the form of graphs, diagrams, and/or numeric values. The television can keep databases for user identity lists and reference facial features, as well as activity and physiological indices.

Fig. 2. Illustration of a home infotainment system with a non-contact IBPS measurement function.

An alternative embodiment of the system features an independent camera device, which can be paired with home infotainment systems utilizing a projector. The device may enable health monitoring features for television systems without a built-in IBPS measurement function.

The patent application states that a smart TV can also further upload the physiological indices to a cloud server, thereby achieving a cloud management service for home healthcare. A growing number of hospitals and clinics are already taking advantage of cloud computing, streamlining medical record-sharing and forming the backbone of telehealth applications. Bringing cloud healthcare to households can grant doctors valuable insight on patients without having to bring them in for testing or monitoring.

FaceHeart specializes in artificial intelligence and deep learning, with a focus on delivering real-time physiological information without physical contact. The company has also been pushing for video-based vital sign monitoring for pandemic prevention. This technology, along with the one described in the patent application, are touted for its non-contact advantage over common health monitoring methods, such as wearing a smart band or smart watch.

Though the patent application does not go into FaceHeart’s method for measuring vital signs, researchers from Microsoft and the University of Washington have come up with similar technology. It involves capturing at least 18 seconds of video of a person, which undergoes a machine learning algorithm, trained using three datasets of videos with corresponding health statistics. The researchers even calibrated the system to adjust for a person’s skin tone, age, and facial hair, as well as factors such as the video’s background and lighting.

The featured patent application, “Home Infotainment System Having Function of Non-contact Imaging-based Physiological Signal Measurement”, was filed with the USPTO on August 27, 2020 and published thereafter on July 22, 2021. The listed applicant is FaceHeart Inc. The listed inventors are Chung-shine Huang, Tsuey-huey Shiue.