Using media for representations in Digital Twins – by Fontys

Fontys

Strawberry Fields project learnings and insights

For the Digital Twin Academy (EUreg research project) the Fontys HTES research group created the ‘Strawberry Fields’ project. The project consists of two learning modules. One to explain some of the concepts of digital twinning and the business opportunities for digital twinning within the agricultural domain, the other containing practical tutorials on how to build a simple digital twin of a Strawberry plant in form of a dashboard using a low-cost sensor for data capturing. The tutorials explain the process of capturing and cleaning data and representing it in the form of data visualisations, 3D, and using it in an augmented reality web-app. Furthermore, it is explained how one could extend the dashboard with a machine learning algorithm to predict the ripeness of the strawberry fruits.

Although the project was developed with an audience in mind that has no digital twinning expertise, it nevertheless led to interesting insights and learnings that can be relevant for more advanced digital twin projects as well – the challenge of data representation and the entropy level of the twinned phenomenon.

Data representation and the perception of media

In the practice of digital twinning, a twin is developed following three steps: the digital shadow capturing data, the digital model providing meaning, and the digital twin as the real-time co-existence of the physical and digital version of the same phenomenon. Sooner or later in this process the digital counterpart needs to ‘communicate’ (representing and expressing itself) with its environment. Especially when this environment includes humans it leads to the use of media as data-to-human interface since humans are not very well equipped to give meaning to the data itself. However, the use of media leads to choices and relies on human perception. As known from media studies, media mediates between truth (data) and its perception. In other words, media affects meaning or as Marshall McLuhan puts it – the medium is the message .

The choice – business case

An example of choice can be found in the streamgraph visualisation in the Strawberry Fields dashboard showing the air and soil humidity over time in a ‘mirrored’ view. From a visualisation point of view the advantage of such a visualisation is that humans are very capable at estimating value expression in one dimension (in this case the height representing the humidity between 0-100%). The streamgraph helps us to understand the total humidity of the system (soil plus air). This choice effects our perception because if the two values would have been printed overlapping each other (air minus soil) the meaning would have been focussed on understanding the difference between soil and air humidity and not on the humidity of the system in total. Although both choices can be regarded as good practices from a visualisation point of view, the value for the digital twin derives from the business case underlying the choice in visualisation. Finally, it is the relevant agricultural domain knowledge that decides whether a visualisation choice is good or not. This is what is known as design for purpose.

Image 1 – the streamgraph of the strawberry fields dashboard showing the air and soil humidity (in percentages) over time.

Models and perception of (3D) reality

Another example we find in the 3D virtual reality part of the dashboard. While the digital model (step two in the process of twinning) aims at a simplification in order to make the phenomenon comprehensible, the 3D model strives to be more and more detailed to fulfil its need for realism. From the theory of semiotics (the theory of signs) we know that the 3D model of the strawberry used in the dashboard is no more than an iconic sign. This means that no matter how much detail we add to the 3D model such as its shape or texture, it remains a representation of ‘a strawberry in general’ and not the representation of ‘this specific strawberry fruit’ we’re looking at in the real world. This might be used as an argument to choose a different kind of technology for the 3D representation and not the polygons 3D modelling based approach of this dashboard. We see alternative technologies being explored and developed but still with a lot of limitations when it comes to the 3D modelling of biological phenomena such as plants.

Image 2 – The strawberry model in VR. The VR represents the light intensity that is measured with the sensor and the prediction of the machine learning algorithm is used to texture the model representing four stages of ripeness.

For the time being it is good to be aware of the foundation of 3D modelling that finds its origin mainly in the production of media products such as 3D games and cinema. These productions do not aim at representing reality but at creating cinematographic reality instead. This means that decisions on what a proficient 3D model is are governed from a different perspective than the decisions made in the digital (twin) model.

Manufactured environments

Our digital twin exploration starts with a single strawberry plant in a pot on our kitchen table which seems like a pretty straightforward environment for twinning. However, as we proceed in creating the digital twin, it turns out that this environment poses several problems when it comes to its digitalisation. An example is the natural light and the daylight cycle. During the development of the VR we use and discuss a mix of technologies such as VR lighting, High Dynamic Ranged texturing, tone-mapping, alpha channel and an external day cycle library to address this challenge. The implementation of the growing lamp is surprisingly simple as we can almost literally use the specifications of an existing growing lamp as arguments for our VR lamp object. The example demonstrates that, at this time, successful digital twinning seems to be related with the level of entropy in the phenomenon. While the phenomenon may be complex or relatively simple, it is the unpredictability that governs the level of complexity when it comes to developing a digital twin.

Conclusion

While digital twinning holds an interesting promise for example in future I4.0 developments, using media in the digital counterpart comes with its challenges especially when it comes to things related with meaning and understanding. Addressing these challenges demands a multi-disciplinary approach and teams bringing the best of ‘both’ worlds together with the objective of creating high-end digital twins.

Acknowledgement: this work is developed by the Fontys research group High Tech Embedded Software, as part of the Interreg EMR project Digital Twin Academy.

Author: Guido Segers

Fontys_logo

Fontys

Share

Others technology sheets

This website uses cookies in order to offer you optimal navigation and to provide us with statistics without knowing your personal data. By continuing to browse this site, you agree to the use of cookies, in accordance with our cookie policy.