Immersive 3D Experience

Carldebono

Comfortably sitting in seat 3F, John is watching one of his favourite operas. This close he can see all the details of the set, costumes, and the movements of the music director as he skilfully conducts the orchestra by careful gestures of his baton. He is immersed in the scene, capturing all the details. Then all of a sudden, the doorbell rings. Annoyed, John has to stop the video to see who it is. This could be the mainstream TV experience of the future.

This scene is called free-viewpoint technology that is part of my research at the University of Malta (UoM). Free-viewpoint television allows the user to select a view from which to watch the scene projected on a 3D television. The technology will allow the audience to change their viewpoint when they want, to where they want to be. By moving a slider or by a hand gesture, the user can change perspective, which is an experience currently used in games with their synthetically generated content ­— synthetically generated by a computer game’s graphics engine.

“For free-viewpoint to work, a scene needs to be captured using many cameras”

Today we are used to seeing a single viewpoint. If there are multiple perspectives we usually don’t have any control over them. Free-viewpoint technology will turn this idea on top of its head. The technology is expected to hit the market in the near future, with some companies and universities already experimenting with content and displays. New auto-stereoscopic displays do not need glasses (pictured next page), these displays ‘automatically’ generate a 3D image depending on which angle you view them. A clear example was the promise made by Japan to deliver 3D free-viewpoint coverage of all football games as part of their bid to host the FIFA World Cup in 2022. The bid was unsuccessful, which might delay the technology by a few years.

Locally, my research (and that of my team) deals with the transmission side of the story (pictured). For free-viewpoint to work, a scene needs to be captured using many cameras. The more cameras there are, the more freedom the user has to select the desired view. So many cameras create a lot of data. All the data captured by the cameras has to be transmitted to a 3D device into people’s homes, smartphones, laptops and so on. This transmission needs to pass over a channel, and whether it is fibre cable or wireless, it will always have a limited capacity. Data transmission also costs money. High costs would keep the technology out of our devices for decades.

My job is to make a large amount of data fit in smaller packages. To fit video in a channel we need to compress it. Current transmission of single view video also uses compression to save space on the channel so that more data can be transmitted and save on price. Note that, for example for high definition we have 24 bits per pixel and an image contains 1280 by 720 pixels (720p HD standard), that’s nearly 100,000 pixels for every frame. Since video is around 24-30 frames per second the amount of data being transmitted every minute starts escalating to unfeasible amounts.

Popupology - double 3d / 3d squared my first attempt at an anaglyph photo of a paper structure. you will need those red/blue glasses to view it properly.
Popupology –
double 3d / 3d squared
my first attempt at an anaglyph photo of a paper structure. you will need those red/blue glasses to view it properly.

Free-viewpoint technology would be another big leap in size. Each camera would be sending their own video, which is the same amount of data as we are now getting. If there are ten cameras, you would need to increase channel size by a factor of ten. This makes it highly expensive and unfeasible. For the example above, the network operator needs ten times more space on the network to get the service to your house, making it ten times more expensive than single view. Therefore, research is needed to drastically reduce the amount of data that needs to be transmitted while still keeping high quality images. These advances will make the technology feasible, cheaper, and available for all.

So the golden question is, how are we going to do that? Research, research, and more research. The first attempts by the video research community to solve this problem were to use its vast knowledge of single view transmission and extend it to the new paradigm. Basic single view algorithms (an algorithm is computer code that can perform a specific function, like Google’s search engine) compress video by searching through the picture and finding similarities in space and in time. Then the algorithms send the change, or the error vector, instead of the actual data. The error vector is a measure of imperfections and how it is used by computer scientists to compress data is explained below.

First let us look at the space component. When looking at a picture, it is quite clear that some areas are very similar. The similar areas can be linked and the data grouped together into one reference point. The reference point has to be transmitted with a mathematical representation (vector) that explains to the computer which areas are similar to each other. This reduces the amount of data that needs to be sent.

Secondly, let us analyse the time aspect. Video is a set of images placed one after another and run at 25 or 30 frames per second that gives the illusion of movement and action. To make a video flow seamlessly images that are right after each other are very similar. If we have two images the second one will be very similar to the first, with only a small movement of some parts of the image. Like we do for space, a mathematical relationship can be calculated for the similar areas from one image to the next. The first image can be used as a reference point and for the second we transmit only the vector that explains which pixels have moved and by how much. This greatly reduces the data that needs to be transmitted.

The above techniques are used in single view transmission, with free-viewpoint technology we have a new dimension. We also need to include the space between cameras shooting the same scene. Since the scene is the same there is a lot of similarity between the videos of each camera. The main difference is that of angle and the problem that some objects might be visible from one camera and not from another. Keeping this in mind, a mathematical equation can be constructed that explains which parts of the scene are the same and which are new. A single camera’s video is used as a reference point while its neighbouring cameras only transmit the ‘extra’ information. The other camera can compress their content drastically. In this way the current standard can be extended to free-viewpoint TV.

Compressing free-viewpoint transmissions is complex work. Its complexity is a drawback, mobile devices simply aren’t fast enough to run computer power intensive algorithms. Our research focuses on reducing the complexity of the algorithms. We modify them so that they are faster to run, need less computing power, and still keep the same quality of video, or with minimal losses.

“The road ahead is steep and a lot of work is needed to bring this technology to homes”

We have also explored new ways of reconstructing high quality 3D views in minimum time, using graphical processing units (GPUs). GPUs are commonly used by high-end video games. Video must be reconstructed with a speed of at least 25 pictures per second. This speed must be maintained if we want to build a smooth continuous video in between two real camera positions (picture). A single computer process cannot handle algrothims that can achieve this feat; instead parallel processing (multiple simultaneous computations) is essential. To remove the strain off a main processing unit in a computer processing can be offloaded to a GPU. Algorithms need to be built that use these alternative processing powers. Ours show that we can obtain the necessary speeds to process free-viewpoint 3D video even on mobile devices.

The autostereoscopic display techinique without glasses. Depending on the position of the viewer, the filter directs the left image to the left eye and the right image to the right eye.
The autostereoscopic display techinique without glasses. Depending on the position of the viewer, the filter directs the left image to the left eye and the right image to the right eye.

Since free-viewpoint takes up a large bandwidth on networks, we researched whether these systems can feasibly handle so much data. We considered the use of next generation mobile telephony networks (4G). Naturally they offer more channel space, we wanted to see how many users they can handle at different screen resolutions. We showed that the technology can be used only using a limited number of cameras. The number of users is directly related to the resolution used, with a lower resolution needing less data and allowing more views or users. This research came up with design solutions for the network’s architecture and broadcasting techniques needed to minimise delays.

The road ahead is steep and a lot of work is needed to bring this technology to homes. My vision is that in the near future we will be consuming 3D content and free-viewpoint technology in a seamless and immersive way in our homes and mobile devices. So for now sit back and imagine what watching an opera or football match on TV would look like in a few years’ time.

Through the Looking Glass

What’s your favourite game? Pacman? Doom? World of Warcraft? Most of us have spent hours immersed in video games, many still do. Prof. Gordon Calleja studies why and how we get so involved in games. Science writer Dr Sedeer El-Showk found out about Calleja’s latest book and game that are gaining worldwide fame.

Continue reading

The key to networking

I first heard about COST (European Cooperation in Science and Technology, a networking platform for scientists www.cost.eu) way back in 1996 during a pharmacokinetics meeting in Athens. Some participants mentioned that their attendance had been funded by COST. So on my return I contacted the Malta Council of Science and Technology to try and obtain more information. When I learnt that COST funds EU networking I quickly applied to become a member of a COST action (this is what COST calls a network). After bureaucratic leaps and bounds I become Malta’s representative on a COST action. It certainly opened new horizons to me and the networks I formed with top researchers in Europe were unique.

By 2010 my enthusiasm resulted in MCST nominating me as Malta national contact point for COST. It has been of huge satisfaction that in these three brief years Malta’s participation has risen from 6 actions to over 100. Over 150 Maltese researchers take part in COST.

Why is COST so important for Malta?

The complaint I hear most often in Malta, not only in academic circles but also among SMEs (small to medium enterprises), is that research in science is only for the elite, that it is too high brow and that it is not relevant to Malta. COST proves otherwise. What else could link disaster bioethics, to colour and space in cultural heritage to the comparison of European prostitution policies, with submerged prehistoric archaeology? Other links include the quality of suburban building stocks, integrated fire engineering and response, and language impairment in a multilingual society. COST also funds networks across a whole spectrum of research from the humanities to the fundamental sciences including string theory to childbirth in various cultures.

Participating in a COST action involves very simple administrative and funding procedures. For once, our small size is an added advantage since every COST country is allowed to nominate two members to participate in each action, putting Malta COST researchers at par with researchers from much larger countries. Achieving these results has not been easy, since many researchers hesitate and require persistent prodding. There are frequent reminders and one to one meetings to persuade them to participate. It has been a real eye-opener meeting researchers in Malta from different disciplines and learning about their research.

Deciding to participate in COST may seem a small step to some, an added administrative burden to others, while some see it as another travel commitment. COST offers the response to the conundrum of how to overcome our physical (and perhaps in some instances also mental) insularity.  You should not let this opportunity pass…

COST in Malta is managed by the Malta Council for Science and Technology. For more information see www.mcst.gov.mt/networking/cost or contact Dr Janet Mifsud, COST CNC, (+356 23402582/2845, janet.mifsud@um.edu.mt) or cost.mcst@gov.mt

A Greener Malta

The facts are clear: Malta has a challenge. It needs to build up a 10% electricity generation from renewables by 2020. Beyond that, it needs cleaner air, cleverer homes, and a consistent power source for its people and economy. The big question is how can we enjoy Malta’s newly won benefits of the developing world without compromising our environment? This challenge motivates researchers worldwide. Malta is doing its bit in environmental engineering: developing green skies, green energy, green homes, and the opportunity for a green Malta.

Greener Skies

Nearly 100,000 commercial flights take off every day worldwide causing 2% of man-made carbon dioxide (CO2) emissions per year. In Malta, over one million tourists visit each year and flights are critical in connecting an island to the outside world. How can we address Malta’s economic needs with a growing public concern on airplane pollution, noise, and contribution towards climate change? A €1.6 billion Clean Sky project funded by the European Commission is trying to make this apparent conflict work. It brings together major European industrial partners, research establishments, and academia to develop breakthrough technologies for the air transport industry. In Malta, the Department of Electronic Systems Engineering is optimising flight paths known as trajectories.

The idea: reduce a plane’s flight path, reduce the time it is airborne, lower CO2 release. Computer algorithms can find a plane’s best trajectory and minimise air pollutants and noise. Malta is handling a part of the software development to optimise flight paths. The purpose of the software platform being developed by the Maltese team is to allow partners to bring together their optimisation models and tools. This will allow the international consortium to solve the complex algorithms that come with flight trajectory problems. 

Another project called Clean Flight is tackling local airplane flight paths. Current flight trajectory calculations are based on lowering costs. This does not necessarily mean that these trajectories provide the least pollution for our skies. Clean Flight’s approach is to lower flight costs by being green. Usually these two qualities match: burn less fuel, spend less, pollute less. Commercial aircrafts should have new flight paths for Malta. 

Harnessing the winds

Every time we switch on an appliance at home or at our workplace we are consuming energy. And energy means money and pollution. Pollution can be nullified by using renewables. Denmark, famous for windmills, has invested heavily in wind energy. By 2011 it generated around 26% of the total electricity demand through wind farms. Denmark also has one of the lowest electricity prices in the EU.

360_windmills_0731
An offshore windmill farm located in the Oeresund, three km from Copenhagen harbour.

Wind energy’s success has come with a backlash. The rapid increase in land use has caused public outcry on despoiling views, animal activists are worried about bird deaths, and increased pressure on limited land availability. In Malta land is of premium value. Green and open spaces are limited. Out at sea these problems could disappear and floating offshore wind structures may provide accessibility to deeper waters. Deeper seas have other plusses. Out there, the wind speeds are higher and more consistent, which makes electricity generation more realistic. The flipside is the expense in sending the precious electrical energy back to shore to power homes.

Floating wind turbines may be the key to fulfil Malta’s renewable energy targets. Malta has agreed to a 10% electricity generation from renewables by 2020 with the EU. If the country fails to meet its target it will be smacked with a huge fine.To give more opportunities for the government to reach this target the Faculty of Engineering is pushing new research into wind energy. There are major differences between floating and fixed offshore structures. Waves cause ever changing stresses on the turbine’s structure and bobbing movement could change the turbine blades’ aerodynamics that reduces power output.

The researchers at the Faculty’s Fluids Lab are testing a model floating wind turbine. Till now, the experiments have examined the change in power experienced by a wind turbine’s rotors when bobbing up and down on waves. The data has been inputted into a computer model to simulate large-scale floating wind farms. By simulating the air flow the drop in generation can be better understood. The simulations are based on the application of free-wake vortex methods. Since the air flow changes with the oscillation of the platform and therefore with time, these methods are capable of capturing the changes in the wake formed by the rotor.

Large wind farms face the problem of having generators and gearboxes mounted on each turbine at great cost. Instead the energy generation could be centralised, with individual turbines pumping seawater towards a central station which makes use of a positive displacement pump. This concept would mean that a centralised hydroelectric power station could be located on our shores.

Replacing every wind turbine’s gearbox and generator with a hydraulic pump offers many advantages. It reduces expense, by minimising the number of moving parts’ maintenance costs, and make a lighter turbine. Wind farms that pump water could also be easier to combine with wave energy, energy storage systems, and reverse osmosis plants that use up a big chunk of Malta’s electricity to make drinking water from the Mediterranean.

“Floating wind turbines may be the key to fulfill Malta’s renewable energy targets”

The only way is up

Normally we think of wind turbines as a tall central structure with a generator on top and propeller-style blades powering everything. But this is not the only possibility. Blades can spin around the turbines’ central structure doing away with expensive maintenance costs and complex gears to turn the blades into the wind. These Vertical Axis Wind Turbines (VAWTs) do not need to be oriented into a specific wind direction. Their problem is a requirement for higher wind speeds before they start to spin.QR5-VAWT

Dr Ing. Pierluigi Mollicone is coordinating a project that is coming up with new design concepts for this type of wind turbine. By working with both local academics and industrial players, a state-of-the-art design has to improve both the starting speed and the capability of controlling the turbine at varying wind speeds. Starting off from a conceptual idea, the design is then detailed and developed in the first step to make a wind turbine. The computer design then needs to be tested for aerodynamics and structural integrity—does it spin well and can it take a beating? The computational model then needs to be translated into a real world structure and tested in a wind tunnel, with further experiments back and forth needed to come up with a new wind turbine.

Malta’s very own windmill: restarted

The Raddiena or Chicago windmill is a well-known sight in rural Malta. These windmills harness the winds to draw water from the water table and irrigate fields. In 2001, 300 windmills were listed across Malta and Gozo. Unfortunately, the introduction of electricity has led farmers to abandon this clean source. Many windmills are gradually deteriorating. Dr Ing. Tonio Sant (Department of Mechanical Engineering) and his team are developing a new wind turbine concept to replace these badly damaged Chicago windmills. Together with the Ministry for Resources and Rural Affairs they are upgrading the rotor design structure’s aerodynamics to improve water-pumping efficiency. At the same time, the researchers want to maintain the original visual appeal of a multi-bladed rotor. The turbine will also produce electricity and be grid-connected. It won’t just pump up water but also provide clean energy.

Green Homes

Malta is covered in houses. Covering their roofs with PV (photovoltaic) panels is a way we could all help by making renewable energy. In the past five years, Malta has seen a drastic increase in PV panel use. Electricity generation is shifting from a centralised power station to our homes. Distributed generation is characterised by small-scale electricity generation, deployed near the point of use: our homes.

Currently our national grid cannot handle large PV installations. The stability of the grid may be compromised leading to power outages. And we all know what being in the dark for a few hours means. No Internet, no TV, no cold drinks in summer. The storage of electric energy can be used to balance the generation and consumption demands for a single household or company. Excess energy generated during periods of high generation can be stored. This stored energy can then be used when supply cannot meet demand, perhaps when using several heaters on a cloudy day.

Microgrids can solve these challenges. These grids are low voltage (electrical distribution within a home) or medium voltage (electrical distribution within a neighborhood) electrical distribution networks designed to supply small electrical loads. They are needed to hook up PV panels to a small community like a housing estate, university, schools, shopping mall, or industrial area. They consist of three major components. The homes equipped with PV or wind turbine installations, systems to store the energy generated, and other electricity users connected to the grid.

Microgrids generate energy near where it will be used. This increases reliability and reduces losses due to long transmission lines. Microgrids can also be used to provide electricity in remote locations unconnected to a main grid. Researchers are developing new methods to reliably operate and control microgrids across an island. They want to implement a low voltage microgrid capable of reliable operation that is connected to the national grid.

Many households have PV panels and solar water heaters. Local researchers are combining the two systems. For a sunny country like Malta, such a system makes perfect sense. One major advantage of a combined system is efficient conversion of concentrated solar energy to heat energy. The homeowner can then flip a switch to either generate electricity or heat water.

A parabolic trough, a curved surface, is used to focus the sun’s rays onto a fluid. The heated fluid can reach temperatures of up to 300°C. Hot enough to power a stirling engine used to generate electricity or pump water.

Greener Seas

6059572064_282c5b10dd_z
il-Qala ta’ San Niklaw, Comino

The Mediterranean has over 150 million people living on its coast. A quarter of a million fishermen live off its fish. Even closer to home, the sea around Malta sees one third of the world’s shipping. The only way this sea can survive is by knowing how much we are polluting and exploiting it by monitoring it. Then the effect needs to be evaluated and the situation managed by administrators around the Sea to balance development and environmental health.

A study that will help gather information about maritime traffic across the Straits of Sicily is being carried out by the Department of Mechanical Engineering and the University of Catania. By quantifying the emissions produced, they will find out the consequences of these emissions on the local plant and animal life. The project aims at supporting the monitoring of pollution at sea in the Straits of Sicily and so contributing to future legislation at national and European level.

Engineers designed a towfish that will be used to monitor a number of pollutants in the Mediterranean Sea. A towfish is an underwater platform that is towed behind a surface boat and can reach a depth of 50m. The towfish will be equipped with an HD camera that can take images of zooplankton and phytoplankton in order to study colonies that exist in the Straits of Sicily. Another HD camera will be used to monitor swarms of jellyfish and their location.

A Green Malta

Pollution bothers everyone with dirty atmospheres and smelly odours. Pollution also makes us sick and causes many health problems including birth defects, and burdens health institutes. We all have a role to play to protect and safeguard our environment. Whether it is our skies, seas, or our homes we all have to do our part. A greener Malta means a greener future for all of us.

In 30 years’ time the electric energy we use in our homes and at the workplace will come mainly from renewable energy sources. We will all be driving electric vehicles. The familiar black clouds of smoke from dirty engines will be a thing of the past. But these advances in technology will not be possible without government funding, industrial collaboration, and the sweat of engineers and researchers to find exciting solutions to power nations and our green homes. 

Watch more here: https://www.youtube.com/watch?v=VwCNHvD2gqY

KSU joins list of donors

In a typical case of charity beginning at home, the Kunsill Studenti Universitarji (KSU) has donated €1,000 towards the research trust of the University of Malta. Speaking during the presentation of the donation, Mr Mario Cachia, former president of the KSU pledged that the KSU shall venture to promote the objectives of the Trust among the students it represents, and encourage them to support it, even by organising activities to raise funds for specific future research. 

Can you die of boredom?

A tricky question since boredom is a mental state. Usually it is triggered by an uninspiring, monotonous environment and/or certain brain chemicals like dopamine (more on dopamine). People who produce less dopamine, or are less sensitive to it, are bored more easily.

People bored for a very long time can suffer from clinical depression, which surely reduces lifespan. Other boredom sufferers have ADHD (attention deficit hyperactive disorder) and extreme ways of combating boredom can be equally dangerous. They could abuse drugs and alcohol or seek thrills in risky sports. Others could become sexually promiscuous. All of these behaviours reduce lifespan. 

Send your questions to think@um.edu.mt and we’ll find out if it’s the truth or just a fib!

An Alternative Currency

Mario Frendo

Malta — a tiny Island, a minute social reality, a precursory canovaccio of European unification — has a unique asset it ought to be prouder of: Culture. For millennia our bonsai place has attracted a continuum of passing civilisations leaving behind them a most colourful and diverse compendium of customs, behaviours, artistic expressions, and intellectual attitudes. Malta’s investment in this unique legacy should not be limited to conservation. It needs to be kept alive through constant support of the contemporary expression of its youth. This attitude will certainly transform our Culture into a most effective and efficient currency of change and growth.

To drive or not to drive

Rush hours, feasts, festivals, beaches in summer, Paceville on Saturday night, all have one thing in common: traffic. Malta has one of the largest traffic problems in the world. Researchers at the University of Malta are trying to figure out what can be done to ease road rage and reduce drivers’ lost time.

Continue reading