Immersive 3D Experience

Carldebono

Comfortably sitting in seat 3F, John is watching one of his favourite operas. This close he can see all the details of the set, costumes, and the movements of the music director as he skilfully conducts the orchestra by careful gestures of his baton. He is immersed in the scene, capturing all the details. Then all of a sudden, the doorbell rings. Annoyed, John has to stop the video to see who it is. This could be the mainstream TV experience of the future.

This scene is called free-viewpoint technology that is part of my research at the University of Malta (UoM). Free-viewpoint television allows the user to select a view from which to watch the scene projected on a 3D television. The technology will allow the audience to change their viewpoint when they want, to where they want to be. By moving a slider or by a hand gesture, the user can change perspective, which is an experience currently used in games with their synthetically generated content ­— synthetically generated by a computer game’s graphics engine.

“For free-viewpoint to work, a scene needs to be captured using many cameras”

Today we are used to seeing a single viewpoint. If there are multiple perspectives we usually don’t have any control over them. Free-viewpoint technology will turn this idea on top of its head. The technology is expected to hit the market in the near future, with some companies and universities already experimenting with content and displays. New auto-stereoscopic displays do not need glasses (pictured next page), these displays ‘automatically’ generate a 3D image depending on which angle you view them. A clear example was the promise made by Japan to deliver 3D free-viewpoint coverage of all football games as part of their bid to host the FIFA World Cup in 2022. The bid was unsuccessful, which might delay the technology by a few years.

Locally, my research (and that of my team) deals with the transmission side of the story (pictured). For free-viewpoint to work, a scene needs to be captured using many cameras. The more cameras there are, the more freedom the user has to select the desired view. So many cameras create a lot of data. All the data captured by the cameras has to be transmitted to a 3D device into people’s homes, smartphones, laptops and so on. This transmission needs to pass over a channel, and whether it is fibre cable or wireless, it will always have a limited capacity. Data transmission also costs money. High costs would keep the technology out of our devices for decades.

My job is to make a large amount of data fit in smaller packages. To fit video in a channel we need to compress it. Current transmission of single view video also uses compression to save space on the channel so that more data can be transmitted and save on price. Note that, for example for high definition we have 24 bits per pixel and an image contains 1280 by 720 pixels (720p HD standard), that’s nearly 100,000 pixels for every frame. Since video is around 24-30 frames per second the amount of data being transmitted every minute starts escalating to unfeasible amounts.

Popupology - double 3d / 3d squared my first attempt at an anaglyph photo of a paper structure. you will need those red/blue glasses to view it properly.
Popupology –
double 3d / 3d squared
my first attempt at an anaglyph photo of a paper structure. you will need those red/blue glasses to view it properly.

Free-viewpoint technology would be another big leap in size. Each camera would be sending their own video, which is the same amount of data as we are now getting. If there are ten cameras, you would need to increase channel size by a factor of ten. This makes it highly expensive and unfeasible. For the example above, the network operator needs ten times more space on the network to get the service to your house, making it ten times more expensive than single view. Therefore, research is needed to drastically reduce the amount of data that needs to be transmitted while still keeping high quality images. These advances will make the technology feasible, cheaper, and available for all.

So the golden question is, how are we going to do that? Research, research, and more research. The first attempts by the video research community to solve this problem were to use its vast knowledge of single view transmission and extend it to the new paradigm. Basic single view algorithms (an algorithm is computer code that can perform a specific function, like Google’s search engine) compress video by searching through the picture and finding similarities in space and in time. Then the algorithms send the change, or the error vector, instead of the actual data. The error vector is a measure of imperfections and how it is used by computer scientists to compress data is explained below.

First let us look at the space component. When looking at a picture, it is quite clear that some areas are very similar. The similar areas can be linked and the data grouped together into one reference point. The reference point has to be transmitted with a mathematical representation (vector) that explains to the computer which areas are similar to each other. This reduces the amount of data that needs to be sent.

Secondly, let us analyse the time aspect. Video is a set of images placed one after another and run at 25 or 30 frames per second that gives the illusion of movement and action. To make a video flow seamlessly images that are right after each other are very similar. If we have two images the second one will be very similar to the first, with only a small movement of some parts of the image. Like we do for space, a mathematical relationship can be calculated for the similar areas from one image to the next. The first image can be used as a reference point and for the second we transmit only the vector that explains which pixels have moved and by how much. This greatly reduces the data that needs to be transmitted.

The above techniques are used in single view transmission, with free-viewpoint technology we have a new dimension. We also need to include the space between cameras shooting the same scene. Since the scene is the same there is a lot of similarity between the videos of each camera. The main difference is that of angle and the problem that some objects might be visible from one camera and not from another. Keeping this in mind, a mathematical equation can be constructed that explains which parts of the scene are the same and which are new. A single camera’s video is used as a reference point while its neighbouring cameras only transmit the ‘extra’ information. The other camera can compress their content drastically. In this way the current standard can be extended to free-viewpoint TV.

Compressing free-viewpoint transmissions is complex work. Its complexity is a drawback, mobile devices simply aren’t fast enough to run computer power intensive algorithms. Our research focuses on reducing the complexity of the algorithms. We modify them so that they are faster to run, need less computing power, and still keep the same quality of video, or with minimal losses.

“The road ahead is steep and a lot of work is needed to bring this technology to homes”

We have also explored new ways of reconstructing high quality 3D views in minimum time, using graphical processing units (GPUs). GPUs are commonly used by high-end video games. Video must be reconstructed with a speed of at least 25 pictures per second. This speed must be maintained if we want to build a smooth continuous video in between two real camera positions (picture). A single computer process cannot handle algrothims that can achieve this feat; instead parallel processing (multiple simultaneous computations) is essential. To remove the strain off a main processing unit in a computer processing can be offloaded to a GPU. Algorithms need to be built that use these alternative processing powers. Ours show that we can obtain the necessary speeds to process free-viewpoint 3D video even on mobile devices.

The autostereoscopic display techinique without glasses. Depending on the position of the viewer, the filter directs the left image to the left eye and the right image to the right eye.
The autostereoscopic display techinique without glasses. Depending on the position of the viewer, the filter directs the left image to the left eye and the right image to the right eye.

Since free-viewpoint takes up a large bandwidth on networks, we researched whether these systems can feasibly handle so much data. We considered the use of next generation mobile telephony networks (4G). Naturally they offer more channel space, we wanted to see how many users they can handle at different screen resolutions. We showed that the technology can be used only using a limited number of cameras. The number of users is directly related to the resolution used, with a lower resolution needing less data and allowing more views or users. This research came up with design solutions for the network’s architecture and broadcasting techniques needed to minimise delays.

The road ahead is steep and a lot of work is needed to bring this technology to homes. My vision is that in the near future we will be consuming 3D content and free-viewpoint technology in a seamless and immersive way in our homes and mobile devices. So for now sit back and imagine what watching an opera or football match on TV would look like in a few years’ time.

Etna

The ancients saw volcanoes as the wrath of their mighty gods. Volcanoes have been blamed for clearing whole towns, even planet-wide extinctions. A local team based in Gozo has just found out if Etna affects the Maltese Islands. Words by Dr Edward Duca.

Continue reading

Racing into the Future

LogoWay back in 2007, a dedicated group of six people put together a formula-style race car in just six months to compete in a prestigious international competition called FSAE. Since then no other team has participated. Students were always interested to build a racing car but found it too hard to actually carry out — the underlying logistics were simply too much.

In December 2012, a group of motivated university students founded the University of Malta Racing (UoMR) team. Their mission statement: ‘To encourage and facilitate students of the University of Malta to unite together as a team in the planning, design and construction of a Formula-style race car and to participate in the Formula SAE, or similar competitions.” They were brought together by a love of cars, engines, speed and a competitive spirit.

Welding-BenchThe 2007 team placed 17th out of 20 teams. The new team has stiff competition and huge challenges to overcome for the upcoming competition in July 2014. Foreign universities compete every year and build a database of knowledge and experience which students use to continue improving their cars. For the UoM to compete eff
ectively with top-class international universities, there must be a strong framework which supports and encourages students from every faculty, especially the Faculty of Engineering. To overcome this challenge the team extensively researched the parts, materials needed and procedure to build a competitive vehicle. The PR and Finance team of the UoMR also drew up a sponsorship proposal, which was used to attract sponsors and collaborators. Without them the project would not be possible.

The team is currently working on the car’s design. At the same time they are fabricating some parts and structures inside their workshop at University. They are looking for financial or in kind assistance from driving enthusiasts and organisations. •


For more information on UoMR and contact details visit: uomracing.com. The University of Malta’s research trust, RIDT, fully supports the UoM racing team initiative. The trust aims to sustain and grow the UoM’s research activity. Please consider making a contribution at www.ridt.eu

Through the Looking Glass

What’s your favourite game? Pacman? Doom? World of Warcraft? Most of us have spent hours immersed in video games, many still do. Prof. Gordon Calleja studies why and how we get so involved in games. Science writer Dr Sedeer El-Showk found out about Calleja’s latest book and game that are gaining worldwide fame.

Continue reading

Titanium: Smoother and shapelier

Electron Beam Melting (EBM) is a state of the art manufacturing process. Using this process product designers will be limited only by their imagination. This technology uses an additive manufacturing approach where parts are built layer by layer — think of ‘3D printing’. It can use exotic metals like titanium alloys, however it does have limitations such as producing a rough surface finish that hampers its functions.

Christian Spiteri (supervised by Dr Arif Rochman) investigated whether the Electric Discharge Machining process can be used to finish parts produced using EBM to give a smooth surface (difference pictured). He treated titanium products with the Electric Discharge Machining process and microscopy showed that the finished surface consists of a set of micro-craters instead of rough grains. By adding the finishing process, more complex geometries could be created. Overall, adding Electric Discharge Machining to EBM had many benefits.

Dr Arif Rochman (University of Malta) said, ‘EBM parts can be used for a wide range of applications such as implants in the medical sector or complex tool inserts, […] which cannot be manufactured using conventional methods. Understanding the synergy between both processes is a must for product designers and process engineers to be able to manufacture high quality EBM products.’

 This research was performed as part of a M.Sc. (Res) in Mechanical Engineering at the Faculty of Engineering. This research was partially funded by the Strategic Educational Pathways Scholarship (Malta). This Scholarship is part-financed by the European Union – European Social Fund (ESF) under Operational Programme II – Cohesion Policy 2007-2013, ‘Empowering People for More Jobs and a Better Quality Of Life’.

titanium1

Music for Clean Food

Everyone eats. Eating food straight out of a packet is the norm in our fast-paced world — a simple fact that makes food science ever more important. We need safe food. THINK editor Dr Edward Duca met up with researcher Dr Vasilis Valdramidis to find out about the latest tech.

Food safety is serious business. In Germany during 2011 a single bug hospitalised over 4,000 people causing 53 deaths. Scientists learnt afterwards that a strain of E. coli had picked up the ability to produce Shiga toxins. These natural chemicals cause dysentery or bloody diarrhoea. The bacteria were living on fresh vegetables and it took German health officials over a month to figure out which farm was responsible.

On the 2 May, German health authorities announced a deadly strain of bacteria in food. By the 26 May, they pointed their finger at cucumbers coming from Spain. They were wrong. The mistake cost the EU over €300 million in farmer reimbursements. Genetic tests found that the bacterium on cucumbers was different than the one which was killing people. The researchers continued to ask people who were infected what they ate: raw tomatoes, cucumbers, and lettuce remained the prime suspects. Till they tested organic local bean sprouts from a farm in Bienenbüttel, Lower Saxony. By the 10 June, the farm was forced to shut down after it was pinpointed as the source. The sprouts were contaminated from the seeds’ source in Egypt.

‘These bean sprouts are found in several ready-to-eat foods, you could have it in your sandwich and not realise that you’re eating it,’ said food scientist Dr Vasilis Valdramidis (University of Malta). This is the reason why it took German officials so long to find the source. Having to rely on people’s memory of what they ate before becoming sick, something as inconspicuous and mild tasting as a bean sprout can be forgotten. Precisely why industrial food safety is so important: it saves lives.

food6

Cleaning food

‘There is no natural sterile environment,’ stated Dr Valdramidis who studies new ways to disinfect ready-to-eat lettuce, cabbage, and bean sprouts to make our food safe. Most bacteria come from nature or during handling. ‘After harvesting, there are 3 different steps for processing fresh produce. First, they are washed to remove all external material. Second, there is the disinfection process. […] Third, they apply a decontamination treatment that most commonly is chlorinated water.’

Dissolving chlorine dioxide powder into water makes most of the industrial chlorinated water. Chlorine is found in tap water so is relatively harmless at low concentrations, but ‘the less we have of this chemical the better for our health, because there are some side effects,’ explained Dr Valdramidis. ‘It can react with the organic substances of food products and produce some compounds […] that aren’t healthy.’

The environment is another problem. Chlorinated water ends up in ground water or other water sources. Elevated levels of chlorine can decontaminate vegetables but also natural habitats.

Dr Valdramidis’ group works to reduce the amount of chemicals, water, and energy used. Fresh water is a precious resource with less than 3% of the world’s water being fresh. In Malta, pressures on fresh water use are intense and the country is facing a little known water crisis. Worldwide energy efficiency is a hot issue, with both environmentalists and industry pushing for greater efficiency and cheaper energy bills.

From Oregano to Music

The herb oregano can be concentrated with its essential oils extracted. Surprisingly, at the right concentration oregano slows bug growth. Dr Valdramidis’ group is taking advantage of this antimicrobial effect to disinfect vegetables. ‘And it tastes better, but it depends on the amount; if you use too much it is bitter.’

The food industry’s bottom line is cost. ‘The extraction process is quite expensive but now the price is going down. [The food industry already] use oregano oil as antimicrobial agents in feeding products for animals. Their aim is to reduce the use of antibiotics. It [oregano] is becoming more and more accessible.’

Oregano oil might be more expensive, but it is a natural product that is non-toxic. Another advantage is that, ‘if the plant cells are relaxed then these essential oils can penetrate’ into the plant disinfecting it thoroughly. Once optimised, it could easily replace chlorine water, reducing the amount of damaging chemicals used.

Oregano could replace chlorine water, but what about the amount of water? Another technique, which uses sound to clean food, could help. Think about ultrasounds used to scan pregnant women. Those ‘operate on megahertz and create images, this [technology] operates on kilohertz and is powerful enough to create physical changes at a microscale’, which means they are high power systems. It works by pulsing sound waves at your submerged vegetable or fruit of choice. The sound creates bubbles that implode, creating a very high pressure and temperature. This energy can kill the bacteria. When Dr Valdramidis gets it right, it cleans the vegetable.

The process is even more extraordinary. The sound wave causes ‘a molecule of water to split and create [the molecules] hydrogen peroxide and other radicals, which are very unstable’ so they react with everything around them (including bacterial DNA), either becoming water again or attacking cells. ‘They affect the membrane of the bacterial cell,’ said Dr Valdramidis, ‘killing it.’ They can also damage plant cells, so the technique needs fine-tuning to get it right. By measuring the appearance, amount of vitamins, enzymes, and other nutrients lost by the procedure, researchers can tweak it to maximise its antimicrobial value and minimise its damage to the vegetable. To continue improving the technique a lot of his work is spent trying to understand exactly how the procedure works and why the bacteria die.

The ultrasound still needs water to work. Water cannot be removed from the equation because bubbles can only form in water and sound also travels better. Water quantity can be reduced. When using chlorinated water, another step is needed to rinse off the chlorine. In this case, it can be skipped. There is an even more radical technique that might bypass water altogether.

A lightning storm

Plasma is made up of ionised air. In nature, plasma is made by lightning, leaving a tell tale ozone smell. Food scientists can pass high frequency electricity through air to create a bacteria-killing plasma stream.

Ionised air kills bacteria because it forms radicals and ozone. Electric discharges create radicals and turn oxygen into highly reactive oxygen radicals (an unstable oxygen atom) or ozone (3 oxygen atoms joined together). These products can react with bacteria and inactivate them. Like sound waves they can also affect food. ‘High levels of ozone can bleach food by oxidising the product. There is no ideal technology,’ stated Dr Valdramidis. The difficulty in all of this is how to kill the bacteria and not the plant. Everyone wants salads with a nice colour, good flavour, and high nutritional value.

On the other hand, the beauty of this technique is that you can zap the food in its packet. So imagine just rinsing the food with a little water, wrapping it up, and finishing off the cleaning process with an electric pulse. The package can be delivered to your local grocer with minimal use of water and your mind at rest. Both sound waves and plasma could also spell the end of excessive chemical treatments.

food2

A computer model of a fruit

Measuring microbe levels is the only certain way to know if food is safe. Traditional methods are labour intensive, time consuming, and expensive. Scientists first need to remove the bacteria from the product, then dilute the bacteria, then count the cells directly with plate counting techniques or under a microscope. More modern techniques use molecular methods such as PCR (Polymerase Chain Reaction) to find out the specific type of bacteria. This can make a huge difference since not all bugs are created equal.

To reduce costs and speed up the process, Dr Valdramidis uses mathematical models to predict the shelf life of products and apply the right decontamination process. ‘We want to predict the amount of bacteria present, so with these equations we are trying to describe how fast the bacteria are inactivated then [how fast those that survive] grow,’ explained Dr Valdramidis. The number of bacteria predicts food safety and how fast it rots.

For mathematical modelling to work, first ‘data needs to be collected […] by performing some experiments. Then I try to describe how the population responds and behaves using these equations. If I can verify this model, then I can come to you and tell you, ‘look, this product has these specific characteristics, within the range of this model, I can tell you that it will expire in 15 days and you don’t need to do any experiments.’ It’s a very powerful tool but it has to be well validated.’ It saves a ton of money, but you must be sure of the model otherwise people could be harmed.

Current maths has its limits. Scientists are still trying to correctly model a single cell. Plant or bacterial cells are complicated machines, with proteins, DNA, and other molecules all jam-packed together working synchronously for a cell’s survival and reproduction. To make things easier, scientists simplify cells when simulating them then consider a whole group of them, a population. Researchers test the whole population. If Dr Valdramidis’ group attempts to model a single bacterial cell’s growth in Malta, he would have to use the University’s supercomputer called ALBERT. Maths on this level uses a lot of computational power.

Taking the cell modelling idea to its extreme, some food scientists are trying to model every plant cell to make a complete fruit — a virtual fruit. They model, ‘the exchange of gases and so on since fruit is still respiring, still alive after harvesting.’ To control the respiration process, they ‘try to control the amount of [the hormone] ethylene, oxygen gas, and so on.’ They also use these models to simulate modified atmospheres around food seeing how they influence respiration rates. Shelf life is affected by plastic packages with different holes sizes, types of plastic, and other parameters. All of these properties are pumped into the mathematical equations and tweaked to maximise shelf life. ‘If you slow rates down, the food lasts longer and can be stored for a longer period,’ explained Dr Valdramidis, which makes both companies and consumers happy.

food4

Working with industry

Dr Valdramidis is young but has a long career in fundamental research. He has modelled and tested the rate of bacterial growth (and inactivation) at changing temperatures, and even investigated how to decontaminate biofilms in industrial food processing plants. Importantly, he has looked into quantifying and speeding up the analysis of microbial levels on food to give an actual ‘best before’ date. His approach always coupled experiments to test his maths and predictions.

Innovations in food science aim to bring down prices, use less water, fewer chemicals, and less energy. For these reasons, Dr Valdramidis is now at a stage where he can collaborate with industrial partners. In Malta, he has already met with the Chamber of Commerce through the creation of a Food Industrial Advisory Platform. With this platform ‘we plan to organise workshops every 6 months. Once to speak about our activities and another to speak about subjects that are of interest to SMEs [Small to Medium Enterprise, or industry].’ Malta is run by food SMEs; they account for 65% of GDP.

Researchers need to work with industry — a statement on everyone’s bucket list. Its importance cannot be understated, since it is unlikely that universities will receive substantially more research funds unless businesses start seeing these institutions as partners. And, they could save or make big bucks by investing in research. Dr Valdramidis’ work is a clear call for collaboration.

Working with others is what drew Dr Valdramidis to Malta. ‘I firmly believe in collaboration. A lot of my [research] publications are not just from the university I would be working in but others as well.’ By opening arms wide open perhaps we can prevent mistakes, like those of the German health authorities, invest in research that reduces waste, and cleans our food just by playing a song at the right energy.

[ct_divider]

Some of the above research is supported by a Marie Curie FP7-Reintegration-Grant within the 7th European Community Framework Programme under the project ‘Development of novel Disinfection Technologies for Fresh Produce (DiTec)’, and part-funded by the Malta Government Scholarship Scheme.

Hotline Miami

Game Review_Costantino

Push start. Grab a weapon. Get shot. Repeat… ad infinitum. ‘Punishing’ hardly describes a session of Hotline Miami. Typically, within 10 seconds you could die three or four times. It is just as frustrating as it is challenging. Addictively, you will not give up until you pass that sneaky little passage. 

Hotline Miami is an ultra-violent, psychotic game, where your only aim is to kill all the ‘bad guys’.

Yet, every little move counts, and deciding which weapons to use or which door to open first will reveal the deep strategic possibilities of the game’s intense experience. As you make your way through a pile of corpses, the suspense builds up to unbearable levels as you risk losing all in-game progress for just a little mistake. The massacre is only interrupted by brief moments that reveal details of our mysterious ‘hero’s’ back-story. Keeping true to expectation, even these interludes are awkward if not disturbing, and hardly shed light on our displaced, faceless avatar

The excellent game tops it all with an irresistible ‘80s aesthetics and a neurotic electronic soundtrack. You’ll quickly find out why this game has stolen the show winning so many awards, and has hooked fans of Grand Theft Auto and of good old shoot’em up games. Hotline Miami is a joy in repetition, providing that being stuck in a Clockwork Orangelike scenario is your idea of joy. 

Screenshot from Hotline Miami
Screenshot from Hotline Miami

www.hotlinemiami.com 

Hand pose replication using a robotic arm

Robotics is the future. Simple but true. Even today, they support us, make the products we need and help humans to get around. Without robots we would be worse off.  Kirsty Aquilina (supervised by Dr Kenneth Scerri) developed a system where a robotic arm could be controlled just by using one’s hand.

The setup was fed images through a single camera. The camera was pointed towards a person’s hand that held a green square marker. The computer was programmed to detect the corners of the marker. These corners give enough information to figure out the hand’s posture in 3D. By using a Kalman Filter, hand movements are tracked and converted into the angles required by the robotic arm.

The robotic arm looks very different from a human one and has limited movement since it has only five degrees of freedom. Within these limitations, the robotic arm can replicate a person’s hand pose. The arm replicates a person’s movement immediately  so  that a person can easily make the robot move around quickly.Controlling robots from afar is essential when there is no prior knowledge of the environment. It allows humans to work safely in hazardous environments like bomb disposal, or when saving lives performing remote microsurgery. In the future, it could assist disabled people.

This research was performed as part of a Bachelor of Engineering (Honours) at the Faculty of Engineering.

A video of the working project can be found at: http://bit.ly/KkrF39

PHONE STORY

Game Review_Costantino

 

What is the story behind our smart phones? Phone Story retraces the production stages of our favorite products, showing us the dramatic working conditions behind their assembly. It seems like Apple didn’t like it: the game is now banned from the App Store.

Mass riots and frequent suicides shed a dark light on Foxconn, the company that assembles product for the likes of Apple, Microsoft, and Amazon. Afterwards, it became impossible to ignore the conditions of the workers that mass-produce contemporary tech gadgets.

Enter Molleindustria, the software house funded by Paolo Pedercini, responsible for a remarkable series of ferociously satirical, “serious” games (check molleindustria.org for more — it’s worth it). Phone Story takes us through an uncomfortable ride of funny minigames with classic, familiar mechanics juxtaposed to harsh vignettes: underage miners extracting silicon, suicidal workers jumping from the roof of the factory, and an army of Apple — pardon, “Pear” — fans taking a flagship store by storm.

Don’t expect a radical new gaming experience with Phone Story, but pick it up if you’d like to reflect on the process that brought you your mobile, while you’re
actually holding it in the palm of your hand. 

https://www.youtube.com/watch?v=sSMSFLAsNzc