IBM prototype could help manage data deluge from world’s largest telescope

IBM has completed a software prototype that automates data management and provides intelligent search functionality to aid with the management of the deluge of data expected from the world’s largest radio telescope, the Square Kilometer Array.

The Square Kilometer Array (SKA) is a €1.5 billion multi-national project to build a network of satellite dish radio telescopes to answer the fundamental questions of the universe: how the first black holes and stars evolved, whether Einstein’s theory of relativity holds up in extreme conditions and whether there is life outside our solar system.

New Zealand is part of a conjoint bid with Australia to host the telescope which will extend 3000km from its central core, with about two out-stations estimated to be hosted locally.

Our tab for hosting the out-stations is estimated to be about 1 to 2% of the total cost, but the economic and intellectual returns are estimated to far exceed this bill.

The host site is expected to be decided in early 2012 but in the meantime companies are striving to prove they have what it takes to be included in the SKA line-up.

One of these is IBM which has been involved in several SKA Pathfinder projects including a Shared University Research Award to Victoria University of Wellington to support SKA related research.

Click to zoom

IBM’s latest offering is a software prototype that aims to help automate necessary astronomy processes to help deal with the estimated in excess of one exabyte of raw data SKA will produce in a single day, or more than the entire current daily internet traffic.

The software is a subset of a new architecture, the Information Intensive Framework, or blueprint for what IBM intends to build, IBM New Zealand chief technology officer and chair of the New Zealand SKA Industry Consortium Dougal Watt said.

The software uses the International Virtual Observatory Association Ontology to classify the collected data into concepts and categories that astronomers use, Mr Watt said.

“It’s almost like you’ve gone to an astronomer, plucked the knowledge out of their head and you’ve put it into this system.  When it gets data into this system, it’s kind of thinking about that data in a similar way to [what] an astronomer would.”

This involved classifying stars into types, for example, by using logic and reasoning from certain observed factors, such as light levels and frequencies, he said.

The prototype also provided guided search functionality, he said. Where astronomers previously had had to build laborious queries which could sometimes result in a false return of no data due to a fractional parameter error (such as defining a time period for observing a certain type of star that was one second out), the prototype provided guided queries and options.

“It literally does know, ‘I’ve got all of these objects in my database, and they were observed in this part of the sky, at this time, down to the millisecond and they have these parameters.’ So the system can put these options to you, you know ‘What do you want to search for, these are the types of stars I know about, here’s where they were in the sky, here’s the time of day’.”

The prototype met its design goals of automating the classification of astronomical objects and improving the efficiency of astronomers’ searches, Mr Watt said. 

Analysis of the results of the project suggested ways to extend the prototype to meet SKA’s required performance levels.  Mr Watt said this included applying the work alongside an astronomy project’s existing systems, and automating other astronomical processes, as well as building functionality, improving the system and tuning it to real world problems.

Indeed, the results from the prototype were also applicable to other organisations faced with a data deluge, he said. 

“We have identified several local scenarios which would benefit from automated analysis of performance data to uncover trends, identify anomalies and improve decisions.  These range from individual manufacturing plants and telecommunications companies to whole transport networks and healthcare systems.”

IBM worked with Victoria University radio astronomer Dr Melanie Johnston-Hollitt to build the IIF.  Dr Johnston-Hollitt said research on exa-scale datasets would force radio astronomers into an as yet unexplored regime of automated processing, imaging and analysis.  She said surveys on SKA precursor telescopes were expected to produce catalogues of tens of millions of radio sources and how astronomers organised and classified the data available in the next three years, was a significant challenge.

“We will need new solutions to fully realise the vast scientific potential of these datasets and it’s fantastic that organisations like IBM are prepared to take up that challenge.”