Here I discuss some of the challenges of applying machine learning to real-world scientific data, in particular noisy and sparse periodic time-series data.

Exoplanets

Our knowledge of exoplanets, or planets that exist outside our Solar System, has advanced drastically over the last few decades. In fact, until relatively recently one could have called exoplanets a theoretical concept. The first confirmed detection of a real exoplanet wasn’t until 1992 (Wolszczan et al. 1992), and even by 2004 only about a hundred exoplanets had been detected. This all changed with the launch of the Kepler space telescope in 2009 (Borucki et al. 2010). Since then, thousands of exoplanets have been detected with the ``transit” method (i.e., detecting an exoplanet by observing the drop in brightness of a star as the orbiting exoplanet crosses our line-or-sight to the star). This year (2018) a new but related space telescope was launched — the Transiting Exoplanet Survey Satellite (TESS) (Ricker et al. 2014). TESS, which also uses the transit method, will concentrate on finding exoplanets closer to Earth and around brighter stars than Kepler allowed, which will be important for follow-up observations necessary to help us learn more details about these exoplanets, such as their compositions and atmospheres.

TESS is already collecting new data with the potential to improve our knowledge of exoplanets, but there is a bottleneck to accessing this knowledge: humans. Each candidate planet identified by TESS must be confirmed by a scientist with follow-up observations from the ground, making it vital that these planet candidates be as reliable as possible, meaning that false positives must be minimized. At the same time, we must avoid missing real planets and minimize the false negatives.

The Kepler/TESS Pipeline

The raw data returned by a space telescope (i.e. Kepler/TESS) is essentially a very noisy, low-framerate video of a patch of the sky. For both Kepler and TESS, the data is a set of Target Pixel Files (TPFs), which are small (e.g. \(11 \times 11\) pixels) image frames roughly centered on the target stars, collected at a regular time interval over a given amount of time — every 2 minutes over 27 days for TESS. From these TPFs, the pipeline extracts time-series photometry of the target stars, called a “light curve”, which is essentially the 1D signal of the brightness of a star over time. The pipeline then must remove the systematic and random noise introduced by the instrument, as well as real stellar phenomena that can look similar to noisy exoplanet transit signatures. Only then can the pipeline search for the exoplanet transit signals — the characteristic drop in light when an exoplanet passes in front of its star (see fig. 1). This is no easy task, and a team at the NASA Ames Research Center has spent the greater part of a decade creating a pipeline to do all this for the Kepler data (Jenkins, et al. 2017), and is now also applying using it to process the new TESS data.

Challenges of the Kepler/TESS Exoplanet Datasets

The challenges present in the detection and classification of exoplanets are different than those typically seen in typical supervised machine learning problems. The quality and completeness of the “ground truth” is limited, being both noisy and intrinsically incomplete. Only the already discovered exoplanets are labelled and a large number of undiscovered planets are present in the data, sometimes incorrectly labelled as false negatives. Those planets that are labelled are biased towards the easier planets to detect — that is large planets. The nature of the TESS mission requires a quick “response time” in order to follow-up the most promising planets with other instruments without wasting limited telescope time and resources.

Transit signals are periodic dips in the light curves, but these are often near the noise floor of the data. Without the pipeline’s removal of systematic noise, transit signals are rarely apparent at all. Transit signals are also extremely sparse however, the occultation of the star due to the planet lasts only a very small fraction of the period and, depending on the planet’s orbital period, this might occur only a handful of times. What makes it possible to observe the transit signals at all is the high precision of the period of the dips, allowing the signals to be folded, averaged over the planetary period. The planetary period is unknown a priori, and must be exhaustively searched. All of these characteristics of the signals make the problem challenging, and different from problems typically solved with machine learning methods.

Learning Approaches

Lightcurves

Recently a deep learning approach to exoplanet classification from detection candidate light curves output by the Kepler pipeline was proposed (Shallue, et al. 2018). A CNN is trained on the light curves folded by the candidate period. Our main approach was to expand upon their work by incorporating more domain knowledge, and extending the method to the TESS dataset which is quite different than that of Kepler. We also addressed the severe class imbalance present in TESS — few candidates are labelled as planets vs. non-planets – by using mini-batch class balancing. Our approach improved in both performance and efficiency over that of (Shallue, et al. 2018). We also tried approaches of folding the candidate light curves on an exhaustive range of possible periods, avoiding the need for the pipeline’s detection.

Target Pixel Files

In a more ambitious on-going approach, we attempt to train on the raw TPF images, bypassing the Kepler/TESS pipeline. With the intrinsic systematic noise, and given the sparsity and low signal-to-noise ratio, we have found a very challenging dataset to train with however. A typical TESS TPF time series has dimensions 11\(\times\)11\(\times\)19815 and there may be as few as two transits. It is clear that novel machine learning approaches may be required to learn to classify exoplanets in the face of extremely sparse periodic signals.

Conclusion

In exoplanet detection, many of the challenges we encountered were typical of applying machine learning to any real-world problem, such as class imbalance and noisy labels. However, the problems we encountered in trying to learn from the raw TPF images highlighted real-world data that is not well addressed by current machine learning methods. In particular learning sparse periodic signals with a low signal-to-noise ratio, and in the presence of strong systematic noise is challenging.

For more information, view our recent presentation on the work we did over the summer, or see the FDL exoplanets challenge website.

Acknowledgments

I’d like to thank the whole TESS team at SETI/NASA who were a massive help and proposed this challenge, in particular Jeffery Smith, Jon Jenkins and Douglas Caldwell. I’d also like to thank the NASA Frontier Development Lab organizers and mentors for giving us this unique opportunity. And finally I thank our challenge’s industry partners: Google Cloud and Kx Systems, in particular Google Cloud’s generous donation of compute resources without the support of which our work would not have been possible.

The vast majority of deep learning researchers had explained away filter groups as an engineering hack, until the initial publication of the Deep Roots paper in May 2016. Indeed it was clear that this was the primary reason for their invention, and by removing parameters surely accuracy was decreased?

Not just an Engineering Hack!

However, even the AlexNet authors noted, back in 2012, that there was an interesting side-effect to this engineering hack - the conv1 filters being easily interpreted, it was noted that filter groups seemed to consistently divide conv1 into two separate and distinct tasks: black and white filters and colour filters.

What wasn’t noted explicitly in the AlexNet paper was the more important side-effect of convolutional groups, that they learn better representations. This seems like quite the extraordinary claim, however this is backed up by one simple experiment: train AlexNet with and without filter groups and observe the difference in accuracy/computational efficiency. This is illustrated in the graph above, and as can be seen, not only is AlexNet without filter groups less efficient (both in parameters and compute), but it is also slightly less accurate!

How do Filter Groups Work?

Above is shown a normal convolutional layer, with no filter groups. Unlike most illustrations of CNNs, including that of AlexNet, here we explicitly show the channel dimension. This is the third dimension of a convolutional feature map, where the output of each filter is represented by one channel. In illustrations like this it is clear that the spatial dimension of a featuremap is often the tip of the iceberg, as we get deeper in a CNN, the number of channels rapidly increases (with the increase in the number of filters), while the spatial dimensions decrease (with pooling/strided convolution). Thus in much of the network, the channel dimension will dominate.

Above is illustrated a convolutional layer with 2 filter groups, where each the filters in each filter group are convolved with only half the previous layer’s featuremaps. Unlike in the AlexNet illustration, with the third dimension shown it is immediately obvious that the grouped convolutional filters are much smaller than their normal counterparts. With two filter groups, as used in most of AlexNet, each filter is exactly half the number of parameters (yellow) of the equivalent normal convolutional layer.

Why do Filter Groups Work?

This is where it gets a big more complicated. It’s not immediately obvious that filter groups should be of any benefit, but they are often able to learn more efficient and better representations. This is because filter relationships are sparse.

We can show this by looking at the correlation across filters of adjacent layers. As shown above, the correlations are generally quite low, although in a standard network there is no discernable ordering of these filter relationships, they are also different between models trained with different random initializations. What about with filter groups?

The effect of filter groups is to learn with a block-diagonal structured sparsity on the channel dimension. As can be seen in the correlation images, the filters with high correlation are learned in a more structured way in the networks with filter groups. In effect, filter relationships that don’t have to be learned are no longer parameterized. In reducing the number of parameters in the network in this salient way, it is not as easy to over-fit, and hence a regularization-like effect allows the optimizer to learn more accurate, more efficient deep networks.

Unanswered Questions

How do we decide the number of filter groups to use? Can filter groups overlap? Do all groups have to be the same size, what about heterogeneous filter groups?

Unfortunately for the moment these are questions yet to be fully answered, although the latter has recently received some attention from Tae Lee et al., at KAIST.

Grouped Convolutions for models such as ResNeXt and Xception and CTC (Connectionist Temporal Classification) loss layer for temporal classification.

For more information, see the CuDNN release notes. Thanks to Michael Figurnov for pointing this out to me!

With what must be one of the best locations for a computer vision conference ever, CVPR 2017 was always going to be one of the best, but even considering that this conference was notably better organized than most of it’s predessesors. This was despite a massive increase in attendance - over 37% more than CVPR 2016 in Los Vegas see here for the slides describing this year’s stats, and a 40% increase in paper submissions. Notable changes this year included moving to a 3 track conference instead of 2 track. Even so, one of the days had a free afternoon, and the organizers noted that this was at least partially because reviewers had not nominated enough orals this year to fill the schedule.

Orals

Of the few papers that were orals, I was impressed with the following:

• “Unsupervised Learning of Depth and Ego-Motion from Video”: This work is simply impressive (full disclaimer, one of the co-authors is my former PhD supervisor). It learns depth simply from unlabelled data of car driving around urban scenes. The weaknesses of the method seemed to be that it learned a bias in the data not to expect any close depth in the center of the frame (caused by correctly cautious drivers leaving a gap between themselves and the car in front), and a lack of good perforamance on non-urban scenes. In both cases the authors claimed that these were solvable by having more data. However, I believe there simply aren’t enough features outside of urban canyons for this method to suceed outside of urban environments.
• “Learning From Simulated and Unsupervised Images””: Although the work is interesting, and potentially very useful, this oral stood out for another reason. It was explicitly an oral by Apple researchers, marking a significant shift for a company that only a few years ago wouldn’t even let it’s CVPR attending employees admit their affiliation - nevermind publish it’s research. Hopefully this newfound openness continues.
• “Densely Connected Convolutional Networks”: This work presented some very impressive results in a domain I’m particularly interested in. I’m surprised such a network can be more computationally efficient while reducing error significantly. I will be trying to repeat these results for sure.
• “Global Optimality in Neural Network Training”: With little theoretical progress in understanding the optimization of deep networks, I like this work because it has relatively few assumptions compared to many theoretical analysis, and yet has relatively large claims. Whether it can be put to effect in practice however remains to be seen.
• “YOLO9000: Better, Faster, Stronger”: This oral surprisingly suceeded despite (because?) of the ridiculous title, and the theme of the presentation (daft punk’s stronger, faster). The presentation was engaging, and the results very impressive. A live demo of their realtime object detection system not only worked, but seemed to go beyond even the author’s expectations when the object detection system identified most of the audience in view along with the planned foreground objects.

Industrial Presence

As with every year, the industrial presence at CVPR grew more this year. While the number of companies increased, this was mostly due to the large number of new startups with a booth presence, rather than the larger companies. If anything the larger US companies had a slightly reduced prescense overall, especially when I compare with NIPS. This had the notable exception of Apple however, which organized three separate events, two `mixers’ and one technical session, and Nvidia, whose CEO attended the conference and made a major new GPU announcement. Notably this year there were many more asian companies with a presence, Chinese (Tencent, and too many to list), and Korean companies (Naver, Samsung).

Machine learning is not magic

And this is a very important lesson indeed. Perhaps the best intuition behind the theorem is to be gained by the following simple example. Assume we see a sequence of random numbers,

and we are asked to predict the next number in the sequence. Most of us would probably predict, \(x = { 1, 3, 9, 27, \ldots}\), assuming that the sequence at each time step is being generated by \(x_t = 3 x_{t-1}\).

However, there is no reason to not instead believe the hypothesis that this sequence is simply the output of a random number generator, in which case the next item of the sequence can not easily predicted. Even if it seems very unlikely (and Occam’s razor is a good argument here), we cannot completely disprove this hypothesis without seeing all the data points.

``Everything, but the data, is an assumption’’

Zoubin Gahramani MSR AI Summer School 2017

The only reason machine learning works at all are the assumptions we make about the problem. We call these assumptions the model. Whatever assumptions we make in our model will, of course, only help predictions with the types of problems where those assumptions hold, while hindering prediction with other types of problems. Wolpert et al. show that this means at best, over all possible input data distributions, we can not expect any model to do better than random.

This is often cited as the nail in the coffin for the idea of a universal learning algorithm, and algorithm that can learn any problem. Theoretically this is certainly true, however we are interested only in learning real-word problems, of which the data lies in a specific subset of all possible distributions.