The Big Data Problem

Out latest spacecraft which observes the Sun has give us a new headache.

This video by Dean Pesnell explains the problem pretty well

At 2 Tb per day we’re constantly trying to sift through these data to do science that we expect to do and, perhaps more importantly, science we may not have expected. It’s pretty easy to come up with a program to sift through data and detect features or events that you expect to see. But how do you go about detecting the unexpected? A quick look at the eye and brain may tell us the answer. How does our brain tell us that something unexpected or new is happening? The brain is very good at taking data from the eye and ignoring most of it. But if something unusual appears it tells us immediately.