Exploring Petabytes of the Night Sky — Jupyter Notebooks at NOIRLab’s Astro Data Lab Science Platform
By Robert Nikutta & Stéphanie Juneau (NSF NOIRLab)
Imagine querying 420+ billion rows of astronomical catalog data — spanning 30 major sky surveys, observed over decades with telescopes on three continents — from a Jupyter notebook in your browser in seconds. No download. No HPC allocation request. No waiting.
That is what 4,800+ astronomers in over 90 countries can do every day at the Astro Data Lab science platform. Data Lab is operated by NSF NOIRLab, the National Optical-Infrared Astronomy Research Laboratory, headquartered in Tucson, Arizona, with observatories in Arizona, Hawai’i, and Chile. Since its public launch in June 2017, Astro Data Lab has quietly become one of the largest deployments of Jupyter notebooks in professional science — and a case study in what happens when you bring the compute to the data instead of the other way around.
The Data Problem Astronomy Had to Solve
Modern sky surveys are data machines. The Dark Energy Survey cataloged 690 million objects. Gaia measured positions and motions for 1.8 billion stars. The DESI Legacy Surveys cover 20,000 square degrees, nearly half of the full sky, in three optical bands. And the upcoming Rubin Observatory’s Legacy Survey of Space and Time (LSST) will generate roughly 10 million transient alerts per night starting later this year.
Traditional astronomy workflows begin with downloading relevant data to a local computer and to use locally installed specialized software tools to process and analyze the data. However, downloading these catalogs to a local machine is now often physically impossible. A single survey’s measurements table can exceed the combined disk space of an entire research group. And even if you could download it, the computing resources needed to query it efficiently at scale requires infrastructure most astronomers don’t have.
The answer the community converged on, like many industries dealing with big data: bring the compute to the data. Host the catalogs in databases, co-locate a computing environment next door, and give scientists a familiar interface to work in. That interface, increasingly, is a Jupyter notebook.
Astro Data Lab: Jupyter at the Observatory
Astro Data Lab was conceived in 2014 and went public in June 2017, originally built to support data releases from the Dark Energy Survey — a few terabytes of catalogs and tens of terabytes of images. We imagined a ceiling of roughly 500 users. We were wrong, in the best way.
Today the platform hosts:
- About 420 billion catalog rows across 30+ major astronomical surveys (DES, Legacy Surveys, DESI, NOIRLab Source Catalog, SDSS, Gaia, unWISE, SMASH, S-PLUS, VHS, 2MASS, and dozens more)
- 31 million spectra via SPARCL, our spectral access service (DESI DR1+EDR, SDSS/BOSS DR17)
- Petabytes of images, accessible through a Simple Image Access service and cutout API
- Over 4,800 registered users from over 90 countries, who submit tens of millions of data queries each year
Every registered user gets a persistent JupyterHub environment with the full astronomy Python stack pre-loaded — Astropy, NumPy, SciPy, Matplotlib, Pandas, Scikit-learn — and our own astro-datalab client library. The library provides core services, for instance auth and DB queries:
from dl import authClient, queryClient
from getpass import getpass
# Log in
token = authClient.login(input("username: "),getpass("password: "))
# Query 10 objects from the NOIRLab Source Catalog near a sky position
# Right Ascension (RA) = 150.12 degrees
# Declination (Dec) = 2.21 degrees
# Search radius = 0.05 degrees
# q3c (Quad Tree Cube) is a spatial indexing scheme for Postgres
# gmag and rmag are the g-band and r-band magnitudes of objects
# in the NOIRLab Source Catalog Data Release 2, ‘object’ table.
result = queryClient.query(
"""SELECT ra, dec, gmag, rmag FROM nsc_dr2.object
WHERE q3c_radial_query(ra, dec, 150.12, 2.21, 0.05) LIMIT 10""",
fmt="pandas"
)
It’s as simple as that. The query runs on the database server next to the data; only the result set crosses the network.
Use Case 1 — Seeing the Sky Inside a Notebook with AladinLite
One of the most immediate joys of working with astronomical data is visualization: not just numbers in a table, but where things are in the sky, what the images look like, and how your query results relate to the underlying survey footprint.
We’ve integrated AladinLite v3 — the interactive sky atlas from Centre de Données Astronomiques de Strasbourg (CDS) — directly into the notebook environment via the ipyaladin widget. With a handful of lines, astronomers can embed a fully interactive sky viewer in a notebook cell or next to their notebook in a “sidecar”, and overlay their own data on top of real survey imagery:
import time
from astropy import units as u
from astropy.table import Table
from astropy.coordinates import SkyCoord
from ipyaladin import Aladin
from sidecar import Sidecar
# Instantiate the Aladin interactive sky viewer
aladin = Aladin(full_screen=True)
with Sidecar(title="aladin_output",anchor='split-right'):
display(aladin)
# globular cluster NGC 1851 (RA, Dec)
aladin.target = SkyCoord(78.52809*u.deg, -40.04656*u.deg)
aladin.coo_frame = "ICRSd" # set coordinate frame to ICRS, angles in deg
time.sleep(1) # race condition
aladin.fov = 0.4 # set field of view to 0.4 degrees
# Overlay catalog query results as circle markers
t = Table.from_pandas(df) # e.g., from a previous query around NGC 1851
aladin.add_table(t,shape='circle',source_size=15,color='green')
The result is a pannable, zoomable sky viewer — right in the notebook — with your query results overlaid as green circles on the actual sky image of a globular cluster (see figure below). Users can overlay MOCs (Multi-Order Coverage maps, which encode survey footprints), user-generated catalogs from a prior query, or any Virtual Observatory-standard data source.
This capability turns what was once a static plot into an exploratory tool: zoom into a cluster, click on a source, cross-match on the fly. For students and scientists unfamiliar with a dataset, it is often the fastest path from “I have a list of objects” to “I understand where they are and what I’m looking at.”
AladinLite integration is now active in our notebook library, with full deployment into the new Data Lab Web Portal on the roadmap for later this year.
Use Case 2 — Stacking Galaxy Spectra with SPARCL
Spectroscopy — measuring how much light a star or a galaxy emits at each wavelength — is one of astronomy’s most powerful tools. But individual spectra are often noisy. The signal-to-noise ratio of a single optical spectrum for a faint galaxy can be too low to measure the emission lines that encode star formation rate, gas chemical content (Oxygen, Nitrogen, etc.) or the even more subtle absorption lines that create small wiggles in the shape of the spectrum, yet encapsulate crucial information such as the mass and age of the stars making up a galaxy.
One trick that astronomers have used for decades: combining or “stacking” spectra. Average hundreds of spectra together, and the noise level reduces while the signal builds up. What was invisible in a single spectrum becomes unmistakable in the stack. While the concept is simple, reading and manipulating large numbers of spectra can be time consuming or cumbersome.
SPARCL (SPectra Analysis and Retrievable Catalog Lab) makes this possible at scale directly in a notebook. With `sparclclient`, users can currently search 31 million spectra by redshift range, target type, and survey, then retrieve flux arrays and wavelength grids ready for stacking:
from sparcl.client import SparclClient
# Instantiate the SPARCL client (connected to production server)
client = SparclClient()
# Find SDSS spectra of galaxies in a redshift slice 0.1<z<0.3
found = client.find(
outfields=['sparcl_id', 'ra', 'dec', 'redshift', 'spectype'],
constraints={'spectype': ['GALAXY'],
'redshift': [0.1, 0.3],
'data_release': ['SDSS-DR17']},
)
# Retrieve flux, wavelength, and inverse-variance arrays
retrieved = client.retrieve(found.ids,
include=['flux', 'wavelength', 'ivar'])
In our SpectralStacking_SDSS science example notebook, users first stack a small number of galaxy spectra (N=5) in eight bins of astrophysical color g−r (green and red filters), revealing trends from blue spectra with emission lines to red spectra with absorption lines but with noisy spectra. Then users stack hundreds of galaxy spectra for the same bins of color g−r and obtain much cleaner spectra where the small wiggles are now real astrophysical features and no longer buried in the noise.
The spectral rainbows below — N=5 then N=200 stacked galaxy spectra color-coded in bins of astrophysical color g−r — are each a single output cell from this notebook, generated entirely within the Data Lab environment.
Use Case 3 — Variable Stars and the Coming Flood of Time-Domain Data
Not all astronomical data is a static snapshot of the sky. Many of the most scientifically rich phenomena — pulsating stars, transiting exoplanets, exploding supernovae, gravitational lensing events — reveal themselves through change over time.
Among the most useful calibration tools in astrophysics are RR Lyrae stars: old, low-mass stars that pulsate with periods of 0.2–1 day and a brightness variation that traces their distance. Finding and characterizing them across millions of square degrees of sky requires querying multi-epoch photometry catalogs, computing period statistics, and folding light curves — all tasks that fit naturally in a notebook workflow.
Our TimeSeriesAnalysisRrLyraeStar notebook demonstrates the full pipeline: query the SMASH DR2 catalog for stars with high photometric variability, run a Lomb-Scargle periodogram on the light curve, identify the dominant period, and phase-fold the observations to reveal the characteristic sawtooth pulsation profile:
from astropy.timeseries import LombScargle
import numpy as np
ls = LombScargle(t, y) # time and magnitude from a previous query
frequency, power = ls.autopower()
period = 1./frequency # period is the inverse of frequency
best_period = period[np.argmax(power)]
phase = (t / best_period) % 1 # folded timeseries = light curve
The resulting phase-folded light curve shown in the figure below is clean, precise, and immediately recognizable to any variable-star astronomer — produced entirely from archival survey data without a single new observation.
This kind of workflow is also a proving ground for the upcoming Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST). When Rubin begins operations and delivers 10 million nightly alerts, the only workflows that will scale are ones already designed to run against large databases or specialized file systems, in shared computing environments, with notebook-native tooling. Astro Data Lab users are building those workflows today.
The Notebook Ecosystem
The three use cases above are drawn from our library of 80+ open-source Jupyter notebooks at github.com/astro-datalab/notebooks-latest. The library is organized into six sections:
Directory Contents
---------------------------------------------------------------------------
01_GettingStartedWithDataLab/ Authentication, dataset discovery, first
queries
02_DataAccessOverview/ More advanced queries, image searches, etc.
03_ScienceExamples/ Many complete science cases (stellar
streams, dwarf galaxies, large-scale
structure, SED fitting, ...)
04_HowTos/ Service-specific tutorials (SPARCL, SIA
image cutouts, cross-matching, file storage)
05_Contrib/ Community-contributed notebooks (ANTARES
alert broker, user science cases, etc.)
06_EPO/ Education & public outreach (Teen Astronomy
Cafe, La Serena School for Data Science)
All notebooks are open-source and community contributions are welcome via pull request. We use them as living teaching materials in workshops at Astronomical Data Analysis Software & Systems (ADASS) and American Astronomical Society (AAS) conferences, summer schools, and university courses around the world. We have also recently translated most of our notebooks to the Spanish language.
In the coming months we will launch a tagged, searchable notebook gallery — filterable by science topic, Unified Astronomy Thesaurus (UAT) keywords, target audience, and difficulty level. The pilot framework was developed by two summer students working with the team.
Looking Ahead
Nine years in, the Astro Data Lab science platform is evolving on several fronts simultaneously.
GPU computing. We are deploying a GPU node, which will be connected to the Jupyter notebook service. This opens deep learning and large-scale ML workflows in the same notebook environment where the data lives.
An AI assistant. Our first-ever user survey, conducted in September 2025, ranked an in-notebook AI assistant as one of the top requested features. We are actively exploring what responsible, science-aware AI assistance looks like in this context — helping users construct SQL/ADQL queries, navigate datasets, and debug notebook code, without hallucinating catalog column names. jupyter-ai might come in very handy here.
New integrated Web Portal. Our Data Explorer — an integrated web interface combining catalog browsing, query execution, image cutouts, spectral search, and job status monitoring — was rolled out last year. Some of the next milestones include integration of AladinLite into the portal, bringing the sky-visualization capability described above out of the notebook and into the browser-native interface, and a new integrated positional cross-matching service.
Try It
The full notebook library is open-source: github.com/astro-datalab/notebooks-latest. Community notebook contributions are welcome — see CONTRIBUTING.md in the repository. You can also run all notebooks locally, after installing the Data Lab command-line client and Python module: `pip install astro-datalab`
Astro Data Lab also offers a JupyterLab environment as a service to the broad astronomy community — students, researchers, educators, and citizen scientists. Registration takes just a moment at datalab.noirlab.edu.
Questions and feedback: datalab@noirlab.edu
Robert Nikutta is a scientist at NSF NOIRLab’s Community Science and Data Center, and lead of the Astro Data Lab science platform. Stéphanie Juneau is an associate astronomer at CSDC and lead of the SPARCL spectroscopy initiative. The platform is the work of the full Astro Data Lab team, past and present.
<hr /><p>Exploring Petabytes of the Night Sky — Jupyter Notebooks at NOIRLab’s Astro Data Lab Science… was originally published in Jupyter Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>