Obtaining Uncalibrated JWST Data
In this tutorial we will use astropy.fits.open and/or jwst_mast_query to obtain uncal JWST data files which are needed for the BREADS analysis.
Downloading public data (Easy)
If you know the exact filenames you would like to download and the datasets have completed their proprietary period to become public, it is straightforward to obtain them directly through the MAST API using astropy.fits.open.
[2]:
import astropy.io.fits as fits
import os
This cell must be modified! Specify the directory you would like to store data products in
[3]:
base_path = '/astro/epsig/tutorial/'
These subdirectories will be useful for organizing data products.
[4]:
data_path = base_path+'data/'
raw_path = data_path+'raw/'
create_paths = [base_path,data_path,raw_path]
for path in create_paths:
if not os.path.exists(path):
os.mkdir(path)
The next cell only downloads 4 files (2 for nrs1 and 2 for nrs2) for testing purposes. See next
[5]:
filelist = ['jw01414013001_02101_00001_nrs1_uncal.fits',
'jw01414013001_02101_00001_nrs2_uncal.fits',
'jw01414013001_02101_00002_nrs1_uncal.fits',
'jw01414013001_02101_00002_nrs2_uncal.fits']
for file in filelist:
print('downloading {} -> {}'.format(file,raw_path))
mast_file_url = f"https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/{file}"
hdul = fits.open(mast_file_url)
hdul.writeto(raw_path+file,overwrite=True)
downloading jw01414013001_02101_00001_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414013001_02101_00001_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414013001_02101_00002_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414013001_02101_00002_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
This loop iterates over a predefined sequence of observations from GTO program 1414, specifically sequences 12, 13, and 14, observations numbered 1-9, and both nrs1 and nrs2 detectors. The resulting files are downloaded and saved under the “raw” subdirectory.
[5]:
for seq_num in ['012','013','014']:
for obs_num in [str(x) for x in range(1,9+1)]:
for det_string in ['nrs1','nrs2']:
file = 'jw01414'+seq_num+'001_02101_0000'+obs_num+'_'+det_string+'_uncal.fits'
print('downloading {} -> {}'.format(file,raw_path))
mast_file_url = f"https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/{file}"
hdul = fits.open(mast_file_url)
hdul.writeto(raw_path+file,overwrite=True)
downloading jw01414012001_02101_00001_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414012001_02101_00001_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414012001_02101_00002_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414012001_02101_00002_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414012001_02101_00003_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414012001_02101_00003_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414012001_02101_00004_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
downloading jw01414012001_02101_00004_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/
---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
Cell In[5], line 7
5 print('downloading {} -> {}'.format(file,raw_path))
6 mast_file_url = f"https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/{file}"
----> 7 hdul = fits.open(mast_file_url)
8 hdul.writeto(raw_path+file,overwrite=True)
File ~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/io/fits/hdu/hdulist.py:220, in fitsopen(name, mode, memmap, save_backup, cache, lazy_load_hdus, ignore_missing_simple, use_fsspec, fsspec_kwargs, decompress_in_memory, **kwargs)
217 if not name:
218 raise ValueError(f"Empty filename: {name!r}")
--> 220 return HDUList.fromfile(
221 name,
222 mode,
223 memmap,
224 save_backup,
225 cache,
226 lazy_load_hdus,
227 ignore_missing_simple,
228 use_fsspec=use_fsspec,
229 fsspec_kwargs=fsspec_kwargs,
230 decompress_in_memory=decompress_in_memory,
231 **kwargs,
232 )
File ~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/io/fits/hdu/hdulist.py:484, in HDUList.fromfile(cls, fileobj, mode, memmap, save_backup, cache, lazy_load_hdus, ignore_missing_simple, **kwargs)
465 @classmethod
466 def fromfile(
467 cls,
(...) 475 **kwargs,
476 ):
477 """
478 Creates an `HDUList` instance from a file-like object.
479
(...) 482 documentation for details of the parameters accepted by this method).
483 """
--> 484 return cls._readfrom(
485 fileobj=fileobj,
486 mode=mode,
487 memmap=memmap,
488 save_backup=save_backup,
489 cache=cache,
490 ignore_missing_simple=ignore_missing_simple,
491 lazy_load_hdus=lazy_load_hdus,
492 **kwargs,
493 )
File ~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/io/fits/hdu/hdulist.py:1186, in HDUList._readfrom(cls, fileobj, data, mode, memmap, cache, lazy_load_hdus, ignore_missing_simple, use_fsspec, fsspec_kwargs, decompress_in_memory, **kwargs)
1183 if fileobj is not None:
1184 if not isinstance(fileobj, _File):
1185 # instantiate a FITS file object (ffo)
-> 1186 fileobj = _File(
1187 fileobj,
1188 mode=mode,
1189 memmap=memmap,
1190 cache=cache,
1191 use_fsspec=use_fsspec,
1192 fsspec_kwargs=fsspec_kwargs,
1193 decompress_in_memory=decompress_in_memory,
1194 )
1195 # The Astropy mode is determined by the _File initializer if the
1196 # supplied mode was None
1197 mode = fileobj.mode
File ~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/io/fits/file.py:220, in _File.__init__(self, fileobj, mode, memmap, overwrite, cache, use_fsspec, fsspec_kwargs, decompress_in_memory)
214 # Handle raw URLs
215 if (
216 isinstance(fileobj, (str, bytes))
217 and mode not in ("ostream", "append", "update")
218 and _is_url(fileobj)
219 ):
--> 220 self.name = download_file(fileobj, cache=cache)
221 # Handle responses from URL requests that have already been opened
222 elif isinstance(fileobj, http.client.HTTPResponse):
File ~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/utils/data.py:1548, in download_file(remote_url, cache, show_progress, timeout, sources, pkgname, http_headers, ssl_context, allow_insecure)
1546 for source_url in sources:
1547 try:
-> 1548 f_name = _download_file_from_source(
1549 source_url,
1550 timeout=timeout,
1551 show_progress=show_progress,
1552 cache=cache,
1553 remote_url=remote_url,
1554 pkgname=pkgname,
1555 http_headers=http_headers,
1556 ssl_context=ssl_context,
1557 allow_insecure=allow_insecure,
1558 )
1559 # Success!
1560 break
File ~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/utils/data.py:1375, in _download_file_from_source(source_url, show_progress, timeout, remote_url, cache, pkgname, http_headers, ftp_tls, ssl_context, allow_insecure)
1373 bytes_read += len(block)
1374 p.update(bytes_read)
-> 1375 block = remote.read(conf.download_block_size)
1376 if size is not None and bytes_read > size:
1377 raise urllib.error.URLError(
1378 f"File was supposed to be {size} bytes but "
1379 f"server provides more, at least {bytes_read} "
1380 "bytes. Download failed."
1381 )
File ~/anaconda3/envs/breads_2026/lib/python3.12/http/client.py:479, in HTTPResponse.read(self, amt)
476 if self.length is not None and amt > self.length:
477 # clip the read to the "end of response"
478 amt = self.length
--> 479 s = self.fp.read(amt)
480 if not s and amt:
481 # Ideally, we would raise IncompleteRead if the content-length
482 # wasn't satisfied, but it might break compatibility.
483 self._close_conn()
File ~/anaconda3/envs/breads_2026/lib/python3.12/socket.py:720, in SocketIO.readinto(self, b)
718 while True:
719 try:
--> 720 return self._sock.recv_into(b)
721 except timeout:
722 self._timeout_occurred = True
File ~/anaconda3/envs/breads_2026/lib/python3.12/ssl.py:1251, in SSLSocket.recv_into(self, buffer, nbytes, flags)
1247 if flags != 0:
1248 raise ValueError(
1249 "non-zero flags not allowed in calls to recv_into() on %s" %
1250 self.__class__)
-> 1251 return self.read(nbytes, buffer)
1252 else:
1253 return super().recv_into(buffer, nbytes, flags)
File ~/anaconda3/envs/breads_2026/lib/python3.12/ssl.py:1103, in SSLSocket.read(self, len, buffer)
1101 try:
1102 if buffer is not None:
-> 1103 return self._sslobj.read(len, buffer)
1104 else:
1105 return self._sslobj.read(len)
KeyboardInterrupt:
Thats it! If you just want to run the tutorial series, you can move onto notebook #2. If you are looking to download exclusive access data which is not public, the process is slightly more involved, but explained below.
Downloading exclusive access data with jwst_mast_query (Medium)
In order to obtain uncalibrated data files for observations which are not yet public, it is required (to my knowledge) to first install jwst_mast_query (https://github.com/spacetelescope/jwst_mast_query). Additionally, one must have acess to a MAST account with the proper exclusive access dataset authorization (ask your PI for help.) Once this is in order, you can genereate a specific exclusive access token at (https://auth.mast.stsci.edu/token) to properly communicate your privileged
status.
This cell must be modified! Replace the #### with your token.
[ ]:
token = '####'
os.environ["MAST_API_TOKEN"] = token
This cell will check your token is properly set as an environment varialbe.
[ ]:
import subprocess
return_token = subprocess.check_output('echo $MAST_API_TOKEN',shell=True)
assert token == str(return_token)[2:-3]
This cell must be modified! Replace ‘n’ with ‘y’ to proceed with the download. Otherwise it will simply print a list of files matching the search criteria.
[ ]:
DL = 'n'
This cell must be modified! Replace the path to your installation of jwst_mast_query, specifically the directory where you can find jwst_dowload.py
[ ]:
JWSTDL_path = '/home/amadurowicz/miniconda3/envs/stenv/bin/'
The following cell uses the shell to call jwst_download.py with specific arguments for our purpose. You can modify the propID and date_string to select other sequences of observations. The current configuration will download the tutorial dataset from GTO 1414 on HD 19467 B.
[ ]:
def download(target):
match target:
case 'HD 19467':
date_string = '2024-01-23 2024-01-26'
cmd = """\
while true; do echo {}; sleep 1; done | \
'{}jwst_download.py' \
--propID 1414 \
--instrument nirspec \
--filetypes 'uncal' \
--outrootdir '{}' \
--outsubdir 'data/raw' \
--skip_propID2outsubdir \
--date_select {} \
""".format(DL, JWSTDL_path, base_path, date_string)
subprocess.call(cmd,shell=True)
download('HD 19467')
[ ]: