{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a842c4c6-a4a2-42c4-9317-52765d37d13f",
   "metadata": {},
   "source": [
    "# Obtaining Uncalibrated JWST Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cd261e6-1ff0-4d5d-9418-054085c49f5e",
   "metadata": {},
   "source": [
    "In this tutorial we will use ```astropy.fits.open``` and/or ```jwst_mast_query``` to obtain ```uncal``` JWST data files which are needed for the BREADS analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10f2399b-ef04-4f58-b077-088a68cf81c9",
   "metadata": {},
   "source": [
    "### Downloading public data (Easy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1302560a-00de-41c5-8891-fda63f3bef1c",
   "metadata": {},
   "source": [
    "If you know the exact filenames you would like to download and the datasets have completed their proprietary period to become public, it is straightforward to obtain them directly through the MAST API using ```astropy.fits.open```."
   ]
  },
  {
   "cell_type": "code",
   "id": "b2244d80-490c-42b3-9e6d-b958d0fc08a6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-02-20T23:53:09.383197Z",
     "start_time": "2026-02-20T23:53:09.314224Z"
    }
   },
   "source": [
    "import astropy.io.fits as fits\n",
    "import os"
   ],
   "outputs": [],
   "execution_count": 2
  },
  {
   "cell_type": "markdown",
   "id": "6e9ae0df-ca66-427b-a16e-d87b22767a85",
   "metadata": {},
   "source": [
    "**This cell must be modified!** Specify the directory you would like to store data products in"
   ]
  },
  {
   "cell_type": "code",
   "id": "0b4955d1-337a-4741-b43a-3c7e04b193f4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-02-20T23:53:09.688271Z",
     "start_time": "2026-02-20T23:53:09.685885Z"
    }
   },
   "source": "base_path = '/astro/epsig/tutorial/'",
   "outputs": [],
   "execution_count": 3
  },
  {
   "cell_type": "markdown",
   "id": "1d428d63-c2d6-42d9-98cd-db99574ff0dd",
   "metadata": {},
   "source": [
    "These subdirectories will be useful for organizing data products."
   ]
  },
  {
   "cell_type": "code",
   "id": "afee1f3f-c6b8-4722-a7a3-189a86efbf2d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-02-20T23:53:10.336354Z",
     "start_time": "2026-02-20T23:53:10.333399Z"
    }
   },
   "source": [
    "data_path = base_path+'data/'\n",
    "raw_path = data_path+'raw/'\n",
    "create_paths = [base_path,data_path,raw_path]\n",
    "for path in create_paths:\n",
    "    if not os.path.exists(path):\n",
    "        os.mkdir(path)"
   ],
   "outputs": [],
   "execution_count": 4
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "The next cell only downloads 4 files (2 for nrs1 and 2 for nrs2) for testing purposes. See next",
   "id": "a71c12626d03447e"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2026-02-20T23:53:59.904909Z",
     "start_time": "2026-02-20T23:53:11.796079Z"
    }
   },
   "cell_type": "code",
   "source": [
    "filelist = ['jw01414013001_02101_00001_nrs1_uncal.fits',\n",
    "            'jw01414013001_02101_00001_nrs2_uncal.fits',\n",
    "            'jw01414013001_02101_00002_nrs1_uncal.fits',\n",
    "            'jw01414013001_02101_00002_nrs2_uncal.fits']\n",
    "for file in filelist:\n",
    "    print('downloading {} -> {}'.format(file,raw_path))\n",
    "    mast_file_url = f\"https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/{file}\"\n",
    "    hdul = fits.open(mast_file_url)\n",
    "    hdul.writeto(raw_path+file,overwrite=True)"
   ],
   "id": "f7bf696f0782dbf8",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "downloading jw01414013001_02101_00001_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414013001_02101_00001_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414013001_02101_00002_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414013001_02101_00002_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n"
     ]
    }
   ],
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "id": "87c4e304-a0ae-4195-8369-024026415d26",
   "metadata": {},
   "source": [
    "This loop iterates over a predefined sequence of observations from GTO program 1414, specifically sequences 12, 13, and 14, observations numbered 1-9, and both nrs1 and nrs2 detectors. The resulting files are downloaded and saved under the \"raw\" subdirectory."
   ]
  },
  {
   "cell_type": "code",
   "id": "b5554c5d-529f-439a-9242-6445e38a3920",
   "metadata": {
    "scrolled": true,
    "ExecuteTime": {
     "end_time": "2026-02-20T04:53:02.254064Z",
     "start_time": "2026-02-20T04:50:53.745490Z"
    }
   },
   "source": [
    "for seq_num in ['012','013','014']:\n",
    "    for obs_num in [str(x) for x in range(1,9+1)]:\n",
    "        for det_string in ['nrs1','nrs2']:\n",
    "            file = 'jw01414'+seq_num+'001_02101_0000'+obs_num+'_'+det_string+'_uncal.fits'\n",
    "            print('downloading {} -> {}'.format(file,raw_path))\n",
    "            mast_file_url = f\"https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/{file}\"\n",
    "            hdul = fits.open(mast_file_url)\n",
    "            hdul.writeto(raw_path+file,overwrite=True)"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "downloading jw01414012001_02101_00001_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414012001_02101_00001_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414012001_02101_00002_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414012001_02101_00002_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414012001_02101_00003_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414012001_02101_00003_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414012001_02101_00004_nrs1_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n",
      "downloading jw01414012001_02101_00004_nrs2_uncal.fits -> /stow/jruffio/data/JWST/tutorial/data/raw/\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001B[31m---------------------------------------------------------------------------\u001B[39m",
      "\u001B[31mKeyboardInterrupt\u001B[39m                         Traceback (most recent call last)",
      "\u001B[36mCell\u001B[39m\u001B[36m \u001B[39m\u001B[32mIn[5]\u001B[39m\u001B[32m, line 7\u001B[39m\n\u001B[32m      5\u001B[39m \u001B[38;5;28mprint\u001B[39m(\u001B[33m'\u001B[39m\u001B[33mdownloading \u001B[39m\u001B[38;5;132;01m{}\u001B[39;00m\u001B[33m -> \u001B[39m\u001B[38;5;132;01m{}\u001B[39;00m\u001B[33m'\u001B[39m.format(file,raw_path))\n\u001B[32m      6\u001B[39m mast_file_url = \u001B[33mf\u001B[39m\u001B[33m\"\u001B[39m\u001B[33mhttps://mast.stsci.edu/api/v0.1/Download/file?uri=mast:JWST/product/\u001B[39m\u001B[38;5;132;01m{\u001B[39;00mfile\u001B[38;5;132;01m}\u001B[39;00m\u001B[33m\"\u001B[39m\n\u001B[32m----> \u001B[39m\u001B[32m7\u001B[39m hdul = \u001B[43mfits\u001B[49m\u001B[43m.\u001B[49m\u001B[43mopen\u001B[49m\u001B[43m(\u001B[49m\u001B[43mmast_file_url\u001B[49m\u001B[43m)\u001B[49m\n\u001B[32m      8\u001B[39m hdul.writeto(raw_path+file,overwrite=\u001B[38;5;28;01mTrue\u001B[39;00m)\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/io/fits/hdu/hdulist.py:220\u001B[39m, in \u001B[36mfitsopen\u001B[39m\u001B[34m(name, mode, memmap, save_backup, cache, lazy_load_hdus, ignore_missing_simple, use_fsspec, fsspec_kwargs, decompress_in_memory, **kwargs)\u001B[39m\n\u001B[32m    217\u001B[39m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m name:\n\u001B[32m    218\u001B[39m     \u001B[38;5;28;01mraise\u001B[39;00m \u001B[38;5;167;01mValueError\u001B[39;00m(\u001B[33mf\u001B[39m\u001B[33m\"\u001B[39m\u001B[33mEmpty filename: \u001B[39m\u001B[38;5;132;01m{\u001B[39;00mname\u001B[38;5;132;01m!r}\u001B[39;00m\u001B[33m\"\u001B[39m)\n\u001B[32m--> \u001B[39m\u001B[32m220\u001B[39m \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[43mHDUList\u001B[49m\u001B[43m.\u001B[49m\u001B[43mfromfile\u001B[49m\u001B[43m(\u001B[49m\n\u001B[32m    221\u001B[39m \u001B[43m    \u001B[49m\u001B[43mname\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    222\u001B[39m \u001B[43m    \u001B[49m\u001B[43mmode\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    223\u001B[39m \u001B[43m    \u001B[49m\u001B[43mmemmap\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    224\u001B[39m \u001B[43m    \u001B[49m\u001B[43msave_backup\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    225\u001B[39m \u001B[43m    \u001B[49m\u001B[43mcache\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    226\u001B[39m \u001B[43m    \u001B[49m\u001B[43mlazy_load_hdus\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    227\u001B[39m \u001B[43m    \u001B[49m\u001B[43mignore_missing_simple\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    228\u001B[39m \u001B[43m    \u001B[49m\u001B[43muse_fsspec\u001B[49m\u001B[43m=\u001B[49m\u001B[43muse_fsspec\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    229\u001B[39m \u001B[43m    \u001B[49m\u001B[43mfsspec_kwargs\u001B[49m\u001B[43m=\u001B[49m\u001B[43mfsspec_kwargs\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    230\u001B[39m \u001B[43m    \u001B[49m\u001B[43mdecompress_in_memory\u001B[49m\u001B[43m=\u001B[49m\u001B[43mdecompress_in_memory\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    231\u001B[39m \u001B[43m    \u001B[49m\u001B[43m*\u001B[49m\u001B[43m*\u001B[49m\u001B[43mkwargs\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    232\u001B[39m \u001B[43m\u001B[49m\u001B[43m)\u001B[49m\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/io/fits/hdu/hdulist.py:484\u001B[39m, in \u001B[36mHDUList.fromfile\u001B[39m\u001B[34m(cls, fileobj, mode, memmap, save_backup, cache, lazy_load_hdus, ignore_missing_simple, **kwargs)\u001B[39m\n\u001B[32m    465\u001B[39m \u001B[38;5;129m@classmethod\u001B[39m\n\u001B[32m    466\u001B[39m \u001B[38;5;28;01mdef\u001B[39;00m\u001B[38;5;250m \u001B[39m\u001B[34mfromfile\u001B[39m(\n\u001B[32m    467\u001B[39m     \u001B[38;5;28mcls\u001B[39m,\n\u001B[32m   (...)\u001B[39m\u001B[32m    475\u001B[39m     **kwargs,\n\u001B[32m    476\u001B[39m ):\n\u001B[32m    477\u001B[39m \u001B[38;5;250m    \u001B[39m\u001B[33;03m\"\"\"\u001B[39;00m\n\u001B[32m    478\u001B[39m \u001B[33;03m    Creates an `HDUList` instance from a file-like object.\u001B[39;00m\n\u001B[32m    479\u001B[39m \n\u001B[32m   (...)\u001B[39m\u001B[32m    482\u001B[39m \u001B[33;03m    documentation for details of the parameters accepted by this method).\u001B[39;00m\n\u001B[32m    483\u001B[39m \u001B[33;03m    \"\"\"\u001B[39;00m\n\u001B[32m--> \u001B[39m\u001B[32m484\u001B[39m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mcls\u001B[39;49m\u001B[43m.\u001B[49m\u001B[43m_readfrom\u001B[49m\u001B[43m(\u001B[49m\n\u001B[32m    485\u001B[39m \u001B[43m        \u001B[49m\u001B[43mfileobj\u001B[49m\u001B[43m=\u001B[49m\u001B[43mfileobj\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    486\u001B[39m \u001B[43m        \u001B[49m\u001B[43mmode\u001B[49m\u001B[43m=\u001B[49m\u001B[43mmode\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    487\u001B[39m \u001B[43m        \u001B[49m\u001B[43mmemmap\u001B[49m\u001B[43m=\u001B[49m\u001B[43mmemmap\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    488\u001B[39m \u001B[43m        \u001B[49m\u001B[43msave_backup\u001B[49m\u001B[43m=\u001B[49m\u001B[43msave_backup\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    489\u001B[39m \u001B[43m        \u001B[49m\u001B[43mcache\u001B[49m\u001B[43m=\u001B[49m\u001B[43mcache\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    490\u001B[39m \u001B[43m        \u001B[49m\u001B[43mignore_missing_simple\u001B[49m\u001B[43m=\u001B[49m\u001B[43mignore_missing_simple\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    491\u001B[39m \u001B[43m        \u001B[49m\u001B[43mlazy_load_hdus\u001B[49m\u001B[43m=\u001B[49m\u001B[43mlazy_load_hdus\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    492\u001B[39m \u001B[43m        \u001B[49m\u001B[43m*\u001B[49m\u001B[43m*\u001B[49m\u001B[43mkwargs\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m    493\u001B[39m \u001B[43m    \u001B[49m\u001B[43m)\u001B[49m\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/io/fits/hdu/hdulist.py:1186\u001B[39m, in \u001B[36mHDUList._readfrom\u001B[39m\u001B[34m(cls, fileobj, data, mode, memmap, cache, lazy_load_hdus, ignore_missing_simple, use_fsspec, fsspec_kwargs, decompress_in_memory, **kwargs)\u001B[39m\n\u001B[32m   1183\u001B[39m \u001B[38;5;28;01mif\u001B[39;00m fileobj \u001B[38;5;129;01mis\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m \u001B[38;5;28;01mNone\u001B[39;00m:\n\u001B[32m   1184\u001B[39m     \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m \u001B[38;5;28misinstance\u001B[39m(fileobj, _File):\n\u001B[32m   1185\u001B[39m         \u001B[38;5;66;03m# instantiate a FITS file object (ffo)\u001B[39;00m\n\u001B[32m-> \u001B[39m\u001B[32m1186\u001B[39m         fileobj = \u001B[43m_File\u001B[49m\u001B[43m(\u001B[49m\n\u001B[32m   1187\u001B[39m \u001B[43m            \u001B[49m\u001B[43mfileobj\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1188\u001B[39m \u001B[43m            \u001B[49m\u001B[43mmode\u001B[49m\u001B[43m=\u001B[49m\u001B[43mmode\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1189\u001B[39m \u001B[43m            \u001B[49m\u001B[43mmemmap\u001B[49m\u001B[43m=\u001B[49m\u001B[43mmemmap\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1190\u001B[39m \u001B[43m            \u001B[49m\u001B[43mcache\u001B[49m\u001B[43m=\u001B[49m\u001B[43mcache\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1191\u001B[39m \u001B[43m            \u001B[49m\u001B[43muse_fsspec\u001B[49m\u001B[43m=\u001B[49m\u001B[43muse_fsspec\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1192\u001B[39m \u001B[43m            \u001B[49m\u001B[43mfsspec_kwargs\u001B[49m\u001B[43m=\u001B[49m\u001B[43mfsspec_kwargs\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1193\u001B[39m \u001B[43m            \u001B[49m\u001B[43mdecompress_in_memory\u001B[49m\u001B[43m=\u001B[49m\u001B[43mdecompress_in_memory\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1194\u001B[39m \u001B[43m        \u001B[49m\u001B[43m)\u001B[49m\n\u001B[32m   1195\u001B[39m     \u001B[38;5;66;03m# The Astropy mode is determined by the _File initializer if the\u001B[39;00m\n\u001B[32m   1196\u001B[39m     \u001B[38;5;66;03m# supplied mode was None\u001B[39;00m\n\u001B[32m   1197\u001B[39m     mode = fileobj.mode\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/io/fits/file.py:220\u001B[39m, in \u001B[36m_File.__init__\u001B[39m\u001B[34m(self, fileobj, mode, memmap, overwrite, cache, use_fsspec, fsspec_kwargs, decompress_in_memory)\u001B[39m\n\u001B[32m    214\u001B[39m \u001B[38;5;66;03m# Handle raw URLs\u001B[39;00m\n\u001B[32m    215\u001B[39m \u001B[38;5;28;01mif\u001B[39;00m (\n\u001B[32m    216\u001B[39m     \u001B[38;5;28misinstance\u001B[39m(fileobj, (\u001B[38;5;28mstr\u001B[39m, \u001B[38;5;28mbytes\u001B[39m))\n\u001B[32m    217\u001B[39m     \u001B[38;5;129;01mand\u001B[39;00m mode \u001B[38;5;129;01mnot\u001B[39;00m \u001B[38;5;129;01min\u001B[39;00m (\u001B[33m\"\u001B[39m\u001B[33mostream\u001B[39m\u001B[33m\"\u001B[39m, \u001B[33m\"\u001B[39m\u001B[33mappend\u001B[39m\u001B[33m\"\u001B[39m, \u001B[33m\"\u001B[39m\u001B[33mupdate\u001B[39m\u001B[33m\"\u001B[39m)\n\u001B[32m    218\u001B[39m     \u001B[38;5;129;01mand\u001B[39;00m _is_url(fileobj)\n\u001B[32m    219\u001B[39m ):\n\u001B[32m--> \u001B[39m\u001B[32m220\u001B[39m     \u001B[38;5;28mself\u001B[39m.name = \u001B[43mdownload_file\u001B[49m\u001B[43m(\u001B[49m\u001B[43mfileobj\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mcache\u001B[49m\u001B[43m=\u001B[49m\u001B[43mcache\u001B[49m\u001B[43m)\u001B[49m\n\u001B[32m    221\u001B[39m \u001B[38;5;66;03m# Handle responses from URL requests that have already been opened\u001B[39;00m\n\u001B[32m    222\u001B[39m \u001B[38;5;28;01melif\u001B[39;00m \u001B[38;5;28misinstance\u001B[39m(fileobj, http.client.HTTPResponse):\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/utils/data.py:1548\u001B[39m, in \u001B[36mdownload_file\u001B[39m\u001B[34m(remote_url, cache, show_progress, timeout, sources, pkgname, http_headers, ssl_context, allow_insecure)\u001B[39m\n\u001B[32m   1546\u001B[39m \u001B[38;5;28;01mfor\u001B[39;00m source_url \u001B[38;5;129;01min\u001B[39;00m sources:\n\u001B[32m   1547\u001B[39m     \u001B[38;5;28;01mtry\u001B[39;00m:\n\u001B[32m-> \u001B[39m\u001B[32m1548\u001B[39m         f_name = \u001B[43m_download_file_from_source\u001B[49m\u001B[43m(\u001B[49m\n\u001B[32m   1549\u001B[39m \u001B[43m            \u001B[49m\u001B[43msource_url\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1550\u001B[39m \u001B[43m            \u001B[49m\u001B[43mtimeout\u001B[49m\u001B[43m=\u001B[49m\u001B[43mtimeout\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1551\u001B[39m \u001B[43m            \u001B[49m\u001B[43mshow_progress\u001B[49m\u001B[43m=\u001B[49m\u001B[43mshow_progress\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1552\u001B[39m \u001B[43m            \u001B[49m\u001B[43mcache\u001B[49m\u001B[43m=\u001B[49m\u001B[43mcache\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1553\u001B[39m \u001B[43m            \u001B[49m\u001B[43mremote_url\u001B[49m\u001B[43m=\u001B[49m\u001B[43mremote_url\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1554\u001B[39m \u001B[43m            \u001B[49m\u001B[43mpkgname\u001B[49m\u001B[43m=\u001B[49m\u001B[43mpkgname\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1555\u001B[39m \u001B[43m            \u001B[49m\u001B[43mhttp_headers\u001B[49m\u001B[43m=\u001B[49m\u001B[43mhttp_headers\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1556\u001B[39m \u001B[43m            \u001B[49m\u001B[43mssl_context\u001B[49m\u001B[43m=\u001B[49m\u001B[43mssl_context\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1557\u001B[39m \u001B[43m            \u001B[49m\u001B[43mallow_insecure\u001B[49m\u001B[43m=\u001B[49m\u001B[43mallow_insecure\u001B[49m\u001B[43m,\u001B[49m\n\u001B[32m   1558\u001B[39m \u001B[43m        \u001B[49m\u001B[43m)\u001B[49m\n\u001B[32m   1559\u001B[39m         \u001B[38;5;66;03m# Success!\u001B[39;00m\n\u001B[32m   1560\u001B[39m         \u001B[38;5;28;01mbreak\u001B[39;00m\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/site-packages/astropy/utils/data.py:1375\u001B[39m, in \u001B[36m_download_file_from_source\u001B[39m\u001B[34m(source_url, show_progress, timeout, remote_url, cache, pkgname, http_headers, ftp_tls, ssl_context, allow_insecure)\u001B[39m\n\u001B[32m   1373\u001B[39m bytes_read += \u001B[38;5;28mlen\u001B[39m(block)\n\u001B[32m   1374\u001B[39m p.update(bytes_read)\n\u001B[32m-> \u001B[39m\u001B[32m1375\u001B[39m block = \u001B[43mremote\u001B[49m\u001B[43m.\u001B[49m\u001B[43mread\u001B[49m\u001B[43m(\u001B[49m\u001B[43mconf\u001B[49m\u001B[43m.\u001B[49m\u001B[43mdownload_block_size\u001B[49m\u001B[43m)\u001B[49m\n\u001B[32m   1376\u001B[39m \u001B[38;5;28;01mif\u001B[39;00m size \u001B[38;5;129;01mis\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m \u001B[38;5;28;01mNone\u001B[39;00m \u001B[38;5;129;01mand\u001B[39;00m bytes_read > size:\n\u001B[32m   1377\u001B[39m     \u001B[38;5;28;01mraise\u001B[39;00m urllib.error.URLError(\n\u001B[32m   1378\u001B[39m         \u001B[33mf\u001B[39m\u001B[33m\"\u001B[39m\u001B[33mFile was supposed to be \u001B[39m\u001B[38;5;132;01m{\u001B[39;00msize\u001B[38;5;132;01m}\u001B[39;00m\u001B[33m bytes but \u001B[39m\u001B[33m\"\u001B[39m\n\u001B[32m   1379\u001B[39m         \u001B[33mf\u001B[39m\u001B[33m\"\u001B[39m\u001B[33mserver provides more, at least \u001B[39m\u001B[38;5;132;01m{\u001B[39;00mbytes_read\u001B[38;5;132;01m}\u001B[39;00m\u001B[33m \u001B[39m\u001B[33m\"\u001B[39m\n\u001B[32m   1380\u001B[39m         \u001B[33m\"\u001B[39m\u001B[33mbytes. Download failed.\u001B[39m\u001B[33m\"\u001B[39m\n\u001B[32m   1381\u001B[39m     )\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/http/client.py:479\u001B[39m, in \u001B[36mHTTPResponse.read\u001B[39m\u001B[34m(self, amt)\u001B[39m\n\u001B[32m    476\u001B[39m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;28mself\u001B[39m.length \u001B[38;5;129;01mis\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m \u001B[38;5;28;01mNone\u001B[39;00m \u001B[38;5;129;01mand\u001B[39;00m amt > \u001B[38;5;28mself\u001B[39m.length:\n\u001B[32m    477\u001B[39m     \u001B[38;5;66;03m# clip the read to the \"end of response\"\u001B[39;00m\n\u001B[32m    478\u001B[39m     amt = \u001B[38;5;28mself\u001B[39m.length\n\u001B[32m--> \u001B[39m\u001B[32m479\u001B[39m s = \u001B[38;5;28;43mself\u001B[39;49m\u001B[43m.\u001B[49m\u001B[43mfp\u001B[49m\u001B[43m.\u001B[49m\u001B[43mread\u001B[49m\u001B[43m(\u001B[49m\u001B[43mamt\u001B[49m\u001B[43m)\u001B[49m\n\u001B[32m    480\u001B[39m \u001B[38;5;28;01mif\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m s \u001B[38;5;129;01mand\u001B[39;00m amt:\n\u001B[32m    481\u001B[39m     \u001B[38;5;66;03m# Ideally, we would raise IncompleteRead if the content-length\u001B[39;00m\n\u001B[32m    482\u001B[39m     \u001B[38;5;66;03m# wasn't satisfied, but it might break compatibility.\u001B[39;00m\n\u001B[32m    483\u001B[39m     \u001B[38;5;28mself\u001B[39m._close_conn()\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/socket.py:720\u001B[39m, in \u001B[36mSocketIO.readinto\u001B[39m\u001B[34m(self, b)\u001B[39m\n\u001B[32m    718\u001B[39m \u001B[38;5;28;01mwhile\u001B[39;00m \u001B[38;5;28;01mTrue\u001B[39;00m:\n\u001B[32m    719\u001B[39m     \u001B[38;5;28;01mtry\u001B[39;00m:\n\u001B[32m--> \u001B[39m\u001B[32m720\u001B[39m         \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[43m.\u001B[49m\u001B[43m_sock\u001B[49m\u001B[43m.\u001B[49m\u001B[43mrecv_into\u001B[49m\u001B[43m(\u001B[49m\u001B[43mb\u001B[49m\u001B[43m)\u001B[49m\n\u001B[32m    721\u001B[39m     \u001B[38;5;28;01mexcept\u001B[39;00m timeout:\n\u001B[32m    722\u001B[39m         \u001B[38;5;28mself\u001B[39m._timeout_occurred = \u001B[38;5;28;01mTrue\u001B[39;00m\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/ssl.py:1251\u001B[39m, in \u001B[36mSSLSocket.recv_into\u001B[39m\u001B[34m(self, buffer, nbytes, flags)\u001B[39m\n\u001B[32m   1247\u001B[39m     \u001B[38;5;28;01mif\u001B[39;00m flags != \u001B[32m0\u001B[39m:\n\u001B[32m   1248\u001B[39m         \u001B[38;5;28;01mraise\u001B[39;00m \u001B[38;5;167;01mValueError\u001B[39;00m(\n\u001B[32m   1249\u001B[39m           \u001B[33m\"\u001B[39m\u001B[33mnon-zero flags not allowed in calls to recv_into() on \u001B[39m\u001B[38;5;132;01m%s\u001B[39;00m\u001B[33m\"\u001B[39m %\n\u001B[32m   1250\u001B[39m           \u001B[38;5;28mself\u001B[39m.\u001B[34m__class__\u001B[39m)\n\u001B[32m-> \u001B[39m\u001B[32m1251\u001B[39m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[43m.\u001B[49m\u001B[43mread\u001B[49m\u001B[43m(\u001B[49m\u001B[43mnbytes\u001B[49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mbuffer\u001B[49m\u001B[43m)\u001B[49m\n\u001B[32m   1252\u001B[39m \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[32m   1253\u001B[39m     \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28msuper\u001B[39m().recv_into(buffer, nbytes, flags)\n",
      "\u001B[36mFile \u001B[39m\u001B[32m~/anaconda3/envs/breads_2026/lib/python3.12/ssl.py:1103\u001B[39m, in \u001B[36mSSLSocket.read\u001B[39m\u001B[34m(self, len, buffer)\u001B[39m\n\u001B[32m   1101\u001B[39m \u001B[38;5;28;01mtry\u001B[39;00m:\n\u001B[32m   1102\u001B[39m     \u001B[38;5;28;01mif\u001B[39;00m buffer \u001B[38;5;129;01mis\u001B[39;00m \u001B[38;5;129;01mnot\u001B[39;00m \u001B[38;5;28;01mNone\u001B[39;00m:\n\u001B[32m-> \u001B[39m\u001B[32m1103\u001B[39m         \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28;43mself\u001B[39;49m\u001B[43m.\u001B[49m\u001B[43m_sslobj\u001B[49m\u001B[43m.\u001B[49m\u001B[43mread\u001B[49m\u001B[43m(\u001B[49m\u001B[38;5;28;43mlen\u001B[39;49m\u001B[43m,\u001B[49m\u001B[43m \u001B[49m\u001B[43mbuffer\u001B[49m\u001B[43m)\u001B[49m\n\u001B[32m   1104\u001B[39m     \u001B[38;5;28;01melse\u001B[39;00m:\n\u001B[32m   1105\u001B[39m         \u001B[38;5;28;01mreturn\u001B[39;00m \u001B[38;5;28mself\u001B[39m._sslobj.read(\u001B[38;5;28mlen\u001B[39m)\n",
      "\u001B[31mKeyboardInterrupt\u001B[39m: "
     ]
    }
   ],
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "id": "1f0fb1ad-6fb7-4bea-b07f-b43f889c1bb5",
   "metadata": {},
   "source": [
    "Thats it! If you just want to run the tutorial series, you can move onto notebook #2. If you are looking to download exclusive access data which is not public, the process is slightly more involved, but explained below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a78e81d-413a-4102-8c12-762f70d35bbd",
   "metadata": {},
   "source": [
    "### Downloading exclusive access data with jwst_mast_query (Medium)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cd364ad-ac8f-40fc-a9ea-a0a81989949a",
   "metadata": {},
   "source": [
    "In order to obtain uncalibrated data files for observations which are not yet public, it is required (to my knowledge) to first install ```jwst_mast_query``` (https://github.com/spacetelescope/jwst_mast_query). Additionally, one must have acess to a MAST account with the proper exclusive access dataset authorization (ask your PI for help.) Once this is in order, you can genereate a specific exclusive access token at (https://auth.mast.stsci.edu/token) to properly communicate your privileged status."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0f6f430-8049-4e45-8afe-5979cf56cd2f",
   "metadata": {},
   "source": [
    "**This cell must be modified!** Replace the #### with your token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "451368f9-5c3e-49f9-8b50-30da4fa35064",
   "metadata": {},
   "outputs": [],
   "source": [
    "token = '####'\n",
    "\n",
    "os.environ[\"MAST_API_TOKEN\"] = token"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bde2e2a-3c90-4ede-b011-8e37594de0a4",
   "metadata": {},
   "source": [
    "This cell will check your token is properly set as an environment varialbe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ff457e2-5506-4cff-84b2-9cea9c5cf287",
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess\n",
    "return_token = subprocess.check_output('echo $MAST_API_TOKEN',shell=True)\n",
    "assert token == str(return_token)[2:-3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b93fc25-e6ee-40c4-a57a-7fed96c2db8d",
   "metadata": {},
   "source": [
    "**This cell must be modified!** Replace 'n' with 'y' to proceed with the download. Otherwise it will simply print a list of files matching the search criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83f48506-af88-4d5c-acf8-137870b2faf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "DL = 'n'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20d9600a-5572-4953-8a6e-ba46e243d991",
   "metadata": {},
   "source": [
    "**This cell must be modified!** Replace the path to your installation of ```jwst_mast_query```, specifically the directory where you can find ```jwst_dowload.py```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f4ba7fb-cfb6-4601-a094-6f98c6561536",
   "metadata": {},
   "outputs": [],
   "source": [
    "JWSTDL_path = '/home/amadurowicz/miniconda3/envs/stenv/bin/'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78067e5e-bc77-40e6-98cf-c83af48b8728",
   "metadata": {},
   "source": [
    "The following cell uses the shell to call ```jwst_download.py``` with specific arguments for our purpose. You can modify the ```propID``` and ```date_string``` to select other sequences of observations. The current configuration will download the tutorial dataset from GTO 1414 on HD 19467 B."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e057eef-d975-4110-9dc7-b8a1fc0886d5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def download(target):\n",
    "    \n",
    "    match target:\n",
    "        case 'HD 19467':\n",
    "            date_string = '2024-01-23 2024-01-26'\n",
    "    \n",
    "    cmd = \"\"\"\\\n",
    "    while true; do echo {}; sleep 1; done | \\\n",
    "    '{}jwst_download.py' \\\n",
    "    --propID 1414 \\\n",
    "    --instrument nirspec \\\n",
    "    --filetypes 'uncal' \\\n",
    "    --outrootdir '{}' \\\n",
    "    --outsubdir 'data/raw' \\\n",
    "    --skip_propID2outsubdir \\\n",
    "    --date_select {} \\\n",
    "    \"\"\".format(DL, JWSTDL_path, base_path, date_string)\n",
    "    \n",
    "    subprocess.call(cmd,shell=True)\n",
    "\n",
    "download('HD 19467')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4041da55-a3b9-41b8-a485-b93b7dc660c0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}