YONDER

A pYthON package for Data dEnoising and Reconstruction

Main paper：J-PLUS: A catalogue of globular cluster candidates around the M81/M82/NGC3077 triplet of galaxies

YONDER is a package that uses singular value decomposition to perform low-rank data denoising and reconstruction. It takes a tabular data matrix and an error matrix as input and returns a denoised version of the original dataset as output. The approach enables a more accurate data analysis in the presence of uncertainties. Consequently, this package can be used as a simple toolbox to perform astronomical data cleaning.

How to install YONDER

The YONDER can be installed via the PyPI and pip:

pip install yonder

If you download the repository, you can also install it in the yonder directory:

git clone https://github.com/pengchzn/yonder
cd yonder
python setup.py install

How to use YONDER

Here is a simple example for the use of YONDER

from yonder import yonder
import numpy as np

#import the data
X = pd.read_csv('./datasets/Xobs.csv')
Xsd = pd.read_csv('./datasets/Xsd.csv')

# put the data into the algorithm
# Get the value
U, S, V = yonder.yonder(X, Xsd, 2)

# Get the denoised data
result = U @ S @ V.T

After the YONDER procedure, you can connect any additional algorithms or models to the denoised data.

Here is the distribution of noisy data and the distribution of denoised data in our test case:

_images/Noisy_data.png _images/Denoised_data.png

In addition, we simulate how the data is used on a daily basis, run the HDBScan on both sets of data, and show the findings. It is obvious from the figures below that YONDER may effectively reduce noise. When it comes to classification, denoised data can be quite beneficial, resulting in a superior outcome.

_images/Classification.png

You can test the test example in this notebook locally by yourself! If you are new to Python or don’t know how to run YONDER locally, you can click here to create a new Colaboratory notebook, so you can run YONDER in the cloud!

Requirements

  • python 3

  • numpy >= 1.21.5

  • Scipy >= 1.7.3

YONDER primarily uses the most recent version of Scipy for single value decomposition. Make sure your Scipy installation is up to date before using YONDER.

References