Low-light and blurring issues are prevalent when capturing photos at night, often due to the use of long exposure to address dim environments. Addressing these joint problems can be challenging and error-prone if an end-to-end model is trained without incorporating an appropriate physical model. In this paper, we introduce JUDE, a Deep Joint Unrolling for Deblurring and Low-Light Image Enhancement, inspired by the image physical model. Based on Retinex theory and the blurring model, the low-light blurry input is iteratively deblurred and decomposed, producing sharp low-light reflectance and illuminance through an unrolling mechanism. Additionally, we incorporate various modules to estimate the initial blur kernel, enhance brightness, and eliminate noise in the final image. Comprehensive experiments on LOL-Blur and Real-LOL-Blur demonstrate that our method outperforms existing techniques both quantitatively and qualitatively.

We propose an indoor place recognition method using only a visitor map. A visitor map image serves as both the map and the database. No additional mapping or exploration is needed. OCR(Optical Character Recognition) can be an effective tool for extracting information from both camera images and map images. However, the camera image is cluttered with miscellaneous and distracting words, which disrupts accurate place recognition. The key contribution of our work is the enhancement of OCR-based place recognition performance through the aggregation of multiple likelihoods, which effectively addresses the issue of distracting words.

Meent is a Python-based Electromagnetic (EM) simulation package consisting of three main components: Modeling, EM Simulation, and Optimization. It employs Rigorous Coupled-Wave Analysis (RCWA) and supports Automatic Differentiation (AD), enabling seamless integration of machine learning (ML) and optics research.To demonstrate its versatility as a research platform, we present three applications: (1) generating datasets for training neural operators,(2) serving as an environment for reinforcement learning-based nanophotonic device optimization, and(3) solving inverse problems using gradient-based optimizers.

Finding an optimal device structure in the vast combinatorial design space of freeform nanophotonic design has been an enormous challenge. In this study, we propose physics-informed reinforcement learning (PIRL) that combines the adjoint-based method with reinforcement learning to improve the sample efficiency by an order of magnitude compared to conventional reinforcement learning and overcome the issue of local minima. To illustrate these advantages of PIRL over other conventional optimization algorithms, we design a family of one-dimensional metasurface beam deflectors using PIRL, exceeding most reported records. We also explore the transfer learning capability of PIRL that further improves sample efficiency and demonstrate how the minimum feature size of the design can be enforced in PIRL through reward engineering. With its high sample efficiency, robustness, and ability to seamlessly incorporate practical device design constraints, our method offers a promising approach to highly combinatorial freeform device optimization in various physical domains.

One way to understand and debug IR optimization process is to visualize Control Flow Graphs (CFGs) before and after optimization and compare them. However, comparing the CFGs can be challenging since they can vary significantly. LLVM-FLOW, an open-source interactive CFG visualization web app, is developed to ease the difficulty by highlighting the same components in two graphs. With this tool, users can easily find corresponding nodes in another graph by clicking the highlighted node. LLVM-FLOW is a useful tool not only for experienced LLVM developers seeking to better understand the IR flow when writing custom passes, but also for newcomers to the LLVM ecosystem who wish to study the behavior of IR patterns.

The increasing demand on a versatile high-performance metasurface requires a freeform design method that can handle a huge design space, which is many orders of magnitude larger than that of conventional fixed-shape optical structures. In this work, we formulate the designing process of one-dimensional freeform Si metasurface beam deflectors as a reinforcement learning problem to find their optimal structures consistently without requiring any prior metasurface data. During training, a deep Q-network-based agent stochastically explores the device design space around the learned trajectory optimized for deflection efficiency. The devices discovered by the agents show overall improvements in maximum efficiency compared to the ones that state-of-the-art baseline methods find at various wavelengths and deflection angles.

Reinforcement Learning (RL) has become one of the major categories in the field of machine learning in the recent years via the breakthroughs such as approximation of complex non-linear function through deep neural networks. Among these, a distributional perspective on the value function estimation has contributed on taking a big jump in the performance of RL algorithms. However, proper discussions of distributional RL (DRL) are still limited to specific algorithms or network architectures such as Q-learning or deterministic policy gradient. In the situation at hand, we have worked to address some of the critical aspects in RL that was left out in the distributional perspective. The details and the findings of this journey can be found in our recent work 'GMAC: A Distributional'