论文网址：MindPainter: Efficient Brain-Conditioned Painting of Natural Images via Cross-Modal Self-Supervised Learning | Proceedings of the AAAI Conference on Artificial Intelligence

英文是纯手打的！论文原文的summarizing and paraphrasing。可能会出现难以避免的拼写错误和语法错误，若有发现欢迎评论指正！文章偏向于笔记，谨慎食用

目录

1. 心得

2. 论文逐段精读

2.1. Abstract

2.2. Introduction

2.3. Related Work

2.4. Method

2.4.1. Our Method

2.5. Experiment

2.5.1. Dataset and Implementation

2.5.2. Qualitative Illustration

2.5.3. Comparisons

2.5.4. Ablation Study

2.5.5. Effectiveness of Designed Modules

2.6. Conclusion

1. 心得

（1）不会真有人放暑假吧？

（2）任务确实特别，用脑信号修改/定制图片，类似于脑信号P图。难绷

2. 论文逐段精读

2.1. Abstract

        ①MindPainter aims to ahieve brain-conditioned image painting

2.2. Introduction

        ①Two steps of image editing, including reconstruction and prompt, is inefficient due to simple combination, large modality gap and limited representation ablity

2.3. Related Work

        ①Lists other generation model, and points out no one utilizes brain signal as supervision

2.4. Method

        ①The shape of input image: , where  denotes height and  denotes width

        ②Edited region is , where 1 denotes editable position

        ③Brain condition: 

        ④Goal: for input , generate a image  that all the region of  contains the semantics of 

        ⑤Limitations of MindEye: a) inefficient in models stacking, b) decoding accuracy dependent, c) limited generation style

2.4.1. Our Method

        ①Over all pipeline:

        ②For input , they get masked  and mask itself 

        ③The pseudo-brain condition  is obtained by  with Pseudo Brain Generator (PBG)  and parameter . PBG is constructed by several residual linear layers

        ④Input: paired  where  denotes brain signal with batch size of  and voxel of , and  denotes image

        ⑤Fed  to CLIP ViT/L-14, obtaining feature  with the shape of . The loss is:

        ⑥Brain Adapter (BA) is a MLP (residual linear layers), process real and simulated brain signals  and  to embedding  and 

        ⑦Calculate the similarity matrix between two embeddings and employ CLIP constrastive loss:

        ⑧Loss of diffusion model:

        ⑨Probability of inpaint, outpaint and random masking: 0.5, 0.3, 0.2

2.5. Experiment

2.5.1. Dataset and Implementation

        ①Dataset: NSD for fMRI and image, OpenImages for image

        ②Test: random matching of 100 fMRI in NSD and 100 images in OpenImages

2.5.2. Qualitative Illustration

        ①Concatenation results:

        ②The same image meets different fMRI:

        ③When mask all of the image, MindPainter will reconstruct image only rely on fMRI:

2.5.3. Comparisons

        ①Comparison:

        ②Real evaluation on 1200 results of 20 reviewers (1 to 3, 3 is the best):

2.5.4. Ablation Study

        ①Ablation results:

2.5.5. Effectiveness of Designed Modules

        ①The effectiveness of pseudo-fmri:

2.6. Conclusion

        ~