The direct application of this method to VAE

[lavinal712]

Great work!

I have a couple of questions regarding the potential extension of your work:

Application to VAE:
Given that continuous modeling often outperforms VQ (Vector Quantization), do you think your approach could be applied to Autoencoder-KL (Variational Autoencoder with Kullback-Leibler divergence)? Specifically, could the continuous latent representations in your framework be adapted to improve the performance or efficiency of VAE models?

Unifying High-Level and Low-Level Representations via Distillation:
Have you considered using distillation techniques to unify high-level and low-level representations within your framework? For instance, could a teacher model with advanced representations guide a student model to learn both high-level semantic features and low-level perceptual details, thereby creating a more cohesive and efficient multimodal system?

I believe addressing these questions could further enhance the versatility and impact of your already groundbreaking work. Thank you for your time, and I look forward to your insights!

[QuLiao1117]

@lavinal712 Thanks for your attention to TokenFlow.

Regarding application to VAE:

• The approach could potentially be adapted to continuous latent space by learning a joint continuous representation encompassing both semantic and low-level features. This could function as a general visual encoder and be applied to other downstream tasks.

Regarding unification through distillation:

• We have not yet explored using distillation techniques to unify high-level and low-level representations. It would be promising to guide a student model to simultaneously learn both high-level semantic features and low-level perceptual details.