I work on diffusion-based controllable image and video generation at UIUC, advised by Yu-Xiong Wang. I design the supervision, training data, and curricula that teach pretrained generative models new capabilities, including new visual controls, multi-view and temporal consistency, and minute-scale identity-consistent video.
I am now extending this work toward unified multimodal models in which reasoning guides generation and generated outcomes improve reasoning. My ongoing projects develop visual world models that predict the next screen after a UI action, physical events from observed video, and robot–object interactions under a prescribed motion.
Graduating Dec. 2026|Open to Research Scientist / Research Engineer roles starting 2027
First or co-first author at NeurIPS 2025, NeurIPS 2024, CVPR 2024 (×2), 3DV 2026, and WACV 2027, across controllable image, video, 3D, and 4D generation.
Virtual Fitting Room: identity-consistent try-on video up to 90 seconds long from a single image, trained only on short clips. NeurIPS 2025
Dress&Dance: teaches pretrained video diffusion new garment, identity, and motion controls. 20–59% lower FVD than a Kling 1.6 try-on-and-animation pipeline. WACV 2027
Research internships at Meta Reality Labs (Zurich 2023, Bay Area 2025) and SpreeAI. At Meta in 2025, designed a data-processing and training pipeline at 128–256-GPU scale.
Jun-Kun Chen, Aayush Bansal, Minh Phuoc Vo, Yu-Xiong Wang
Builds on Dress&Dance (D&D). Learns reference-conditioned generation from ordinary pairs of short clips. At inference, generated 360° anchors keep identity consistent in try-on videos up to 90 seconds long. On 30–50 s hard-motion tests it scores highest on VBench subject consistency, background consistency, and motion smoothness against D&D+FramePack and D&D+Kling 2.0 (93.2 / 95.3 / 99.4 vs. 88.3 / 91.0 / 98.2 and 92.7 / 94.1 / 98.9), with comparable GPT-judged try-on quality.
Jun-Kun Chen, Aayush Bansal, Minh Phuoc Vo, Yu-Xiong Wang
A general framework for conditional video generation: CondNet maps text, layout, pose, garment, and identity into one conditioning space, and target-steered supervision with staged training adds these controls to pretrained video diffusion. Demonstrated on high-resolution virtual try-on, it generates 1152×720, 24-FPS videos and reaches FVD (lower is better) of 157 vs. 197 on Internet videos and 20 vs. 49 on captured videos against a Kling 1.6 try-on-and-animation pipeline, and lower FVD than all four evaluated modular baselines on both benchmarks. Followed by Virtual Fitting Room, which extends this setting to minute-long videos.
Jun-Kun Chen, Samuel Rota Bulò, Norman Müller, Lorenzo Porzi, Peter Kontschieder, Yu-Xiong Wang
Teaches 2D diffusion to edit 3D scenes consistently across views. Uses cross-view self-supervision and staged adaptation, with consistency losses along the full denoising trajectory. A 4-GPU asynchronous pipeline overlaps diffusion training and image generation with scene fitting, for a 10× training speed-up.
The first instruction-driven 4D editing framework. Treating a dynamic scene as pseudo-3D breaks long 4D edits into tractable problems for 2D diffusion.
Yanming Zhang*, Jun-Kun Chen*, Jipeng Lyu, Yu-Xiong Wang
General-purpose, training-free editing of videos and 3D scenes. Hierarchical task decomposition separates instruction following from content preservation.
* Equal contribution. Publications appear under Jun-Kun Chen; earlier papers may use Junkun Chen or J. Chen.
Current Research: Generative World Models
Each project adapts a unified multimodal model, a vision-language reasoner paired with a diffusion generator, to predict what happens next on a screen, in a physical scene, or under a robot's motion. In computer use, feedback derived from generated screens already measurably improves the reasoner. Physics and robotics move to video, where feedback must judge motion and the generator must get contact and its consequences right.
One training-set example: the teacher sees the render beside the real next screen and edits one phrase of the Reasoner's text. Rendered from the corrected text, the canvas fills like the real screen.
A language Reasoner predicts what a UI action will change, and a post-trained diffusion Generator produces the next screenshot conditioned on that prediction. We distill generative feedback into the Reasoner: a teacher compares the generated next screen with the real one and locally corrects the Reasoner's own prediction. The Reasoner is then trained offline on these corrections in a single round, so inference needs no teacher or real next screen. In a controlled comparison with the conditioning branch and Generator fixed, the distillation raised VLM-judged screen similarity (0–100) from 74.09 to 78.57 on 92 held-out AndroidWorld transitions; distilling corrections made without seeing the generated screen reached 74.74.
Diagnostic examples at half speed. From the same first second of real video, two samples of one model (seeds 42, 43) go wrong at contact: dominoes cross a gap that stops the real chain, and a ball vanishes or rolls away.
Physical-event world model
Video diffusion · Cosmos3-Nano · 2025–26 · project lead, supervising an undergraduate intern
We adapt Cosmos3-Nano video diffusion to predict the next short window of a physical event from observed RGB frames and a structured event description, chaining windows into autoregressive video rollouts. Controlled tests examine contact responses, object persistence, and how much predictions rely on the text conditioning. Comparing rollouts from real and generated prefixes helps separate errors in the learned dynamics from errors that compound as generated frames are fed back. Feedback on video has to judge motion, not just appearance, so we also built a benchmark of 3,571 simulated rigid-body clips for diagnosing video judges, with simulator labels that separate physical errors from visual flaws.
On a held-out DROID scene, each imagined clip follows the motion it is given. In a synthetic registration test, the later curriculum applies the paired example's offset instead of copying the rendering's position.
Robot-interaction world model
Motion-conditioned video diffusion · Cosmos3-Nano · 2025–26 · solo project
I adapt only the video Generator of Cosmos3-Nano, training on simulated and real robot data (MimicGen, RoboTwin, DROID, RH20T, AgiBot) to predict robot–object interaction video from a starting scene, recent frames, and a supplied robot motion. The motion enters through a visual conditioning interface: its rendering, plus paired render–observation examples from the target setup. Correspondence curricula deliberately shift and scale the two sides of these examples differently, so the model must apply the demonstrated render-to-camera mapping instead of copying the rendering's position. The next goals are transfer to new camera setups without retraining and predicting a motion's consequences for objects, such as release when the gripper opens.
Next, I plan to extend the reasoner–generator interface beyond text to visual plans that the generator reads directly, train motion-sensitive reasoning that can critique generated video, and move the feedback from an external teacher into the model itself.
Experience
Meta Reality Labs
Research Scientist InternBay Area, CA, U.S. · Zurich, Switzerland
May–Dec. 2025Bay Area · part-time continuation Aug.–Dec.
Studied joint 3D and video generation, then explored video-to-3D and 3D-to-video generation. Designed and implemented a GPU-role-separated pipeline for data processing and model training at 128–256-GPU scale on Meta's distributed infrastructure.
May–Aug. 2023Zurich
Took ConsistDreamer (CVPR 2024) from method design through implementation and evaluation. Designed cross-view self-supervision, staged diffusion adaptation, and a 4-GPU asynchronous pipeline that overlaps diffusion training and image generation with scene fitting, for a 10× training speed-up.
SpreeAI
AI Research InternRemote, U.S.
May 2024 – May 2025
Dress&Dance (WACV 2027): taught pretrained video diffusion new garment, identity, and motion controls. Designed and implemented target-steered supervision, training triplets with synthetic conditions and real targets, and staged image/video curricula. Evaluated garment and identity fidelity, motion, and video quality.
Virtual Fitting Room (NeurIPS 2025): designed training on arbitrary pairs of short clips to learn reference appearance and prefix continuation, without canonical 360° training anchors. At inference, a generated 360° reference and anchored autoregression maintain minute-scale consistency, while a refiner trained with targeted augmentation removes generator artifacts.
Mila – Quebec AI Institute
Research InternMontreal, Canada
Feb.–Aug. 2020Knowledge-graph reasoning with Prof. Jian Tang. Co-first author of RNNLogic (ICLR 2021).
Baidu, NLP Department
Research / Development InternBeijing, China
July 2019 – Jan. 2020Scene-aware dialogue generation. Top-3 system at DSTC8, presented at the DSTC8 Workshop at AAAI 2020.
Models: image/video diffusion, diffusion Transformers (DiT), video world models (NVIDIA Cosmos), unified multimodal models (mixture-of-transformers), vision-language models, LoRA
Practice: training diffusion models with rectified-flow / flow-matching objectives, diffusion post-training (supervised fine-tuning, LoRA, reward-based tuning with DiffusionNFT), distilling teacher corrections from generative feedback into a language Reasoner, action and multimodal conditioning, video and robot-data pipelines, multi-GPU training with exact checkpoint resume, held-out benchmark design, human and VLM-judge evaluation, paired-bootstrap statistics
Selected Honors
ICPC EC-Final — Gold Medal, 4th place (2018)
ICPC Qingdao Regional — Gold Medal, 2nd place (2018)
CCPC Harbin Regional — Gold Medal (2017)
National Olympiad in Informatics — Gold Medal (2016)
Problem setter — 9 contest tasks, including hardest-tier problems at NOI 2018 and NOI 2019 (2018–2019)
