Internship | Language-Guided Long-Horizon Visual Navigation in Photo-Realistic Simulation

About this internship

In this project you will propose an embodied AI agent that follows complex, natural-language navigation instructions inside a large-scale, photo-realistic environment in NVIDIA Isaac Sim (optionally using Isaac Lab). A central part of the assignment is to analyze and select a suitable environment configuration from available options—such as built-in indoor USD scenes, SimReady asset libraries, or imported large custom worlds—and justify your choice in terms of spatial scale, perceptual richness, semantic waypoint density (doors, tables, corners, corridors), and suitability for long-horizon trajectories. The agent will integrate (1) a Large Language Model (LLM) to decompose free-form instructions into structured sub-goals, (2) a Vision-Language Model (VLM) to ground these sub-goals in visual observations, and (3) a navigation controller that executes motion while incorporating perceptual feedback. The system should handle instructions such as: “go through the door on the left, traverse the room, turn right at the corner, move to the table, and tell me what is on it”, producing both a trajectory and a semantic interpretation of the final scene.

Scientifically, this project sits at the intersection of embodied AI, vision-language navigation, and LLM-based planning. Current research typically evaluates language-guided navigation in discrete graphs or moderately sized indoor datasets with predefined annotations, while real-time integration of open-ended LLM reasoning and visual grounding in continuous, high-fidelity simulation remains an open challenge. A key novelty of this assignment is the requirement to build and justify a rich, multi-room environment that enables extended trajectories and semantic waypoints, and to demonstrate closed-loop integration between high-level language reasoning and perception in a realistic simulator. By moving beyond small benchmark scenes toward configurable, long-horizon embodied reasoning in Isaac Sim, the project encourages students to explore how modern foundation models can be transformed from passive predictors into active, situated decision-making systems.

You will perform this assignment within TNO’s Intelligent Imaging department. The Intelligent Imaging department is a passionate, creative, and dedicated team of professionals (60 people) specialized in developing groundbreaking applications in the field of computer vision. Our team members have diverse backgrounds, ranging from the medical field to artificial intelligence. Intelligent Imaging is a young and growing department which has built up a lot of expertise over the past years in AI and deep learning.

What we expect from you

You are in the final stages of your master's degree in artificial intelligence, computer science, physics, mathematics, electrical engineering, or a similar degree. Prior experience with computer vision, image analysis, programming (Python), artificial intelligence/deep learning is appreciated.

What you'll get in return

You will, of course, be properly supervised during your work placement and be given the scope for you to get the best out of yourself. Furthermore, we provide:

• A highly professional, innovative internship environment, within a team of top experts.
• A suitable internship allowance (615 euro for wo-, hbo- and mbo-students, for a full-time internship).
• Possibility of eight hours of free leave per internship month (for a full-time internship).
• A free membership of Jong TNO, where you can meet other TNO professionals and join several activities, such as sports activities, (work-related) courses or the yearly ski-trip.
• Use of a laptop.
• An allowance for travel expenses in case you don’t receive an OV-card.

TNO as an employer

At TNO, we innovate for a healthier, safer and more sustainable life. And for a strong economy. We are driven to push boundaries. There is all the space and support for your talent and ambition. You work with people who will challenge you: who inspire you and want to learn from you. Our state-of-the-art facilities are there to realize your vision. What you do at TNO matters: impact makes the difference. Because with every innovation you contribute to tomorrow’s world.