AI-TD-Facades-Materials
Project Overview
This project explores whether synthetic material images can reduce the need for large real-world datasets in architectural computer vision. The system compares models trained with real images, synthetic images, and mixed datasets across five material classes: brick, glass, concrete, metal panel, and vegetation.
The project connects AI, architectural material recognition and real-time visual systems. A computer vision model classifies façade materials, evaluates performance through metrics and Grad-CAM explanations, and sends predictions into TouchDesigner, where the results become live visual signals for an interactive spatial demo.
Starting Point
The project started from a practical research question: can procedurally generated images help train a model that still performs well on real architectural materials?
Real façade datasets are time-consuming to collect and label. At the same time, architectural visualization tools can generate large amounts of synthetic material images with controlled variations. The challenge was to test whether those synthetic images could become useful training data instead of only visual references.
The first stage focused on defining the material classes, preparing natural and synthetic datasets, and creating a clear experiment structure to compare training strategies under the same evaluation conditions.
Problem Solving
The project was organized around three training conditions: NAT-only, SYN-only, and MIX, where natural and synthetic data were used in different ways to evaluate the sim-to-real gap. A CNN classifier was trained and tested on a held-out natural dataset to understand which strategy generalized better to real façade images.
The evaluation process included accuracy, macro F1-score, per-class performance, confusion matrices, and Grad-CAM overlays to inspect what the model was looking at when making predictions. This was important because the goal was not only to get a number, but to understand whether the model was learning meaningful architectural features or overfitting to texture artifacts.
After the model evaluation, the system was connected to TouchDesigner through Python-based communication. Material predictions, confidence values, and class outputs were translated into real-time visual parameters, allowing AI inference to control an interactive design environment.
More recently, I have also been working with depth maps and point clouds in Blender to develop a final version of the project. This adds a spatial layer to the research, moving the output from classification and data visualization into a more architectural and atmospheric visual result.
Implementation
The final system combines a trained computer vision model, evaluation outputs, and a real-time TouchDesigner interface. The Python side handles image preprocessing, model inference, class prediction, confidence values, and communication. TouchDesigner receives this information and transforms it into visual behavior, allowing the detected material to affect color, motion, texture, or spatial atmosphere.
The project also includes a reproducible structure through GitHub and YouTube. The repository contains the code, setup instructions, training/evaluation workflow, and integration files. The video explains the project as a tutorial, showing how to run it locally and how to connect the AI model with TouchDesigner.
The final visual direction expands the system with Blender, where depth maps and point-cloud workflows are being used to create a more polished “beauty” output. This helps present the project not only as a research experiment, but also as a spatial design system with visual and architectural potential.
Results
The project produced a complete AI-to-real-time-design workflow: from dataset creation and CNN training to evaluation, explainability, and live visual integration. It demonstrates how computer vision can be used as more than an analysis tool; it can become a real-time input for computational design environments.
The research also showed the value and limitations of synthetic data for architectural material recognition. By comparing NAT-only, SYN-only, and MIX training strategies, the project created a framework for understanding how synthetic images can support real-world computer vision tasks in architecture.
As a portfolio project, this work represents the intersection of my current profile: AI, computer vision, computational design, architecture, TouchDesigner, and real-time spatial systems. It connects technical research with visual experimentation and shows how machine learning outputs can be translated into interactive spatial experiences.