Computer vision for buildings, image-to-floorplan, generative ML models, performance prediction, structural analysis ML, energy prediction, natural language to design, and point cloud ML for AEC computational design
36k tokens
context cost
the whole folder, loaded on every use
4
files
instructions only
0
copies elsewhere
how many repositories repackaged it
197
stars on the repo
on the repository, not the skill itself
Install
one command, takes just this skill from the repository
Machine learning is reshaping specific domains within Architecture, Engineering, and Construction, though the transformation is uneven. This skill provides a thorough, practitioner-oriented guide to where ML delivers real value in AEC today, the architectures and methods that work, the data challenges that constrain adoption, and practical pipelines for training, deploying, and maintaining ML models in production AEC workflows.
1. ML in AEC: Current State
1.1 Where ML Actually Works in AEC Today
ML in AEC is most effective where three conditions converge: (a) sufficient training data exists or can be generated, (b) the task is well-defined with measurable performance metrics, and (c) the cost of errors is manageable or human review is in the loop.
Proven, deployed applications:
Construction progress monitoring (photo comparison to BIM schedule)
Safety monitoring on construction sites (PPE detection, exclusion zones)
Defect detection (crack detection in concrete, facade inspections via drone imagery)
Document classification (sorting drawings by discipline, type)
Energy performance prediction (surrogate models replacing full simulation)
Point cloud semantic segmentation (labeling structural elements from LiDAR scans)
Cost estimation from early-stage design parameters
Promising but not yet mature:
Floor plan generation from adjacency programs
Automated scan-to-BIM conversion
Generative massing from site constraints
Structural topology optimization acceleration
Natural language to BIM queries
Overhyped or premature:
Fully autonomous building design from text prompts
AI replacing architectural design judgment
General-purpose design AI that understands building codes, physics, and aesthetics simultaneously
End-to-end text-to-construction-documents
1.2 Data Challenges in AEC
The AEC industry faces unique data challenges that limit ML adoption:
Small datasets: Unlike ImageNet (14M images) or web-scale text corpora, AEC datasets are small. A large architecture firm might have 5,000 floor plans in its portfolio. A structural engineering firm might have 2,000 analyzed buildings. These numbers are 3-4 orders of magnitude below what deep learning models typically require.
Inconsistent labeling: Building elements are labeled differently across firms, software platforms, and regions. A "wall" in one BIM model might be modeled as a "generic model" in another. Room naming conventions vary wildly. There is no universal taxonomy.
Domain complexity: Buildings are multi-physics systems where geometry, structure, thermal behavior, acoustics, daylight, and human experience interact. ML models that capture only one dimension produce solutions that fail on others.
Proprietary data: Most building data is proprietary. Firms are reluctant to share project data. Public datasets are limited in size and diversity.
High-dimensional output: A floor plan is not a single number or a class label; it is a complex geometric arrangement satisfying dozens of constraints simultaneously. This makes supervised learning difficult because the "ground truth" is itself a design decision, not an objective fact.
1.3 ML Maturity by AEC Subdomain
| Subdomain | ML Maturity | Key Applications | Data Availability |
| Urban analysis | Low-Medium | Land use classification, traffic | Moderate (satellite, GIS) |
1.4 Build vs. Buy Decisions
| Approach | When to Use | Examples |
|----------|------------|---------|
| Use off-the-shelf | Standard CV tasks (object detection, segmentation) with fine-tuning | YOLOv8, Detectron2, Segment Anything |
| Fine-tune pre-trained | AEC-specific tasks with moderate data (100-10,000 samples) | Fine-tuned ResNet for facade classification, ControlNet for architectural sketches |
| Train from scratch | Unique data modality or task with no applicable pre-trained model | Custom GNN for floor plan generation, custom PointNet for AEC-specific segmentation |
| Buy commercial | Mature, productized solutions where accuracy matters and in-house ML capacity is limited | OpenSpace (construction monitoring), Buildots, Avvir |
2. Computer Vision for AEC
2.1 Object Detection
Detecting and localizing building elements in images, drawings, or renderings.
Architectures:
| Model | Speed | Accuracy | Best For |
|-------|-------|----------|----------|
| YOLOv8/v9 | Very fast (real-time) | Good | Site safety monitoring, real-time applications |
| Faster R-CNN | Moderate | Very good | Drawing element detection, precise localization |
| DETR (Detection Transformer) | Moderate | Very good | Complex scenes, variable-size objects |
| EfficientDet | Fast | Good | Mobile/edge deployment, drone imagery |
AEC object detection tasks:
Detecting doors, windows, columns, stairs in architectural drawings
Identifying structural elements (beams, columns, braces) in construction photos
Recognizing equipment (HVAC units, electrical panels) in MEP drawings
Detecting construction vehicles and workers on site
Identifying signage, safety barriers, and temporary works
Training data preparation:
Collect images: site photos, drawing scans, BIM screenshots, drone footage
Annotate with bounding boxes using tools like LabelImg, CVAT, Roboflow, Label Studio
Define class taxonomy: start small (5-10 classes), expand as needed
Ensure diversity: different lighting, angles, scales, drawing styles
Split: 70% train, 15% validation, 15% test; ensure no project overlap between splits
Augment: rotation, flipping, brightness, contrast, noise for images; not applicable for drawings where orientation matters
2.2 Semantic Segmentation
Pixel-level classification of every pixel in an image.
Architectures:
| Model | Parameters | Best For |
|-------|-----------|----------|
| U-Net | ~31M | Medical imaging heritage; small datasets; floor plan segmentation |
Symbol detection (valves, pumps, instruments, equipment)
Line detection (process lines, signal lines)
Text recognition (tag numbers, labels)
Topology extraction (connectivity graph)
Drawing annotation extraction:
Title block parsing: project name, sheet number, revision, date, scale
Dimension text extraction
Room name and number extraction
Note and specification text extraction
Models: Combination of object detection (for symbols), line detection (for pipes), and OCR (for text). Tesseract, PaddleOCR, or EasyOCR for text; custom detectors for symbols.
2.5 Construction Progress Monitoring
Comparing as-built photos to BIM model to track construction progress:
Image capture: 360-degree cameras on hard hats or fixed mounts; capture daily
Pose estimation: Determine camera position relative to BIM using visual SLAM or marker-based localization
Element matching: Match detected elements in photos to BIM elements using projected positions
Progress scoring: For each BIM element, determine installation status:
Not started (element not visible)
In progress (partially installed)
Complete (fully installed, matches BIM geometry)
Dashboard: Overlay progress status on BIM model; generate progress reports
Inference: Real-time on edge GPU (Jetson, Intel NCS)
Alert: If no_hard_hat or no_vest detected, trigger alert
Unsafe behavior detection:
Worker in exclusion zone (geofenced dangerous areas)
Worker near heavy equipment operating radius
Working at height without fall protection
Improper lifting posture
Datasets: COCO (general person detection), SODA (Safety Of Drivers and Automobiles), SHEL5K (Safety HElmet), Chi-SID (Construction Safety Image Dataset)
PointFlow: Normalizing flow model generating point clouds
Point-E (OpenAI): Text-to-3D point cloud generation
ShapeNet: Large-scale 3D shape dataset (includes some architectural objects)
Current limitations: Generated shapes lack architectural precision; no structural logic; no floor plates or walls; resolution too low for detailed building geometry. Useful for early-stage massing exploration only.
4.6 Current Limitations of Generative ML for AEC
Physical validity: Generated designs may violate structural, MEP, or code requirements
Resolution: Output resolution insufficient for construction documentation
Geometric precision: ML models produce fuzzy boundaries; not the crisp lines needed for architecture
Constraint enforcement: Difficult to enforce hard constraints (code compliance, structural limits) within the generation process
Evaluation: No universally accepted metric for design quality; human evaluation is expensive and subjective
Data: Small AEC datasets limit generative model quality; models trained on web images do not understand buildings
Integration: Generated outputs do not integrate directly with BIM software without significant post-processing
5. Performance Prediction Models
5.1 Energy Prediction
Predicting building energy use intensity (EUI) from design parameters without running full simulation:
Input features:
Geometry: floor area, surface-to-volume ratio, compactness, number of stories
Envelope: wall U-value, roof U-value, window U-value, WWR by orientation
Orientation: building azimuth, latitude
Climate: HDD, CDD, solar radiation
Systems: HVAC type, lighting power density, equipment load
Schedule: occupancy hours, setpoint temperatures
Target variable: Annual EUI (kWh/m2/yr) or monthly energy consumption
Training data generation:
Create parametric building model (e.g., in OpenStudio or EnergyPlus via eppy)
Define parameter ranges (sampling plan: Latin Hypercube Sampling)
Run 1,000-10,000 simulations
Each simulation = one training example (parameters → EUI)
Model comparison (typical performance on EUI prediction):
| Model | R2 | RMSE (kWh/m2) | Training Time | Interpretability |
Primitive fitting: Fit geometric primitives to segments:
Planes → walls, floors, ceilings
Cylinders → pipes, columns
Boxes → beams, equipment
Custom shapes → MEP fittings
Topology recovery: Determine connections between elements (wall-wall intersection, pipe-fitting-pipe)
BIM element creation: Map primitives to BIM elements with attributes (type, material, dimensions)
Model assembly: Create IFC or Revit model from elements
Current state: Steps 1-3 are increasingly automated with ML. Steps 4-6 still require significant manual intervention. Full end-to-end scan-to-BIM automation is 3-5 years away for typical buildings.
7.5 As-Built vs. As-Designed Comparison
Comparing point cloud (as-built) to BIM model (as-designed):
Registration: Align point cloud to BIM coordinate system (ICP, feature matching)
Point-to-surface distance: For each point, compute distance to nearest BIM surface
Deviation mapping: Color-code deviations (green = within tolerance, yellow = marginal, red = out of tolerance)
Tolerance checking: Flag elements exceeding tolerance (typically ±25mm for structural, ±50mm for architectural)
Missing element detection: BIM elements with no nearby points may be missing or not yet installed
Extra element detection: Point clusters not corresponding to any BIM element indicate field additions
| BuildingNet | 513K meshes | 31 | 3D building models | Public |
8. NLP for AEC
8.1 Building Code Parsing and Querying
Using NLP to make building codes searchable and machine-readable:
Approaches:
Information retrieval: Index code text; retrieve relevant sections for a query (e.g., "What is the maximum travel distance for a sprinklered business occupancy?")
Named entity recognition: Extract entities from code text (dimensions, occupancy types, construction types, materials)
Relation extraction: Identify relationships between entities (occupancy + sprinkler status → travel distance)
Question answering: LLM fine-tuned on building code corpus; answer natural language questions about code requirements
Semantic parsing: Convert code text to structured rules (IF-THEN format) for automated compliance checking
Challenges: Building codes use dense legal language with complex cross-references, exceptions, and conditional clauses. Accuracy requirements are high (incorrect code interpretation has liability implications).
8.2 Design Brief Analysis
Extracting structured information from narrative design briefs:
Room program extraction: identify room types, areas, counts from text
Coordinate system alignment: Align all data to common coordinate system
Missing data handling: AEC data is often incomplete; impute or flag missing values
Outlier detection: Identify and handle anomalous values (e.g., room with 0 area, wall with 100m thickness)
Class balancing: AEC datasets are often imbalanced (many walls, few stairs); use oversampling, undersampling, or class weights
9.2 Feature Engineering for AEC Data
See Section 5.6 for detailed feature types. Key principles:
Use domain knowledge to create meaningful features (architects and engineers know what matters)
Normalize features to similar scales (StandardScaler, MinMaxScaler)
Handle categorical features (one-hot encoding for room types, occupancy types)
Create interaction features (WWR * orientation = directional solar gain proxy)
Dimensionless ratios often work better than raw dimensions (compactness, aspect ratio vs. raw width/height)
9.3 Model Selection Decision Tree
Is the output a category or a number?
├── Category (classification)
│ ├── Structured/tabular data → XGBoost or Random Forest
│ ├── Image data → CNN (ResNet, EfficientNet) or ViT
│ ├── Point cloud data → PointNet++ or KPConv
│ └── Graph data → GNN (GCN, GAT)
│
└── Number (regression)
├── Structured/tabular data
│ ├── Small dataset (<1000) → Gaussian Process or Random Forest
│ ├── Medium dataset (1K-100K) → XGBoost or MLP
│ └── Large dataset (>100K) → Deep learning (MLP, CNN)
├── Image/field output → U-Net or encoder-decoder CNN
├── Sequence output → Transformer or LSTM
└── Generative (create new designs)
├── Image-like output → GAN, VAE, or Diffusion Model
├── Graph-conditioned → GNN + Decoder
└── Point cloud output → PointFlow or Point-E
| Development | Workstation GPU (A5000, A6000) | $3000-7000 one-time | Medium models, production training |
| Production | Cloud GPU (A100, H100) | $2-5/hr (AWS, GCP, Azure) | Large models, multi-GPU training |
| Inference | CPU or edge GPU (Jetson) | $200-500 per device | Deployed model serving |
9.5 Validation Methodology
Standard k-fold cross-validation: Split data into k folds (typically 5 or 10); train on k-1, test on 1; rotate and average.
Spatial cross-validation (important for AEC): Buildings from the same project or site should not appear in both training and test sets. Group by project, not by individual samples.
Temporal cross-validation: For construction monitoring or operational data, use time-based splits (train on earlier data, test on later data).
Domain shift testing: Test on data from a different building type, climate zone, or country to assess generalization.
9.6 Deployment Options
| Deployment | Method | Latency | Integration |
|-----------|--------|---------|-------------|
| REST API | Flask/FastAPI + Docker | 100-500ms | Any platform via HTTP |
Floor plan datasets are dominated by specific cultures (Chinese residential in RPLAN, Finnish in CubiCasa5K). Models trained on these produce culturally biased layouts.
Construction safety datasets may underrepresent certain demographics, leading to biased PPE detection.
Energy models trained on one climate zone do not generalize to others.
Mitigation: Diversify training data, test across populations and regions, document training data composition.
10.2 Liability for ML-Driven Design Decisions
Who is liable when an ML model suggests a non-compliant design? The architect? The software vendor? The ML developer?
Professional responsibility still rests with the licensed professional who signs and seals the documents.
ML outputs must be reviewed and validated by qualified professionals before use in construction documents.
Document the role of ML in the design process; maintain audit trails.
10.3 Explainability Requirements
Regulatory bodies may require explanations for design decisions (especially for code compliance and structural safety).
Black-box models (deep neural networks) are difficult to explain. Prefer interpretable models (Random Forest, decision trees) for safety-critical applications.
Use SHAP values, partial dependence plots, and attention visualization to explain model behavior.
For generative models, provide constraint satisfaction scores alongside generated designs.
10.4 Building Code Compliance of ML Outputs
ML-generated designs are not automatically code-compliant. All generated layouts, structures, and systems must be checked against applicable codes.
Do not represent ML outputs as code-compliant without explicit verification.
ML can assist code compliance checking but should not be the sole authority.
10.5 Professional Responsibility
Architects and engineers have professional and legal obligations that cannot be delegated to ML.
ML is a tool; professional judgment is the final authority.
Training in ML literacy should be part of professional education.
Firms should develop AI/ML use policies aligned with professional practice standards.
10.6 Data Privacy
Building data may contain personally identifiable information (occupant data, access patterns, energy usage).
Construction site photos may capture worker faces and activities.
Comply with GDPR, CCPA, and local privacy regulations when collecting and using AEC data.
Anonymize data where possible; limit data retention; obtain consent for data collection.
Key Takeaways for AEC Practitioners
Start with the problem, not the model: Define the design or engineering problem precisely before selecting an ML approach.
Data is the bottleneck: Invest in data collection, cleaning, and annotation. Without good data, no model will perform well.
Use simulation to generate training data: Parametric simulation (energy, structural, daylight) can generate thousands of training examples.
Tabular models first: For structured AEC data, XGBoost and Random Forest often outperform deep learning with less effort.
Validate rigorously: Use domain-appropriate cross-validation; test on projects not in the training set.
Keep humans in the loop: ML augments; it does not replace professional judgment.
Deploy simply: A FastAPI endpoint or Grasshopper component is often sufficient; avoid over-engineering the deployment.
Monitor in production: Track prediction quality and retrain when performance degrades.
Document everything: Training data, model version, validation results, and deployment configuration.
10. Ethical awareness: Understand bias, liability, and privacy implications of ML in AEC.
How to use it
Copy the folder
Take abhinavbwj/ml-for-aec from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
Check the name does not clash
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.