HVAC load forecasting is a critical component for future deployments of energy sources and grid integration. However, there are few studies on developing the integrated models that allow users to start from original solar and weather data to predict and differentiate thermal and electrical loads so that different strategies to save or control energy consumptions can be developed for individual buildings. Here, the authors introduce an integrated framework to applying lidar data and using a lumped thermal network model with Extended Kalman Filter (EKF) to provide a relatively accurate way to forecast the electrical and thermal load for buildings. Image processing techniques to generate the solar and geometry inputs using lidar data is developed. A hybrid model based on lidar data to forecast the load is tested and parameter optimization is evaluated. Results are validated by a commercial building on a university campus. The final results show that 4% amount of accuracy is achieved by comparing measure and simulated chilled water consumption.