This study presents a novel non-invasive approach for breast cancer detection and tumor localization by developing an optimized Physics-Informed Neural Network (PINN) model integrated with infrared (IR) thermal imaging and 3D physical breast modeling. The proposed method leverages thermal data from an IR camera and anatomical information from a 3D scanner to train a PINN, incorporating the bioheat equation to perform both forward and inverse predictions. The PINN is uniquely optimized to estimate bio-physical parameters, such as tumor radius and depth, enabling accurate tumor diagnosis. Validation against Finite Element Method (FEM) simulations from ANSYS demonstrates that the PINN model achieves high accuracy, with error margins as low as 0.70% for radius and 1.38% for depth after training optimizations. Compared to traditional FEM solvers, the PINN model offers 10 times faster simulation post training, highlighting its computational efficiency. This work underscores the potential of PINNs as a promising tool for non-invasive breast cancer diagnostics, combining physical constraints, anatomical accuracy, and machine learning for enhanced tumor detection and localization.