A key challenge in using probabilistic programming for data science applications is learning accurate models from complex data and performing fast and accurate probabilistic inference. In this talk, I will present new Bayesian program learning techniques for domain-specific languages that make it possible to automatically discover entire probabilistic programs from tabular data, relational data, and time series data. Inference in these learned programs can be then performed effectively by compiling them in the Sum-Product Probabilistic Language (SPPL), a new specialized PPL that automatically delivers fast exact answers to a range of user queries. Applications and runtime/accuracy improvements will be shown for a variety of data science problems such as forecasting flu rates, modeling 3D zebrafish trajectories, and analyzing the fairness of decision tree classifiers.