There are two main regions that are perfect for surfing: the southern coast and the eastern coast. The southern coast, which includes Hikkaduwa, Mirissa, Ahangama, and Galle are popular tourist spots, and not just among surfers. While the waves here range from those perfect for beginners to those for advanced surfers, it also means that you will be in the middle of a crowded season.
The eastern coast, on the other hand, seems a fewer number of tourists. Those who visit the area are often focused on surfing, especially at Arugam Bay. The best time to visit with regard to the swells is between April and October. As this is the off-season with regard to general tourism, you would see less of a crowd in the eastern coast.