Lines to get into the Wizarding World are now gone by 1pm most days and it becomes its own free flow island. However, before 4pm you can expect about a 90 minute wait for journey (which shockingly is about a full queue at max capacity) If you wait until after 4 I've seen it at about 45-60. Olivanders is the same story but add an hour to that
Tip for butterbeer I have learned: Go to Three Broomsticks after 8pm. They offer a bigger plastic cup then you can get anywhere else and its much less of a wait. So... MORE butterbeer, less wait...

and drinking butter beer looking over the patio at the lagoon at night is a great way to end a day!