With the Hubble Frontier Fields program, gravitational lensing has provided a powerful way to extend the study of the ultraviolet luminosity function (LF) of galaxies at $z \sim 6$ down to unprecedented magnitude limits. At the same time, significant discrepancies between different studies were found at the very faint end of the LF. In an attempt to understand such disagreements, we present a comprehensive assessment of the uncertainties associated with the lensing models and the size distribution of galaxies. We use end-to-end simulations from the source plane to the final LF that account for all lensing effects and systematic uncertainties by comparing several mass models. In addition to the size distribution, the choice of lens model leads to large differences at magnitudes fainter than $M_{UV} = -15~$ AB mag, where the magnification factor becomes highly uncertain. We perform MCMC simulations that include all these uncertainties at the individual galaxy level to compute the final LF, allowing, in particular, a crossover between magnitude bins. The best LF fit, using a modified Schechter function that allows for a turnover at faint magnitudes, gives a faint-end slope of $\alpha = -2.01_{-0.14}^{+0.12}$, a curvature parameter of $\beta = 0.48_{-0.25}^{+0.49}$, and a turnover magnitude of $M_{T} = -14.93_{-0.52}^{+0.61}$. Most importantly our procedure shows that robust constraints on the LF at magnitudes fainter than $M_{UV} = -15~$ AB remain unrealistic. More accurate lens modeling and future observations of lensing clusters with the James Webb Space Telescope can reliably extend the UV LF to fainter magnitudes.